diff --git a/doc/.vitepress/config.mts b/doc/.vitepress/config.mts index 85c00f5aa..846127ae7 100644 --- a/doc/.vitepress/config.mts +++ b/doc/.vitepress/config.mts @@ -27,13 +27,18 @@ export default defineConfig({ items: [ { text: 'Docker', link: '/docker.md' }, { text: 'Configuration', link: '/configuration.md' }, + { text: 'Deployment', link: '/deployment.md' }, + { text: 'Database', link: '/database.md' }, { text: 'Localization', link: '/localization.md' }, { text: 'Cookies', link: '/cookies.md' }, { text: 'Plugins', link: '/plugins.md' }, { text: 'Stats', link: '/stats.md' }, {text: 'Skins', link: '/skins.md' }, + { text: 'Accessibility', link: '/accessibility.md' }, {text: 'Demo', link: '/demo.md' }, {text: 'CLI', link: '/cli.md'}, + { text: 'Development', link: '/development.md' }, + { text: 'FAQ', link: '/faq.md' }, ] }, { diff --git a/doc/accessibility.md b/doc/accessibility.md new file mode 100644 index 000000000..24e1269b4 --- /dev/null +++ b/doc/accessibility.md @@ -0,0 +1,91 @@ +# Accessibility + +Etherpad aims to be usable by everyone, including people who rely on a +keyboard, a screen reader, or other assistive technology. The editor follows +common conventions so that selecting, formatting, and navigating text works the +way you would expect in other applications, and the toolbar can be reached and +operated without a mouse. + +If you find a feature that is not accessible, please let us know by opening an +issue so it can be improved. + +## Keyboard shortcuts + +The following shortcuts are built into the editor. On macOS use the Command +(`Cmd`) key wherever `Ctrl` is listed. + +::: tip +Most shortcuts can be individually enabled or disabled through the +`padShortcutEnabled` settings, so a deployment may have customised which of +these are active. +::: + +### Editor + +| Action | Shortcut | +| --- | --- | +| Bold | `Ctrl` + `B` | +| Italic | `Ctrl` + `I` | +| Underline | `Ctrl` + `U` | +| Strikethrough | `Ctrl` + `5` | +| Ordered (numbered) list | `Ctrl` + `Shift` + `N` or `Ctrl` + `Shift` + `1` | +| Unordered (bulleted) list | `Ctrl` + `Shift` + `L` | +| Indent line or selection | `Tab` | +| Outdent line or selection | `Shift` + `Tab` | +| Undo | `Ctrl` + `Z` | +| Redo | `Ctrl` + `Y` or `Ctrl` + `Shift` + `Z` | +| Save a named revision | `Ctrl` + `S` | +| Duplicate the current line(s) | `Ctrl` + `Shift` + `D` | +| Delete the current line(s) | `Ctrl` + `Shift` + `K` | +| Clear authorship colors on the pad or selection | `Ctrl` + `Shift` + `C` | +| Show the authors of the current line | `Ctrl` + `Shift` + `2` | +| Focus the toolbar (see below) | `Alt` + `F9` | +| Focus the chat input | `Alt` + `C` | + +Text selection, cut (`Ctrl` + `X`), copy (`Ctrl` + `C`), paste +(`Ctrl` + `V`), and the arrow keys behave as they do in any standard text +editor. + +### Timeslider + +The timeslider (revision history) provides its own shortcuts: + +| Action | Shortcut | +| --- | --- | +| Play / pause history playback | `Space` | +| Step back one revision | `Left Arrow` | +| Step forward one revision | `Right Arrow` | +| Jump back to the previous starred revision | `Shift` + `Left Arrow` | +| Jump forward to the next starred revision | `Shift` + `Right Arrow` | + +## Toolbar navigation + +The toolbar holds the formatting controls (bold, italic, lists, and so on) and +can be reached and operated entirely from the keyboard: + +* Press `Alt` + `F9` from the editor to move focus to the first button in the + toolbar. +* Use the `Left Arrow` and `Right Arrow` keys to move between buttons. `Tab` + also moves to the next focusable control. +* Press `Enter` to activate the focused button. +* Press `Alt` + `F9` again, or `Escape`, to return focus to the pad. + +Pressing `Escape` while a toolbar dropdown (such as the settings or color +picker) is open closes that dropdown first. + +## Screen readers + +Etherpad provides as much screen reader support as possible. Support quality +varies between platforms and browsers, so the following combinations are +recommended: + +* On Windows, Firefox with [NVDA](https://www.nvaccess.org/) currently gives the + best experience. + +To reduce verbose feedback while typing collaboratively in NVDA, open the +keyboard settings (`NVDA` + `Ctrl` + `K`) and turn off **Speak typed characters** +and **Speak typed words**. + +Support in other screen readers and browsers (for example Orca on Linux, or +Chrome) is more limited. Contributions to improve coverage on these platforms +are very welcome. diff --git a/doc/api/changeset_library.md b/doc/api/changeset_library.md index 9785f2b47..dbf3c9224 100644 --- a/doc/api/changeset_library.md +++ b/doc/api/changeset_library.md @@ -1,7 +1,7 @@ # Changeset Library The [changeset -library](https://github.com/ether/etherpad-lite/blob/develop/src/static/js/Changeset.ts) +library](https://github.com/ether/etherpad/blob/develop/src/static/js/Changeset.ts) provides tools to create, read, and apply changesets. ## Changeset @@ -21,6 +21,42 @@ A transmitted changeset looks like this: 'Z:z>1|2=m=b*0|1+1$\n' ``` +### Reading a changeset + +`unpack()` splits a changeset string into its parts: + +```javascript +const unpacked = Changeset.unpack('Z:z>1|2=m=b*0|1+1$\n'); +// { oldLen: 35, newLen: 36, ops: '|2=m=b*0|1+1', charBank: '\n' } +``` + +`oldLen` is the document length before the change and `newLen` the length after. +`ops` is the list of operations, and `charBank` holds the characters inserted by +those operations. + +Iterate the operations with `deserializeOps()`, which yields one `Op` at a time: + +```javascript +for (const op of Changeset.deserializeOps(unpacked.ops)) { + console.log(op); +} +// Op { opcode: '=', chars: 22, lines: 2, attribs: '' } +// Op { opcode: '=', chars: 11, lines: 0, attribs: '' } +// Op { opcode: '+', chars: 1, lines: 1, attribs: '*0' } +``` + +There are three kinds of operation, each applied starting from the current +position in the text: + +- `=` keeps text (it may still change the text's attributes, e.g. make it bold). +- `-` removes text. +- `+` inserts text (taking the characters from the changeset's `charBank`). + +`opcode` is the operation type; `chars` and `lines` are how much text it covers; +and `attribs` are the attributes applied, written as `*` references into the +pad's attribute pool. In the example above the final op inserts one character +(the newline from `charBank`) carrying attribute `*0`. + ## Attribute Pool ```javascript @@ -36,6 +72,59 @@ are used many times. There is one attribute pool per pad, and it includes every current and historical attribute used in the pad. +A pool can be serialized to and from a plain object with `toJsonable()` and +`fromJsonable()`: + +```javascript +const pool = new AttributePool(); +pool.fromJsonable({ + numToAttrib: { + 0: ['author', 'a.kVnWeomPADAT2pn9'], + 1: ['bold', 'true'], + 2: ['italic', 'true'], + }, + nextNum: 3, +}); + +pool.getAttrib(1); // [ 'bold', 'true' ] +pool.getAttribKey(1); // 'bold' +pool.getAttribValue(1); // 'true' +``` + +Each attribute is a `[key, value]` pair — `['bold', 'true']`, or +`['author', '']`. A character can carry several attributes (bold *and* +italic), but only one value per key (so it cannot belong to two authors). + +## Attributed text (atext) + +A pad's content is stored as *attributed text* (`atext`): the plain text plus an +attribute string describing which attributes apply to each span. + +```javascript +const atext = { + text: 'bold text\nitalic text\nnormal text\n\n', + attribs: '*0*1+9*0|1+1*0*1*2+b|1+1*0+b|2+2', +}; +``` + +The attribute string is a sequence of `+` operations — the same encoding used by +changesets — which you can read with `deserializeOps()`: + +```javascript +for (const op of Changeset.deserializeOps(atext.attribs)) { + console.log(op); +} +// Op { opcode: '+', chars: 9, lines: 0, attribs: '*0*1' } +// Op { opcode: '+', chars: 1, lines: 1, attribs: '*0' } +// Op { opcode: '+', chars: 11, lines: 0, attribs: '*0*1*2' } +// Op { opcode: '+', chars: 1, lines: 1, attribs: '' } +// Op { opcode: '+', chars: 11, lines: 0, attribs: '*0' } +// Op { opcode: '+', chars: 2, lines: 2, attribs: '' } +``` + +Read against the pool above, the first nine characters (`bold text`) carry +attributes `*0*1` (author + bold), the following newline carries `*0`, and so on. + ## Further Reading Detailed information about the changesets & Easysync protocol: diff --git a/doc/database.md b/doc/database.md new file mode 100644 index 000000000..9d73eb8e4 --- /dev/null +++ b/doc/database.md @@ -0,0 +1,204 @@ +# Database structure + +## Keys and their values + +### groups +A list of all existing groups (a JSON object with groupIDs as keys and `1` as values). + +### pad:$PADID +Contains all information about pads + +- **atext** - the latest attributed text +- **pool** - the attribute pool +- **head** - the number of the latest revision +- **chatHead** - the number of the latest chat entry +- **public** - flag that disables security for this pad +- **passwordHash** - string that contains a salted sha512 sum of this pad's password + +### pad:$PADID:revs:$REVNUM +Saves a revision $REVNUM of pad $PADID + +- **meta** + - **author** - the autorID of this revision + - **timestamp** - the timestamp of when this revision was created +- **changeset** - the changeset of this revision + +### pad:$PADID:chat:$CHATNUM +Saves a chat entry with num $CHATNUM of pad $PADID + +- **text** - the text of this chat entry +- **userId** - the authorID of this chat entry +- **time** - the timestamp of this chat entry + +### pad2readonly:$PADID +Translates a padID to a readonlyID + +### readonly2pad:$READONLYID +Translates a readonlyID to a padID + +### token2author:$TOKENID +Translates a token to an authorID + +### globalAuthor:$AUTHORID +Information about an author + +- **name** - the name of this author as shown in the pad +- **colorID** - the colorID of this author as shown in the pad + +### mapper2group:$MAPPER +Maps an external application identifier to an internal group + +### mapper2author:$MAPPER +Maps an external application identifier to an internal author + +### group:$GROUPID +a group of pads + +- **pads** - object with pad names in it, values are 1 + +### session:$SESSIONID +a session between an author and a group + +- **groupID** - the groupID the session belongs too +- **authorID** - the authorID the session belongs too +- **validUntil** - the timestamp until this session is valid + +### author2sessions:$AUTHORID +saves the sessions of an author + +- **sessionsIDs** - object with sessionIDs in it, values are 1 + +### group2sessions:$GROUPID + +- **sessionsIDs** - object with sessionIDs in it, values are 1 + +# Connecting to a database backend + +Etherpad stores everything in a single key/value table through +[ueberDB](https://www.npmjs.com/package/ueberdb2), so the same data model works +across many backends. The backend is selected with `dbType` in `settings.json`, +and backend-specific connection options go in `dbSettings`. + +The default `dirty` backend writes to a local file (`var/dirty.db`) and needs no +setup, which is convenient for development but not recommended for production. +For a production instance, point Etherpad at a real database such as MySQL/MariaDB, +PostgreSQL or Redis. Etherpad creates its own table on first run; you only need +to provision an empty database and a user with access to it. + +## MySQL / MariaDB + +Create the database and a user, then grant access: + +```sql +CREATE DATABASE `etherpad` CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; +CREATE USER 'etherpad'@'localhost' IDENTIFIED BY 'a-secure-password'; +GRANT CREATE,ALTER,SELECT,INSERT,UPDATE,DELETE ON `etherpad`.* TO 'etherpad'@'localhost'; +``` + +Then configure `settings.json`: + +```json +"dbType": "mysql", +"dbSettings": { + "user": "etherpad", + "host": "localhost", + "port": 3306, + "password": "a-secure-password", + "database": "etherpad", + "charset": "utf8mb4" +} +``` + +Setting `charset` to `utf8mb4` is strongly recommended so that the full range of +Unicode (including emoji) is stored correctly. To connect over a local socket +instead of TCP, replace `host`/`port` with `"socketPath": "/var/run/mysqld/mysqld.sock"`. + +## PostgreSQL + +Create the user and a database owned by it: + +```sql +CREATE USER etherpad WITH PASSWORD 'a-secure-password'; +CREATE DATABASE etherpad OWNER etherpad; +``` + +Then configure `settings.json`: + +```json +"dbType": "postgres", +"dbSettings": { + "user": "etherpad", + "host": "localhost", + "port": 5432, + "password": "a-secure-password", + "database": "etherpad" +} +``` + +The `dbSettings` object is passed straight to the `node-postgres` connection +pool, so any option it accepts (including a single `"connectionString"`) works. +On Debian/Ubuntu you can use peer authentication over the local socket by +setting `"host": "/var/run/postgresql"` and an empty password, provided the +operating-system user that runs Etherpad matches the PostgreSQL role. + +## Redis + +Install Redis and make sure it persists data to disk. Configure `settings.json` +with either discrete fields or a single connection URL: + +```json +"dbType": "redis", +"dbSettings": { + "host": "localhost", + "port": 6379, + "password": "a-secure-redis-password" +} +``` + +```json +"dbType": "redis", +"dbSettings": { + "url": "redis://:a-secure-redis-password@localhost:6379" +} +``` + +## Migrating from MySQL to PostgreSQL + +[pgloader](https://pgloader.io/) can copy an existing Etherpad database from +MySQL to PostgreSQL. Stop Etherpad first so the source database is quiescent. + +```bash +sudo apt-get install postgresql pgloader + +# Create the target role and database +sudo -u postgres createuser etherpad +sudo -u postgres createdb -O etherpad etherpad + +# Describe and run the migration +cat > pgloader.load <<'EOF' +LOAD DATABASE + FROM mysql://etherpad:MYSQL_PASSWORD@127.0.0.1/etherpad + INTO postgresql:///etherpad +WITH preserve index names, prefetch rows = 100 +ALTER SCHEMA 'etherpad' RENAME TO 'public'; +EOF + +pgloader --verbose pgloader.load +``` + +Afterwards set the PostgreSQL user's password and make sure it can read and +write the migrated table: + +```sql +ALTER USER etherpad WITH PASSWORD 'a-secure-password'; +GRANT pg_read_all_data TO etherpad; +GRANT pg_write_all_data TO etherpad; +``` + +Then point `settings.json` at PostgreSQL as shown above and start Etherpad. + +::: tip +To move data between *any* two backends supported by ueberDB, you can also +use the `migrateDB` CLI tool, which reads every record from a source database +descriptor and writes it to a target one. See the [CLI chapter](./cli.md). +::: diff --git a/doc/deployment.md b/doc/deployment.md new file mode 100644 index 000000000..2c5ab5d0c --- /dev/null +++ b/doc/deployment.md @@ -0,0 +1,359 @@ +# Deployment + +This page collects working configurations for deploying Etherpad in production: +running it behind a reverse proxy, hosting it under a subdirectory, terminating +HTTPS natively, running it as a system service, and deploying it on Kubernetes. + +Etherpad listens on port `9001` by default. Throughout this page the upstream +Etherpad server is assumed to be reachable at `http://127.0.0.1:9001`. + +## Running behind a reverse proxy + +The recommended production setup is to run Etherpad on `127.0.0.1:9001` and put a +reverse proxy in front of it to terminate TLS, serve a virtual host, and forward +requests. + +Etherpad uses WebSockets (via socket.io). The load-bearing part of every proxy +config below is the WebSocket upgrade: the proxy **must** forward the `Upgrade` +and `Connection` headers, or real-time editing will silently fail back to slow +long-polling (or break entirely). + +When Etherpad runs behind a proxy you should also set `trustProxy: true` in your +settings so that Etherpad honours the `X-Forwarded-*` headers (correct client IP, +secure-cookie flag, etc.). See the `trustProxy` section in the [Configuration documentation](./configuration.md) for the full details of which headers are trusted. + +### Nginx + +```nginx +# Map the Upgrade header so WebSockets work. Place this in the http context. +map $http_upgrade $connection_upgrade { + default upgrade; + '' close; +} + +server { + listen 443 ssl; + listen [::]:443 ssl; + server_name pad.example.com; + + ssl_certificate /etc/nginx/ssl/etherpad.crt; + ssl_certificate_key /etc/nginx/ssl/etherpad.key; + + location / { + proxy_pass http://127.0.0.1:9001; + proxy_buffering off; + proxy_set_header Host $host; + proxy_pass_header Server; + + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # WebSocket support + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + } +} + +# Redirect plain HTTP to HTTPS +server { + listen 80; + listen [::]:80; + server_name pad.example.com; + return 301 https://$host$request_uri; +} +``` + +### Apache + +Enable `mod_proxy`, `mod_proxy_http`, `mod_proxy_wstunnel` and `mod_headers`. +The `mod_proxy_wstunnel` `upgrade=websocket` syntax requires Apache 2.4.47 or +newer. + +```apache + + ServerName pad.example.com + + SSLEngine on + SSLCertificateFile /etc/ssl/etherpad/etherpad.crt + SSLCertificateKeyFile /etc/ssl/etherpad/etherpad.key + + ProxyVia On + ProxyRequests Off + ProxyPreserveHost On + + # WebSocket traffic (socket.io) must be matched first. + + ProxyPass "ws://127.0.0.1:9001/socket.io" upgrade=websocket timeout=30 + ProxyPassReverse "ws://127.0.0.1:9001/socket.io" + + + + ProxyPass "http://127.0.0.1:9001/" retry=0 timeout=30 + ProxyPassReverse "http://127.0.0.1:9001/" + + +``` + +### Caddy + +Caddy v2 proxies WebSocket connections automatically and obtains/renews a +certificate for you, so the configuration is minimal: + +```caddy +pad.example.com { + reverse_proxy 127.0.0.1:9001 +} +``` + +### Traefik + +Traefik v2 also proxies WebSockets transparently. For a Docker deployment, attach +these labels to the Etherpad container: + +```yaml +labels: + - "traefik.enable=true" + - "traefik.http.routers.etherpad.rule=Host(`pad.example.com`)" + - "traefik.http.routers.etherpad.entrypoints=websecure" + - "traefik.http.routers.etherpad.tls.certresolver=myresolver" + - "traefik.http.services.etherpad.loadbalancer.server.port=9001" + - "traefik.http.services.etherpad.loadbalancer.passhostheader=true" +``` + +### HAProxy + +HAProxy detects the `Connection: Upgrade` exchange automatically and switches to +tunnel mode once the WebSocket is established. The important value is +`timeout tunnel`, which governs the lifetime of the upgraded connection. + +```haproxy +frontend http + mode http + bind *:80 + bind *:443 ssl crt /etc/haproxy/certs/etherpad.pem alpn h2,http/1.1 + http-request redirect scheme https code 301 unless { ssl_fc } + http-request add-header X-Forwarded-Proto https if { ssl_fc } + default_backend etherpad + +backend etherpad + mode http + option forwardfor + timeout client 25s + timeout server 25s + timeout tunnel 3600s + server pad 127.0.0.1:9001 +``` + +## Hosting under a subdirectory + +To serve Etherpad from a path such as `https://example.com/pad` rather than from +the root of a domain, the proxy must send the `X-Proxy-Path` header so that +Etherpad rewrites its own asset and API URLs to include the prefix. This header +is honoured regardless of the `trustProxy` setting — see the [Configuration documentation](./configuration.md). + +```nginx +location /pad/ { + rewrite ^/pad/(.*)$ /$1 break; + proxy_pass http://127.0.0.1:9001; + proxy_buffering off; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Proxy-Path /pad; + + # WebSocket support + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; +} +``` + +## Native HTTPS without a proxy + +Etherpad can terminate TLS itself using Node's native HTTPS server, with no +reverse proxy required. Configure the `ssl` block in `settings.json`: + +```json +"ssl": { + "key": "/path-to-your/etherpad-server.key", + "cert": "/path-to-your/etherpad-server.crt", + "ca": ["/path-to-your/intermediate-cert1.crt", "/path-to-your/intermediate-cert2.crt"] +} +``` + +* `key` — path to the private key file. +* `cert` — path to the certificate file. +* `ca` — an (optional) array of intermediate/chain certificate paths. + +Restart Etherpad after editing the settings. It will now serve HTTPS on its +configured port. + +For local testing you can generate a self-signed certificate with a single +command: + +```bash +openssl req -x509 -newkey rsa:4096 -nodes -days 365 \ + -keyout etherpad-server.key -out etherpad-server.crt \ + -subj "/CN=localhost" +``` + +Make sure the files are readable only by the user that runs Etherpad: + +```bash +chmod 400 etherpad-server.key etherpad-server.crt +chown etherpad etherpad-server.key etherpad-server.crt +``` + +::: tip +Self-signed certificates trigger browser warnings and are only suitable for +testing. For production, obtain a free, trusted certificate from +[Let's Encrypt](https://letsencrypt.org/), or terminate TLS at a reverse proxy +(see above) and let it manage certificate issuance and renewal. +::: + +## Running as a service (systemd) + +On a modern Linux distribution, run Etherpad as a `systemd` service so it starts +on boot and restarts automatically on failure. + +Create a dedicated unprivileged user and install Etherpad into its home +directory (for example `/opt/etherpad`), owned by that user. Etherpad refuses to +start as root. + +Create `/etc/systemd/system/etherpad.service`: + +```ini +[Unit] +Description=Etherpad collaborative editor +After=network.target + +[Service] +Type=simple +User=etherpad +Group=etherpad +WorkingDirectory=/opt/etherpad +Environment=NODE_ENV=production +ExecStart=/usr/bin/pnpm run prod +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +``` + +Adjust `WorkingDirectory` to your install path and the `ExecStart` path to +wherever `pnpm` lives (`which pnpm`). Then enable and start the service: + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now etherpad.service + +# check status and follow logs +sudo systemctl status etherpad.service +sudo journalctl -u etherpad.service -f +``` + +## Kubernetes (Istio) + +The following manifest deploys Etherpad behind an Istio ingress gateway. It +defines three resources: a `Gateway` (TLS + hostname), a `VirtualService` +(routing with WebSocket-friendly timeouts), and a `DestinationRule` (sticky +sessions via the socket.io `io` cookie). + +It assumes: + +* Istio >= 1.18 +* A `Service` named `etherpad` in the `etherpad` namespace, on port `9001` +* A TLS secret `etherpad-tls` provisioned in the gateway namespace +* You replace `` with your own hostname + +::: warning +Sticky sessions are necessary but **not** sufficient for a multi-replica +Etherpad deployment. Multi-replica also needs the socket.io Redis adapter so +that pad state is shared across pods. Without it, two clients editing the same +pad but routed to different pods will see divergent state. + +Recommendation: start with `replicas: 1` plus good failover, and only go +multi-replica once the Redis adapter is wired up. +::: + +```yaml +apiVersion: networking.istio.io/v1beta1 +kind: Gateway +metadata: + name: etherpad + namespace: etherpad +spec: + selector: + istio: ingressgateway + servers: + - port: + number: 443 + name: https + protocol: HTTPS + tls: + mode: SIMPLE + credentialName: etherpad-tls + hosts: + - + - port: + number: 80 + name: http + protocol: HTTP + hosts: + - + tls: + httpsRedirect: true + +--- +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: etherpad + namespace: etherpad +spec: + hosts: + - + gateways: + - etherpad + http: + - match: + - uri: + prefix: / + route: + - destination: + host: etherpad + port: + number: 9001 + # No per-request timeout — websockets and long-polling sit on the + # connection indefinitely. The default of 15s kills WS upgrades. + timeout: 0s + +--- +apiVersion: networking.istio.io/v1beta1 +kind: DestinationRule +metadata: + name: etherpad + namespace: etherpad +spec: + host: etherpad + trafficPolicy: + loadBalancer: + # Sticky sessions on the socket.io session cookie. Required so that + # long-polling fallback requests land on the same pod that owns the + # session state. + consistentHash: + httpCookie: + name: io + ttl: 0s # session cookie, expires with the browser tab + connectionPool: + tcp: + maxConnections: 10000 + http: + # Must comfortably exceed socket.io's pingInterval (25s) + + # pingTimeout (20s). 1h is conservative. + idleTimeout: 3600s + h2UpgradePolicy: UPGRADE + http1MaxPendingRequests: 1000 +``` diff --git a/doc/development.md b/doc/development.md new file mode 100644 index 000000000..4a0780ed7 --- /dev/null +++ b/doc/development.md @@ -0,0 +1,240 @@ +# Development + +This page is a contributor-oriented tour of the Etherpad source tree and of a +few internals that plugin authors and core contributors commonly need to +understand: how the source is laid out, how pads are converted to and from +other formats, and how to access the database from server-side code. + +The Etherpad server is written in TypeScript (`.ts`). Most server code lives +under `src/node/` and most client code under `src/static/js/`. + +## Source tree overview + +The repository root contains, among others, the following directories: + +``` +etherpad/ +|- bin/ # maintenance and build scripts (run.sh, pad tools, docs, release) +|- doc/ # this manual, in AsciiDoc and Markdown +|- src/ # the Etherpad source code +|- packaging/ # OS/distribution packaging helpers +|- var/ # runtime data (e.g. the dirty.db database file) +``` + +`bin/` contains scripts for running and maintaining Etherpad. For example +`bin/run.sh` starts the server, and there are TypeScript utilities such as +`bin/checkPad.ts`, `bin/deletePad.ts`, `bin/repairPad.ts`, +`bin/rebuildPad.ts`, `bin/migrateDB.ts` and `bin/make_docs.ts`. + +The HTML manual is built from the AsciiDoc sources in `doc/` by +`bin/make_docs.ts` (exposed as the `makeDocs` script), which shells out to +`asciidoctor` and writes the result to `out/doc/`. From the repository root you +can run it with `pnpm run makeDocs`. (`asciidoctor` must be installed.) + +The `src/` directory looks like this: + +``` +src/ +|- locales/ # translations, managed via https://translatewiki.net +|- node/ # server-side code +|- static/ # client-side code, CSS and fonts +|- templates/ # server-rendered page templates +|- ep.json # core plugin/hook registration +|- package.json # package name: ep_etherpad-lite +``` + +### src/node/ (server side) + +``` +src/node/ +|- db/ # database access and pad/author/group/session state +|- eejs/ # server-side embedded-JS templating +|- handler/ # import/export and collaboration message handling +|- hooks/ # express route registration and i18n +|- security/ # crypto, OAuth2/OIDC, secret rotation +|- types/ # shared TypeScript types +|- updater/ # in-place self-update machinery +|- utils/ # settings, import/export format helpers, toolbar, minification +|- server.ts # entry point +``` + +`db/` contains the modules that read and write pad state. `Pad.ts` manages an +individual pad; `PadManager.ts`, `AuthorManager.ts`, `GroupManager.ts`, +`SessionManager.ts` and `ReadOnlyManager.ts` manage the corresponding records; +`DB.ts` exposes the low-level key/value store (see +[Accessing the database from server code / plugins](#accessing-the-database-from-server-code-plugins)); and `API.ts` implements +the public HTTP API. + +`handler/` contains the request and message handlers. `PadMessageHandler.ts` +drives real-time collaboration, while `ImportHandler.ts` and `ExportHandler.ts` +handle import and export. + +`hooks/` contains mostly Express-related code. `i18n.ts` builds the translation +files and registers routes to serve them, and `hooks/express/` registers the +routes that serve pads, the timeslider, static assets and the admin pages. + +`utils/` contains the import/export format converters (`ImportHtml.ts`, +`ExportHtml.ts`, `ExportTxt.ts`, `ExportEtherpad.ts`, `ImportEtherpad.ts`, +`ExportHelper.ts`, and native converters such as `ExportPdfNative.ts` and +`ImportDocxNative.ts`), the settings parser (`Settings.ts`), the toolbar builder +(`toolbar.ts`) and the asset minifier (`Minify.ts`). + +### src/static/ (client side) + +``` +src/static/ +|- css/ # stylesheets, including css/pad/icons.css +|- font/ # web fonts, including the fontawesome-etherpad icon font +|- img/ +|- js/ # client-side TypeScript +|- skins/ # bundled UI skins +|- vendor/ +``` + +`js/` contains the client-side editor code. Notable modules include +`ace2_inner.ts` and `ace2_common.ts` (the editor core), `contentcollector.ts`, +`linestylefilter.ts` and `domline.ts` (content/attribute processing, shared +with the server import/export pipeline), `Changeset.ts` and `AttributePool.ts` +(the changeset and attribute model), and `collab_client.ts` (the +client side of real-time collaboration). + +### src/templates/ + +`templates/` contains the server-rendered page templates for the index, the +pad, the timeslider and the admin pages, plus the bootstrap scripts that load +the client bundles. The templates expose named `eejs` blocks that plugins can +hook into to inject custom HTML. + +## How Etherpad converts pads to and from other formats + +Internally a pad is not stored as HTML. A pad is a sequence of lines, and each +line carries **attributes** (for example `heading1`, `bullet` or a list number). +The set of attributes that a pad can use is stored in its **attribute pool**; the +pool only records which attributes exist, not where they are applied. The +pool grows over the history of the pad. + +Where an attribute is applied to a line is recorded in an **attribute string**, +and a line that carries a line-level attribute is prefixed with a **line marker** +(`lmkr`). Attribute strings and changesets are defined by +`src/static/js/Changeset.ts` and `src/static/js/AttributePool.ts`. + +### Collecting content + +`src/static/js/contentcollector.ts` is the shared starting point for both the +client (when content is typed or pasted) and the server (when content is +imported). It walks the incoming DOM/HTML, decides which attributes apply to +each line, adds the discovered attributes to the attribute pool, and emits the +resulting attribute strings. On import, `src/node/utils/ImportHtml.ts` calls +`contentcollector.makeContentCollector(...)` to do exactly this, and the HTML +import path in `src/node/handler/ImportHandler.ts` ultimately drives it. + +### From attributes to HTML/text (export) + +On export the flow is, conceptually: + +``` +contentcollector.ts + -> linestylefilter.ts + -> ExportHtml.ts / ExportTxt.ts (helped by ExportHelper.ts) + -> ExportHandler.ts + -> the HTTP API / /export/* route +``` + +- `src/static/js/linestylefilter.ts` walks each line, reads its attributes, + and turns them into the classes/markup the line should render with. +- `src/node/utils/ExportHelper.ts` adds export-only logic that does not belong + in the live editor. The clearest example is lists: in the editor each list + item is rendered as its own line-level block, but a clean export needs the + items collapsed into a single properly nested list. The helper performs that + reshaping for export only. +- `src/node/utils/ExportHtml.ts` and `src/node/utils/ExportTxt.ts` (and + `ExportEtherpad.ts` for the native `.etherpad` format) turn the attributed + text (`atext`) into the final HTML or plain text. +- `src/node/handler/ExportHandler.ts` receives the export request and dispatches + on the requested format — for instance, office formats such as `.docx` and + `.pdf` are routed through the native converters / LibreOffice rather than + through the plain HTML/text path. + +On the client side, edits are turned into changesets by the editor, attributes +are translated into CSS classes (so `heading2` becomes +`class="heading2"`), and `src/static/js/domline.ts` (`createDomLine`) renders +the final DOM for each line. + +## Accessing the database from server code / plugins + +Etherpad stores everything in a single key/value store backed by +[ueberDB](https://www.npmjs.com/package/ueberdb2), which abstracts over the +configured database (dirtyDB, MySQL/MariaDB, PostgreSQL, SQLite, MongoDB, Redis, +and others). Server-side code and plugins access it through +`src/node/db/DB.ts`. + +The package name of the core module is, for historical reasons, still +`ep_etherpad-lite`, so plugins import the database module like this: + +```javascript +const db = require('ep_etherpad-lite/node/db/DB'); +``` + +The exposed methods are asynchronous and return promises (use `await`), not the +old callback style. The available methods are `get`, `set`, `remove`, `getSub`, +`setSub`, `findKeys` and `findKeysPaged`: + +```javascript +// Read a record (returns undefined/null if it does not exist) +const value = await db.get('record_key'); + +// Create or replace a record +await db.set('record_key', data); + +// Read or write a nested value inside a record +const colorId = await db.getSub('author_key', ['colorId']); +await db.setSub('author_key', ['email'], 'tutti@frutti.org'); + +// Delete a record +await db.remove('record_key'); +``` + +For example, given the author record: + +```json +{"colorId":"#79d9d9","name":"tutti","timestamp":1364832712430,"padIDs":{"mypad":1}} +``` + +calling `await db.setSub('author_key', ['email'], 'tutti@frutti.org')` yields: + +```json +{"colorId":"#79d9d9","name":"tutti","timestamp":1364832712430,"padIDs":{"mypad":1},"email":"tutti@frutti.org"} +``` + +::: warning +Keys are namespaced (for example `pad:`, +`pad::revs:`, `globalAuthor:`). Prefer the high-level +managers (`Pad.ts`, `AuthorManager.ts`, etc.) over direct `DB` access where one +exists; reach for `DB` directly only for data your plugin owns, and use a key +prefix unique to your plugin to avoid collisions. +::: + +## Adding a toolbar icon + +Etherpad's toolbar icons come from the bundled `fontawesome-etherpad` icon +font in `src/static/font/`. Toolbar buttons reference an icon by a +`buttonicon-` CSS class (see `src/node/utils/toolbar.ts`, which builds +each button's class as `buttonicon buttonicon-`), and those classes are +defined in `src/static/css/pad/icons.css`. The font itself is generated with +[Fontello](http://fontello.com) from `src/static/font/config.json` (whose +`css_prefix_text` is `buttonicon-`). + +To add a new icon: + +1. Go to [Fontello](http://fontello.com) and import the existing + `src/static/font/config.json` (Fontello's "import" loads the current icon + set and pre-selects the icons it contains). +2. Select the additional icon(s) you want, then click **Download webfont**. +3. From the unzipped download, copy `config.json` and the + `font/fontawesome-etherpad.*` files over the ones in `src/static/font/`. +4. From the unzipped `css/fontawesome-etherpad.css`, copy the new + `.buttonicon-:before { content: '\\eXXX'; }` rules into + `src/static/css/pad/icons.css`, replacing the existing block of icon rules. + +The icon is then available wherever a `buttonicon-` class can be used, +including toolbar button definitions. diff --git a/doc/faq.md b/doc/faq.md new file mode 100644 index 000000000..8f2a81a58 --- /dev/null +++ b/doc/faq.md @@ -0,0 +1,204 @@ +# FAQ + +This page answers common operational questions about running and maintaining +an Etherpad instance. It collects material previously kept on the project wiki. + +## How do I install Etherpad? + +There are several supported ways to install Etherpad. Pick whichever suits your +environment. + +### Docker + +The official image is published to Docker Hub (`etherpad/etherpad`) and to the +GitHub Container Registry (`ghcr.io/ether/etherpad`) with identical tags. + +```bash +docker pull etherpad/etherpad +docker run -p 9001:9001 etherpad/etherpad +``` + +See the [Docker chapter](./docker.md) for building personalized images, enabling plugins, and +configuring office-format import/export. + +### One-line installer (macOS / Linux / WSL) + +```bash +curl -fsSL https://raw.githubusercontent.com/ether/etherpad/master/bin/installer.sh | sh +``` + +On Windows (PowerShell): + +```powershell +irm https://raw.githubusercontent.com/ether/etherpad/master/bin/installer.ps1 | iex +``` + +The installer clones Etherpad, installs dependencies and builds the frontend. +Set `ETHERPAD_RUN=1` to also start it once the install finishes. + +### apt repository (Debian / Ubuntu) + +Etherpad publishes a signed APT repository (`stable` channel). Import the signing +key, add the repository and install: + +```bash +curl -fsSL https://etherpad.org/key.asc \ + | sudo gpg --dearmor -o /usr/share/keyrings/etherpad.gpg + +echo "deb [signed-by=/usr/share/keyrings/etherpad.gpg] https://etherpad.org/apt stable main" \ + | sudo tee /etc/apt/sources.list.d/etherpad.list + +sudo apt-get update +sudo apt-get install etherpad +``` + +The repository provides `amd64` and `arm64` builds. Etherpad depends on +Node.js >= 24, so on older distributions you may also need NodeSource's apt +repository to satisfy that dependency. + +### From source + +Etherpad requires [Node.js](https://nodejs.org/) >= 24 and `pnpm`. + +```bash +git clone -b master https://github.com/ether/etherpad +cd etherpad +pnpm i +pnpm run build:etherpad +pnpm run prod +``` + +Then open `http://localhost:9001`. + +## What URL paths does Etherpad serve? + +| Path | Description | +|------|-------------| +| `/admin` | Administration dashboard (requires admin login). | +| `/admin/plugins` | Install, update and remove plugins from the web UI. | +| `/admin/settings` | Edit `settings.json` from the web UI. | +| `/p/:padID` | Open (or create) the pad with the given `padID`, e.g. `/p/foo`. | +| `/p/:padID/timeslider` | Open the pad's history/timeslider view. Append `#N` to jump to a specific revision, e.g. `/p/foo/timeslider#5`. | +| `/p/:padID/export/:type` | Export the pad in the given format, e.g. `/p/foo/export/html`. Append `?revs=N` to export a specific revision. | + +Supported export types: + +- **Native (no extra dependencies):** `txt`, `html`, `etherpad`, `docx`, `pdf`. +- **Via LibreOffice:** `odt`, `doc`, `rtf` — these require the `soffice` setting +to point at a LibreOffice executable. See the office-format notes in the +[Docker chapter](./docker.md). + +## How do I list all pads? + +The recommended way is the HTTP API method `listAllPads`, combined with `jq`: + +```bash +ETHERPAD_HOST='https://pad.example.com' +ETHERPAD_API_KEY='...' # the APIKEY.txt file in the Etherpad root +ETHERPAD_API_VERSION='...' # see https://pad.example.com/api + +curl -s "${ETHERPAD_HOST}/api/${ETHERPAD_API_VERSION}/listAllPads?apikey=${ETHERPAD_API_KEY}" \ + | jq -r '.data.padIDs[]' +``` + +For an interactive list with management actions, install the `ep_adminpads2` +plugin and browse to `/admin/pads`. + +As a last resort you can query the database directly. The exact query depends on +your configured backend; pad records use keys of the form `pad:` and +`pad::revs:`. For example, with SQLite: + +```bash +sqlite3 ./var/sqlite.db "select key from store where key like 'pad:%'" \ + | grep -Eo '^pad:[^:]+' \ + | sed -e 's/pad://' \ + | sort -u +``` + +Prefer the API or admin plugin over direct SQL: the schema is an implementation +detail and may change. + +## How do I delete or manage pads? + +Use the HTTP API `deletePad` method: + +```bash +curl -s "${ETHERPAD_HOST}/api/${ETHERPAD_API_VERSION}/deletePad?apikey=${ETHERPAD_API_KEY}&padID=foo" +``` + +The API also offers `copyPad`, `movePad`, `getRevisionsCount` and more — see the +[HTTP API chapter](./api/http_api.md). + +For a web UI, install the `ep_adminpads2` plugin and manage pads from +`/admin/pads`, where you can search, view and delete pads. + +The `deletePad` CLI tool is also available for operators: + +```bash +pnpm run --filter bin deletePad +``` + +## How do I back up and restore pads? + +### Back up the whole instance + +All pad data lives in the configured database. Back it up using the tool +appropriate to your backend (for example `mysqldump` for MySQL/MariaDB, +`pg_dump` for PostgreSQL, or a file copy of `var/*.db` for the file-based +`dirty`/`rusty` engines while Etherpad is stopped). A regular, automated dump of +the database is the canonical backup for a production instance. + +### Back up a single pad + +Export the pad over HTTP by appending `/export/` to its URL. Plain text, +HTML and the round-trippable `etherpad` format are most useful for backups: + +```bash +curl -o mypad.txt https://pad.example.com/p/foo/export/txt +curl -o mypad.html https://pad.example.com/p/foo/export/html +curl -o mypad.etherpad https://pad.example.com/p/foo/export/etherpad +``` + +The `etherpad` export preserves the pad's full history and can be re-imported, +making it the best choice for migrating or archiving an individual pad. + +### Restore or inspect an old revision + +Every state the pad has been in is stored in the database, so you can retrieve +an earlier revision without a separate backup: + +- Open `/p/:padID/timeslider` to browse the history and find the revision +number you want. +- Export a specific revision directly with the `?revs=N` query parameter, e.g. +`https://pad.example.com/p/foo/export/html?revs=1000`. + +### Repairing a damaged pad + +If a pad is corrupt, use the CLI repair tools (`checkPad`, `repairPad`, +`rebuildPad`) documented in the [CLI chapter](./cli.md). Always back up the database before +running write operations. + +## How do I limit history or prune revisions? + +Etherpad keeps the full revision history of every pad, so the database grows +over time. To reclaim space, use the pad-compaction CLI tools, which collapse or +trim revision history for one pad, every pad, or only stale pads: + +```bash +# Collapse all history of one pad +pnpm run --filter bin compactPad + +# Keep only the last 50 revisions of one pad +pnpm run --filter bin compactPad --keep 50 + +# Compact every pad on the instance +pnpm run --filter bin compactAllPads + +# Compact only pads not edited in the last 90 days, keeping the last 50 revisions +pnpm run --filter bin compactStalePads --older-than 90 --keep 50 +``` + +These tools require `cleanup.enabled = true` in `settings.json` and are +**destructive** — history is collapsed or trimmed. Export anything you can't +afford to lose via the pad's `/export/etherpad` route first. The same primitive +is available over the wire as the `compactPad` HTTP API method. See the [CLI chapter](./cli.md) for full details.