MessageWeave/README.md

# MessageWeave

**[中文文档 / Chinese documentation → README.zh-CN.md](README.zh-CN.md)**

> One Telegram bot per mailbox. New mail arrives as metadata; the original is always
> fetched read-only from your own JMAP server.

## 1. What it is

MessageWeave sits between a JMAP mail server (for example Stalwart) and a Telegram bot.

- **Inbound.** A JMAP push subscription tells the service that a message changed. Nothing
  is polled at rest.
- **Outbound.** Each event becomes a short, **metadata-only** Telegram notification:
  sender, subject, timestamp. Mail bodies are never sent to Telegram.
- **On demand.** When you ask, the service reads the original through a fresh JMAP read
  call. Nothing is cached in between.
- **Optionally summarised.** If you explicitly opt in, an OpenAI-compatible LLM returns a
  short summary. Requests, responses and attachment bytes are never persisted.

State is an architectural boundary here, not a preference.

- **Redis is the only state store.** No database, no embedded store, no second store.
- **The process writes nothing to disk.** No log files, no data files, no local volume.
- **No long-lived connections.** No SSE, no WebSocket, no long polling.
- **Logs go to stdout only.** The platform's collector owns log storage and retention.
- **What it is not.** It is not a mail client, not a proxy, and not a relay. It forwards
  metadata to one bot; it does not read arbitrary URLs, open tunnels, or broker traffic. It has
  no resident `server` mode flag, holds no EventSource/SSE or Telegram long-polling connection,
  persists nothing to SQLite or a local volume, and is not bound to a specific serverless platform
  — every notification path is a short request-response.

## 2. Two ways to run it

One image, one Redis, one SPA. The only difference is what sits in front and which URL you hand
to Telegram and Stalwart.

**The Cloudflare Worker load balancer is optional.** Nothing in the backend requires it, and the
backend runs identically with or without it. It exists for one reason: Cloudflare's Load
Balancer product is not available on the free plan, so a Worker is the way a free-plan account
puts load sharing and failover in front of several backends.

**It is also not a security layer, and it is not meant to be one.** There is no intent to hide
the backends and no intent to harden them; the origins stay directly reachable whether the
load balancer is present or not, and no access control is added at the edge. The fixed route safelist
is what makes it a load balancer for a bounded set of paths, not a firewall.

| | Form A: backend origin only | Form B: behind the Cloudflare Worker load balancer |
|---|---|---|
| What answers requests | one backend host | one Worker URL, N backend origins |
| Failover | none — a down origin stays down | bounded: 2 attempts by default, only timeout and `5xx` retry |
| Callback URL you register | `https://a.example` | `https://lb.example` |
| Backend access | direct from the network | equally direct — the load balancer adds no access control |
| Load balancer route set | n/a — there is no load balancer | fixed at 19 paths; unknown paths answer `404` |
| Health to watch | `GET /ready` on the origin | `GET /ready` proxied, plus `GET /healthz-worker` for the load balancer aggregate |
| Extra moving parts | none | one Worker deploy, one origin list |
| Docs | §3 below | §2 above, §6, `cloudflare-worker/README.md` |

Pick Form A for a single origin with no HA requirement. Pick Form B for two or more origins, or
whenever you want one stable callback URL while you roll or rebuild an instance.

Switching is a re-registration, not a migration. Registration is idempotent and registering a
new address retires the previous subscription, so moving from Form A to Form B, or back, is one
field in the SPA. Do not register both origins at once: there is a single registration slot, and
the platform entrance must stay internal (`C-LB-SINGLE-REG-URL`).

No secret moves to the load balancer in Form B. It holds no business credential and checks no auth
header. Instead every backend must carry the same `SAF-AUTH-*` secrets and the same encrypted
business config, because a callback can land on any instance and no instance can prove it was
the one asked for (`C-LB-SHARED-SECRETS`). Mis-matched secrets show up as a random 401, not as
a routing error.

`POST /api/bootstrap` and `/debug/*` are not in the load balancer's route set, so requests addressed
to the Worker URL never reach them. That is the safelist working as designed for the paths that
need to be load balanced; it is not hiding anything, because the same origins are reachable at
their own addresses in both forms.

## 3. Setup, from zero

This section is a hand-holding walkthrough: create a Telegram bot, deploy one backend, fill
the configuration page, register the callbacks, get the first notification. Field semantics,
defaults, TTLs and error codes live in `docs/reference.md`; production, scaling and
load-balancer operation live in §6 below.

### 3.0 What you need

- A mailbox on a JMAP server you administer (this guide assumes Stalwart), HTTPS only.
- A Telegram account.
- A Redis endpoint with persistence (Upstash's free tier works; it is TLS-only, so the URL
  starts with `rediss://`).
- A machine or platform that can run one container and expose one HTTP port.
- One public HTTPS origin that can receive webhooks — your own domain, a tunnel, or a
  Cloudflare route. Telegram and Stalwart both require HTTPS.
- Optional: an OpenAI-compatible LLM key, a Cloudflare account if you want the load balancer.

Nothing else is required: no Postgres, no message broker, no Kubernetes.

The platform or proxy terminates TLS; the process listens on one HTTP port (`PORT`, default
`8080`) and exposes no other TCP port, and the public HTTPS URL is provided by the platform, not
the bot. The Docker image uses a Rust builder and a Debian slim runtime, links `rustls` with
native roots for outbound TLS, and carries no secret in its layers or build arguments. Back up
Redis with AOF persistence — it is the only state store.

### 3.1 Create the Telegram bot

1. In Telegram open a chat with `@BotFather` and send `/newbot`.
2. Pick a display name, then a username that ends in `bot` or `_bot`.
3. BotFather replies with a token that looks like `1234567890:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsaw1`.
   That string is the bot token. Treat it like a password: anyone holding it can read your
   webhook traffic and send messages as your bot. It is never echoed back by the API once saved.
4. Send the new bot any message (for example `/start`) from the account or group that should
   receive notifications. Until you do, Telegram has no chat to deliver to.

### 3.2 Find the chat id

1. Open `https://api.telegram.org/bot<TOKEN>/getUpdates` in a browser, replacing `<TOKEN>` with
   the token from step 1.
2. In the JSON, find `result[].message.chat.id`. Private chats give a positive number such as
   `123456789`; groups and supergroups give a negative one such as `-1001234567890`.
3. For a group, add the bot to the group first, then send a message in the group before you call
   `getUpdates`. The bot only sees messages in groups after you disable privacy mode under
   `@BotFather` → `/setprivacy` → `Disable`, otherwise group messages never reach it.

You will use this id twice in step 6: as the notification target and as a member of the inbound
allowlist. The allowlist must contain at least one id — an empty allowlist is rejected as invalid
configuration rather than treated as "allow everyone".

### 3.3 Get the JMAP credentials

1. In your Stalwart admin, create an application password for the mailbox — an app password, not
   the account login password. The backend authenticates with HTTP Basic over JMAP.
2. The session URL is your mail origin: `https://mail.example.com`. You may also paste
   `https://mail.example.com/.well-known/jmap`; the backend collapses both to the origin, and
   rejects any URL with embedded credentials, a query string or a fragment.
3. The JMAP account id is optional. Leave it empty: after the first successful save the backend
   reads it back from the server and the page shows it.

### 3.4 Run the backend

The container needs exactly two environment variables:

```sh
# .env — never commit this file
REDIS_URL=rediss://default:<password>@<host>:6379
CONFIG_ENCRYPTION_KEY=<64 hex characters, from: openssl rand -hex 32>
```

- `REDIS_URL` is your only state store. Back it up like a database.
- `CONFIG_ENCRYPTION_KEY` doubles as the SPA admin credential and the encryption key for stored
  business secrets. If you lose it you can no longer read the configuration page or decrypt the
  stored secrets; if you rotate it, previously stored secrets become unreadable.

Then build and start:

```sh
docker build -t messageweave .
docker run --rm --env-file .env -p 8080:8080 messageweave
curl localhost:8080/healthz   # → ok
```

For local builds without Docker: `cargo build --locked` produces the `message-weave` binary.

`GET /healthz` is a static liveness reply and never touches Redis. The readiness report is
`GET /ready`: it answers 503 until the configuration exists and every configured upstream is
reachable, and 200 with a JSON component report once they are. Point your HTTPS reverse proxy or
tunnel at this port now — steps 6 and 7 need a public HTTPS origin.

If the service answers requests but does nothing useful, check the logs for a startup warning:
a missing required variable does **not** crash the process. It serves a read-only setup router
over in-memory state instead. See §4.

### 3.5 Sign in to the configuration page

1. Open the page in a browser. If the backend started with the required variables the first card
   is **Create admin session**.
2. Paste `CONFIG_ENCRYPTION_KEY` — not the Redis password. This mints an admin session that lives
   only in the page's memory for 30 minutes; reloading the page logs you out.

### 3.6 Fill in the business configuration

The form is a full replacement, not a patch: whatever you submit becomes the whole stored
configuration. The sections:

- **01 JMAP mailbox** — session URL and username, plus the app password from step 3.3.
- **02 Telegram and access auth** — bot token, notification chat id, allowlist, and three secrets
  you make up yourself: webhook secret, reconcile token and worker token. Generate them with
  `openssl rand -hex 32` each. The webhook secret must match on the Telegram side later; the
  reconcile and worker tokens protect the scheduler endpoints.
- **03 LLM service** — optional. Leave disabled unless you want AI digests; if you enable it, the
  HTTPS base URL, model name and API key all become required.
- **04 Notification display** — the time zone used to render received times in notifications.

Then, in order:

1. Click **Test connection**. It preflights the credentials without saving anything.
2. Click **Save business config and hot-reload**. The first save must contain every required
   field or the server answers 422 and stores nothing.
3. Read the result line under the button: `runtime_applied: true` means the new runtime was
   installed on this backend. `false` means it saved but the runtime was not installed, and the
   `warnings` array says which component failed — most often an unreachable `jmap_session_url`.
   Fix the field and save again; `/ready` stays 503 until it goes green.
4. If the save answers **409 conflict**, someone else (another tab, a second operator) saved in
   between. Reload the page to pick up their values, then re-apply your edit.

Secrets are write-only: the page never shows a stored secret again, and submitting an empty secret
field keeps the previously stored value — clear a secret deliberately in a separate edit.

If you run multiple backends behind a load balancer, configure on **any one node** — all backends
share one Redis and read the same configuration. See [§6.1](#61-multi-instance-prerequisites) for the full model.

### 3.7 Register the callbacks

Still on the same page, under **External callbacks**:

1. Paste the public HTTPS origin that reaches this backend, scheme and host only, no path —
   for example `https://mw.example.com`. If you put the Cloudflare load balancer in front, use its
   origin, not the backend's: callbacks must survive a backend restart, and the load balancer is
   what stays up.
2. Click **Register callbacks**. One click registers both directions: the Telegram webhook at
   `/webhook/tg` and the Stalwart push subscription at `/push/jmap`. Verification is handled
   automatically; the page reports the result.

Registration is explicit and repeatable. Registering a new origin cancels the previous subscription,
so the old origin stops receiving pushes. A failed registration never removes the old one — if it
can't reach Telegram, the previous webhook stays in place. If it did reach Telegram but not
Stalwart, the page reports a partial failure and tells you to click again.

Verify from a terminal:

```sh
curl -s "https://api.telegram.org/bot<TOKEN>/getWebhookInfo"
# expect "url": "https://mw.example.com/webhook/tg" and "pending_update_count": 0
```

### 3.8 Turn on processing

Flip **Enable business processing** at the top of the page. This is the master switch: with it off,
webhooks are accepted but `/worker` and `/reconcile` refuse to drain, so nothing reaches Telegram.
With it on, incoming mail and messages are enqueued and the worker drains them.

### 3.9 Verify end to end

1. Send yourself an email. Within a few seconds the bot should post a notification with the
   sender, subject and received time — never the body. That path alone proves JMAP ingest,
   Redis, the queue and the Telegram send all work.
2. In Telegram try `/search 发票`. Note: the search triggers are the Chinese words `搜索`, `查找`
   and `检索`, plus the `/search` command — English keywords like `search invoice` are not
   recognised as triggers.
3. Try `/summary <email_id>` from the notification's email id. The first call asks for
   confirmation; the summary only goes out after you confirm, and only if the LLM section is
   enabled.

### 3.10 Schedule the drain

Push delivery can be delayed or missed; an external scheduler is the safety net. On any machine
that can reach your origin, every 5–10 minutes:

```sh
MW_APP_DOMAIN=mw.example.com \
MW_WORKER_TOKEN=<worker token> \
MW_RECONCILE_TOKEN=<reconcile token> \
scripts/cron-drain.sh --once
```

The script drains the queue (`/worker`) and rescans for missed pushes (`/reconcile`); both return
204 with an empty body on success. `MW_RECONCILE_TOKEN` is optional — if omitted, the reconcile
step is skipped and only the queue is drained. If you put the load balancer in front, `/worker`
must reach the backend origin directly — see §6.3 for why.

### 3.11 When something is wrong

- **Nothing arrives after saving.** Check the enable toggle (step 3.8), then
  `getWebhookInfo` (step 3.7). A `last_error_message` about a 401 means the webhook secret does not
  match what Telegram sends — re-save section 02 and re-register.
- **`/ready` answers 503.** That body is just an error code — it does not name the failing
  component. When `/ready` returns 200, its JSON has per-field booleans (`jmap`, `telegram`,
  `redis`). Common causes are an unreachable `jmap_session_url`, a wrong app password, or
  `runtime_applied: false` from the last save.
- **The form says conflict.** Two sessions saved at once; reload and redo (step 3.6).
- **The bot ignores you in a group.** Privacy mode is on; disable it under `@BotFather` (step 3.2).
- **`/search` answers with a help hint.** Your query started with a Latin word; the prefix triggers
  are `搜索` / `查找` / `检索`.

## 4. Configuration entry points

Three layers. They are not interchangeable.

**Boot-time environment — 2 required, 1 optional.**

| Variable | Required | Default |
|---|---|---|
| `REDIS_URL` | yes | — |
| `CONFIG_ENCRYPTION_KEY` | yes | — |
| `PORT` | no | `8080` |

`.env.example` ships only the two required
variables; the rest are documented defaults.

**Business configuration — Redis, written by the browser.**
`POST /api/bootstrap` is the one-shot trust bootstrap; `PUT /api/business-config` hot-
reloads individual settings within about one second. The process never reads business
secrets from the environment in normal operation.

**Health.**
`GET /healthz` returns 200 unconditionally — it is liveness, not readiness. `GET /ready`
returns 503 until configuration, Redis, and both upstreams (a JMAP session `GET` and
Telegram `getMe`, 3s each) are reachable. Probe `/ready`, not `/healthz`, when deciding
whether to route traffic — but `/ready` is the heavier probe (~3s worst case), so a monitor
that must stay cheap should watch the load balancer aggregate `GET /healthz-worker` when the load balancer
is enabled.

**Remote debug (opt-in, off by default).**
`/debug/*` is an optional remote-debug surface: live JMAP and Telegram probes, the current
business config, and a single Telegram send. It sits behind a two-factor gate — the process
must be started with `--debug` **and** `DEBUG_TOKEN` must be set; miss either and the routes
do not exist at all (requests fall through to a generic 404). Nothing under `/debug/*` is on
the load balancer allowlist, so it is reachable only on the backend origin itself. The only rule for
running it is to keep it off: leave `--debug` out of the start command and leave `DEBUG_TOKEN`
unset. If you do turn it on for a one-off diagnosis, configure the chat allowlist first — with
an empty allowlist the test send is not restricted to any chat. See `docs/reference.md` §3 for
the endpoint contracts.

## 5. Security boundary, in one sentence

> All state lives in Redis; the process reads only two environment variables and writes
> nothing to disk.

One corollary matters for setup: the SPA admin credential — and therefore the bootstrap
trust root — is `CONFIG_ENCRYPTION_KEY` itself, the single 32-byte hex value you supply at
startup. It is only compared in constant time, and is never echoed, logged, or stored. The
Redis ACL password, if you have one, authenticates the Redis connection alone and is not
the credential for any HTTP endpoint.

The optional Cloudflare Worker load balancer in front of the backends is not part of this boundary.
It holds no credential and checks no auth header, and the backends behind it are directly
reachable either way, so nothing about it changes what stands between an attacker and your
mail. See §2.

## 6. Operations and multi-instance

Production, scaling and load-balancer operation reference. The walkthrough in §3 covers a single
instance; this section covers what differs when more than one backend runs behind one or more
HTTPS origins.

### 6.1 Multi-instance prerequisites

Multiple backends can share traffic because request state is stored in Redis, duplicate
deliveries are deduplicated, and reconciliation repairs missed push events. No sticky session or
application code change is required.

All Redis keys are named globally and carry no instance, hostname, or origin suffix. Sharing one
Redis is not a compromise here: several origins hold a single copy of each item below.

| Data | Key | Semantics |
|---|---|---|
| Business configuration | `config:business` | one global ciphertext blob (AES-256-GCM), with no per-instance slot |
| Configuration revision | `config:business:revision` | one global monotonic counter, bumped in the same Lua transaction as the write (`CONFIG_STORE_REV_SCRIPT`) |
| Enable gate | `config:enabled` | one global boolean |
| Reconcile cursor | `state:jmap:since` | one global cursor, with no account dimension |
| Delivery streams | `stalwart:jmap:events` / `stalwart:telegram:events` | shared through a consumer group; the group is created lazily and `XAUTOCLAIM` reclaims idle messages |
| Reconcile lock | `lock:reconcile` | one global single-flight lock, mutual exclusion across instances |

Two consequences matter in practice.

First, **the business configuration exists once, and an SPA edit from any origin writes the same
copy.** There is no way to change only one instance: no key carries an instance dimension and
`refresh_business_config` takes no instance argument. This is intended behavior. If two genuinely
independent configurations are needed, for example production alongside a pre-production twin,
deploy a second instance pointed at a second Redis rather than distinguishing inside one Redis.

Second, **`CONFIG_ENCRYPTION_KEY` must be byte-identical on every origin.** It is 64 hex
characters (32 bytes) and is used directly as the AES-256-GCM key, with the AAD fixed at
`message-weave/config:business/v1`. It also doubles as the SPA admin credential. An origin holding
a different value can neither decrypt configuration written by another origin nor log in to the
SPA, and both paths fail closed.

Extra origins buy failover rather than throughput. Delivery is spread across the shared consumer
group, but `/reconcile` is serialized globally by `lock:reconcile` and the cursor is singular. If
one instance dies mid-batch, the next instance picks the messages up once they pass the
`XAUTOCLAIM` idle window; the cost is possible duplicate delivery, guarded by the `dedup:*` keys,
never message loss.

### 6.2 Configuration convergence across origins

One SPA edit writes once to Redis. The origins converge **lazily, at the request boundary**: each
entry point that calls `refresh_business_config` re-reads the global revision from Redis and
returns immediately when `remote_revision <= local_revision`. There is no polling loop and no
watch, so an origin that receives no traffic keeps the configuration it already had.

The entry points are `/ready`, `/worker`, `/reconcile`, `/push/jmap`, `/webhook/tg`,
`/api/config`, `/api/business-config` and `/api/telegram/register-webhook`. `GET /healthz` is
**not** among them — it is a static `200 "ok"` and never refreshes — and neither does
`GET /api/status`, `/api/enabled`, `/api/push/register`, `/api/push/disable` or the
`/api/admin/session` pair.

**Known behavior: a failed rebuild is not retried until the next revision.** The revision check
above is an early exit, not a retry guard. When `build_worker` fails for one origin — a JMAP
session URL that is unreachable, or an `llm_api_key` / `llm_base_url` that is missing or invalid —
that origin advances its in-memory revision to the remote value and keeps the runtime it already
had. The early-exit condition then holds forever for that origin, so it does not try again until a
**new** revision is written. Every other failure inside the same function (Redis unreachable, blob
missing, JSON parse failure, validation failure) leaves the local revision untouched and therefore
does retry on the next request; those paths fail before a runtime is built, so no worker is
rebuilt.

The symptom is one origin running the new configuration and another still on the old one, with no
error and no log line: the failed path returns silently. To detect it, compare the field values
reported by `GET /debug/config` on each origin — those come from the live in-memory runtime. Do not
use its `revision` field for this: that value is read from Redis and shows the latest global
revision even when the instance has not converged.

To recover, touch the configuration again from the SPA so that the revision moves; the stuck origin
will attempt the rebuild once. If the origin is still in the Worker's origin list, remove it in the
meanwhile so requests are not split between two configurations.

### 6.3 Reliability and delivery objective

The Worker retries only timeouts and backend `5xx` responses; it returns `4xx` responses directly.
`/reconcile` and `/worker` have longer per-route timeouts and do not fail over to a second backend,
avoiding duplicate work. Backend queue processing is at least once and relies on Redis-backed
idempotency. See the route and state reference for the complete behavior.

The service permits short notification delays and targets at least 99.9% notification availability.
Redis availability and the external scheduling interval are outside the application's control;
exact delivery guarantees are described in `NFR-NOTIFY-SLA` and `NFR-RECONCILE-INTERVAL` in the
project charter.

## 7. Where to read more

| Document | What it answers |
|---|---|
| [`docs/design.md`](docs/design.md) | Why the system is shaped this way: data flow, JMAP semantics, Redis streams, AI consent rules |
| [`docs/reference.md`](docs/reference.md) | **Single source of truth for verifiable facts** — Redis keys and TTLs, error codes, routes, environment layers, budgets |
| [`docs/opengaps.md`](docs/opengaps.md) | Gaps, blockers and the next phase |
| [`docs/retired.md`](docs/retired.md) | What was tried and dropped — abandoned routes, unreleased designs, and names that never existed |
| [`docs/charter.md`](docs/charter.md) | Project charter: goals, locked technology choices, security invariants, prohibitions, and the stable-ID registry |
| [`cloudflare-worker/README.md`](cloudflare-worker/README.md) | The Cloudflare Worker load balancer: route safelist, timeout and retry budget, bounded failover, health aggregate, deploy |
| [`AGENTS.md`](AGENTS.md) | Language-agnostic engineering norms: code style, config and secrets, build environment, testing gates, document governance |

Facts that matter are traceable. If two documents disagree, `docs/reference.md` wins.