docs: 重构文档体系 — README 瘦身中英对等 + 新增 reference.md 权威事实表
- README.md / README.zh-CN.md 瘦身至 ~100 行，五节 1:1 对等（是什么/5分钟/配置入口/安全边界/去哪读更多） - 新增 docs/reference.md：Redis 键+TTL+写者+读者、错误 envelope+Retry-After、后端路由与网关白名单矩阵、env 三层等单一权威表（顶部 verified against ed93358） - 新增 docs/roadmap.md 取代 docs/todo.md（只放缺口/阻塞/阶段目标/待决策） - docs/design.md 保留架构语义，可查表改引 reference.md；修正 /search 未实现、XAUTOCLAIM 阈值矛盾、teloxide 幽灵引用等 9 处标注 - docs/deployment.md §8.1 重写，删除依赖不存在子命令/curl 的必然失败 healthcheck 示例 - AGENTS.md §6 登记 reference.md/roadmap.md + 冲突仲裁「以 reference.md 为准」+ 中英同轮同步/公共 API 变更规则
This commit is contained in:
parent
ed93358e29
commit
24cfbb45b0
21
AGENTS.md
21
AGENTS.md
|
|
@ -145,14 +145,29 @@
|
|||
|
||||
| 文件 | 内容 | 何时更新 |
|
||||
|---|---|---|
|
||||
| `README.md` / `README.zh-CN.md` | 面向最终用户：是什么 / 五分钟跑起来 / 配置入口 / 安全边界一句话 / 去哪读更多（英文版 / 中文版，内容须对等，顶部互链） | 用户可见行为或流程变更时 |
|
||||
| `docs/reference.md` | **可核对事实的唯一权威来源**：Redis 键与 TTL、错误 envelope 与错误码、后端路由与网关白名单矩阵、环境变量三层、出站与预算常量 | 公共 API 或 Redis 键名/TTL 变更时（**必更**） |
|
||||
| `docs/design.md` | 产品行为、架构、模块接口、状态机、数据流、错误处理、测试、实施阶段、渠道抽象（Channel/Notifier/MessageAdapter）、产品/架构类待确认问题 | 行为/接口变更时 |
|
||||
| `docs/deployment.md` | 通用 HTTPS-only Docker、Debian、构建/运行时、Secrets、Redis（状态唯一载体）、Webhook/Push/对账短请求路由、**多实例 LB/HA（Worker 前置，§10）**、健康检查、CI、2 项待确认（`Q-DEP-A` 监控日志落盘 / `Q-DEP-B` 日志采集） | 部署/发布变更时 |
|
||||
| `docs/deployment.md` | 通用 HTTPS-only Docker、Debian、构建/运行时、Secrets、Redis（状态唯一载体）、Webhook/Push/对账短请求路由、**多实例 LB/HA（Worker 前置，§10）**、健康检查、CI、2 项待确认（`Q-DEP-A` 平台 URL/域名与证书配置 / `Q-DEP-B` 外部调度器选型） | 部署/发布变更时 |
|
||||
| `docs/roadmap.md` | **只放缺口、阻塞与阶段目标**（取代 `docs/todo.md`，已删除） | 缺口或阻塞项增减时 |
|
||||
| `AGENTS.md`（本文件） | 目标、硬性安全边界、实现顺序、禁止事项、测试验收、文档引用关系、**跨文档引用索引（§7）** | 安全边界/流程变更时 |
|
||||
| `README.md` / `README.zh-CN.md` | 面向最终用户的项目简介、安全模型、配置/环境变量、首次联调流程、已知限制（英文版 / 中文版，内容须对等，顶部互链） | 用户可见行为或流程变更时 |
|
||||
| `docs/todo.md` | 未完成项与阻塞原因（替代已删除的根目录 `HANDOFF.md`） | 待办项增减时 |
|
||||
|
||||
### 6.1 权威与冲突仲裁
|
||||
|
||||
- **`docs/reference.md` 是可核对事实的唯一权威来源。两份文档表述冲突时，以 `docs/reference.md` 为准。**
|
||||
- 可核对事实（Redis 键名与 TTL、错误码、路由、环境变量分层、预算常量）**只存在于 `docs/reference.md`**。
|
||||
`design.md` 保留"为什么"，`deployment.md` 保留"怎么配"；两者不得复制 reference.md 的表格或数值，只引用它。
|
||||
|
||||
### 6.2 同轮同步规则
|
||||
|
||||
- **`README.md` 与 `README.zh-CN.md` 必须同轮更新。**改了一个就必须同轮改另一个并保持内容对等；只改一份的改动不予合入。
|
||||
- **改动 `src/config.rs`、`src/state.rs`、`src/worker.rs`、`src/notify.rs` 的公共 API，或改动任何 Redis 键名 / TTL，必须同轮更新 `docs/reference.md`。**评审按此拦截。
|
||||
|
||||
### 6.3 边界
|
||||
|
||||
- 改动产品规则必须同步三份文档的相关表述，保持一致、不重复堆砌。
|
||||
- `design.md` 不写部署细节，`deployment.md` 不写产品行为/接口；交叉处用**稳定 ID**（§7 索引）互相引用，不用章节号。
|
||||
- `README.md` 只回答"是什么 / 怎么跑 / 去哪配 / 安全边界 / 去哪读"；**不写缺口、不写 roadmap、不写深水区、不写历史环境变量清单**。缺口一律进 `docs/roadmap.md`。
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
361
README.md
361
README.md
|
|
@ -1,321 +1,96 @@
|
|||
# MessageWeave
|
||||
|
||||
中文文档：[README.zh-CN.md](README.zh-CN.md) · Architecture & design: [docs/design.md](docs/design.md) · Deployment & operations: [docs/deployment.md](docs/deployment.md)
|
||||
**[中文文档 / Chinese documentation → README.zh-CN.md](README.zh-CN.md)**
|
||||
|
||||
A personal email assistant that connects your **Stalwart mailbox** to **Telegram**: you get notified in Telegram when a new email arrives, and you can read or summarise emails on demand.
|
||||
> One Telegram bot per mailbox. New mail arrives as metadata; the original is always
|
||||
> fetched read-only from your own JMAP server.
|
||||
|
||||
> ⚠️ **This is an early-stage project — please complete a small-scale integration test first.**
|
||||
> Real mailboxes and Telegram still need to be verified by the operator. JMAP Push subscription registration and verification write-back are implemented, but the callback URL is never guessed automatically: an administrator must call the registration API explicitly.
|
||||
## 1. What it is
|
||||
|
||||
---
|
||||
MessageWeave sits between a JMAP mail server (for example Stalwart) and a Telegram bot.
|
||||
|
||||
## What this project is
|
||||
- **Inbound.** A JMAP push subscription tells the service that a message changed. Nothing
|
||||
is polled at rest.
|
||||
- **Outbound.** Each event becomes a short, **metadata-only** Telegram notification:
|
||||
sender, subject, timestamp. Mail bodies are never sent to Telegram.
|
||||
- **On demand.** When you ask, the service reads the original through a fresh JMAP read
|
||||
call. Nothing is cached in between.
|
||||
- **Optionally summarised.** If you explicitly opt in, an OpenAI-compatible LLM returns a
|
||||
short summary. Requests, responses and attachment bytes are never persisted.
|
||||
|
||||
- You run your own [Stalwart](https://stalwart.dev) mail server.
|
||||
- This tool connects your mailbox to a Telegram bot: when a new email arrives, it pushes a notification (sender / subject / time only — **no message body**).
|
||||
- In Telegram you can ask to view the raw email, or to have it summarised or translated by an AI.
|
||||
State is an architectural boundary here, not a preference.
|
||||
|
||||
### Privacy model
|
||||
- **Redis is the only state store.** No database, no embedded store, no second store.
|
||||
- **The process writes nothing to disk.** No log files, no data files, no local volume.
|
||||
- **No long-lived connections.** No SSE, no WebSocket, no long polling.
|
||||
- **Logs go to stdout only.** The platform's collector owns log storage and retention.
|
||||
- **What it is not.** It is not a mail client, not a proxy, and not a relay. It forwards
|
||||
metadata to one bot; it does not read arbitrary URLs, open tunnels, or broker traffic.
|
||||
|
||||
- **Viewing the raw email never touches any AI.** It is read straight from your mailbox.
|
||||
- **The body is sent to an AI only when you explicitly ask for it and confirm.**
|
||||
- Analysis results are never stored.
|
||||
- The AI is optional and can be left unconfigured or fully disabled.
|
||||
## 2. 5-minute setup
|
||||
|
||||
---
|
||||
Requires a reachable Redis 7 instance and a JMAP session plus a Telegram bot token.
|
||||
|
||||
## What you need before deploying
|
||||
```sh
|
||||
cp .env.example .env # fill REDIS_URL and CONFIG_ENCRYPTION_KEY
|
||||
docker build -t messageweave:latest .
|
||||
docker run --env-file .env -p 8080:8080 messageweave:latest
|
||||
```
|
||||
|
||||
1. **A Stalwart mailbox account**, with an **App Password** generated for it — not your account's master password.
|
||||
2. **A Telegram bot** (create one via @BotFather), and its **Bot Token**.
|
||||
3. **A Redis database** that you operate (AOF persistence recommended). All of the bot's state lives there. Use `rediss://` for TLS Redis and `redis://` for plain connections; if your password (Upstash etc.) contains `@`, `:`, `/` or `#`, URL-encode it first.
|
||||
4. **A public HTTPS address.** The platform points this address at the bot container (HTTPS is provided by the platform).
|
||||
5. **An external cron / scheduler** that calls `/reconcile` every 5–10 minutes so the bot periodically reconciles against your mailbox. The bot does not run any timers of its own.
|
||||
Then open `http://localhost:8080` in a browser. The first-boot SPA calls
|
||||
`POST /api/bootstrap`, stores the business secrets in Redis, and creates the admin session.
|
||||
From that point the browser configures the service through Redis — no environment variable
|
||||
is read or written again for business configuration.
|
||||
|
||||
### Addresses that must be registered
|
||||
For local builds without Docker: `cargo build --locked` produces the `message-weave` binary.
|
||||
|
||||
- Tell Telegram to send its webhook to `your-public-address/webhook/tg`.
|
||||
If the service answers requests but does nothing useful, check the logs for a startup
|
||||
warning: a missing required variable does **not** crash the process. It serves a
|
||||
read-only setup router over in-memory state instead. See §3.
|
||||
|
||||
> If you use the optional "multiple instances + Cloudflare Worker load balancer" setup (see `docs/deployment.md` §10), point the Telegram webhook and the reconcile URL at the **Worker's stable domain** (`https://<your-worker>.workers.dev/webhook/tg` etc.) instead of the per-instance platform addresses.
|
||||
## 3. Configuration entry points
|
||||
|
||||
---
|
||||
Three layers. They are not interchangeable.
|
||||
|
||||
## Creating the container in your Docker manager
|
||||
**Boot-time environment — 2 required, 2 optional.**
|
||||
|
||||
Create a new container, fill in the usual image / port / memory fields, and inject only the Redis credentials:
|
||||
|
||||
| Environment variable | Required? | Value |
|
||||
| Variable | Required | Default |
|
||||
|---|---|---|
|
||||
| `REDIS_URL` | ✔ | Redis connection string (including password), e.g. `redis://:password@redis.example.com:6379/0` |
|
||||
| `CONFIG_ENCRYPTION_KEY` | ✔ | A 32-byte hex key; generate with `openssl rand -hex 32` |
|
||||
| `REDIS_URL` | yes | — |
|
||||
| `CONFIG_ENCRYPTION_KEY` | yes | — |
|
||||
| `PORT` | no | `8080` |
|
||||
| `RUN_MODE` | no | `webhook` |
|
||||
|
||||
After the first start, submit the Telegram / JMAP / LLM and business-auth configuration through the protected SPA page. Configuration secrets are encrypted into Redis and are never returned by the API; after a successful admin `PUT` the service validates the new configuration and hot-rebuilds the clients atomically, so subsequent requests use it immediately, and on failure the previous instances are kept.
|
||||
`RUN_MODE` accepts `webhook` or `reconcile`. `.env.example` ships only the two required
|
||||
variables; the rest are documented defaults.
|
||||
|
||||
AI summarisation is off by default. A Telegram user must first send one of the recognised consent phrases, then `/summary <email_id>`: `/ai on` or `/ai yes` (1 hour); any message containing `临时` or `一次`, e.g. `临时一次` (one-off, 1 hour); `今天` (24 hours); `7天` (7 days); or `直到我撤销` / `长期` (until revoked, capped at 365 days). These phrases are matched literally, so type them in Chinese exactly as shown — English aliases are not recognised. To revoke: `/ai off` (or `撤销授权`). Authorisation never auto-renews once it expires. Without a valid authorisation only metadata is returned, and if the LLM is unset or fails the bot falls back to a locally truncated summary.
|
||||
**Business configuration — Redis, written by the browser.**
|
||||
`POST /api/bootstrap` is the one-shot trust bootstrap; `PUT /api/business-config` hot-
|
||||
reloads individual settings within about one second. The process never reads business
|
||||
secrets from the environment in normal operation.
|
||||
|
||||
> If `REDIS_URL` or `CONFIG_ENCRYPTION_KEY` is missing, the service enters *configuration-setup* mode: the SPA, `/api/status` and the probes still work. The SPA hides the admin-session section and shows only which variables are missing, together with the note that keys are never stored or echoed. Once the configuration is restored, the authorisation UI appears only after `/api/status` reports `ready=true`.
|
||||
**Health.**
|
||||
`GET /healthz` returns 200 unconditionally — it is liveness, not readiness. `GET /ready`
|
||||
returns 503 until configuration and Redis are available. Probe `/ready`, not `/healthz`,
|
||||
when deciding whether to route traffic.
|
||||
|
||||
### HostStack native deployment
|
||||
## 4. Security boundary, in one sentence
|
||||
|
||||
HostStack builds and runs the Rust service using the repository-root `hoststack.yaml`: it runs `cargo build --release --locked`, executes `./target/release/message-weave`, and uses `/healthz` as a health check every 30 seconds with a 5-second timeout. Provide `REDIS_URL` and `CONFIG_ENCRYPTION_KEY` as HostStack Secrets; the keys are never written into the YAML, the image, or any repository file.
|
||||
> All state lives in Redis; the process reads only two environment variables and writes
|
||||
> nothing to disk.
|
||||
|
||||
The "enable business processing" switch in the admin session is persisted in Redis at `config:enabled`; a missing value or a read failure is treated as disabled. While disabled, the webhook, reconcile and worker routes return HTTP 503 and neither ack nor drop upstream events; the SPA, status and configuration APIs stay available.
|
||||
One corollary matters for setup: the bootstrap one-shot token is the **password component of
|
||||
`REDIS_URL`**. The Redis ACL credential is therefore both the application connection
|
||||
credential and the trust root for bootstrap — keep it private and rotate it as one value.
|
||||
|
||||
---
|
||||
## 5. Where to read more
|
||||
|
||||
## How to check that it is healthy after startup
|
||||
| Document | What it answers |
|
||||
|---|---|
|
||||
| [`docs/design.md`](docs/design.md) | Why the system is shaped this way: data flow, JMAP semantics, Redis streams, AI consent rules |
|
||||
| [`docs/deployment.md`](docs/deployment.md) | How to deploy: Dockerfile, secrets, Redis hosting, webhook/push/reconcile routing, multi-instance load balancing |
|
||||
| [`docs/reference.md`](docs/reference.md) | **Single source of truth for verifiable facts** — Redis keys and TTLs, error codes, routes, environment layers, budgets |
|
||||
| [`docs/roadmap.md`](docs/roadmap.md) | Gaps, blockers and the next phase |
|
||||
| [`AGENTS.md`](AGENTS.md) | Contribution rules, hard boundaries, the stable-ID index for cross-document references |
|
||||
|
||||
- Open **`your-public-address/healthz`** in a browser or with `curl`: a normal response means the bot process is alive.
|
||||
- `/ready` checks configuration completeness and Redis reachability. When ready it returns `200` with a JSON readiness report (`{"status":"ready","configured":...,...}`). When not ready it returns `503` with the standard error envelope `{"error":"service_unavailable","request_id":"<id>"}` and `Retry-After: 30`. It never calls JMAP or Telegram and never triggers a business side effect.
|
||||
- For monitoring, Uptime Kuma is recommended: an HTTP liveness check on `/healthz` and an HTTP readiness check on `/ready` (expect 200, 30-second interval, 5-second timeout). The project ships no Prometheus exporter.
|
||||
- `/reconcile` performs authentication, takes a Redis single-flight lock, and runs a bounded JMAP `Email/changes` reconciliation. Events are written to Redis Streams idempotently, and the Redis `sinceState` cursor is only advanced after all of them are queued; a temporary dependency failure returns `503` so cron can retry. Push events still go through the authenticated, de-duplicated enqueue path, so this must not be described as complete real-time synchronisation.
|
||||
- The cold-start baseline also has page, message and time budgets. When a budget is exhausted the checkpoint is saved and the next `/reconcile` continues from it, so a large mailbox is never re-scanned from the beginning.
|
||||
|
||||
## Configuration management
|
||||
|
||||
Open the root address **`your-public-address/`** and enter the password of the Redis ACL user from `REDIS_URL` to create an admin session valid for 15 minutes. The session and the keys typed into the form live only in the current page's memory; after a refresh or closing the page you must authorise again. The page never stores sessions, keys or configuration in browser storage or cookies.
|
||||
|
||||
The page keeps two kinds of configuration separate: runtime parameters (JMAP, Telegram, LLM timeouts and retry limits) can be read and saved through the API; business configuration (JMAP, Telegram, LLM, chat allowlist and the various auth tokens) is fully replaced through `PUT /api/business-config`. To avoid echoing secrets, the service exposes no read endpoint for business configuration, so every full replacement requires re-entering the mandatory fields. After a 204 response the page reports that the new configuration has been hot-loaded and clears the key inputs.
|
||||
|
||||
While Redis is not connected or initialised, the admin-session and configuration APIs return `503`; the page explains the unavailable state and keeps runtime-parameter saving disabled until the read succeeds. When reached through a Cloudflare Worker, use the Worker's root URL; the admin API paths `/api/config`, `/api/business-config`, `/api/admin/session` and `/api/admin/session/revoke` are on the restricted route allowlist (`POST /api/push/register` is not — see `docs/deployment.md` §10.5).
|
||||
|
||||
## End-to-end first integration
|
||||
|
||||
This is a repeatable path from an empty deployment to a first message in Telegram. Every `<...>` below is a placeholder; do not copy the angle brackets, and never paste real secrets into chats, tickets or the repository.
|
||||
|
||||
### 1. Verify the deployment and Redis
|
||||
|
||||
```bash
|
||||
curl -fsS https://<message-weave-domain>/api/status
|
||||
curl -fsS https://<message-weave-domain>/healthz
|
||||
```
|
||||
|
||||
With correct `REDIS_URL` and `CONFIG_ENCRYPTION_KEY`, `/api/status` should contain:
|
||||
|
||||
```json
|
||||
{"ready":true,"mode":"configured","missing":[]}
|
||||
```
|
||||
|
||||
If a variable is missing, the container does not exit; it enters `configuration-setup`, and the SPA shows only the missing variables without the admin-session section. The Redis URL must use the right scheme: `rediss://` for TLS, `redis://` for plain connections. A TLS Redis must not be verified only with `redis-cli --tls` — the application's own `REDIS_URL` must use `rediss://` too.
|
||||
|
||||
### 2. Create a Telegram bot and a test group
|
||||
|
||||
Open `@BotFather` in Telegram, send `/newbot`, follow the prompts and save the bot token. Then create a **group** (not a channel), e.g. `MessageWeave`, add the bot and make it an administrator.
|
||||
|
||||
Send `/start` in the group, then read the Chat ID from the Bot API's `getUpdates` endpoint:
|
||||
|
||||
```bash
|
||||
curl "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/getUpdates"
|
||||
```
|
||||
|
||||
In the response:
|
||||
|
||||
```json
|
||||
"chat":{"id":-5260770881,"title":"MessageWeave","type":"group"}
|
||||
```
|
||||
|
||||
`-5260770881` is the Chat ID. Group IDs are usually negative — do not drop the minus sign. `getUpdates` is not a group-list endpoint; the group only appears once the bot has actually received a message in it. If it returns an empty array, send `/start` in the group first. If a webhook is already set, delete it first, otherwise messages are consumed by the webhook:
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/deleteWebhook"
|
||||
```
|
||||
|
||||
If the bot cannot see ordinary messages, run `/setprivacy` in `@BotFather`, choose the bot and select `Disable`; or make sure the bot is an administrator. Switch the webhook back on once debugging is done.
|
||||
|
||||
### 3. Generate the application-side auth secrets
|
||||
|
||||
Generate these three values separately. Do not reuse the Redis password, the bot token, or the SPA admin session:
|
||||
|
||||
```bash
|
||||
openssl rand -hex 32 # Telegram Webhook Secret
|
||||
openssl rand -hex 32 # Reconcile Token
|
||||
openssl rand -hex 32 # Worker Token
|
||||
```
|
||||
|
||||
- **Telegram Webhook Secret**: you generate it, submit it through the SPA, and pass it as `secret_token` to `setWebhook`. MessageWeave validates the `X-Telegram-Bot-Api-Secret-Token` request header.
|
||||
- **Reconcile Token**: you generate it; the external cron uses it for `/reconcile`.
|
||||
- **Worker Token**: you generate it; it protects `/worker` and the compatibility admin API.
|
||||
|
||||
### 4. Create the SPA admin session
|
||||
|
||||
Open `https://<message-weave-domain>/`, enter the password of the Redis ACL user from `REDIS_URL`, and click to create the admin session. It is valid for 900 seconds, lives only in the current page's memory, and must be created again after a refresh.
|
||||
|
||||
### 5. Fill in the Telegram business configuration
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
Target Chat ID: -5260770881
|
||||
Chat allowlist: -5260770881
|
||||
Telegram Webhook Secret: <first openssl random value>
|
||||
Reconcile Token: <second openssl random value>
|
||||
Worker Token: <third openssl random value>
|
||||
```
|
||||
|
||||
`Target Chat ID` and `Chat allowlist` are **not** the same field: the former is the default target, the latter is the whitelist of chats allowed to drive the bot. For a single test group they can hold the same ID. The allowlist accepts one entry per line or a comma-separated list; duplicates are rejected.
|
||||
|
||||
### 6. Fill in the JMAP business configuration
|
||||
|
||||
- The `JMAP Session URL` may be either `https://mail.example.com` or `https://mail.example.com/.well-known/jmap`; both are normalised.
|
||||
- HTTPS only. Do not put a username, password, query string or fragment into the URL.
|
||||
- `JMAP Username` is the mailbox account.
|
||||
- `JMAP Password` is the Stalwart **App Password**.
|
||||
- `Account ID` may be left empty for the first integration test; the session's primary account is used.
|
||||
- You do not need to create a PushSubscription by hand in the Stalwart WebUI. The built-in `user` role normally already includes reading, creating, modifying and deleting PushSubscriptions; only if registration actually returns `forbidden` should an administrator check the role under `/admin → Management → Directory → Accounts/Roles`. Stalwart's Push global settings under `Settings → Network → JMAP → Push` are not a per-subscription configuration.
|
||||
- Keep the LLM disabled for the first integration test, so you do not mix mailbox problems with AI network configuration.
|
||||
|
||||
### 7. Register the JMAP PushSubscription
|
||||
|
||||
MessageWeave does not require you to create a PushSubscription by hand in the Stalwart WebUI. After saving the JMAP business configuration, call the registration API with the current SPA admin session or `WORKER_TOKEN`, supplying a public HTTPS callback URL explicitly:
|
||||
|
||||
```bash
|
||||
curl -X POST "https://<messageweave-domain>/api/push/register" \
|
||||
-H "Authorization: Bearer <admin-session-or-worker-token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"callback_url":"https://<messageweave-domain>/push/jmap"}'
|
||||
```
|
||||
|
||||
On success the response contains `push_subscription_id`. The URL must be HTTPS and must not contain a username, password or any other embedded credential. Re-registering the same callback URL reuses the existing subscription instead of creating a duplicate. Stalwart then sends its generated `PushVerification` to `/push/jmap`; MessageWeave writes the verification code back through JMAP `PushSubscription/set`, and only accepts real Push events once verification has succeeded. The registration API never guesses the platform domain and never accepts a user-entered verification code.
|
||||
|
||||
The regular `user` role normally already has read/create/modify/delete permissions on PushSubscriptions, so no manual configuration is needed; only check `/admin → Management → Directory → Accounts/Roles` if you actually get `forbidden`. Stalwart's `Settings → Network → JMAP → Push` page holds global Push parameters, not per-subscription creation.
|
||||
|
||||
References: [Stalwart Push notifications](https://stalw.art/docs/http/jmap/push/), [Stalwart Permissions](https://stalw.art/docs/auth/authorization/permissions/), [Stalwart Roles](https://stalw.art/docs/auth/authorization/roles/), [RFC 8620 §7.2 PushSubscription](https://www.rfc-editor.org/rfc/rfc8620.html).
|
||||
|
||||
When the business configuration is saved the service really instantiates the JMAP client; if that fails it returns `503`, stores no broken configuration, and you can fix and re-save. There is no GET endpoint for business configuration, so after a successful save the keys are not echoed back; every full replacement requires re-entering the mandatory fields.
|
||||
|
||||
### 8. Set the Telegram webhook and enable business processing
|
||||
|
||||
After the business configuration is saved:
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/setWebhook" \
|
||||
--data-urlencode "url=https://<message-weave-domain>/webhook/tg" \
|
||||
--data-urlencode "secret_token=<TELEGRAM_WEBHOOK_SECRET>"
|
||||
|
||||
curl "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/getWebhookInfo"
|
||||
```
|
||||
|
||||
Make sure the returned `url` is correct and that `last_error_message` does not keep growing. Then go back to the SPA and switch on "enable business processing". While the global switch is off, the webhook, reconcile and worker routes return `503 service disabled` and neither enqueue nor ack upstream messages.
|
||||
|
||||
Finally send a test message in the allowlisted group. When it arrives, watch Telegram `getWebhookInfo`, the platform logs and the bot's reply. If there is no reply, check the Chat ID's minus sign, the allowlist, the Webhook Secret, the global switch, and the bot's admin rights first.
|
||||
|
||||
### 9. Test reconcile authentication
|
||||
|
||||
```bash
|
||||
curl -i -X POST "https://<message-weave-domain>/reconcile" \
|
||||
-H "Authorization: Bearer <RECONCILE_TOKEN>"
|
||||
```
|
||||
|
||||
A wrong token must return `401`. The external cron is recommended to call once every 5–10 minutes, as a safety net for email reconciliation when Push events are lost.
|
||||
|
||||
## How email synchronisation works today
|
||||
|
||||
Do not treat PushSubscription as the only reliable source in the current release. Push callbacks still have to be authenticated and enqueued, so run the external cron calling `/reconcile` at the same time. Reconciliation uses a persistent Redis `sinceState` and returns `503` on enqueue failure instead of advancing the cursor early.
|
||||
|
||||
If your platform lets the service sleep, the cron's HTTP request will usually wake the container, but the first request pays the cold-start cost and may exceed the cron's or the platform's request timeout. Therefore:
|
||||
|
||||
- set the cron timeout long enough and allow retries on failure;
|
||||
- do not call more often than the platform's wake-up / request limits allow;
|
||||
- do not use `/healthz` as email synchronisation — it only checks that the process is alive;
|
||||
- if the service cannot be woken while sleeping, reconciliation cannot run either; in that case disable sleeping or use an always-on instance.
|
||||
|
||||
The reconciliation lock is renewed by its holder while the task runs; if the renewal fails the cursor is not advanced. Push registration and reconciliation both use Redis owner-token single-flight locks, so concurrent requests cannot create duplicate subscriptions or advance the same JMAP cursor twice.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Redis: `can't connect with TLS, the feature is not enabled`
|
||||
|
||||
Usually the Redis TLS feature was not enabled when the image was built, or `redis://` / `rediss://` is wrong. Use the latest image with TLS support and make sure `REDIS_URL=rediss://...`. Special characters in an Upstash password must be URL-encoded, e.g. `@` becomes `%40`.
|
||||
|
||||
### Redis: `Multiplexed connection driver unexpectedly terminated`
|
||||
|
||||
First validate the network and credentials with the same URL from outside. Do not validate only `redis-cli --tls` while the application still uses `redis://`. Check that HostStack / the platform allows outbound 6379/TLS, and check the ACL user, password and port in the URL.
|
||||
|
||||
### rustls provider panic
|
||||
|
||||
This is the old image loading more than one TLS provider. You must use the latest image, which installs the explicit rustls `ring` provider; do not keep using cached old image tags.
|
||||
|
||||
### `getUpdates` returns an empty array
|
||||
|
||||
The bot has received no new messages, or the message was already consumed by the webhook. Delete the webhook and send `/start` in the group again; confirm the bot is in the group, is an administrator, and — if needed — that privacy mode is disabled through BotFather.
|
||||
|
||||
### Telegram says it is not allowed to read messages
|
||||
|
||||
Confirm that you created a **group** and not a channel; make the bot an administrator; send `/start` again. With privacy mode enabled, the bot normally only sees commands, replies and mentions.
|
||||
|
||||
### SPA returns `503` when saving business configuration
|
||||
|
||||
This usually means the JMAP URL is unreachable from the deployment platform, the certificate/DNS is broken, the app password is wrong, or the LLM configuration fails HTTPS validation. A failed save keeps the old worker and the old configuration, and does not lock the initialisation slot; fix and retry.
|
||||
|
||||
### SPA shows *configuration-setup*
|
||||
|
||||
Check that the HostStack Secrets or container environment contain both:
|
||||
|
||||
```text
|
||||
REDIS_URL
|
||||
CONFIG_ENCRYPTION_KEY
|
||||
```
|
||||
|
||||
`CONFIG_ENCRYPTION_KEY` must be a 32-byte hex value, e.g.:
|
||||
|
||||
```bash
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
It must never be written into the public `hoststack.yaml`, the Dockerfile, the README or the image.
|
||||
|
||||
## Privacy and operational red lines (transparent to users)
|
||||
|
||||
- **No database, no local files**: all of the bot's state lives in your Redis. The container writes no local files and mounts no local volumes; after a restart it recovers from Redis plus mailbox reconciliation.
|
||||
- **Logs go to stdout only**: the container prints logs to stdout/stderr, and your platform or collector is responsible for collection and storage. The bot **does not write log files**.
|
||||
- **Sensitive content never lands in logs**: mailbox bodies, keys, AI requests/responses and attachment contents are **never** written to logs or Redis. Only structured events, counters, timestamps and redacted summaries are recorded.
|
||||
- See the checklists in `docs/deployment.md` §0 and §8.2.
|
||||
|
||||
---
|
||||
|
||||
## What it can do today / what is still missing
|
||||
|
||||
**Already in place**
|
||||
|
||||
- The entry-point framework, three layers of auth protection, and configuration loading.
|
||||
- A configuration SPA backed by short-lived Redis admin sessions: the full business configuration can be edited and hot-loaded, and `/api/config` runtime parameters can be read and saved.
|
||||
- Basic code for reading your Stalwart mailbox (list folders, list emails, read raw messages).
|
||||
- Telegram webhook, Chat ID allowlist, Reconcile / Worker bearer tokens, and a Redis-persisted global enable switch.
|
||||
- HostStack native Rust deployment configuration and a Docker build.
|
||||
- **Optional**: a free Cloudflare Worker front-load-balancer for multi-instance HA — see `docs/deployment.md` §10 and the `cloudflare-worker/` directory.
|
||||
|
||||
**Still missing / not yet verified**
|
||||
|
||||
- ⚠️ **Real mailbox integration needs to be verified by the operator**: the JMAP client code and configuration validation are provided, but different Stalwart networks, permissions and app passwords still differ.
|
||||
- **JMAP Push registration is implemented**: after an administrator submits a public HTTPS callback URL through `POST /api/push/register`, the backend calls `PushSubscription/set create` and `/push/jmap` writes the verification code back automatically. A real Stalwart environment still needs to be verified, and the external cron calling `/reconcile` remains the reliable compensation channel.
|
||||
- Production-grade multi-instance coordination and sending email are not finished yet; AI summarisation is optional and needs separate configuration and verification.
|
||||
|
||||
Both a Docker image and the HostStack native Rust deployment are provided, but we still recommend running the integration steps above on a small scale before using it in production.
|
||||
|
||||
## Quality gates for contributors
|
||||
|
||||
Everything must be green before a change is released (run inside the Debian `rust:1-slim-bookworm` container — see `AGENTS.md` §5 and `docs/deployment.md` §8):
|
||||
|
||||
```bash
|
||||
cargo fmt --check
|
||||
cargo clippy --all-targets -- -D warnings
|
||||
cargo test
|
||||
```
|
||||
|
||||
- Real-device verification tests are marked `#[ignore]` and driven by environment variables (`cargo test -- --ignored jmap::`); CI does not run them by default. Do not claim "verified on real hardware" before actually running them.
|
||||
- Any change that touches a security boundary must be reflected consistently in all three of `docs/design.md`, `docs/deployment.md` and `AGENTS.md`.
|
||||
|
||||
---
|
||||
|
||||
## For developers
|
||||
|
||||
Architecture, design and deployment details:
|
||||
|
||||
- [Architecture & design](docs/design.md)
|
||||
- [Deployment & operations](docs/deployment.md)
|
||||
- [Rules for AI coding assistants](AGENTS.md)
|
||||
- [Open items and blockers](docs/todo.md)
|
||||
|
||||
## License
|
||||
|
||||
(TBD)
|
||||
Facts that matter are traceable. If two documents disagree, `docs/reference.md` wins.
|
||||
|
|
|
|||
357
README.zh-CN.md
357
README.zh-CN.md
|
|
@ -1,323 +1,88 @@
|
|||
# MessageWeave（中文）
|
||||
# MessageWeave
|
||||
|
||||
English documentation: [README.md](README.md) · 架构与设计见 [docs/design.md](docs/design.md)，部署与运维见 [docs/deployment.md](docs/deployment.md)
|
||||
**[English documentation / 英文文档 → README.md](README.md)**
|
||||
|
||||
一个把 **Stalwart 邮箱**接入 **Telegram** 的个人邮件助手：新邮件来了在 Telegram 提醒你，还能按需查看/总结邮件内容。
|
||||
> 一个邮箱对应一个 Telegram bot。新邮件以元数据形式送达；原文始终从你自己的 JMAP 服务器
|
||||
> 只读拉取。
|
||||
|
||||
> ⚠️ **当前仍属早期阶段，请先按本文完成小范围联调。**
|
||||
> 真实邮箱和 Telegram 仍需要由部署者自行验证；JMAP Push 注册、验证回写已接入，但不会自动猜测公网 callback URL，必须由管理员显式调用注册接口。
|
||||
## 1. 是什么
|
||||
|
||||
---
|
||||
MessageWeave 位于 JMAP 邮件服务器（例如 Stalwart）与 Telegram bot 之间。
|
||||
|
||||
## 这个项目是什么
|
||||
- **入站。** JMAP push subscription 通知本服务"有消息变更"。静止时不做任何轮询。
|
||||
- **出站。** 每个事件变成一条简短的**仅元数据** Telegram 通知：发件人、主题、时间戳。
|
||||
邮件正文永不发往 Telegram。
|
||||
- **按需取原文。** 你询问时，服务通过一次新的 JMAP read 调用取回原文。中间不做缓存。
|
||||
- **可选摘要。** 你显式同意时，由 OpenAI-compatible LLM 返回简短摘要。请求、响应与附件
|
||||
字节一律不持久化。
|
||||
|
||||
- 你在自己的服务器上运行 [Stalwart](https://stalwart.dev) 邮箱。
|
||||
- 用本工具把邮箱接到 Telegram 机器人：收到邮件时推送通知（只含发件人/主题/时间，不含正文）。
|
||||
- 你可以在 Telegram 里要求查看某封邮件的原文、或让 AI 帮忙总结/翻译。
|
||||
在这里，状态是架构边界，不是偏好选择。
|
||||
|
||||
### 隐私原则
|
||||
- **Redis 是唯一状态存储。** 无数据库、无内嵌存储、无第二个存储。
|
||||
- **进程不写任何磁盘。** 无日志文件、无数据文件、无本地卷。
|
||||
- **无长连接。** 无 SSE、无 WebSocket、无长轮询。
|
||||
- **日志只写 stdout。** 日志存储与保留由平台采集器负责。
|
||||
- **它不是什么。** 它不是邮件客户端，不是代理，也不是转发中继。它只把一个 bot 的元数据
|
||||
送达 Telegram；不抓取任意 URL、不开隧道、不中转流量。
|
||||
|
||||
- **查看原文不会经过任何 AI**，直接读邮箱。
|
||||
- **只有在你明确要求并确认后**，才把正文发给 AI 分析。
|
||||
- 分析结果不会被保存。
|
||||
- AI 可以完全不配置、完全关闭。
|
||||
## 2. 五分钟跑起来
|
||||
|
||||
---
|
||||
需要可达的 Redis 7 实例、一个 JMAP session 与一个 Telegram bot token。
|
||||
|
||||
## 部署前，你需要准备这几样东西
|
||||
```sh
|
||||
cp .env.example .env # 填写 REDIS_URL 与 CONFIG_ENCRYPTION_KEY
|
||||
docker build -t messageweave:latest .
|
||||
docker run --env-file .env -p 8080:8080 messageweave:latest
|
||||
```
|
||||
|
||||
1. **一个 Stalwart 邮箱账户**，并为其生成一个 **应用专用密码（App Password）**，而不是账户主密码。
|
||||
2. **一个 Telegram 机器人**（通过 @BotFather 创建），拿到它的 **Bot Token**。
|
||||
3. **一个 Redis 数据库**（你托管，建议开启 AOF 持久化）。机器人所有状态都放在这里。TLS Redis 使用 `rediss://`，普通 Redis 使用 `redis://`；Upstash 等服务的密码如果包含 `@`、`:`、`/` 或 `#`，必须先做 URL 编码。
|
||||
4. **一个公网 HTTPS 地址**。平台会把这个地址指向机器人容器（HTTPS 由平台自动提供）。
|
||||
5. **一个定时任务（外部 Cron / 调度器）**：每隔 5–10 分钟访问一次 `/reconcile` 地址，让机器人定期和邮箱对账。机器人自己不会设置定时器。
|
||||
然后在浏览器打开 `http://localhost:8080`。首启 SPA 调用 `POST /api/bootstrap`，把业务
|
||||
密钥写入 Redis 并创建 admin session。此后浏览器通过 Redis 配置服务——业务配置不再读取或
|
||||
写入任何环境变量。
|
||||
|
||||
### 需要登记的地址
|
||||
不用 Docker 的本地构建：`cargo build --locked` 产出 `message-weave` 二进制。
|
||||
|
||||
- 告诉 Telegram：把 webhook 指向 `你的公网地址/webhook/tg`
|
||||
如果服务有响应但什么都不做，请查看启动日志的告警：**缺少必需变量不会让进程崩溃**，它会
|
||||
退化为运行在内存态上的只读配置路由。见 §3。
|
||||
|
||||
> 如果你使用「多实例 + Cloudflare Worker 负载均衡」（可选，见 `docs/deployment.md` §10），则把上面的 Telegram Webhook 和 Reconcile 地址指向 **Worker 的稳定域名**（`https://<your-worker>.workers.dev/webhook/tg` 等），不再指向各后端的平台地址。
|
||||
## 3. 配置入口
|
||||
|
||||
---
|
||||
共三层，彼此不可替代。
|
||||
|
||||
## 在 Docker 管理器中创建容器
|
||||
**启动期环境变量 —— 2 个必需，2 个可选。**
|
||||
|
||||
创建一个新容器，填好镜像、端口、内存等常规项后，只需注入 Redis 连接凭据：
|
||||
|
||||
| 环境变量 | 必须？ | 填什么 |
|
||||
| 变量 | 必需 | 默认值 |
|
||||
|---|---|---|
|
||||
| `REDIS_URL` | ✔ | Redis 连接串（含密码），例如 `redis://:密码@redis.example.com:6379/0` |
|
||||
| `CONFIG_ENCRYPTION_KEY` | ✔ | 32 字节十六进制密钥；可用 `openssl rand -hex 32` 生成 |
|
||||
| `REDIS_URL` | 是 | — |
|
||||
| `CONFIG_ENCRYPTION_KEY` | 是 | — |
|
||||
| `PORT` | 否 | `8080` |
|
||||
| `RUN_MODE` | 否 | `webhook` |
|
||||
|
||||
首次启动后，通过 SPA 的受保护配置页面提交 Telegram/JMAP/LLM 和业务鉴权配置。配置密钥加密写入 Redis，API 永不回显；管理员 PUT 成功后会校验并原子热重建客户端，后续请求即时生效，失败保留旧实例。
|
||||
`RUN_MODE` 只接受 `webhook` 或 `reconcile`。`.env.example` 只提供 2 个必需变量；其余是
|
||||
文档化默认值。
|
||||
|
||||
AI 摘要默认关闭。Telegram 用户需先发送受支持的授权语，再发送 `/summary <email_id>`：`/ai on`（或 `/ai yes`，1 小时）；含 `临时` 或 `一次` 的任意消息，如 `临时一次`（一次性，1 小时）；`今天`（24 小时）；`7天`（7 天）；`直到我撤销`（或 `长期`，最长 365 天）。这些触发词按原文字匹配，需按上文的中文原文输入，不支持英文别名。撤销：`/ai off`（或 `撤销授权`）。授权到期不会自动续期。未授权时仅返回邮件元数据；未配置 AI 或调用失败则回退为本地截断摘要。
|
||||
**业务配置 —— 存于 Redis，由浏览器写入。**
|
||||
`POST /api/bootstrap` 是一次性信任引导；`PUT /api/business-config` 可在约一秒内热更新单
|
||||
项配置。正常运行中进程从不从环境变量读取业务密钥。
|
||||
|
||||
> 缺少 `REDIS_URL` 或 `CONFIG_ENCRYPTION_KEY` 时服务进入 configuration-setup 模式，仍提供 SPA、`/api/status` 和探针；SPA 隐藏管理会话授权区，只显示缺少的环境变量与密钥不保存、不回显说明。配置恢复后 `/api/status` ready=true 才显示授权入口。
|
||||
**健康检查。**
|
||||
`GET /healthz` 无条件返回 200——它是存活探测，不是就绪探测。`GET /ready` 在配置与 Redis
|
||||
可用之前返回 503。判断是否可路由流量时，请探测 `/ready` 而不是 `/healthz`。
|
||||
|
||||
### HostStack 原生部署
|
||||
## 4. 安全边界，一句话
|
||||
|
||||
HostStack 使用根目录 `hoststack.yaml` 构建并运行 Rust 服务：执行 `cargo build --release --locked`，运行 `./target/release/message-weave`，并以 `/healthz` 进行 30 秒间隔、5 秒超时的健康检查。请在 HostStack Secret 中注入 `REDIS_URL` 与 `CONFIG_ENCRYPTION_KEY`；密钥不写入 YAML、镜像或仓库文件。
|
||||
> 所有状态都在 Redis 里；进程只读两个环境变量，不写任何磁盘。
|
||||
|
||||
管理会话内的“启用业务处理”开关持久化在 Redis `config:enabled`；缺失或读取失败均按关闭处理。关闭时 Webhook、Reconcile 和 Worker 返回 HTTP 503，不确认或丢弃上游事件；SPA、状态和管理配置 API 仍可用。
|
||||
有一条推论与部署直接相关：bootstrap 的一次性 token 就是 **`REDIS_URL` 的密码分量**。因此
|
||||
Redis ACL 凭据同时是应用连接凭据与 bootstrap 信任根——请单独保管并按单一值轮换。
|
||||
|
||||
---
|
||||
## 5. 去哪读更多
|
||||
|
||||
## 启动后怎么看它是否正常
|
||||
| 文档 | 回答什么 |
|
||||
|---|---|
|
||||
| [`docs/design.md`](docs/design.md) | 为什么这样设计：数据流、JMAP 语义、Redis streams、AI 授权规则 |
|
||||
| [`docs/deployment.md`](docs/deployment.md) | 怎么部署：Dockerfile、secrets、Redis 托管、webhook/push/对账路由、多实例负载均衡 |
|
||||
| [`docs/reference.md`](docs/reference.md) | **可核对事实的唯一权威来源**——Redis 键与 TTL、错误码、路由、环境变量分层、预算常量 |
|
||||
| [`docs/roadmap.md`](docs/roadmap.md) | 缺口、阻塞项与下一阶段目标 |
|
||||
| [`AGENTS.md`](AGENTS.md) | 贡献规则、硬边界、跨文档引用的稳定 ID 索引 |
|
||||
|
||||
- 浏览器或命令行访问 **`你的公网地址/healthz`**：能返回正常状态，说明机器人进程活着。
|
||||
- `/ready` 检查配置完整性与 Redis 可访问性：就绪时返回 `200` 与 JSON 就绪报告（`{"status":"ready","configured":…,…}`）；不就绪时返回 `503` 与标准错误 envelope `{"error":"service_unavailable","request_id":"<id>"}`，并带 `Retry-After: 30`。它不会调用 JMAP、Telegram 或触发任何业务副作用。
|
||||
- 运行监控建议使用 Uptime Kuma：为 `/healthz` 建立存活 HTTP 检查，为 `/ready` 建立就绪 HTTP 检查（期望 200，间隔 30 秒、超时 5 秒）。项目不引入 Prometheus 或 exporter。
|
||||
- `/reconcile` 执行鉴权、Redis 单飞锁与有界 JMAP `Email/changes` 对账；事件先幂等入 Redis Streams，全部成功后才推进 Redis `sinceState`，依赖暂时失败返回 `503` 以便 Cron 重试。Push 事件仍按鉴权、去重、入队路径处理，不能将其宣传为完整实时同步。
|
||||
- 冷启动基线也有页数、邮件数和时间预算；超出预算时保存断点，下一次 `/reconcile` 从断点继续，避免大邮箱反复从头扫描。
|
||||
|
||||
## 配置管理
|
||||
|
||||
打开服务根地址 **`你的公网地址/`**，输入 `REDIS_URL` 中 Redis ACL 用户的密码，创建有效期 15 分钟的管理会话。会话与表单中的密钥只保存在当前页面内存；刷新或关闭页面后需要重新授权。页面不使用浏览器存储或 Cookie 保存会话、密钥和配置。
|
||||
|
||||
页面分开管理两类配置：运行参数（JMAP、Telegram、LLM 请求超时与最大重试次数）可通过 API 读取和保存；业务配置（JMAP、Telegram、LLM、chat allowlist 与各鉴权 token）由 `PUT /api/business-config` 完整替换。为避免密钥回显，服务端没有业务配置读取接口，因此每次完整替换都必须重新输入必填项。保存收到 HTTP 204 后页面会提示新配置已热加载，并清空密钥输入框。
|
||||
|
||||
Redis 尚未连接或初始化时，管理会话或配置 API 会返回 `503`；页面会说明不可用状态，并在运行参数读取成功前禁用其保存。经 Cloudflare Worker 访问时使用 Worker 根地址；`/api/config`、`/api/business-config`、`/api/admin/session`、`/api/admin/session/revoke` 等管理 API 路径已纳入受限路由白名单（`POST /api/push/register` 不在其中，需按 `docs/deployment.md` §10.5 直连某个后端实例）。
|
||||
|
||||
## 首次联调完整流程
|
||||
|
||||
下面是一条从空部署到 Telegram 收到第一条消息的可重复流程。所有示例中的 `<...>` 都是占位符，不要把尖括号原样复制；真实密钥不要发到聊天、工单或仓库。
|
||||
|
||||
### 1. 先确认部署和 Redis
|
||||
|
||||
访问：
|
||||
|
||||
```bash
|
||||
curl -fsS https://<message-weave-domain>/api/status
|
||||
curl -fsS https://<message-weave-domain>/healthz
|
||||
```
|
||||
|
||||
当 `REDIS_URL` 与 `CONFIG_ENCRYPTION_KEY` 正确时，`/api/status` 应包含：
|
||||
|
||||
```json
|
||||
{"ready":true,"mode":"configured","missing":[]}
|
||||
```
|
||||
|
||||
如果缺少变量，容器不会退出，而是进入 `configuration-setup`；SPA 只显示缺少的变量，不会显示管理会话授权区。Redis URL 必须包含正确协议：`rediss://` 表示 TLS，`redis://` 表示明文连接。TLS Redis 不能只在 `redis-cli` 命令中加 `--tls`，应用本身的 `REDIS_URL` 也必须使用 `rediss://`。
|
||||
|
||||
### 2. 创建 Telegram Bot 和测试群
|
||||
|
||||
在 Telegram 中打开 `@BotFather`，发送 `/newbot`，按提示创建机器人并保存 Bot Token。然后新建“群组”（不是频道），例如 `MessageWeave`，把机器人加入群并设置为管理员。
|
||||
|
||||
在群里发送 `/start`，然后临时使用 Bot API 的 `getUpdates` 查看 Chat ID：
|
||||
|
||||
```bash
|
||||
curl "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/getUpdates"
|
||||
```
|
||||
|
||||
返回中的：
|
||||
|
||||
```json
|
||||
"chat":{"id":-5260770881,"title":"MessageWeave","type":"group"}
|
||||
```
|
||||
|
||||
其中 `-5260770881` 就是 Chat ID。群 ID 通常是负数，负号不能删除。`getUpdates` 不是群组列表接口；只有机器人实际收到消息后才会返回该群。若返回空数组，先在群里发送 `/start`。若已经设置 Webhook，先删除它，否则消息会被 Webhook 消费：
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/deleteWebhook"
|
||||
```
|
||||
|
||||
如果机器人看不到普通消息，在 `@BotFather` 执行 `/setprivacy`，选择机器人并选择 `Disable`；或者确保机器人是管理员。调试结束后再设置 Webhook。
|
||||
|
||||
### 3. 生成应用侧鉴权密钥
|
||||
|
||||
以下三个值必须分别生成，不能使用 Redis 密码、Bot Token 或 SPA 管理会话：
|
||||
|
||||
```bash
|
||||
openssl rand -hex 32 # Telegram Webhook Secret
|
||||
openssl rand -hex 32 # Reconcile Token
|
||||
openssl rand -hex 32 # Worker Token
|
||||
```
|
||||
|
||||
- **Telegram Webhook Secret**：由你生成，填写 SPA，并在 Telegram `setWebhook` 时作为 `secret_token`。MessageWeave 会校验请求头 `X-Telegram-Bot-Api-Secret-Token`。
|
||||
- **Reconcile Token**：由你生成，供外部 Cron 调用 `/reconcile`。
|
||||
- **Worker Token**：由你生成，供 `/worker` 和兼容管理 API 使用。
|
||||
|
||||
### 4. 创建 SPA 管理会话
|
||||
|
||||
打开 `https://<message-weave-domain>/`，输入部署时 `REDIS_URL` 中 Redis ACL 用户的密码，点击创建管理会话。会话有效期 900 秒，只保存在当前页面内存，刷新页面后需要重新授权。
|
||||
|
||||
### 5. 填写 Telegram 业务配置
|
||||
|
||||
示例：
|
||||
|
||||
```text
|
||||
目标 Chat ID: -5260770881
|
||||
Chat allowlist: -5260770881
|
||||
Telegram Webhook Secret: <第一组 openssl 随机值>
|
||||
Reconcile Token: <第二组 openssl 随机值>
|
||||
Worker Token: <第三组 openssl 随机值>
|
||||
```
|
||||
|
||||
`目标 Chat ID` 和 `Chat allowlist` 不代表同一个字段：前者是默认目标，后者是允许触发机器人的白名单；测试单个群时可以填写相同的 ID。allowlist 支持每行一个或逗号分隔，重复项会被拒绝。
|
||||
|
||||
### 6. 填写 JMAP 业务配置
|
||||
|
||||
- `JMAP Session URL` 可填 `https://mail.example.com` 或 `https://mail.example.com/.well-known/jmap`；两者会被归一化。
|
||||
- 只能使用 HTTPS；URL 中不要放用户名、密码、query 或 fragment。
|
||||
- `JMAP Username` 填邮箱账号。
|
||||
- `JMAP Password` 填 Stalwart 应用专用密码。
|
||||
- `Account ID` 初次联调可留空，使用 JMAP Session 的主账户。
|
||||
- 不需要在 Stalwart WebUI 手工创建 PushSubscription。通常内置 `user` 角色已经包含 PushSubscription 的读取、创建、修改和删除权限；只有实际注册返回 `forbidden` 时，管理员才需要在 `/admin → Management → Directory → Accounts/Roles` 检查角色。Stalwart 的 Push 全局参数位于 `Settings → Network → JMAP → Push`，它不是具体订阅配置。
|
||||
- LLM 初次联调建议关闭，避免把邮件问题和 AI 网络配置混在一起。
|
||||
|
||||
### 7. 注册 JMAP PushSubscription
|
||||
|
||||
MessageWeave 不要求你在 Stalwart WebUI 中手工创建 PushSubscription。保存 JMAP 业务配置后，使用当前 SPA 管理会话或 `WORKER_TOKEN` 调用注册接口，并明确提供公网 HTTPS callback URL：
|
||||
|
||||
```bash
|
||||
curl -X POST "https://<messageweave-domain>/api/push/register" \
|
||||
-H "Authorization: Bearer <admin-session-or-worker-token>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"callback_url":"https://<messageweave-domain>/push/jmap"}'
|
||||
```
|
||||
|
||||
成功返回 `push_subscription_id`。URL 必须是 HTTPS，不能包含用户名、密码或其他内嵌凭据。相同 callback URL 的重复注册请求会复用已有订阅，避免重复创建。Stalwart 随后向 `/push/jmap` 发送它生成的 `PushVerification`；MessageWeave 自动通过 JMAP `PushSubscription/set` 回写验证码，验证成功后才接收正式 Push。注册接口不会猜测平台域名，也不会接受用户填写验证码。
|
||||
|
||||
普通 `user` 角色通常已经具备 PushSubscription 的读取、创建、修改和删除权限，无需手工配置；只有实际返回 `forbidden` 时，管理员才检查 `/admin → Management → Directory → Accounts/Roles`。Stalwart 的 `Settings → Network → JMAP → Push` 是全局 Push 参数，不是具体订阅创建页面。
|
||||
|
||||
依据：[Stalwart Push notifications](https://stalw.art/docs/http/jmap/push/)、[Stalwart Permissions](https://stalw.art/docs/auth/authorization/permissions/)、[Stalwart Roles](https://stalw.art/docs/auth/authorization/roles/)、[RFC 8620 §7.2 PushSubscription](https://www.rfc-editor.org/rfc/rfc8620.html)。
|
||||
|
||||
保存业务配置时服务端会真实建立 JMAP 客户端；建立失败返回 `503`，不会写入坏配置，可以直接修正后再次保存。业务配置没有 GET 接口，保存成功后密钥不会回显；完整替换时必须重新输入必填项。
|
||||
|
||||
### 8. 设置 Telegram Webhook 并打开业务开关
|
||||
|
||||
业务配置保存成功后执行：
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/setWebhook" \
|
||||
--data-urlencode "url=https://<message-weave-domain>/webhook/tg" \
|
||||
--data-urlencode "secret_token=<TELEGRAM_WEBHOOK_SECRET>"
|
||||
|
||||
curl "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/getWebhookInfo"
|
||||
```
|
||||
|
||||
确认返回的 `url` 正确，并且没有持续增长的 `last_error_message`。然后回到 SPA 打开“启用业务处理”。全局开关关闭时，Webhook、Reconcile 和 Worker 都返回 `503 service disabled`，不会入队，也不会 ACK 上游消息。
|
||||
|
||||
最后在白名单群里发送测试消息。收到消息后观察 Telegram `getWebhookInfo`、平台日志和机器人回复；如果没有回复，优先检查 Chat ID 的负号、allowlist、Webhook Secret、全局开关和机器人管理员权限。
|
||||
|
||||
### 9. Reconcile 鉴权测试
|
||||
|
||||
```bash
|
||||
curl -i -X POST "https://<message-weave-domain>/reconcile" \
|
||||
-H "Authorization: Bearer <RECONCILE_TOKEN>"
|
||||
```
|
||||
|
||||
错误 Token 必须返回 `401`。外部 Cron 建议每 5–10 分钟调用一次，用于 Push 丢失时的邮件对账兜底。
|
||||
|
||||
## 当前邮件同步方式
|
||||
|
||||
当前版本不要把 PushSubscription 当作唯一可靠来源；Push 回调仍需鉴权后入队，邮件同步请同时使用外部 Cron 定时调用 `/reconcile`。对账使用 Redis 持久 `sinceState`，并在入队失败时返回 `503`，不会提前推进游标。
|
||||
|
||||
如果部署平台会让服务休眠，Cron 的 HTTP 请求通常可以唤醒容器，但首次请求会承担冷启动延迟，可能超出 Cron 或平台的请求超时时间。因此：
|
||||
|
||||
- 将 Cron 超时时间设置得足够长，并允许失败重试；
|
||||
- 频率不要高于平台允许的唤醒/请求限制；
|
||||
- 不要把 `/healthz` 当作邮件同步，它只检查进程存活；
|
||||
- 如果平台休眠期间无法唤醒服务，Reconcile 也无法执行，此时需要关闭休眠或改用常驻实例。
|
||||
|
||||
对账锁会在长任务运行期间由持有者续租；续租失败不会推进游标。Push 注册和对账都使用 Redis owner-token 单飞锁，避免并发请求重复创建订阅或并行推进同一个 JMAP 游标。
|
||||
|
||||
## 常见踩坑与排查
|
||||
|
||||
### Redis 报 `can't connect with TLS, the feature is not enabled`
|
||||
|
||||
通常是镜像构建时 Redis TLS feature 未启用，或 `redis://` / `rediss://` 写错。使用包含 TLS 支持的最新镜像，并确保 `REDIS_URL=rediss://...`。Upstash 密码中的特殊字符必须 URL 编码，例如 `@` 写成 `%40`。
|
||||
|
||||
### Redis 报 `Multiplexed connection driver unexpectedly terminated`
|
||||
|
||||
先用同一个 URL 在外部验证网络和凭据；不要只验证 `redis-cli --tls` 而应用仍使用 `redis://`。检查 HostStack/平台是否允许出站 6379/TLS，检查 URL 中 ACL 用户、密码和端口。
|
||||
|
||||
### Rustls 报 provider panic
|
||||
|
||||
这是旧镜像同时加载多个 TLS provider 的问题。必须使用包含显式 rustls `ring` provider 安装逻辑的最新镜像，不要继续使用旧镜像标签缓存。
|
||||
|
||||
### `getUpdates` 返回空数组
|
||||
|
||||
机器人没有收到新消息，或消息已经被 Webhook 消费。删除 Webhook 后在群里重新发送 `/start`；确认机器人已加入群、是管理员，并在需要时通过 BotFather 关闭隐私模式。
|
||||
|
||||
### Telegram 提示无权读取消息
|
||||
|
||||
确认创建的是群组而不是频道；把机器人设为管理员；重新发送 `/start`。隐私模式开启时，机器人通常只能看到命令、回复和提及它的消息。
|
||||
|
||||
### SPA 保存业务配置返回 `503`
|
||||
|
||||
这通常表示 JMAP URL 无法从部署平台访问、证书/DNS 有问题、应用密码错误，或 LLM 配置不符合 HTTPS 校验。保存失败会保留旧 worker 和旧配置，不会锁死初始化槽位；修正后可重试。
|
||||
|
||||
### SPA 显示 configuration-setup
|
||||
|
||||
检查 HostStack Secret 或容器环境变量中是否同时存在：
|
||||
|
||||
```text
|
||||
REDIS_URL
|
||||
CONFIG_ENCRYPTION_KEY
|
||||
```
|
||||
|
||||
`CONFIG_ENCRYPTION_KEY` 必须是 32 字节十六进制值，例如：
|
||||
|
||||
```bash
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
它不能写在公开的 `hoststack.yaml`、Dockerfile、README 或镜像中。
|
||||
|
||||
## 隐私与运维红线（对用户透明）
|
||||
|
||||
- **无数据库、无本地文件**：机器人所有状态都放在你的 Redis 里；容器自身不写任何本地文件、不挂本地卷，重启后由 Redis + 邮箱对账恢复。
|
||||
- **日志只走标准输出**：容器只往 stdout/stderr 打印日志，由你的平台/采集器负责收集与落盘；机器人自己**不写日志文件**。
|
||||
- **日志里不会写敏感内容**：邮箱正文、密钥、AI 请求/响应、附件内容**都不会**出现在日志或 Redis 里，只记录结构化事件、计数、时间戳、脱敏摘要。
|
||||
- 详见 `docs/deployment.md` §0 与 §8.2 的检查清单。
|
||||
|
||||
---
|
||||
|
||||
## 目前能做到什么 / 还差什么
|
||||
|
||||
**已经有了**
|
||||
|
||||
- 入口框架、三处安全鉴权、配置加载。
|
||||
- 配置管理 SPA：使用短期 Redis admin session；可编辑完整业务配置并热加载，也可读取和保存 `/api/config` 运行参数。
|
||||
- 读取 Stalwart 邮箱的基础代码（列文件夹、列邮件、读原文）。
|
||||
- Telegram Webhook、Chat ID 白名单、Reconcile/Worker Bearer Token 和 Redis 持久化全局启用开关。
|
||||
- HostStack 原生 Rust 部署配置与 Docker 构建配置。
|
||||
- **可选**：一个免费 Cloudflare Worker 前置负载均衡（多实例 HA），见 `docs/deployment.md` §10 与 `cloudflare-worker/` 子目录。
|
||||
|
||||
**还差 / 尚未验证**
|
||||
|
||||
- ⚠️ **真实邮箱联调需要部署者验证**：JMAP 客户端代码和配置校验已提供，但不同 Stalwart 网络、权限和应用密码仍可能不同。
|
||||
- **JMAP Push 注册已接入**：通过 `POST /api/push/register` 显式提交 HTTPS callback URL 后，后端调用 `PushSubscription/set create`，并由 `/push/jmap` 自动完成验证码回写。当前仍需真实 Stalwart 环境验证；外部 Cron 调用 `/reconcile` 继续作为可靠补偿通道。
|
||||
- 生产级多实例协调和发送邮件能力仍未完成；AI 总结属于可选能力，需另行配置并验证。
|
||||
|
||||
Docker 镜像和 HostStack 原生 Rust 部署均已提供，但仍建议先按本文完成小范围联调，再考虑生产使用。
|
||||
|
||||
## 对开发者的门禁
|
||||
|
||||
任何改动发布前须全绿（在 Debian `rust:1-slim-bookworm` 容器内执行，见 `AGENTS.md` §5 与 `docs/deployment.md` §8）：
|
||||
|
||||
```bash
|
||||
cargo fmt --check
|
||||
cargo clippy --all-targets -- -D warnings
|
||||
cargo test
|
||||
```
|
||||
|
||||
- 真机验证用例以 `#[ignore]` 标记、由环境变量驱动（`cargo test -- --ignored jmap::`），CI 默认不跑；未实际运行前不得声称“真机通过”。
|
||||
- 涉及安全边界的改动需同步 `docs/design.md`、`docs/deployment.md`、`AGENTS.md` 三份文档。
|
||||
|
||||
---
|
||||
|
||||
## 给开发/技术人员
|
||||
|
||||
架构、设计、部署细节见：
|
||||
|
||||
- [架构与设计](docs/design.md)
|
||||
- [部署与运维](docs/deployment.md)
|
||||
- [给 AI 编码助手的规则](AGENTS.md)
|
||||
- [未完成项与阻塞原因](docs/todo.md)
|
||||
|
||||
## 许可证
|
||||
|
||||
（待定）
|
||||
重要的事实都可追溯。两份文档不一致时，以 `docs/reference.md` 为准。
|
||||
|
|
|
|||
|
|
@ -246,25 +246,43 @@ GET /ready 公开就绪探针；检查配置完整性与 Redis 可访
|
|||
|
||||
容器以 `--user 1000:1000` 运行，并把 `RUSTUP_HOME`/`CARGO_HOME` 挂载到仓库内持久缓存目录（`-v "$PWD/.gate-cache/rustup:$RUSTUP_HOME" -v "$PWD/.gate-cache/cargo:$CARGO_HOME"`），否则缓存写入因权限失败、工具链每次重建；`.gate-cache/` 已在 `.gitignore` 中，不得提交。门禁结果必须基于**当前仓库快照**重跑得到，不得复用先前快照的通过结果。
|
||||
|
||||
（其余 CI：镜像构建 → 镜像扫描 → 非 root → health 端点到容器验证；compose 示例收敛为单端口 + 外部 Redis URL + 无 volume）
|
||||
（其余 CI：镜像构建 → 镜像扫描 → 非 root → `/ready` 端点到容器验证；运行方式见 8.1）
|
||||
|
||||
### 8.1 docker-compose 示例（收敛）
|
||||
```yaml
|
||||
services:
|
||||
bot:
|
||||
image: messageweave:latest
|
||||
ports:
|
||||
- "8080:8080" # 平台 ingress/反代映射 HTTPS → 8080（C-HTTPS-INBOUND）
|
||||
environment:
|
||||
PORT: "8080"
|
||||
RUN_MODE: webhook
|
||||
# BOT_TOKEN / JMAP_* / CHAT_ALLOWLIST / REDIS_URL
|
||||
# RECONCILE_TOKEN / TG_WEBHOOK_SECRET（写入口鉴权；Push verification 由 Stalwart 动态生成）
|
||||
healthcheck: { test: ["CMD", "message-weave", "health", "--addr", "127.0.0.1:8080"], interval: 30s }
|
||||
# REDIS_URL 指向外部、已认证的 Redis；不在此 compose 中运行 Redis。
|
||||
### 8.1 运行方式与就绪探测
|
||||
|
||||
> **本仓库没有 `docker-compose.yml`。**此前本节给过一段 compose 片段，其中的 `healthcheck` 写成
|
||||
> `message-weave health --addr 127.0.0.1:8080` —— 这个子命令不存在（`src/main.rs` 无任何 CLI
|
||||
> 参数解析，二进制只起 axum 服务），而且 runtime 阶段是 `debian:bookworm-slim`，只装了
|
||||
> `ca-certificates tini`，镜像里没有 curl/wget。照抄那段示例必然得到两个结果之一：healthcheck
|
||||
> 永久失败，或容器被判定 healthy 却什么都不做。示例已删除，改为下面的事实约定。
|
||||
|
||||
**运行**
|
||||
|
||||
```sh
|
||||
cp .env.example .env # 仅需 REDIS_URL 与 CONFIG_ENCRYPTION_KEY
|
||||
docker build -t messageweave:latest .
|
||||
docker run --env-file .env -p 8080:8080 messageweave:latest
|
||||
```
|
||||
|
||||
> 注意：本示例对接 Compose；真实部署可去掉 `ports` 直接挂到平台 ingress（HTTPS-only）。
|
||||
- 单端口 8080；平台 ingress/反代映射 HTTPS → 8080（`C-HTTPS-INBOUND`）。
|
||||
- runtime 阶段为 `debian:bookworm-slim`，非 root（uid 1000）；无本地 volume、无数据库引擎
|
||||
（`C-NO-LOCAL-WRITE` / `C-NO-DB`）。
|
||||
- Redis 由外部已认证实例提供，不与此服务同容器运行（`C-REDIS-EXTERNAL`）。
|
||||
|
||||
**就绪探测：用 `/ready`，不要用 `/healthz`**
|
||||
|
||||
| 端点 | 语义 | 用途 |
|
||||
|---|---|---|
|
||||
| `GET /healthz` | 无条件 200 | 存活探测（进程还活着） |
|
||||
| `GET /ready` | 配置与 Redis 就绪前 503 | 就绪探测、负载均衡摘除、Uptime Kuma |
|
||||
| `GET /api/status` | 缺必需环境变量时 503 + `{"status":"configuration-setup","missing":[...]}` | 排查"起来了但没干活" |
|
||||
|
||||
> **由平台 ingress 探测 `/ready`。**镜像内没有 curl/wget、也没有 CLI 子命令，不要依赖容器内
|
||||
> healthcheck。
|
||||
|
||||
**缺必需环境变量不会崩溃，但也不会干活。**缺 `REDIS_URL` 或 `CONFIG_ENCRYPTION_KEY` 时，进程
|
||||
降级为只读配置路由（内存态、空 token、`NoopWorker`），容器继续应答请求但不做任何业务。排查这类
|
||||
现象：看启动日志告警，或查 `GET /api/status` 返回的 `missing` 列表。
|
||||
|
||||
### 8.2 镜像与部署验证检查项（红线自检）
|
||||
|
||||
|
|
|
|||
|
|
@ -95,7 +95,7 @@ client_ws/ WebSocket 客户端（feature = "websockets"）
|
|||
| 读附件 | `Blob/get`（blobId）或 `email_parse` | 大附件需分片/流式下载 |
|
||||
| 发邮件（2 步） | ① `email_set`/`email_import` 建 draft ② `email_submission_set` 发送 | submission 关联 identityId |
|
||||
| 删除/归档 | `email_set`（keywords `$seen`/`$flagged`）、`mailbox_destroy` | JMAP 无真"删除"，靠 keyword/搬家 |
|
||||
| 搜索 | `email_query` Filter + `SearchSnippet/get` 高亮 | |
|
||||
| 搜索（未实现） | `email_query` Filter + `SearchSnippet/get` 高亮 | |
|
||||
| 实时通知（Push + 对账兜底） | Push HTTPS 回调 → `StateChange`；外部 Cron 调用 `/reconcile` 使用 `Email/changes` 补差 | 需公网 HTTPS 入口（deployment.md `C-HTTPS-INBOUND`/`FLOW-NEW-MAIL`）；Push 不是唯一可靠来源 |
|
||||
| SSE / WebSocket（非目标） | `event_source` / `client_ws` | 本部署**不使用**（deployment.md `NG-POLLING-SSE`/`NG-LONG-POLLING`/`C-NO-LONG-CONN`）；仅列 crate 能力供调研 |
|
||||
|
||||
|
|
@ -154,7 +154,7 @@ WebSocket(...) ws 错误（feature 开启时）
|
|||
| `read_email(id, want_body)` | → `EmailBody { text, html, attachments }` | `email_get`([BodyStructure, BodyValues, BlobIds])；**多 part 原文**（`REQ-JMAP-RAW-MULTIPART`）：按 `text_body` 顺序筛选"有 `part_id` 且 `bodyValue`"的 part 后**拼接**为 `text`；若无可用部分 → 返回**明确错误**（不静默返回空串）。附件用 `Blob/get` |
|
||||
| `send_email(to, subject, body, attachments)` | → `EmailId` | ① `email_import`/`email_set` 建 draft ② `email_submission_set`(onSend) |
|
||||
| `set_flag(id, keyword)` | → () | `email_set` keywords |
|
||||
| `search(query)` | → `Vec<EmailSummary>` | `email_query`(Filter::and[…]) + `SearchSnippet/get` |
|
||||
| `search(query)`（未实现） | → `Vec<EmailSummary>` | `email_query`(Filter::and[…]) + `SearchSnippet/get` |
|
||||
|
||||
> 设计要点：`email_query` 的 `anchor`+`position` 分页是 JMAP 标准做法，比传统 offset 更稳；`sinceState` + `changes` 用于增量同步，避免重复拉全量。
|
||||
|
||||
|
|
@ -169,7 +169,7 @@ WebSocket(...) ws 错误（feature 开启时）
|
|||
| grammers | `grammers` / `grammerslib` | 维护一般 | MTProto（非 Bot API），无需 Telegram Bot Token | ★★ | 仅在不能用 Bot API 时 |
|
||||
| telegram-bot (旧) | `telegram-bot` | 基本停更 | reqwest + futures | ★ | 不推荐 |
|
||||
|
||||
### 4.2 选定 teloxide 的理由
|
||||
### 4.2 曾评估 teloxide 的理由（曾评估，未采纳 —— 未采纳理由见下）
|
||||
1. 与 jmap-client 同为 tokio + reqwest 生态，运行时与 TLS 栈（rustls）可复用。
|
||||
2. 内建 Dispatcher + UpdateKind 枚举匹配命令，与 Bot 的命令路由天然契合。
|
||||
3. 支持 `webhooks-axum`（生产 Webhook 形态；**不使用长轮询** `NG-LONG-POLLING`）与 Redis 会话存储（记住用户上下文：当前选中的文件夹/分页游标；不用 SQLite）。
|
||||
|
|
@ -246,8 +246,8 @@ teloxide = { version = "0.17", features = [
|
|||
|
||||
| 抽象 | 职责 | 首个实现 |
|
||||
|---|---|---|
|
||||
| `Channel` | 渠道生命周期：接收输入、分发命令、配置端点 | `TelegramChannel`（teloxide Dispatcher） |
|
||||
| `Notifier` | 主动推送：把领域 `Notification` 发送到用户 | `TelegramNotifier`（teloxide requester） |
|
||||
| `Channel` | 渠道生命周期：接收输入、分发命令、配置端点 | `TelegramChannel`（teloxide Dispatcher；teloxide 未引入，计划） |
|
||||
| `Notifier` | 主动推送：把领域 `Notification` 发送到用户 | `TelegramNotifier`（teloxide requester；teloxide 未引入，计划） |
|
||||
| `MessageAdapter` | 领域数据 ↔ 渠道消息渲染（文本/按钮/转义） | `TelegramMessageAdapter`（`render.rs`） |
|
||||
|
||||
- 领域与渠道之间用**领域 Command / 领域 Notification** 数据结构传递，渠道只在边缘做适配（解析→领域 Command；领域 Notification→渲染）。
|
||||
|
|
@ -343,10 +343,10 @@ message-weave/
|
|||
│ ├── channel/ # 渠道层（每渠道一个 submodule，见 §5.2）
|
||||
│ │ ├── mod.rs # Channel / Notifier / MessageAdapter traits（thin）—— 阶段0 边界已落地
|
||||
│ │ └── telegram/ # 首个也是当前唯一渠道（adapter #1）—— 阶段2+
|
||||
│ │ ├── mod.rs # TelegramChannel / TelegramNotifier 装配（teloxide Dispatcher）
|
||||
│ │ ├── commands.rs # 领域 Command ↔ teloxide 命令映射（/folders /list /read /send /search /flag）
|
||||
│ │ ├── mod.rs # TelegramChannel / TelegramNotifier 装配（teloxide Dispatcher；teloxide 未引入，计划）
|
||||
│ │ ├── commands.rs # （未实现，文件不存在）领域 Command ↔ teloxide 命令映射（/folders /list /read /send /search /flag）
|
||||
│ │ ├── render.rs # TelegramMessageAdapter：领域 Notification → TG 消息/行内按钮（HTML escape！）
|
||||
│ │ └── session.rs # teloxide 会话键（当前 folder/page/last_email_id）
|
||||
│ │ └── session.rs # teloxide 会话键（当前 folder/page/last_email_id；teloxide 未引入，计划）
|
||||
│ │ # 预留扩展位：dingtalk/、feishu/（不实现，见 §5.2）
|
||||
│ ├── notify/
|
||||
│ │ ├── core.rs # 新邮件处理：JMAP 增量 → 领域 Notification（渠道无关）
|
||||
|
|
@ -359,7 +359,7 @@ message-weave/
|
|||
└── tests/
|
||||
├── jmap_mock.rs # 对 JmapService 的契约测试（mock Push 回调 + changes 响应）
|
||||
├── channel_mock.rs # MockChannel/Notifier 契约：领域流程不依赖 Telegram
|
||||
└── telegram_dispatch.rs # teloxide 测试模式（mock Bot，仅 channel::telegram）
|
||||
└── telegram_dispatch.rs # teloxide 测试模式（mock Bot，仅 channel::telegram；teloxide 未引入，计划）
|
||||
```
|
||||
|
||||
### 6.1 模块职责矩阵
|
||||
|
|
@ -640,23 +640,23 @@ pub enum BotError {
|
|||
|
||||
Push 事件经 Streams 消费并投递到 Telegram，其关键路径交付语义如下：
|
||||
|
||||
**键与 TTL**
|
||||
**键与 TTL 的唯一权威来源是 `docs/reference.md`（Redis 键与 TTL 各节）。**本节不再重复表格——重复的表
|
||||
会被改坏：此前曾把 `push:subscription:{id}` 标成 7d、把 `push:registration:{...}` 标成 360s 注册单飞锁，
|
||||
两处都错，而真正的 360s 单飞锁 `lock:push-register:{sha256(callback_url)}` 当时整行缺失。现已收敛到
|
||||
单一权威表，本节只保留影响设计判断的四条"为什么"：
|
||||
|
||||
| 键 | 作用 | TTL | 说明 |
|
||||
|---|---|---|---|
|
||||
| `delivery:inflight:{stream}:{id}` | 处理中租约 | 60s | 崩溃后可被 XAUTOCLAIM 回收重试 |
|
||||
| `delivery:committed:{stream}:{id}` | 幂等/重放标记 | 7d | 已成功投递的哨兵，防止重放重复打扰 |
|
||||
| `dedup:jmap:{account}:{email}` | 对账去重 | 24h | 入队时原子抢占，同封邮件 24h 内只通知一次（有意设计，见 11.5） |
|
||||
| `config:enabled` 等配置 | — | — | 缺键视为关闭（fail-closed），不进业务 |
|
||||
| `push:subscription:{id}` | Push 订阅的验证码摘要 | 7d | 只存摘要不存明文；`push:disable` 一并清理 |
|
||||
| `push:orphan:{subscription_id}` | 注册失败后的 JMAP destroy 补偿记录 | 7d | 订阅已创建但回写/销毁失败时记录，供后续清理 |
|
||||
| `push:registration:{...}` | 注册单飞锁 | 360s | 幂等创建，owner-token 防并发重复注册 |
|
||||
- **对账锁 TTL 300s，刻意大于单页 120s 上限**，避免持锁期间锁过期导致同一账号重复进入对账
|
||||
（`REQ-RECONCILE-IDEMPOTENCY`、`SAF-RECONCILE-LOCK`）。
|
||||
- **push 注册单飞锁 TTL 360s，刻意大于单条出站请求上限 300s**，否则一次慢注册会放进重复请求
|
||||
（`SAF-AUTH-JMAP-PUSH`）。
|
||||
- **对账去重 24h 是有意设计**：同一封邮件 24h 内只通知一次（`MOD-DEDUP`，范围见本文 11.5）。
|
||||
- **配置键缺键即视为关闭（fail-closed）**，不进入业务路径。
|
||||
|
||||
**投递流程**（`notify::worker`，一次 XREADGROUP 批量 ≤10 条）
|
||||
|
||||
1. 读批 → 逐条 `process` → XACK。
|
||||
2. 处理中写入 `delivery:inflight`（60s）作租约；成功后写 `delivery:committed`（7d）。
|
||||
3. 崩溃于 `inflight` 租约窗口内的条目，由另一实例经 XAUTOCLAIM（idle 阈值 300s）回收重试 —— 至多重复、不丢。
|
||||
3. 崩溃于 `inflight` 租约窗口内的条目，由另一实例经 XAUTOCLAIM 回收重试（空闲阈值 = 批大小 × 单条上限 300s，**不是**固定 300s）—— 至多重复、不丢。
|
||||
4. `retry_or_dlq`（`state.rs`）：重试计数（`max_attempts.max(1)`）未到上限留在源流重试；达到上限则以**单个 Lua 脚本**原子地 `INCR`+`XADD`（入 DLQ）+`XACK`（源流确认），保证不会出现"源已 ACK 但既不在源也不在 DLQ"的缝隙。
|
||||
|
||||
**告警边界（当前实现，见 §8.3 监控约定）**
|
||||
|
|
@ -665,7 +665,7 @@ Push 事件经 Streams 消费并投递到 Telegram，其关键路径交付语义
|
|||
- DLQ 条数、inflight 积压深度、对账补齐条数等指标尚未自动上报（§8.3 标记为可选）。当前运维需通过 Redis 直接查看：`XLEN messageweave:dlq:*`、`XPENDING` 等。
|
||||
- 若需自动化告警，建议在 Uptime Kuma 增加对 `/ready` 或对账入口的健康检查，并手动复核 DLQ 深度。
|
||||
|
||||
**已知边界（当前实现仍存在，见 `docs/todo.md`）**
|
||||
**已知边界（当前实现仍存在，见 `docs/roadmap.md`）**
|
||||
|
||||
- Redis 错误映射：`read_batch` 与 `retry_or_dlq` 在 Redis 出错时仍可能返回 `Ok(())`，消费循环因此不会因单次失败而退出；这类故障只能靠 Redis 侧告警发现。
|
||||
- 多实例重复投递窗口：XAUTOCLAIM 空闲阈值已由固定 300s 改为「批大小 × 单条上限 300s」；单实例不受影响，仅当单条事件处理耗时接近 300s 上限时，多实例部署下另一实例仍可能提前认领，导致**重复投递（仅重复，不丢）**。
|
||||
|
|
@ -673,9 +673,9 @@ Push 事件经 Streams 消费并投递到 Telegram，其关键路径交付语义
|
|||
**审计意见 → 收口（2026-09-26）**
|
||||
|
||||
- 【应修-2】入队失败时 dedup 释放 best-effort 曾可能造成 24h 静默丢事件 → 已修复为 `claim_dedup_and_enqueue`（Lua 原子：`SET NX EX` 成功才 `XADD`），claim 与入队之间无中间失败窗口。
|
||||
- 【应修-1】`Email/changes` 依赖 `newState` 续传，`jmap-client 0.4.2` 无 `upToId` → 已改为「同 `sinceState` 下逐次翻倍 `maxChanges` 扩窗（上限 4096），仅在无法扩窗时才推进 `new_state`」，避免按页推进时漏批；`newState` 语义本身仍需真实 Stalwart 复验（见 `docs/todo.md`）。
|
||||
- 【应修-1】`Email/changes` 依赖 `newState` 续传，`jmap-client 0.4.2` 无 `upToId` → 已改为「同 `sinceState` 下逐次翻倍 `maxChanges` 扩窗（上限 4096），仅在无法扩窗时才推进 `new_state`」，避免按页推进时漏批；`newState` 语义本身仍需真实 Stalwart 复验（见 `docs/roadmap.md`）。
|
||||
- 其余低风险项均已收口：未知 stream 的空值改为 `Err`（fail-closed，进重试/DLQ）；`push:disable` 经 `forget_push_subscription` 清理验证码摘要键；`SET NX EX` TTL 下限收紧为 `.max(1)`；XAUTOCLAIM 空闲阈值按批大小缩放；无 payload 的畸形流条目由 `ack_malformed` 经 `XACK` 移出 PEL；CSPRNG 兜底 owner-token 改为「时间 + PID + 计数器」，不再使用常量。
|
||||
- 未排期待办（阶段 5「搜索 + 搜索片段 + 打磨」）见 `docs/todo.md`「未排期」。
|
||||
- 未排期待办（阶段 5「搜索 + 搜索片段」）见 `docs/roadmap.md`「代码缺口」。
|
||||
|
||||
### 10.6 总估时
|
||||
~10.5 人日（不含等待用户确认与真实联调排障）。
|
||||
|
|
|
|||
|
|
@ -0,0 +1,383 @@
|
|||
# MessageWeave — Reference
|
||||
|
||||
> **This file is the single source of truth for verifiable facts.**
|
||||
> When any other document disagrees with this one, this one wins.
|
||||
>
|
||||
> **Verified against commit `ed93358`.** Line numbers in this file were read from that
|
||||
> commit with the working tree clean.
|
||||
>
|
||||
> **Maintenance responsibility.** Any change to the public API of `src/config.rs`,
|
||||
> `src/state.rs`, `src/worker.rs`, or `src/notify.rs`, or to **any Redis key name or TTL**,
|
||||
> must update this file in the same change. Reviewers: reject a change that touches those
|
||||
> files without touching this one.
|
||||
|
||||
Conventions used throughout:
|
||||
|
||||
- **TTL** is in seconds. "No EX" means the key never expires and lives until explicitly deleted.
|
||||
- **Writer / Reader** name the code symbol that performs the operation; the line number is
|
||||
an annotation for fast navigation, not an identity.
|
||||
- Only this file carries line numbers. All other documents reference symbols by name.
|
||||
|
||||
---
|
||||
|
||||
## 1. Redis keys
|
||||
|
||||
### 1.1 Configuration
|
||||
|
||||
| Key | TTL | Writer | Reader |
|
||||
|---|---|---|---|
|
||||
| `config:business` | No EX | `set_business_config` (state.rs:530, key at state.rs:534); first-time write via `SET NX` in `initialize_business_config` (state.rs:545) | `get_business_config` (state.rs:519) |
|
||||
| `config:business:revision` | No EX | atomic `INCR` on every config write (state.rs:538); `SET 1 NX` on first-time init (state.rs:558) | `business_config_revision` (state.rs:593) |
|
||||
| `config:outbound` | No EX | `set_outbound_config` (state.rs:506) | `get_outbound_config` (state.rs:490) |
|
||||
| `config:enabled` | No EX | `set_enabled` (state.rs:611) | `is_enabled` (state.rs:602) |
|
||||
| `config:admin_session` | EX 900 | `put_admin_session` (state.rs:568, key at state.rs:571) | `admin_session_valid` (state.rs:580); cleared by `revoke_admin_session` (state.rs:589) |
|
||||
|
||||
Notes:
|
||||
|
||||
- `config:business:revision` is an **atomic Redis `INCR`**, not a read-modify-write. Hot reload
|
||||
compares it: the refresh loop only rebuilds the worker when the remote revision exceeds the
|
||||
cached local value, which is what keeps a single bad poll from turning into a hot retry loop
|
||||
(notify.rs:627-650).
|
||||
- `config:admin_session` carries no TTL metadata beyond the literal `EX 900` passed at write
|
||||
time; the handler reports the same window in its response body as `expires_in: 900`
|
||||
(notify.rs:700).
|
||||
- `config:enabled` is consulted as a gate; missing or `false` keeps business processing off.
|
||||
|
||||
### 1.2 AI consent
|
||||
|
||||
| Key | TTL | Writer | Reader |
|
||||
|---|---|---|---|
|
||||
| `consent:ai:{chat_id}` | EX 3600 / 86_400 / 604_800 / 31_536_000 | `set_ai_consent` (state.rs:458; key at state.rs:460, `EX` at state.rs:466) | `ai_consent_until` (state.rs:472); cleared by `clear_ai_consent` (state.rs:481) |
|
||||
|
||||
The stored value is an **absolute Unix expiry timestamp**, not a duration; the key's `EX`
|
||||
carries the same duration, so the key removes itself (state.rs:461-466).
|
||||
|
||||
Trigger words are exact Chinese literals with **no English aliases** — the in-app help text
|
||||
enforces this itself (`worker.rs:569`):
|
||||
|
||||
| Input | TTL | Label |
|
||||
|---|---|---|
|
||||
| `/ai on`, `临时一次`, `临时`, `一次` | 3600 | `1小时` |
|
||||
| `今天` | 86_400 | `今天` |
|
||||
| `7天` | 604_800 | `7天` |
|
||||
| `直到我撤销`, `长期` | 31_536_000 (365 d) | `直到我撤销` |
|
||||
| `/ai off`, `关闭 ai`, `撤销授权`, `停止摘要` | — | revoke |
|
||||
|
||||
`1小时` is a **category label, not a trigger word**: typing `1小时` alone grants nothing.
|
||||
Consent never auto-renews.
|
||||
|
||||
### 1.3 Delivery pipeline
|
||||
|
||||
| Key | TTL | Writer | Reader |
|
||||
|---|---|---|---|
|
||||
| `stalwart:jmap`, `stalwart:telegram` | Stream, no EX | `State::enqueue` (state.rs:241, `XADD` at state.rs:243); call sites notify.rs:123 (TG webhook) and notify.rs:772 (JMAP push); reconcile path via `claim_dedup_and_enqueue` (worker.rs:373) | `read_batch` over consumer group `stalwart-workers`, consumer `http-worker` (notify.rs:265-266, :280) |
|
||||
| `delivery:inflight:{stream}:{id}` | EX 60 | key built at notify.rs:315, `claim_dedup` at notify.rs:316 | `release_dedup` on completion (notify.rs:329) |
|
||||
| `delivery:committed:{stream}:{id}` | EX 604_800 | key built at notify.rs:294, `claim_dedup` at notify.rs:328 | `dedup_exists` before send (notify.rs:295) |
|
||||
| `retry:{stream}:{id}` | EX 86_400 | `retry_or_dlq` Lua script (key state.rs:434, `INCR` state.rs:438, `EXPIRE` state.rs:439) | reclaim path |
|
||||
| `stalwart:jmap:dlq`, `stalwart:telegram:dlq` | Stream, no EX | Lua `XADD` at state.rs:441; name built at notify.rs:348, `max_attempts` 3 at notify.rs:349 | **nothing in code reads it** |
|
||||
|
||||
There is **no `delivery:pending:{stream}` key**. "Pending" refers to the Redis Streams
|
||||
pending-entries list of the `stalwart:*` streams, which Redis tracks internally; it is not a
|
||||
writable key and nothing in the codebase names one.
|
||||
|
||||
Delivery is at-most-once per event: the `delivery:inflight` key is claimed with a 60 s window
|
||||
before the outbound send, and `delivery:committed` extends the guarantee to seven days after
|
||||
acknowledgement.
|
||||
|
||||
The DLQ is **append-and-ack**: the same Lua script that appends to `stalwart:jmap:dlq` /
|
||||
`stalwart:telegram:dlq` also `XACK`s the message out of the source stream, so increment, DLQ
|
||||
append and source acknowledgement are atomic (state.rs:438-442). No code path reads the DLQ
|
||||
back — replay is an operator action, not a service feature.
|
||||
|
||||
`{stream}` in the keys above is one of the two concrete stream names; there are no other
|
||||
streams in the service.
|
||||
|
||||
### 1.4 Idempotency and rate limits
|
||||
|
||||
| Key | TTL | Writer | Reader |
|
||||
|---|---|---|---|
|
||||
| `dedup:tg:{update_id}` | EX 86_400 | key at notify.rs:118, `claim_dedup` at notify.rs:119 | — |
|
||||
| `dedup:jmap:{account}:{email}` | EX 86_400 | key at notify.rs:767, `claim_dedup` at notify.rs:768; same key rebuilt at worker.rs:360 and claimed with `claim_dedup_and_enqueue` at worker.rs:373 | — |
|
||||
| `ratelimit:push-verify:{subscription}` | EX 30 | key at notify.rs:728, `claim_dedup` at notify.rs:729 | — |
|
||||
| `state:jmap:since` | **No EX** | `set_reconcile_state` (state.rs:629; key at state.rs:632, `SET` without `EX`) | `get_reconcile_state` (state.rs:621) |
|
||||
|
||||
`state:jmap:since` is the only state key that survives indefinitely without an explicit
|
||||
delete, by design: it is the reconciliation cursor, and expiring it would force a full
|
||||
re-baseline on the next restart.
|
||||
|
||||
### 1.5 Locks
|
||||
|
||||
| Key | TTL | Writer | Reader |
|
||||
|---|---|---|---|
|
||||
| `lock:reconcile` | 300 | `acquire_lock` (notify.rs:159) | `renew_lock` heartbeat 90 (notify.rs:172) |
|
||||
| `lock:push-register:{sha256(callback_url)}` | 360 | `acquire_lock` (notify.rs:819) | `release_lock` (notify.rs:835, 907) |
|
||||
|
||||
**Why the registration lock is 360 s and not 300 s.** The lock must outlive the longest
|
||||
configured JMAP request timeout (300 s), otherwise a slow registration could admit a
|
||||
duplicate. Source comment, notify.rs:816:
|
||||
|
||||
> The lock must outlive the configured 300s maximum JMAP request timeout; this prevents a
|
||||
> slow create from admitting a duplicate.
|
||||
|
||||
### 1.6 Push subscription state
|
||||
|
||||
| Key | TTL | Writer | Reader |
|
||||
|---|---|---|---|
|
||||
| `push:subscription:{id}` | EX 300 (caller-supplied, min 1) | `remember_push_subscription` (state.rs:647, called at notify.rs:746) | `push_subscription_verified` (state.rs:663) |
|
||||
| `push:subscription:id` | No EX | `remember_push_subscription_id` (state.rs:672) | current-subscription lookup |
|
||||
| `push:subscription:{id}:status` | 900 / 300 / 86_400 (min 1 enforced) | `set_push_subscription_status` (state.rs:679, key at state.rs:687); pending at notify.rs:873, verified at notify.rs:758, disabled at notify.rs:944 | **none** |
|
||||
| `push:registration:{sha256(callback_url)}` | EX 604_800 | `remember_push_subscription_for_callback` (state.rs:732, key at state.rs:739, `EX` at state.rs:742; called at notify.rs:890) | `get_push_subscription_for_callback` (state.rs:721); removed by `remove_push_subscription_for_callback` (state.rs:748) |
|
||||
| `push:orphan:{subscription_id}` | EX 604_800 | `record_push_orphan` (state.rs:712) | orphan sweep |
|
||||
|
||||
**Two distinct keys that are frequently confused. Read this before editing either.**
|
||||
|
||||
- `push:registration:{sha256(callback_url)}` is a **mapping from callback URL to
|
||||
subscription id**, TTL seven days. It is not a lock.
|
||||
- `lock:push-register:{sha256(callback_url)}` is the **single-flight registration lock**,
|
||||
TTL 360 s. It is not a mapping.
|
||||
|
||||
They use the same digest but are separate keys with unrelated lifetimes and purposes.
|
||||
|
||||
**Annotation: `push:subscription:{id}:status` is write-only by design.**
|
||||
The source carries an explicit note at state.rs:169:
|
||||
|
||||
> `push:subscription:{id}:status` is currently write-only
|
||||
|
||||
It is an ops / observability trail intended for `redis-cli` inspection; it is **not** a gate
|
||||
and nothing reads it for authorization. The three values ever written are `pending` (900 s),
|
||||
`verified` (300 s) and `disabled` (86400 s) — see the table above for their call sites.
|
||||
Do not infer correctness from its presence.
|
||||
|
||||
---
|
||||
|
||||
## 2. Error envelope
|
||||
|
||||
Every error response uses one envelope shape:
|
||||
|
||||
```json
|
||||
{ "error": { "code": "<machine_code>", "message": "<human readable>" } }
|
||||
```
|
||||
|
||||
| Status | Code | Retry-After |
|
||||
|---|---|---|
|
||||
| 400 | `invalid_request`, `missing` | — |
|
||||
| 401 | `unauthorized` | — |
|
||||
| 403 | `forbidden` | — |
|
||||
| 409 | `conflict` | — |
|
||||
| 422 | `invalid_configuration` | — |
|
||||
| 500 | `internal_error` | — |
|
||||
| 503 | `service_unavailable`, `disabled`, `push_subscription_not_found`, `push_verify_rate_limited` | `30` when retryable |
|
||||
|
||||
`Retry-After` is set only when the `retry` flag is true, inside `error_response`
|
||||
(notify.rs:243-244), and is emitted unconditionally by the readiness handler for a not-ready
|
||||
503 (notify.rs:1003). The header value is always the literal string `"30"`.
|
||||
|
||||
`GET /api/status` is the exception that reports setup state with a non-envelope body.
|
||||
|
||||
---
|
||||
|
||||
## 3. Backend routes
|
||||
|
||||
All registered in `router_with_worker_state_runtime_bootstrap_config` (notify.rs:1187-1201).
|
||||
|
||||
| Method | Path | Handler |
|
||||
|---|---|---|
|
||||
| POST | `/webhook/tg` | `telegram_webhook` |
|
||||
| POST | `/push/jmap` | `jmap_push` |
|
||||
| POST | `/api/push/register` | `register_push` |
|
||||
| POST | `/api/push/disable` | `disable_push` |
|
||||
| POST | `/reconcile` | `reconcile` |
|
||||
| POST | `/worker` | `worker` |
|
||||
| GET, PUT | `/api/config` | `get_config` / `put_config` |
|
||||
| GET, PUT | `/api/enabled` | `get_enabled` / `put_enabled` |
|
||||
| PUT | `/api/business-config` | `put_business_config` |
|
||||
| POST | `/api/bootstrap` | `bootstrap` |
|
||||
| POST | `/api/admin/session/revoke` | `revoke_admin_session` |
|
||||
| POST | `/api/admin/session` | `create_admin_session` |
|
||||
| GET | `/healthz` | `healthz` |
|
||||
| GET | `/ready` | `ready` |
|
||||
| GET | `/api/status` | `setup_status` |
|
||||
|
||||
Response conventions:
|
||||
|
||||
- `GET /healthz` returns **200 unconditionally**. It is a liveness probe and must not be
|
||||
used to decide whether to route traffic.
|
||||
- `GET /ready` returns **503** when configuration or dependencies are not ready, and carries
|
||||
`Retry-After: 30` for not-ready responses.
|
||||
- `GET /api/status` returns **503** with body `{"status":"configuration-setup","missing":[...]}`
|
||||
when a required environment variable is absent.
|
||||
- Static assets are served with a strict CSP; see §7.
|
||||
|
||||
---
|
||||
|
||||
## 4. Gateway vs backend route matrix
|
||||
|
||||
> **Independently verified.** Source: `cloudflare-worker/src/backends.js` `SAFE_ROUTES`
|
||||
> (backends.js:9-22), 12 entries. The worker entry point is `src/index.js`
|
||||
> (wrangler.toml:20); `src/lb.js` performs forwarding and bounded failover (`SAF-LB-PASSTHRU`,
|
||||
> `C-NO-LONG-CONN`).
|
||||
|
||||
**Forwarded by the worker (12):**
|
||||
|
||||
`/` · `/assets/config.js` · `/assets/styles.css` · `/api/status` · `/api/config` ·
|
||||
`/api/business-config` · `/api/admin/session` · `/api/admin/session/revoke` ·
|
||||
`/webhook/tg` · `/push/jmap` · `/reconcile` · `/ready`
|
||||
|
||||
**Registered on the backend but NOT forwarded (6):**
|
||||
|
||||
| Path | Why it is absent from the gateway |
|
||||
|---|---|
|
||||
| `POST /api/push/register` | Push registration requires the callback URL exchange to complete directly against the backend |
|
||||
| `POST /api/push/disable` | Same reasoning as registration |
|
||||
| `GET, PUT /api/enabled` | Internal operational switch, not exposed through the public gateway |
|
||||
| `POST /api/bootstrap` | One-shot trust bootstrap; kept off the public path |
|
||||
| `POST /worker` | Internal worker endpoint |
|
||||
| `GET /healthz` | Liveness is aggregated by the gateway itself |
|
||||
|
||||
Consequence: the SPA's first-boot flow cannot drive `/api/bootstrap` through the worker.
|
||||
Bootstrap must be performed against the backend origin directly, or the bootstrap path must
|
||||
be added to the gateway allowlist.
|
||||
|
||||
---
|
||||
|
||||
## 5. Environment variables
|
||||
|
||||
Three distinct layers. They are not interchangeable.
|
||||
|
||||
### 5.1 Boot-time (2 required, 2 optional with defaults)
|
||||
|
||||
| Variable | Required | Default | Source |
|
||||
|---|---|---|---|
|
||||
| `REDIS_URL` | yes | — | read at main.rs:39 |
|
||||
| `CONFIG_ENCRYPTION_KEY` | yes | — | `encryption_key_from_env` at main.rs:46 |
|
||||
| `PORT` | no | `8080` | `unwrap_or(8080_u16)` at main.rs:33 |
|
||||
| `RUN_MODE` | no | `webhook` | `unwrap_or_else` at config.rs:338 |
|
||||
|
||||
`RUN_MODE` accepts only `webhook` or `reconcile`; anything else is rejected at boot
|
||||
(main.rs:71-78). Both modes currently route to the same no-op router; the distinction is
|
||||
load-bearing for future splits, not for behaviour today.
|
||||
|
||||
**Missing a required variable does not crash the process.** It logs a warning and serves
|
||||
`router_configuration_setup`, which builds a router over `MemoryState` with empty tokens and
|
||||
a `NoopWorker` (notify.rs:1097-1109). The container stays up and answers requests while doing
|
||||
no business work. This is a deliberate fail-closed-to-setup posture, and it is the single
|
||||
most likely cause of "the container is healthy but nothing happens".
|
||||
|
||||
### 5.2 Redis-resident business configuration
|
||||
|
||||
Written by the SPA through `PUT /api/bootstrap`, then hot-reloaded by
|
||||
`PUT /api/business-config` (effective within 1 s). The process never reads these from the
|
||||
environment in normal operation.
|
||||
|
||||
### 5.3 Legacy compatibility (read but not required)
|
||||
|
||||
`config.rs` still reads an environment-shaped business configuration for backward
|
||||
compatibility. Recognised names, unprefixed unless noted:
|
||||
|
||||
`BOT_TOKEN` · `CHAT_ALLOWLIST` · `TG_WEBHOOK_SECRET` · `TELEGRAM_CHAT_ID` ·
|
||||
`JMAP_*` · `RECONCILE_TOKEN`
|
||||
|
||||
The single prefixed name is `TELEGRAM_CHAT_ID`. Anything else documented as
|
||||
`TELEGRAM_BOT_TOKEN` or similar is a documentation error, not a supported variable.
|
||||
|
||||
**Trust root.** The bootstrap one-shot token is the password component of `REDIS_URL`
|
||||
(main.rs:87-91). The Redis ACL credential is therefore simultaneously the application
|
||||
connection credential and the trust root for bootstrap.
|
||||
|
||||
---
|
||||
|
||||
## 6. Outbound and runtime parameters
|
||||
|
||||
### 6.1 Outbound
|
||||
|
||||
| Parameter | Default | Range / cap | Source |
|
||||
|---|---|---|---|
|
||||
| `jmap_timeout_ms` | 15_000 | 100..=300_000 | default state.rs:55; validated notify.rs:411 |
|
||||
| `telegram_timeout_ms` | 10_000 | 100..=300_000 | default state.rs:56; validated notify.rs:412 |
|
||||
| `llm_timeout_ms` | 30_000 | 100..=300_000 | default state.rs:57; validated notify.rs:413 |
|
||||
| `max_retries` | 3 | hard cap 5 | default state.rs:58; rejected at notify.rs:414; re-clamped at channel.rs:87 |
|
||||
|
||||
There is **no** `LLM_MAX_RETRIES` environment variable; retry count lives in
|
||||
`config:outbound` and is bounded at 5 regardless of what is written.
|
||||
|
||||
### 6.2 Reconcile budgets
|
||||
|
||||
| Constant | Value | Source |
|
||||
|---|---|---|
|
||||
| `MAX_BASELINE_PAGES` | 100 | worker.rs:248 |
|
||||
| `MAX_BASELINE_EMAILS` | 10_000 | worker.rs:249 |
|
||||
| `RECONCILE_BUDGET` | 20 s | worker.rs:250 |
|
||||
| `CHANGE_WINDOW_CAP` | 4_096 | worker.rs:306 |
|
||||
| Lock TTL / heartbeat | 300 s / 90 s | notify.rs:159 / 172 |
|
||||
|
||||
`max_changes` is a function parameter, not a constant.
|
||||
|
||||
### 6.3 Idle threshold
|
||||
|
||||
XAUTOCLAIM's idle threshold is **batch size × 300 s**, not a fixed 300 s.
|
||||
|
||||
---
|
||||
|
||||
## 7. Static assets and response headers
|
||||
|
||||
Served from `src/web.rs`.
|
||||
|
||||
- Three routes only: `GET /`, `GET /assets/config.js`, `GET /assets/styles.css` (web.rs:21-23).
|
||||
Everything else returns 404.
|
||||
- Assets are embedded at compile time with `include_str!` (web.rs:15-17). There is no
|
||||
filesystem access at runtime.
|
||||
- Every response carries `Content-Type`, `Cache-Control: no-store`,
|
||||
`X-Content-Type-Options: nosniff` and `Referrer-Policy: no-referrer` (web.rs:43-51).
|
||||
- CSP is applied **only to the HTML document**: the constant is at web.rs:39, inserted inside
|
||||
an `if html` block (web.rs:53-58):
|
||||
|
||||
```
|
||||
default-src 'none'; script-src 'self'; style-src 'self'; connect-src 'self';
|
||||
form-action 'self'; base-uri 'none'; frame-ancestors 'none'
|
||||
```
|
||||
|
||||
- `X-Frame-Options: DENY` is set alongside the CSP, on the HTML response only (web.rs:58).
|
||||
|
||||
`default-src 'none'` is deliberately stricter than `'self'`: the page is closed by default
|
||||
and only `script-src`, `style-src` and `connect-src` re-open a same-origin channel.
|
||||
|
||||
---
|
||||
|
||||
## 8. Not implemented
|
||||
|
||||
Recorded here so that references elsewhere cannot be mistaken for shipped features.
|
||||
|
||||
- `/search` is not implemented. `SearchSnippet` and `search_snippet` have zero occurrences in
|
||||
`src/`.
|
||||
- There is no `Search` variant in the worker's `Intent` enum, so the search path cannot be
|
||||
entered even indirectly.
|
||||
- The Cloudflare Worker contains no Rust: `cloudflare-worker/src/` holds only `index.js`,
|
||||
`lb.js`, `backends.js` and `health.js`. Any `*.rs` path written under `cloudflare-worker/`
|
||||
is a documentation error, not a source file.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
External dependencies, all reachable and returning HTTP 200 at the time of verification:
|
||||
|
||||
- axum — https://docs.rs/axum
|
||||
- redis (Rust crate, 0.27) — https://docs.rs/redis/0.27/redis/
|
||||
- reqwest (Rust crate, 0.13) — https://docs.rs/reqwest/0.13/reqwest/
|
||||
- jmap-client (0.4.2) — https://docs.rs/jmap-client/0.4.2
|
||||
- JMAP specification — https://jmap.io/spec/
|
||||
- JMAP RFC 8620 — https://datatracker.ietf.org/doc/rfc8620/
|
||||
- Redis Streams — https://redis.io/docs/latest/develop/data-types/streams/
|
||||
- Redis persistence — https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/
|
||||
- Cloudflare Workers — https://developers.cloudflare.com/workers/
|
||||
- Cloudflare Workers runtime APIs — https://developers.cloudflare.com/workers/runtime-apis/
|
||||
- RFC 8030 (Web Push) — https://www.rfc-editor.org/rfc/rfc8030
|
||||
- RFC 7231 §7.2.3 — https://www.rfc-editor.org/rfc/rfc7231#section-7.2.3
|
||||
- Dockerfile reference — https://docs.docker.com/reference/dockerfile/
|
||||
- rust Docker image — https://hub.docker.com/_/rust
|
||||
- Uptime Kuma — https://uptime.kuma.pet/
|
||||
- OWASP Logging Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
|
||||
|
||||
Not linked because they do not resolve to documentation: `crates.io/crates/jmap-client`
|
||||
(404) and `platform.openai.com` documentation paths (403). The project describes the LLM
|
||||
dependency as OpenAI-compatible and documents it through `reqwest` instead.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
# MessageWeave — Roadmap
|
||||
|
||||
> 只放**缺口**、**阻塞**与**阶段目标**。已实现能力见 `README.md`、`docs/design.md`、`docs/deployment.md`；
|
||||
> 可核对的事实（Redis 键、TTL、错误码、路由）见 `docs/reference.md`。
|
||||
> 验证基线：`ed93358`。门禁命令见 `docs/deployment.md` 的 Gate 一节。
|
||||
|
||||
## 阶段目标
|
||||
|
||||
`GATE-P0` 已过（fmt / clippy / check / test，Debian 容器内）。下一阶段目标：
|
||||
在真实邮箱 + 真实 Telegram Bot 上完成端到端联调，消掉下面的阻塞项。
|
||||
|
||||
## 阻塞（需要真实环境，当前无法验证）
|
||||
|
||||
- **真实 Stalwart 邮箱联调** — 未做任何真实邮箱端到端集成。JMAP adapter 代码与配置校验已完成并通过本地 `cargo test`，但 `Email/changes`、`PushSubscription` 路径未在真实账号上验证。`REQ-JMAP-SESSION-URL` 已实现（`D-G1-1`）但待真机验证。**阻塞原因**：无可用 Stalwart 账号/凭据；禁止使用测试账号或伪造结果。
|
||||
- **真实 Telegram Bot 联调** — `MOD-TELEGRAM-NOTIFY` 的签名校验、`getUpdates` 迁移、`setWebhook` 注册、`sendMessage` 出站均未在真实 Bot 上验证。**阻塞原因**：无 Bot Token；禁止向真实用户发消息。
|
||||
- **真实 Redis TLS 连接** — `rediss://` 握手、密码特殊字符 URL 编码、`CONFIG_ENCRYPTION_KEY` 热更新未实测。**阻塞原因**：无托管 Redis 实例。
|
||||
- **`Email/changes` 的 `newState` 语义** — 客户端已用「同 `sinceState` 翻倍 `maxChanges` 扩窗」消除按页漏批，但 `newState` 是否表示"全部待报变更之后"仍需真实 Stalwart 复验，否则停机积压边界无法判定。**阻塞原因**：无可用 Stalwart 账号/凭据。
|
||||
- **Push callback 公网映射** — 单飞锁 `lock:push-register:{sha256(callback_url)}`（360s）与映射 `push:registration:{sha256(callback_url)}`（7d）的 TTL 需真实回调时序验证。**阻塞原因**：无真实推送回调可观察。
|
||||
|
||||
## 代码缺口（已定位，待实现）
|
||||
|
||||
- **`/search` + `SearchSnippet/get` 高亮** — 未实现。`src/` 中 `SearchSnippet` / `search_snippet` 零命中，worker 的 `Intent` 枚举也无 `Search` 变体，因此该路径无法进入。设计文档中的接口表已标注「（未实现）」。
|
||||
- **Telegram 429 退避实测** — `src/channel.rs` 的 `max_retries`（默认 3，硬上限 5）不针对 Telegram 服务端 30 msg/s 限流做专门退避；429 行为无实测数据。需真实 Bot 压测后再决定是否引入指数退避。
|
||||
- **Redis 错误映射** — `read_batch` 与 `retry_or_dlq` 在 Redis 出错时仍可能返回 `Ok(())`，消费循环不会因单次失败退出，这类故障只能靠 Redis 侧告警发现。（可选改进）
|
||||
- **多实例重复投递窗口（残余）** — XAUTOCLAIM 空闲阈值已按批大小缩放（批大小 × 单条上限 300s）；仅当单条事件处理耗时接近 300s 上限时，多实例部署下另一实例仍可能提前认领（仅重复不丢）。单实例不受影响。
|
||||
- **Worker safelist 与注册流程不一致** — 6 条后端路由不经 Worker 转发（见 `docs/reference.md` 的网关矩阵）。其中 `POST /api/push/register` 与 `POST /api/push/disable` 需直连后端 origin。**待决策**：加入 safelist，或维持直连后端。
|
||||
|
||||
## 决策待定
|
||||
|
||||
- **平台 URL / 域名与证书配置方**（`Q-DEP-A`，关联 `C-HTTPS-URL`）— 平台自动证书 or 自管反代，以及 4 条路径的可达性验证方式。
|
||||
- **外部调度器选型**（`Q-DEP-B`，关联 `NFR-RECONCILE-INTERVAL`）— 系统 crontab / k8s CronJob / CI scheduled / 第三方 cron。仅影响运维方式，不改变架构。
|
||||
|
||||
## 验收待办
|
||||
|
||||
- **真实平台日志采集验收**（`Q-DEP-B` 允许延后）— 容器不写日志文件、只走 stdout 已由代码审查确认（`C-LOG-STDOUT-ONLY`）；结构化日志字段与脱敏已由代码审查确认（`SAF-LOG-PURITY`）。未验证项：真实平台采集器是否落盘、是否可检索。
|
||||
- **`/ready` 端到端依赖探测**（`GATE-READY-DEPS`）— 当前 `ARCH-READY-BASELINE` 仅覆盖配置完整性 + Redis 可达性；JMAP session / Telegram getMe 等端到端探测尚未接入。
|
||||
- **JMAP 只读 adapter 真机验证**（`GATE-G1-JMAP-READONLY`）— 代码已实现（mock + `#[ignore]` 真机测试），待 `cargo test -- --ignored jmap::`。
|
||||
34
docs/todo.md
34
docs/todo.md
|
|
@ -1,34 +0,0 @@
|
|||
# 未完成项与阻塞原因
|
||||
|
||||
> 本文件取代已删除的根目录 `HANDOFF.md`。只保留未完成项与阻塞原因；已完成能力见 `README.md`、`docs/design.md`、`docs/deployment.md`。门禁命令见 `docs/deployment.md` §8.0。
|
||||
|
||||
## 阻塞（需要真实环境，当前无法验证）
|
||||
|
||||
- **真实 Stalwart 邮箱联调**：未运行任何真实邮箱端到端集成。JMAP 客户端代码与配置校验已完成并通过本地 `cargo test`，但 `Email/changes`、`PushSubscription` 等路径没有用真实邮箱账号验证过。→ 阻塞原因：无可用 Stalwart 账号/凭据，禁止使用测试账号或伪造结果。
|
||||
- **真实 Telegram Bot 联调**：`/webhook/tg` 的签名校验、`getUpdates` 迁移、`setWebhook` 注册、`sendMessage` 出站均未在真实 Bot 上验证。→ 阻塞原因：无 Bot Token，禁止向真实用户发消息。
|
||||
- **真实 Redis TLS 连接**：`REDIS_URL` 使用 `rediss://` 的 TLS 握手、密码特殊字符 URL 编码、`CONFIG_ENCRYPTION_KEY` 热更新均未实测。→ 阻塞原因：无 Upstash/托管 Redis 实例。
|
||||
- **`Email/changes` 的 `newState` 语义**（design.md §10.5.1 应修-1）：客户端已用「同 `sinceState` 翻倍 `maxChanges` 扩窗」消除按页漏批，但 `newState` 是否表示"全部待报变更之后"仍需真实 Stalwart 复验，否则停机积压边界无法判定。→ 阻塞原因：无可用 Stalwart 账号/凭据。
|
||||
- **Push callback 公网映射**：`push:registration` 注册锁（360s）与 callback 映射（7d）TTL 依赖真实 Stalwart 的回调时序。→ 阻塞原因：无真实推送回调可观察。
|
||||
|
||||
## 代码缺口（已定位，待实现）
|
||||
|
||||
- **Redis 错误映射**：`read_batch` 与 `retry_or_dlq` 在 Redis 出错时仍可能返回 `Ok(())`，消费循环不会因单次失败退出，这类故障只能靠 Redis 侧告警发现（可选改进）。
|
||||
- **多实例重复投递窗口（残余）**：XAUTOCLAIM 空闲阈值已按批大小缩放（批大小 × 单条上限 300s）；仅当单条事件处理耗时接近 300s 上限时，多实例部署下另一实例仍可能提前认领（仅重复不丢），单实例不受影响。
|
||||
- **Worker safelist 与注册流程不一致**：`POST /api/push/register` 不在 `cloudflare-worker/src/backends.js` 的 `SAFE_ROUTES` 中，Worker 对其返回 404；`docs/deployment.md` §10.4 已修正为直连后端实例。→ 待决策：加入 safelist，或保持直连后端。
|
||||
|
||||
## 未排期（阶段 5「搜索 + 搜索片段 + 打磨」）
|
||||
|
||||
- `/search` + `SearchSnippet/get` 高亮。
|
||||
- 错误消息打磨、速率/退避实测、`clippy`/`fmt`、README 复核。
|
||||
|
||||
## 监控验收（Q-DEP-B）
|
||||
|
||||
- 容器不写日志文件、只走 stdout，已由 `docker`/`docker-compose.yml` 检查项覆盖。
|
||||
- 业务事件、计数、时间戳、脱敏摘要的结构化日志已由代码审查确认。
|
||||
- 未验证项：真实平台的日志采集器是否落盘、是否可检索。**→ 阻塞原因：Q-DEP-B 明确允许延后到真实部署验收。**
|
||||
|
||||
## 本轮已收口（2026-09-26）
|
||||
|
||||
- **错误 envelope 统一**：全部公开路由错误响应（含 `GET /ready` 不就绪时的 503）统一为 `{"error":<code>,"request_id":<id>}`，可重试的 503 追加 `Retry-After: 30`；`/ready` 就绪时 `200` 仍返回结构化诊断报告。Uptime Kuma 按 HTTP 状态码（`/ready` 期望 200）监控，不受响应体变化影响。
|
||||
- **Web SPA i18n**：按浏览器语言判断 `zh`/`en`，默认英文，保持纯静态 SPA（`data-i18n` 属性 + 内联消息字典，无新增资源文件）。
|
||||
- **文档**：`README.md` 改英文默认，新增 `README.zh-CN.md`，两版章节与关键事实 1:1 对齐；`docs/design.md`、`docs/deployment.md`、`AGENTS.md` 同步修正滞后描述。
|
||||
Loading…
Reference in New Issue