docs: retire HANDOFF-push-401.md; record incident lessons; fix stale baseline claims

- Delete HANDOFF-push-401.md (fully superseded: TTL fix chain closed,
  rate-limiter ordering fixed in 3b36fe3, live chain confirmed 2026-10-16,
  two-path table superseded by in-process drain ac0f291). Retirement row
  added to docs/retired.md / retired.zh-CN.md.
- docs/opengaps.md + opengaps.zh-CN.md: new closing-record sections
  §4 (push-401 incident, TTL 0 lesson) and §5 (untimely delivery, sole
  consumer was out-of-process cron) plus process lessons (handoff docs
  live outside validators; live user test is final authority).
- docs/design.md + design.zh-CN.md: replace stale 106-passed baseline and
  phantom TEST_BASELINE anchor with the 111/0/4 (2026-10-16) value citing
  the test-baseline line under GATE-P0 in docs/charter.md.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
anonsaber 2026-10-16 23:44:02 +08:00
parent 509e44a9a2
commit 5339656c9b
7 changed files with 78 additions and 215 deletions

View File

@ -1,213 +0,0 @@
# HANDOFF — 邮件 Push 通知投递问题
> **日期**: 2026-10-13
> **状态**: 代码修复已完成,**未部署**
> **接手方优先级**: P0 — 用户阻塞中,邮件 push 收不到
---
## 1. 问题描述
**症状**: 用户配置了 Stalwart 邮件服务器 + Cloudflare Worker + Telegram Bot,通过 **cron 定时任务**(`/reconcile` 路径)可以正常收到邮件通知,但 **邮件 Push**(Stalwart → `/push/jmap` 路径)**完全收不到**。
**用户已确认的现象**:
- `/help` 等 Telegram 命令秒回
- cron 路径(`/reconcile` → JMAP `/changes` → worker → Telegram)工作正常
- 邮件 push 通知(Stalwart → `/push/jmap`)从未到达 Telegram
---
## 2. 根因分析(已完成,结论确认)
### 2.1 根因: 验证码 key 5 分钟后过期(旧代码 `TTL=300`)
Stalwart 按 RFC 8620 协议向我们注册 Push 订阅时:
1. **注册时**: Stalwart 发第一次带 `verificationCode` 的验证请求,我们的后端把验证码存进 Redis(key = `push:subscription:{id}`,value = 验证码)。**旧代码 TTL = 300 秒(5 分钟)**。
2. **事件推送时**: 每封新邮件到达后,Stalwart 发带 `emailId`/`changeType` 但**不带** `verificationCode` 的事件推送。我们的后端去 Redis 查 `push:subscription:{id}` 确认订阅已验证 — 但 key 已在 5 分钟后过期 → 返回 **401 unauthorized** → 邮件通知被丢弃。
### 2.2 时间线证据(来自 Stalwart 日志)
| 时间 (UTC) | 事件 |
|---|---|
| 02:52:30 | 用户执行 register,Stalwart 分配 `push_subscription_id: "b"`,向 `/push/jmap` 发第一次验证请求,后端存 key(TTL=300) |
| ~02:57:30 | Redis key 过期被自动删除 |
| 02:58:49 | 测试邮件到达 Stalwart,changeId=305,documentId=167 |
| 02:58:50 | Stalwart 向 `/push/jmap` 推送事件 → 后端查 key → **401 unauthorized** |
**注册到 401 恰好间隔 ~6 分钟,与 TTL=300(5 分钟)完全吻合。**
### 2.3 为什么 cron 能收到
cron 走 `/reconcile` → 主动调 JMAP `/changes` → 拉新邮件 → 发 Telegram。这条路径**完全不依赖验证码 key**,所以不受 TTL 影响。push 路径依赖 key 核对,key 过期就 401。
### 2.4 次要问题: 限流器 429
注册日志里出现 `429 push_verify_rate_limited` — Stalwart 在 30 秒内重复发了验证请求,第二个被限流器(`ratelimit:push-verify:{subscription}`,TTL=30s)拦住。但**第一个请求已成功存了 key**(只是 5 分钟后过期),所以 429 不是当前 401 的原因,只是鲁棒性问题。
---
## 3. 已完成的修复(3 个 commit,均已 push 双 remote)
| Commit | 修复内容 | 关键代码位置 |
|---|---|---|
| `d452f95` | `PUSH_SUBSCRIPTION_SECONDS` 300 → 604800(7 天) | `src/state.rs` ttl 模块 |
| `7331cb7` | `PUSH_SUBSCRIPTION_SECONDS` 604800 → **0(永不过期)** | `src/state.rs` ttl 模块 |
| `f9160c0` | ① 验证响应回 `{"verificationCode":"..."}`(原返回空 204,违反 RFC 8620 §7.2);② `verify_push_subscription` JMAP API 改为 best-effort(失败不阻塞) | `src/notify.rs` `jmap_push` 验证分支 |
### 3.1 关键代码变化
**`src/state.rs`** — TTL 常量:
```rust
pub const PUSH_SUBSCRIPTION_SECONDS: u64 = 0; // 永不过期
```
`remember_push_subscription` 处理 `ttl_seconds == 0` 时用 `SET key value`(不带 `EX`),MemoryState 用 `expiry = i64::MAX`。
**`src/notify.rs`** — 验证流程(当前代码,已修但未部署):
```
if 带 verificationCode(验证请求):
1. rate limiter claim_dedup → 429 拦截(❗这在 remember 之前,有问题)
2. verify_push_subscription JMAP API → best-effort(warn 不阻塞)
3. remember_push_subscription → 存 key(TTL=0 永不过期)
4. set_push_subscription_status → "verified"
else(事件推送):
查 push:subscription:{id} 是否存在 → 不存在 401
```
**RFC 8620 §7.2 响应**(`notify.rs` 的 `jmap_push` handler,验证请求分支末尾):
```rust
if let Some(code) = push.verification_code.as_deref() {
return axum::Json(serde_json::json!({
"verificationCode": code
})).into_response();
}
return StatusCode::NO_CONTENT.into_response();
```
---
## 4. 未完成的工作(接手方要做)
### 4.1 【P0】部署最新代码到服务器
**当前服务器跑的是旧代码(TTL=300),三个修复 commit 都在 git 里但没部署。**
部署方式(Docker):
```bash
# 在服务器上
git pull origin main # 拉到 f9160c0 或更新
docker build -t messageweave .
docker stop <容器名>
docker run --rm --env-file .env -p 8080:8080 messageweave
```
或者如果不用 Docker:
```bash
git pull origin main
cargo build --locked --release
# 重启运行中的 message-weave 进程
```
### 4.2 【P0】部署后重新注册 Push 订阅
部署新代码后,旧的 key 可能还残留(或已过期),需要让 Stalwart 重新走一次验证流程:
```bash
# 变量
BACKEND="https://messageweave.motofans.club"
TOKEN="你的 worker_token(业务配置里的)"
# 1. 禁用旧订阅
curl -X POST "$BACKEND/api/push/disable" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"callback_url":"https://messageweave.motofans.club/push/jmap"}'
# 2. 重新注册(Stalwart 会发新验证请求,新代码存 key 且永不过期)
curl -X POST "$BACKEND/api/push/register" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"callback_url":"https://messageweave.motofans.club/push/jmap"}'
# 3. 发测试邮件,观察 Stalwart 日志确认不再 401
```
### 4.3 【P1】修复限流器顺序(代码改动)
**问题**: `notify.rs` `jmap_push` 的验证分支中,rate limiter(`claim_dedup("ratelimit:push-verify:{subscription}")`)在 `remember_push_subscription` **之前**。如果 Stalwart 在 30 秒内重发验证请求,第二个请求被 429 拦住 → `remember_push_subscription` 不会被调用 → 那次尝试的 key 没存。虽然第一个请求通常已存了 key,但这不鲁棒。
**修法**: 把 `remember_push_subscription` 移到 rate limiter **之前**,rate limiter 只控制**出站** `verify_push_subscription` JMAP API 调用(防止频繁调外部 API),**不拦验证请求本身**。
修改后流程应该是:
```
if 带 verificationCode:
1. remember_push_subscription(先存 key,TTL=0) ← 移到最前
2. set_push_subscription_status → "verified"
3. rate limiter claim_dedup → 只拦 JMAP API 调用,不拦整个请求
4. verify_push_subscription JMAP API → best-effort
```
**需要同步更新的文档**(改代码后必须一起改,否则 GATE-DOCS 会挂):
- `docs/reference.md` / `docs/reference.zh-CN.md` — `ratelimit:push-verify:{subscription}` 键说明行 + 429 `push_verify_rate_limited` 错误码表行
- `docs/opengaps.md` / `docs/opengaps.zh-CN.md` — TTL 表 `PUSH_VERIFY_LIMIT_SECONDS` 行
- `docs/design.md` / `docs/design.zh-CN.md` — "Push verification code writing current limiting" 段落
### 4.4 【P2】验证完整链路
部署 + 重新注册后:
1. 发测试邮件
2. 看 Stalwart 日志: 验证请求应该返回 200 + `{"verificationCode":"..."}`,事件推送应该返回 2xx(不再 401)
3. 看 Telegram 是否收到通知
4. 等 10+ 分钟再发第二封(确认 key 不再过期)
---
## 5. 两条投递路径说明
| 路径 | 触发方式 | 依赖验证码 key | 当前状态 |
|---|---|---|---|
| **Cron** | 定时调 `/reconcile` → JMAP `/changes` → 拉新邮件 → worker → Telegram | ❌ 不依赖 | ✅ 正常工作 |
| **Push** | Stalwart → `/push/jmap`(事件推送)→ 查 key → worker → Telegram | ✅ 依赖(401 的原因) | ❌ 旧代码 key 过期 → 401 |
---
## 6. 环境信息
- **Stalwart**: `0.16.26`,Docker 运行在 `mail.motofans.club`
- **Push 订阅 ID**: `"b"`(Stalwart 分配的单字符 ID)
- **回调 URL**: `https://messageweave.motofans.club/push/jmap`
- **后端仓库**: `/home/okabe/Repo/messageweave`
- **双 remote**: `origin`(motofans.club)+ `upstream`(GitHub fork)
- **测试基线**: Rust 106 passed / 0 failed / 4 ignored,GATE-DOCS 6/6 PASS
- **最新 commit**: `f9160c0`(已 push 双 remote)
---
## 7. 已知的文档残留(非阻塞)
- `docs/opengaps.md` 声称"当前无未完成项",但限流器顺序问题(§4.3)还没修 — 接手方修完后应更新
- `docs/reference.md` / `docs/reference.zh-CN.md` 的 `ratelimit:push-verify` 与 429 错误码表行,在限流器顺序修完后需要改
- `docs/roadmap.md` 不存在(之前有任务提到要创建,没落地)
- `audit_anchor_semantics.py` 有 7 个候选(历史遗留,非本轮引入)
- `docs/reference.zh-CN.md` 机翻质量问题(历史遗留)
---
## 8. 快速诊断命令
```bash
# 查看部署版本
curl -s https://messageweave.motofans.club/api/status | jq .
# 查看 push 注册状态(需要 worker_token)
TOKEN="你的token"
curl -s https://messageweave.motofans.club/api/push/status \
-H "Authorization: Bearer $TOKEN" | jq .
# 查看 Redis 中验证码 key 是否存在(需要 Redis 访问)
# key 格式: push:subscription:{subscription_id}
redis-cli GET "push:subscription:b"
# 查看 Stalwart 日志(在 mail.motofans.club 上)
docker logs <stalwart容器名> --since 10m 2>&1 | grep -i push
```

View File

@ -560,7 +560,7 @@ The design target form once included six additional variants: `Jmap(#[from] jmap
- axum adopts **0.8** (`ARCH-AXUM-08`); if subsequent review decides to adjust the version, Cargo.toml will prevail and this section will be synchronized. - axum adopts **0.8** (`ARCH-AXUM-08`); if subsequent review decides to adjust the version, Cargo.toml will prevail and this section will be synchronized.
- Currently SSE/WebSocket/long polling/SQLite/local volumes (`C-NO-LONG-CONN`/`NG-SQLITE-PERSIST`/`NG-LOCAL-VOLUME`) are not implemented; entrance authentication (`R1`/`SAF-AUTH-*`) is implemented as a fail-closed hard access control, and the Push, Streams worker and `/reconcile` business paths after authentication have been implemented. - Currently SSE/WebSocket/long polling/SQLite/local volumes (`C-NO-LONG-CONN`/`NG-SQLITE-PERSIST`/`NG-LOCAL-VOLUME`) are not implemented; entrance authentication (`R1`/`SAF-AUTH-*`) is implemented as a fail-closed hard access control, and the Push, Streams worker and `/reconcile` business paths after authentication have been implemented.
**Phase 0 P0 Gate (`GATE-P0`) - Passed. It was 45 passed / 0 failed / 1 ignored when it was first frozen, and has continued to grow in subsequent rounds ④–⑥. The current baseline is 106 passed / 0 failed / 4 ignored (2026-10-14 retest; see the `TEST_BASELINE` line of `docs/charter.md` for the authoritative value). The following are the criteria at the time of stage 0, which are retained for historical records: ** **Phase 0 P0 Gate (`GATE-P0`) - Passed. It was 45 passed / 0 failed / 1 ignored when it was first frozen, and has continued to grow in subsequent rounds ④–⑥, most recently 111 passed / 0 failed / 4 ignored after the 3 in-process stream drain tests (2026-10-16 retest; see the test-baseline line under `GATE-P0` in `docs/charter.md` for the authoritative value). The following are the criteria at the time of stage 0, which are retained for historical records: **
1. `cargo fmt --check` passes (no format difference). 1. `cargo fmt --check` passes (no format difference).
2. `cargo clippy --all-targets -- -D warnings` passes (zero warnings; crate-level `allow` is disabled). 2. `cargo clippy --all-targets -- -D warnings` passes (zero warnings; crate-level `allow` is disabled).
3. `cargo test` passed (including routing/configuration minimum test). 3. `cargo test` passed (including routing/configuration minimum test).

View File

@ -560,7 +560,7 @@ pub enum BotError {
- axum 采用 **0.8**(`ARCH-AXUM-08`);如后续审核决定调整版本,以 Cargo.toml 为准并同步本节。 - axum 采用 **0.8**(`ARCH-AXUM-08`);如后续审核决定调整版本,以 Cargo.toml 为准并同步本节。
- 当前不实现 SSE/WebSocket/长轮询/SQLite/本地卷(`C-NO-LONG-CONN`/`NG-SQLITE-PERSIST`/`NG-LOCAL-VOLUME`);入口鉴权(`R1`/`SAF-AUTH-*`)作为 fail-closed 硬门禁落地,鉴权之后的 Push、Streams worker 和 `/reconcile` 业务路径已实现。 - 当前不实现 SSE/WebSocket/长轮询/SQLite/本地卷(`C-NO-LONG-CONN`/`NG-SQLITE-PERSIST`/`NG-LOCAL-VOLUME`);入口鉴权(`R1`/`SAF-AUTH-*`)作为 fail-closed 硬门禁落地,鉴权之后的 Push、Streams worker 和 `/reconcile` 业务路径已实现。
**阶段0 P0 门禁(`GATE-P0`)——已通过。首次冻结时为 45 passed / 0 failed / 1 ignored,此后随 ④–⑥ 轮实现持续增长,当前基线为 106 passed / 0 failed / 4 ignored(2026-10-14 复测;权威值见 `docs/charter.md` 的 `TEST_BASELINE` 行)。以下为阶段0 当时的判据,保留作历史记录:** **阶段0 P0 门禁(`GATE-P0`)——已通过。首次冻结时为 45 passed / 0 failed / 1 ignored,此后随 ④–⑥ 轮实现持续增长,最近一次为进程内 stream drain 新增 3 个单测后的 111 passed / 0 failed / 4 ignored(2026-10-16 复测;权威值见 `docs/charter.md` `GATE-P0` 下的测试基线行)。以下为阶段0 当时的判据,保留作历史记录:**
1. `cargo fmt --check` 通过(无格式差异)。 1. `cargo fmt --check` 通过(无格式差异)。
2. `cargo clippy --all-targets -- -D warnings` 通过(零告警;禁 crate 级 `allow`)。 2. `cargo clippy --all-targets -- -D warnings` 通过(零告警;禁 crate 级 `allow`)。
3. `cargo test` 通过(含路由/配置最小测试)。 3. `cargo test` 通过(含路由/配置最小测试)。

View File

@ -111,6 +111,43 @@ Single accounts are fixed by requirement (`REQ-SINGLE-ACCOUNT`) and never enter
This item is not skipped, but explicitly rejected: the deployment unit saved by single instance and multiple accounts cannot be exchanged for the routing and cursor complexity it introduces. This item is not skipped, but explicitly rejected: the deployment unit saved by single instance and multiple accounts cannot be exchanged for the routing and cursor complexity it introduces.
### 4. Push 401: the push-verification key carried a TTL (closed 2026-10-13)
Symptom: Stalwart event pushes (`changed` batches hitting `jmap_push`) were rejected with 401 while the verification requests still passed; the cron reconcile path kept delivering, so mail arrived late instead of not at all.
Root cause: the JMAP push verification code is stored under `push:subscription:{id}` with a TTL that had started at 300 s. Stalwart verifies a subscription exactly once, at registration, and never re-verifies: once the key expired, every subsequent event push was rejected with 401, silently.
Fix chain (all committed):
- `d452f95`: TTL 300 s → 604_800 s (a week).
- `7331cb7`: → 0 (never expires). `ttl::PUSH_SUBSCRIPTION_SECONDS` is now 0, and the key is explicitly deleted by `forget_push_subscription` when a subscription is retired. A finite TTL here is a time bomb, not a cleanup mechanism.
- `f9160c0`: the verification response conforms to RFC 8620 §7.2 (JSON `{"verificationCode": …}`); the outbound verification API call is best-effort — a failure still records the subscription.
- `3b36fe3`: ordering fixed — `remember_push_subscription` runs before the rate limiter, so `ttl::PUSH_VERIFY_LIMIT_SECONDS` (30 s) now gates only the outbound API call and never the verification writeback.
Lesson: state that an external server checks only once, at registration, must not carry an expiry. The correct posture is TTL 0 + an explicit delete path, not "let Redis sweep it later".
### 5. Untimely delivery: the sole stream consumer was an out-of-process cron (closed 2026-10-16)
Symptom: mail reached JMAP but the Telegram notification did not arrive for days; the stream backlog only drained when someone invoked the external `/worker` by hand.
Root cause: the stream's only consumer was the external cron's `POST /reconcile` + `POST /worker`. When that cron was paused, enqueued messages sat unprocessed. (The deployment docs described `/worker` as "manual ops trigger" and deliberately kept it out of the Worker whitelist — that defect is recorded in the section "Deployment document defects discovered and fixed in this round".)
Fix (`ac0f291`, in-process stream drain):
- `notify::run_stream_drain` is spawned from `src/main.rs` as a 10 s loop (`ttl::INPROC_DRAIN_INTERVAL_SECONDS`), using the consumer `notify::INPROC_CONSUMER`; after each successful push reconcile it also does an immediate one-shot drain, so a push event is delivered within one drain interval instead of waiting for the next cron tick.
- The external `/worker` shares the same Redis consumer group `stalwart-workers`, so overlapping triggers are no-ops, not double-sends. It is demoted to an optional safety net, and `scripts/cron-drain.sh` was demoted accordingly.
- Charter invariant 21 ("no long polling scheduling") was re-stated in the same batch: the drain is a loop of short Redis round trips, not a long-lived connection.
Confirmation: 2026-10-16, real-machine user test — the full mail→Telegram chain was confirmed working by the operator ("现在收到邮件了").
Lesson: a sole delivery path must not depend on an out-of-process scheduler — cron can be paused or lost on restart. A Redis consumer group makes dual triggering idempotent, so the in-process loop and the external trigger can safely coexist.
## Process lessons from the two incidents
- **One-off handoff documents live outside every validator's document list.** `HANDOFF-push-401.md` (root, retired this round with a row in `docs/retired.md` "Deleted documents") carried "known residuals" that had already been fixed upstream, and no gate saw the drift. The stance is: retire such files, move reusable facts into the permanent documents, and verify on a real machine before closing the incident.
- **When a fix inverts an assumption, docs move in the same batch.** The in-process drain made "external cron is the only trigger" false in `README.md` §3.10, `docs/design.md`, `docs/reference.md`, this file, and `docs/charter.md` (tech-choice "Scheduling" row + invariant 21); all of it was synced in the same batch as `ac0f291`.
- **Real traffic is the final authority for "does it work".** Both incidents were closed by the operator's on-device confirmation, not by unit tests or doc-reading.
## Deployment document defects discovered and fixed in this round ## Deployment document defects discovered and fixed in this round
`README.md` §3.10 Originally, only the scheduling example of `/reconcile` was given, and `/worker` - the only queue consumption entry in the entire code base - was described as "manual triggering of operation and maintenance" and was deliberately not included in the Worker whitelist. **Copying the deployment according to the document = emails continue to be queued and notifications are never sent out**. This is the reason why the joint debugging environment is backlogged for several days before being emptied manually. `README.md` §3.10 The description of `/worker`’s scheduling steps, sequence requirements, batch limit and "204 cannot be used as a success signal" has been added. Later the in-process stream drain (`notify::run_stream_drain`, a spawned 10-second loop plus an immediate drain after each successful push reconcile) made the external `/worker` trigger an optional safety net instead of the only consumption path; `scripts/cron-drain.sh` was demoted accordingly. `README.md` §3.10 Originally, only the scheduling example of `/reconcile` was given, and `/worker` - the only queue consumption entry in the entire code base - was described as "manual triggering of operation and maintenance" and was deliberately not included in the Worker whitelist. **Copying the deployment according to the document = emails continue to be queued and notifications are never sent out**. This is the reason why the joint debugging environment is backlogged for several days before being emptied manually. `README.md` §3.10 The description of `/worker`’s scheduling steps, sequence requirements, batch limit and "204 cannot be used as a success signal" has been added. Later the in-process stream drain (`notify::run_stream_drain`, a spawned 10-second loop plus an immediate drain after each successful push reconcile) made the external `/worker` trigger an optional safety net instead of the only consumption path; `scripts/cron-drain.sh` was demoted accordingly.

View File

@ -111,6 +111,43 @@ TTL 那条真机绿灯已经用你给的 Upstash URL 跑过了(见 §1),
这条不是被跳过,是被明确拒绝:单实例多账户省下的一个部署单元,换不来它引入的路由与游标复杂度。 这条不是被跳过,是被明确拒绝:单实例多账户省下的一个部署单元,换不来它引入的路由与游标复杂度。
### 4. push 401:push 校验码带了 TTL(2026-10-13 已关闭)
现象:Stalwart 事件推送(命中 `jmap_push` 的 `changed` 批次)被 401 拒绝,而校验请求本身仍然通过;cron 对账路径继续投递,所以表现为「邮件迟到」而不是「邮件不到」。
根因:JMAP push 校验码存在 `push:subscription:{id}` 下,TTL 原为 300 s。Stalwart 只在订阅注册时校验一次,之后永不重新校验:键过期后,每次事件推送都被 401 拒绝,且没有任何告警。
修复链(均已提交):
- `d452f95`:TTL 300 s → 604_800 s(一周)。
- `7331cb7`:→ 0(永不过期)。`ttl::PUSH_SUBSCRIPTION_SECONDS` 现为 0,键在订阅退役时由 `forget_push_subscription` 显式删除。这里的有限 TTL 是定时炸弹,不是清理机制。
- `f9160c0`:校验响应符合 RFC 8620 §7.2(JSON `{"verificationCode": …}`);出站校验 API 调用改为尽力而为——调用失败仍记录订阅。
- `3b36fe3`:顺序修正——`remember_push_subscription` 移到速率限制器之前,`ttl::PUSH_VERIFY_LIMIT_SECONDS`(30 s)现在只限制出站 API 调用,不会挡校验码落库。
经验:外部服务器只在注册时校验一次的状态,绝不能带过期时间。正确姿态是 TTL 0 + 显式删除路径,而不是「让 Redis 以后扫掉」。
### 5. 投递不及时:唯一的 stream 消费者是进程外 cron(2026-10-16 已关闭)
现象:邮件到达 JMAP,但 Telegram 通知几天都没来;stream 积压直到有人手工触发外部 `/worker` 才排空。
根因:stream 的唯一消费者是外部 cron 的 `POST /reconcile` + `POST /worker`。cron 被暂停时,入队消息无人处理。(部署文档当时把 `/worker` 描述为「运维手工触发」并刻意排除在 Worker 白名单之外——该文档缺陷记录在下文「本轮发现并修复的部署文档问题」。)
修复(`ac0f291`,进程内 stream drain):
- `notify::run_stream_drain` 从 `src/main.rs` spawn 成一个 10 秒循环(`ttl::INPROC_DRAIN_INTERVAL_SECONDS`),消费者为 `notify::INPROC_CONSUMER`;每次成功的 push 对账之后立即排空一次,所以 push 事件在一次 drain 间隔内投递,不必等下一个 cron tick。
- 外部 `/worker` 与它共用同一个 Redis 消费者组 `stalwart-workers`,重叠触发是空操作而不是重复发送。`/worker` 降级为可选安全网,`scripts/cron-drain.sh` 相应降级。
- 章程不变量 21(「无长轮询调度」)在同一批里重新表述:drain 是短 Redis 往返的循环,不是长连接。
确认:2026-10-16 真机用户测试,邮件→Telegram 全链路由运维在设备上确认通过(「现在收到邮件了」)。
经验:唯一投递路径不能依赖进程外调度器——cron 会被暂停、会随重启丢失。Redis 消费者组让双重触发幂等,所以进程内循环与外部触发可以安全共存。
## 两次事故沉淀的流程经验
- **一次性交接文档不在任何校验器的文档清单里。** `HANDOFF-push-401.md`(仓库根,本轮退役并在 `docs/retired.md`「已删除文档」表留了一行)里的「已知残留」实际已在上游修复,而没有任何门禁发现这个漂移。对策:这类文件退役掉,可复用事实搬进常驻文档,事故收口前用真机验证。
- **修改变了一个假设时,文档必须同批改。** 进程内 drain 让 `README.md` §3.10、`docs/design.md`、`docs/reference.md`、本文件、`docs/charter.md`(技术选型「调度」行 + 不变量 21)里「外部 cron 是唯一触发器」的表述都变成了假的;这几处在 `ac0f291` 同一批内同步完毕。
- **真流量是「到底通不通」的最终权威。** 两次事故都以运维在设备上的确认为收口依据,而不是单测或读文档。
## 本轮发现并修复的部署文档问题 ## 本轮发现并修复的部署文档问题
`README.md` §3.10 原来只给了 `/reconcile` 的调度示例,而 `/worker`——全代码库唯一的队列消费入口——被描述成「运维手工触发」,且刻意不在 Worker 白名单里。**照文档照抄部署 = 邮件持续进队列、通知永远发不出去**,这正是联调环境积压了数天才被手工排空的成因。`README.md` §3.10 已补上 `/worker` 的调度步骤、顺序要求、批量上限与「204 不能当成功信号」的说明。后来进程内 stream drain(`notify::run_stream_drain`,一个被 spawn 的 10 秒循环 + 每次成功的 push 对账之后立即排空一次)把外部 `/worker` 触发从唯一消费路径降级成了**可选安全网**,`scripts/cron-drain.sh` 也相应降级。 `README.md` §3.10 原来只给了 `/reconcile` 的调度示例,而 `/worker`——全代码库唯一的队列消费入口——被描述成「运维手工触发」,且刻意不在 Worker 白名单里。**照文档照抄部署 = 邮件持续进队列、通知永远发不出去**,这正是联调环境积压了数天才被手工排空的成因。`README.md` §3.10 已补上 `/worker` 的调度步骤、顺序要求、批量上限与「204 不能当成功信号」的说明。后来进程内 stream drain(`notify::run_stream_drain`,一个被 spawn 的 10 秒循环 + 每次成功的 push 对账之后立即排空一次)把外部 `/worker` 触发从唯一消费路径降级成了**可选安全网**,`scripts/cron-drain.sh` 也相应降级。

View File

@ -110,6 +110,7 @@ These names never exist in the code. Some of them appear in the directory tree o
|---|---|---|---| |---|---|---|---|
| `docs/todo.md` | Superseded (deleted) | Unclear structure, overlapping with design/deployment, and once carrying historical narratives such as "this round has been closed" | First replaced by `docs/roadmap.md`, which was renamed and narrowed to `docs/opengaps.md` on 2026-09-29 (only unfinished/unverified gaps, blockages and next-stage goals are included) | | `docs/todo.md` | Superseded (deleted) | Unclear structure, overlapping with design/deployment, and once carrying historical narratives such as "this round has been closed" | First replaced by `docs/roadmap.md`, which was renamed and narrowed to `docs/opengaps.md` on 2026-09-29 (only unfinished/unverified gaps, blockages and next-stage goals are included) |
| Temporary handover files in the root directory (no such files are left) | Temporary handover files, **never entered the git history** | Handover content should be returned to the permanent document, no temporary files in the root directory are left | Contents are merged into `docs/design.md`, `docs/reference.md`, `docs/opengaps.md` | | Temporary handover files in the root directory (no such files are left) | Temporary handover files, **never entered the git history** | Handover content should be returned to the permanent document, no temporary files in the root directory are left | Contents are merged into `docs/design.md`, `docs/reference.md`, `docs/opengaps.md` |
| `HANDOFF-push-401.md` | Retired handover document (deleted 2026-10-16; committed as `09a108b`, still in git history) | One-off handover for the 2026-10-13 push-401 incident. All follow-ups closed: TTL of the push-verification key `push:subscription:{id}` 300s→0 (commits `d452f95` / `7331cb7`), verification reply / best-effort API call (`f9160c0`), verification-writeback vs. rate-limiter ordering (`3b36fe3`), and the full chain was confirmed by a real-machine user test on 2026-10-16. Its "two delivery paths" table is superseded by the in-process drain (`ac0f291`); the file was outside every validator's document list | Closing records and lessons in `docs/opengaps.md` §4 / §5; current facts in `docs/reference.md` |
--- ---

View File

@ -110,6 +110,7 @@
|---|---|---|---| |---|---|---|---|
| `docs/todo.md` | 已取代(已删除) | 结构不清,与 design/deployment 重叠,且曾承载「本轮已收口」这类历史叙述 | 先由 `docs/roadmap.md` 取代,后者于 2026-09-29 更名并收窄为 `docs/opengaps.md`(只放仍未完成/未验证的缺口、阻塞与下一阶段目标) | | `docs/todo.md` | 已取代(已删除) | 结构不清,与 design/deployment 重叠,且曾承载「本轮已收口」这类历史叙述 | 先由 `docs/roadmap.md` 取代,后者于 2026-09-29 更名并收窄为 `docs/opengaps.md`(只放仍未完成/未验证的缺口、阻塞与下一阶段目标) |
| 根目录临时交接件(不留此类文件) | 临时交接件,**从未进入 git 历史** | 交接内容应归位到常驻文档,不留根目录临时件 | 内容并入 `docs/design.md`、`docs/reference.md`、`docs/opengaps.md` | | 根目录临时交接件(不留此类文件) | 临时交接件,**从未进入 git 历史** | 交接内容应归位到常驻文档,不留根目录临时件 | 内容并入 `docs/design.md`、`docs/reference.md`、`docs/opengaps.md` |
| `HANDOFF-push-401.md` | 一次性交接文档(2026-10-16 删除;由 `09a108b` 提交,git 历史可证) | 2026-10-13 push-401 事故的一次性交接件。后续项全部关闭:push 校验键 `push:subscription:{id}` 的 TTL 300s→0(提交 `d452f95` / `7331cb7`)、验证回执 / 尽力 API 调用(`f9160c0`)、验证写回与限流器的先后顺序(`3b36fe3`),且 2026-10-16 已由用户真机测试确认全链路可用。其「两条投递路径」表已被进程内 drain(`ac0f291`)取代;该文件不在任何校验器的文档清单里 | 关闭记录与经验见 `docs/opengaps.md` §4 / §5;现行事实见 `docs/reference.md` |
--- ---