From 5339656c9bd03b8e57e25488b63202c2d41d4282 Mon Sep 17 00:00:00 2001 From: anonsaber Date: Fri, 16 Oct 2026 23:44:02 +0800 Subject: [PATCH] docs: retire HANDOFF-push-401.md; record incident lessons; fix stale baseline claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Delete HANDOFF-push-401.md (fully superseded: TTL fix chain closed, rate-limiter ordering fixed in 3b36fe3, live chain confirmed 2026-10-16, two-path table superseded by in-process drain ac0f291). Retirement row added to docs/retired.md / retired.zh-CN.md. - docs/opengaps.md + opengaps.zh-CN.md: new closing-record sections §4 (push-401 incident, TTL 0 lesson) and §5 (untimely delivery, sole consumer was out-of-process cron) plus process lessons (handoff docs live outside validators; live user test is final authority). - docs/design.md + design.zh-CN.md: replace stale 106-passed baseline and phantom TEST_BASELINE anchor with the 111/0/4 (2026-10-16) value citing the test-baseline line under GATE-P0 in docs/charter.md. Co-Authored-By: Claude Code --- HANDOFF-push-401.md | 213 ----------------------------------------- docs/design.md | 2 +- docs/design.zh-CN.md | 2 +- docs/opengaps.md | 37 +++++++ docs/opengaps.zh-CN.md | 37 +++++++ docs/retired.md | 1 + docs/retired.zh-CN.md | 1 + 7 files changed, 78 insertions(+), 215 deletions(-) delete mode 100644 HANDOFF-push-401.md diff --git a/HANDOFF-push-401.md b/HANDOFF-push-401.md deleted file mode 100644 index 1ffc9df..0000000 --- a/HANDOFF-push-401.md +++ /dev/null @@ -1,213 +0,0 @@ -# HANDOFF — 邮件 Push 通知投递问题 - -> **日期**: 2026-10-13 -> **状态**: 代码修复已完成，**未部署** -> **接手方优先级**: P0 — 用户阻塞中，邮件 push 收不到 - ---- - -## 1. 问题描述 - -**症状**: 用户配置了 Stalwart 邮件服务器 + Cloudflare Worker + Telegram Bot，通过 **cron 定时任务**（`/reconcile` 路径）可以正常收到邮件通知，但 **邮件 Push**（Stalwart → `/push/jmap` 路径）**完全收不到**。 - -**用户已确认的现象**: -- `/help` 等 Telegram 命令秒回 -- cron 路径（`/reconcile` → JMAP `/changes` → worker → Telegram）工作正常 -- 邮件 push 通知（Stalwart → `/push/jmap`）从未到达 Telegram - ---- - -## 2. 根因分析（已完成，结论确认） - -### 2.1 根因: 验证码 key 5 分钟后过期（旧代码 `TTL=300`） - -Stalwart 按 RFC 8620 协议向我们注册 Push 订阅时: - -1. **注册时**: Stalwart 发第一次带 `verificationCode` 的验证请求，我们的后端把验证码存进 Redis（key = `push:subscription:{id}`，value = 验证码）。**旧代码 TTL = 300 秒（5 分钟）**。 -2. **事件推送时**: 每封新邮件到达后，Stalwart 发带 `emailId`/`changeType` 但**不带** `verificationCode` 的事件推送。我们的后端去 Redis 查 `push:subscription:{id}` 确认订阅已验证 — 但 key 已在 5 分钟后过期 → 返回 **401 unauthorized** → 邮件通知被丢弃。 - -### 2.2 时间线证据（来自 Stalwart 日志） - -| 时间 (UTC) | 事件 | -|---|---| -| 02:52:30 | 用户执行 register，Stalwart 分配 `push_subscription_id: "b"`，向 `/push/jmap` 发第一次验证请求，后端存 key（TTL=300） | -| ~02:57:30 | Redis key 过期被自动删除 | -| 02:58:49 | 测试邮件到达 Stalwart，changeId=305，documentId=167 | -| 02:58:50 | Stalwart 向 `/push/jmap` 推送事件 → 后端查 key → **401 unauthorized** | - -**注册到 401 恰好间隔 ~6 分钟，与 TTL=300（5 分钟）完全吻合。** - -### 2.3 为什么 cron 能收到 - -cron 走 `/reconcile` → 主动调 JMAP `/changes` → 拉新邮件 → 发 Telegram。这条路径**完全不依赖验证码 key**，所以不受 TTL 影响。push 路径依赖 key 核对，key 过期就 401。 - -### 2.4 次要问题: 限流器 429 - -注册日志里出现 `429 push_verify_rate_limited` — Stalwart 在 30 秒内重复发了验证请求，第二个被限流器（`ratelimit:push-verify:{subscription}`，TTL=30s）拦住。但**第一个请求已成功存了 key**（只是 5 分钟后过期），所以 429 不是当前 401 的原因，只是鲁棒性问题。 - ---- - -## 3. 已完成的修复（3 个 commit，均已 push 双 remote） - -| Commit | 修复内容 | 关键代码位置 | -|---|---|---| -| `d452f95` | `PUSH_SUBSCRIPTION_SECONDS` 300 → 604800（7 天） | `src/state.rs` ttl 模块 | -| `7331cb7` | `PUSH_SUBSCRIPTION_SECONDS` 604800 → **0（永不过期）** | `src/state.rs` ttl 模块 | -| `f9160c0` | ① 验证响应回 `{"verificationCode":"..."}`（原返回空 204，违反 RFC 8620 §7.2）；② `verify_push_subscription` JMAP API 改为 best-effort（失败不阻塞） | `src/notify.rs` `jmap_push` 验证分支 | - -### 3.1 关键代码变化 - -**`src/state.rs`** — TTL 常量: -```rust -pub const PUSH_SUBSCRIPTION_SECONDS: u64 = 0; // 永不过期 -``` -`remember_push_subscription` 处理 `ttl_seconds == 0` 时用 `SET key value`（不带 `EX`），MemoryState 用 `expiry = i64::MAX`。 - -**`src/notify.rs`** — 验证流程（当前代码，已修但未部署）: -``` -if 带 verificationCode（验证请求）: - 1. rate limiter claim_dedup → 429 拦截（❗这在 remember 之前，有问题） - 2. verify_push_subscription JMAP API → best-effort（warn 不阻塞） - 3. remember_push_subscription → 存 key（TTL=0 永不过期） - 4. set_push_subscription_status → "verified" -else（事件推送）: - 查 push:subscription:{id} 是否存在 → 不存在 401 -``` - -**RFC 8620 §7.2 响应**（`notify.rs` 的 `jmap_push` handler，验证请求分支末尾）: -```rust -if let Some(code) = push.verification_code.as_deref() { - return axum::Json(serde_json::json!({ - "verificationCode": code - })).into_response(); -} -return StatusCode::NO_CONTENT.into_response(); -``` - ---- - -## 4. 未完成的工作（接手方要做） - -### 4.1 【P0】部署最新代码到服务器 - -**当前服务器跑的是旧代码（TTL=300），三个修复 commit 都在 git 里但没部署。** - -部署方式（Docker）: -```bash -# 在服务器上 -git pull origin main # 拉到 f9160c0 或更新 -docker build -t messageweave . -docker stop <容器名> -docker run --rm --env-file .env -p 8080:8080 messageweave -``` - -或者如果不用 Docker: -```bash -git pull origin main -cargo build --locked --release -# 重启运行中的 message-weave 进程 -``` - -### 4.2 【P0】部署后重新注册 Push 订阅 - -部署新代码后，旧的 key 可能还残留（或已过期），需要让 Stalwart 重新走一次验证流程: - -```bash -# 变量 -BACKEND="https://messageweave.motofans.club" -TOKEN="你的 worker_token（业务配置里的）" - -# 1. 禁用旧订阅 -curl -X POST "$BACKEND/api/push/disable" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"callback_url":"https://messageweave.motofans.club/push/jmap"}' - -# 2. 重新注册（Stalwart 会发新验证请求，新代码存 key 且永不过期） -curl -X POST "$BACKEND/api/push/register" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"callback_url":"https://messageweave.motofans.club/push/jmap"}' - -# 3. 发测试邮件，观察 Stalwart 日志确认不再 401 -``` - -### 4.3 【P1】修复限流器顺序（代码改动） - -**问题**: `notify.rs` `jmap_push` 的验证分支中，rate limiter（`claim_dedup("ratelimit:push-verify:{subscription}")`）在 `remember_push_subscription` **之前**。如果 Stalwart 在 30 秒内重发验证请求，第二个请求被 429 拦住 → `remember_push_subscription` 不会被调用 → 那次尝试的 key 没存。虽然第一个请求通常已存了 key，但这不鲁棒。 - -**修法**: 把 `remember_push_subscription` 移到 rate limiter **之前**，rate limiter 只控制**出站** `verify_push_subscription` JMAP API 调用（防止频繁调外部 API），**不拦验证请求本身**。 - -修改后流程应该是: -``` -if 带 verificationCode: - 1. remember_push_subscription（先存 key，TTL=0） ← 移到最前 - 2. set_push_subscription_status → "verified" - 3. rate limiter claim_dedup → 只拦 JMAP API 调用，不拦整个请求 - 4. verify_push_subscription JMAP API → best-effort -``` - -**需要同步更新的文档**（改代码后必须一起改，否则 GATE-DOCS 会挂）: -- `docs/reference.md` / `docs/reference.zh-CN.md` — `ratelimit:push-verify:{subscription}` 键说明行 + 429 `push_verify_rate_limited` 错误码表行 -- `docs/opengaps.md` / `docs/opengaps.zh-CN.md` — TTL 表 `PUSH_VERIFY_LIMIT_SECONDS` 行 -- `docs/design.md` / `docs/design.zh-CN.md` — "Push verification code writing current limiting" 段落 - -### 4.4 【P2】验证完整链路 - -部署 + 重新注册后: -1. 发测试邮件 -2. 看 Stalwart 日志: 验证请求应该返回 200 + `{"verificationCode":"..."}`，事件推送应该返回 2xx（不再 401） -3. 看 Telegram 是否收到通知 -4. 等 10+ 分钟再发第二封（确认 key 不再过期） - ---- - -## 5. 两条投递路径说明 - -| 路径 | 触发方式 | 依赖验证码 key | 当前状态 | -|---|---|---|---| -| **Cron** | 定时调 `/reconcile` → JMAP `/changes` → 拉新邮件 → worker → Telegram | ❌ 不依赖 | ✅ 正常工作 | -| **Push** | Stalwart → `/push/jmap`（事件推送）→ 查 key → worker → Telegram | ✅ 依赖（401 的原因） | ❌ 旧代码 key 过期 → 401 | - ---- - -## 6. 环境信息 - -- **Stalwart**: `0.16.26`，Docker 运行在 `mail.motofans.club` -- **Push 订阅 ID**: `"b"`（Stalwart 分配的单字符 ID） -- **回调 URL**: `https://messageweave.motofans.club/push/jmap` -- **后端仓库**: `/home/okabe/Repo/messageweave` -- **双 remote**: `origin`（motofans.club）+ `upstream`（GitHub fork） -- **测试基线**: Rust 106 passed / 0 failed / 4 ignored，GATE-DOCS 6/6 PASS -- **最新 commit**: `f9160c0`（已 push 双 remote） - ---- - -## 7. 已知的文档残留（非阻塞） - -- `docs/opengaps.md` 声称"当前无未完成项"，但限流器顺序问题（§4.3）还没修 — 接手方修完后应更新 -- `docs/reference.md` / `docs/reference.zh-CN.md` 的 `ratelimit:push-verify` 与 429 错误码表行，在限流器顺序修完后需要改 -- `docs/roadmap.md` 不存在（之前有任务提到要创建，没落地） -- `audit_anchor_semantics.py` 有 7 个候选（历史遗留，非本轮引入） -- `docs/reference.zh-CN.md` 机翻质量问题（历史遗留） - ---- - -## 8. 快速诊断命令 - -```bash -# 查看部署版本 -curl -s https://messageweave.motofans.club/api/status | jq . - -# 查看 push 注册状态（需要 worker_token） -TOKEN="你的token" -curl -s https://messageweave.motofans.club/api/push/status \ - -H "Authorization: Bearer $TOKEN" | jq . - -# 查看 Redis 中验证码 key 是否存在（需要 Redis 访问） -# key 格式: push:subscription:{subscription_id} -redis-cli GET "push:subscription:b" - -# 查看 Stalwart 日志（在 mail.motofans.club 上） -docker logs --since 10m 2>&1 | grep -i push -``` diff --git a/docs/design.md b/docs/design.md index 9628752..8cd4662 100644 --- a/docs/design.md +++ b/docs/design.md @@ -560,7 +560,7 @@ The design target form once included six additional variants: `Jmap(#[from] jmap - axum adopts **0.8** (`ARCH-AXUM-08`); if subsequent review decides to adjust the version, Cargo.toml will prevail and this section will be synchronized. - Currently SSE/WebSocket/long polling/SQLite/local volumes (`C-NO-LONG-CONN`/`NG-SQLITE-PERSIST`/`NG-LOCAL-VOLUME`) are not implemented; entrance authentication (`R1`/`SAF-AUTH-*`) is implemented as a fail-closed hard access control, and the Push, Streams worker and `/reconcile` business paths after authentication have been implemented. -**Phase 0 P0 Gate (`GATE-P0`) - Passed. It was 45 passed / 0 failed / 1 ignored when it was first frozen, and has continued to grow in subsequent rounds ④–⑥. The current baseline is 106 passed / 0 failed / 4 ignored (2026-10-14 retest; see the `TEST_BASELINE` line of `docs/charter.md` for the authoritative value). The following are the criteria at the time of stage 0, which are retained for historical records: ** +**Phase 0 P0 Gate (`GATE-P0`) - Passed. It was 45 passed / 0 failed / 1 ignored when it was first frozen, and has continued to grow in subsequent rounds ④–⑥, most recently 111 passed / 0 failed / 4 ignored after the 3 in-process stream drain tests (2026-10-16 retest; see the test-baseline line under `GATE-P0` in `docs/charter.md` for the authoritative value). The following are the criteria at the time of stage 0, which are retained for historical records: ** 1. `cargo fmt --check` passes (no format difference). 2. `cargo clippy --all-targets -- -D warnings` passes (zero warnings; crate-level `allow` is disabled). 3. `cargo test` passed (including routing/configuration minimum test). diff --git a/docs/design.zh-CN.md b/docs/design.zh-CN.md index aa0cee2..bb745c2 100644 --- a/docs/design.zh-CN.md +++ b/docs/design.zh-CN.md @@ -560,7 +560,7 @@ pub enum BotError { - axum 采用 **0.8**（`ARCH-AXUM-08`）；如后续审核决定调整版本，以 Cargo.toml 为准并同步本节。 - 当前不实现 SSE/WebSocket/长轮询/SQLite/本地卷（`C-NO-LONG-CONN`/`NG-SQLITE-PERSIST`/`NG-LOCAL-VOLUME`）；入口鉴权（`R1`/`SAF-AUTH-*`）作为 fail-closed 硬门禁落地，鉴权之后的 Push、Streams worker 和 `/reconcile` 业务路径已实现。 -**阶段0 P0 门禁（`GATE-P0`）——已通过。首次冻结时为 45 passed / 0 failed / 1 ignored，此后随 ④–⑥ 轮实现持续增长，当前基线为 106 passed / 0 failed / 4 ignored（2026-10-14 复测；权威值见 `docs/charter.md` 的 `TEST_BASELINE` 行）。以下为阶段0 当时的判据，保留作历史记录：** +**阶段0 P0 门禁（`GATE-P0`）——已通过。首次冻结时为 45 passed / 0 failed / 1 ignored，此后随 ④–⑥ 轮实现持续增长，最近一次为进程内 stream drain 新增 3 个单测后的 111 passed / 0 failed / 4 ignored（2026-10-16 复测；权威值见 `docs/charter.md` `GATE-P0` 下的测试基线行）。以下为阶段0 当时的判据，保留作历史记录：** 1. `cargo fmt --check` 通过（无格式差异）。 2. `cargo clippy --all-targets -- -D warnings` 通过（零告警；禁 crate 级 `allow`）。 3. `cargo test` 通过（含路由/配置最小测试）。 diff --git a/docs/opengaps.md b/docs/opengaps.md index c1d0150..e178055 100644 --- a/docs/opengaps.md +++ b/docs/opengaps.md @@ -111,6 +111,43 @@ Single accounts are fixed by requirement (`REQ-SINGLE-ACCOUNT`) and never enter This item is not skipped, but explicitly rejected: the deployment unit saved by single instance and multiple accounts cannot be exchanged for the routing and cursor complexity it introduces. +### 4. Push 401: the push-verification key carried a TTL (closed 2026-10-13) + +Symptom: Stalwart event pushes (`changed` batches hitting `jmap_push`) were rejected with 401 while the verification requests still passed; the cron reconcile path kept delivering, so mail arrived late instead of not at all. + +Root cause: the JMAP push verification code is stored under `push:subscription:{id}` with a TTL that had started at 300 s. Stalwart verifies a subscription exactly once, at registration, and never re-verifies: once the key expired, every subsequent event push was rejected with 401, silently. + +Fix chain (all committed): + +- `d452f95`: TTL 300 s → 604_800 s (a week). +- `7331cb7`: → 0 (never expires). `ttl::PUSH_SUBSCRIPTION_SECONDS` is now 0, and the key is explicitly deleted by `forget_push_subscription` when a subscription is retired. A finite TTL here is a time bomb, not a cleanup mechanism. +- `f9160c0`: the verification response conforms to RFC 8620 §7.2 (JSON `{"verificationCode": …}`); the outbound verification API call is best-effort — a failure still records the subscription. +- `3b36fe3`: ordering fixed — `remember_push_subscription` runs before the rate limiter, so `ttl::PUSH_VERIFY_LIMIT_SECONDS` (30 s) now gates only the outbound API call and never the verification writeback. + +Lesson: state that an external server checks only once, at registration, must not carry an expiry. The correct posture is TTL 0 + an explicit delete path, not "let Redis sweep it later". + +### 5. Untimely delivery: the sole stream consumer was an out-of-process cron (closed 2026-10-16) + +Symptom: mail reached JMAP but the Telegram notification did not arrive for days; the stream backlog only drained when someone invoked the external `/worker` by hand. + +Root cause: the stream's only consumer was the external cron's `POST /reconcile` + `POST /worker`. When that cron was paused, enqueued messages sat unprocessed. (The deployment docs described `/worker` as "manual ops trigger" and deliberately kept it out of the Worker whitelist — that defect is recorded in the section "Deployment document defects discovered and fixed in this round".) + +Fix (`ac0f291`, in-process stream drain): + +- `notify::run_stream_drain` is spawned from `src/main.rs` as a 10 s loop (`ttl::INPROC_DRAIN_INTERVAL_SECONDS`), using the consumer `notify::INPROC_CONSUMER`; after each successful push reconcile it also does an immediate one-shot drain, so a push event is delivered within one drain interval instead of waiting for the next cron tick. +- The external `/worker` shares the same Redis consumer group `stalwart-workers`, so overlapping triggers are no-ops, not double-sends. It is demoted to an optional safety net, and `scripts/cron-drain.sh` was demoted accordingly. +- Charter invariant 21 ("no long polling scheduling") was re-stated in the same batch: the drain is a loop of short Redis round trips, not a long-lived connection. + +Confirmation: 2026-10-16, real-machine user test — the full mail→Telegram chain was confirmed working by the operator ("现在收到邮件了"). + +Lesson: a sole delivery path must not depend on an out-of-process scheduler — cron can be paused or lost on restart. A Redis consumer group makes dual triggering idempotent, so the in-process loop and the external trigger can safely coexist. + +## Process lessons from the two incidents + +- **One-off handoff documents live outside every validator's document list.** `HANDOFF-push-401.md` (root, retired this round with a row in `docs/retired.md` "Deleted documents") carried "known residuals" that had already been fixed upstream, and no gate saw the drift. The stance is: retire such files, move reusable facts into the permanent documents, and verify on a real machine before closing the incident. +- **When a fix inverts an assumption, docs move in the same batch.** The in-process drain made "external cron is the only trigger" false in `README.md` §3.10, `docs/design.md`, `docs/reference.md`, this file, and `docs/charter.md` (tech-choice "Scheduling" row + invariant 21); all of it was synced in the same batch as `ac0f291`. +- **Real traffic is the final authority for "does it work".** Both incidents were closed by the operator's on-device confirmation, not by unit tests or doc-reading. + ## Deployment document defects discovered and fixed in this round `README.md` §3.10 Originally, only the scheduling example of `/reconcile` was given, and `/worker` - the only queue consumption entry in the entire code base - was described as "manual triggering of operation and maintenance" and was deliberately not included in the Worker whitelist. **Copying the deployment according to the document = emails continue to be queued and notifications are never sent out**. This is the reason why the joint debugging environment is backlogged for several days before being emptied manually. `README.md` §3.10 The description of `/worker`’s scheduling steps, sequence requirements, batch limit and "204 cannot be used as a success signal" has been added. Later the in-process stream drain (`notify::run_stream_drain`, a spawned 10-second loop plus an immediate drain after each successful push reconcile) made the external `/worker` trigger an optional safety net instead of the only consumption path; `scripts/cron-drain.sh` was demoted accordingly. diff --git a/docs/opengaps.zh-CN.md b/docs/opengaps.zh-CN.md index 1767b5d..ba7527d 100644 --- a/docs/opengaps.zh-CN.md +++ b/docs/opengaps.zh-CN.md @@ -111,6 +111,43 @@ TTL 那条真机绿灯已经用你给的 Upstash URL 跑过了（见 §1）， 这条不是被跳过，是被明确拒绝：单实例多账户省下的一个部署单元，换不来它引入的路由与游标复杂度。 +### 4. push 401：push 校验码带了 TTL（2026-10-13 已关闭） + +现象：Stalwart 事件推送（命中 `jmap_push` 的 `changed` 批次）被 401 拒绝，而校验请求本身仍然通过；cron 对账路径继续投递，所以表现为「邮件迟到」而不是「邮件不到」。 + +根因：JMAP push 校验码存在 `push:subscription:{id}` 下，TTL 原为 300 s。Stalwart 只在订阅注册时校验一次，之后永不重新校验：键过期后，每次事件推送都被 401 拒绝，且没有任何告警。 + +修复链（均已提交）： + +- `d452f95`：TTL 300 s → 604_800 s（一周）。 +- `7331cb7`：→ 0（永不过期）。`ttl::PUSH_SUBSCRIPTION_SECONDS` 现为 0，键在订阅退役时由 `forget_push_subscription` 显式删除。这里的有限 TTL 是定时炸弹，不是清理机制。 +- `f9160c0`：校验响应符合 RFC 8620 §7.2（JSON `{"verificationCode": …}`）；出站校验 API 调用改为尽力而为——调用失败仍记录订阅。 +- `3b36fe3`：顺序修正——`remember_push_subscription` 移到速率限制器之前，`ttl::PUSH_VERIFY_LIMIT_SECONDS`（30 s）现在只限制出站 API 调用，不会挡校验码落库。 + +经验：外部服务器只在注册时校验一次的状态，绝不能带过期时间。正确姿态是 TTL 0 + 显式删除路径，而不是「让 Redis 以后扫掉」。 + +### 5. 投递不及时：唯一的 stream 消费者是进程外 cron（2026-10-16 已关闭） + +现象：邮件到达 JMAP，但 Telegram 通知几天都没来；stream 积压直到有人手工触发外部 `/worker` 才排空。 + +根因：stream 的唯一消费者是外部 cron 的 `POST /reconcile` + `POST /worker`。cron 被暂停时，入队消息无人处理。（部署文档当时把 `/worker` 描述为「运维手工触发」并刻意排除在 Worker 白名单之外——该文档缺陷记录在下文「本轮发现并修复的部署文档问题」。） + +修复（`ac0f291`，进程内 stream drain）： + +- `notify::run_stream_drain` 从 `src/main.rs` spawn 成一个 10 秒循环（`ttl::INPROC_DRAIN_INTERVAL_SECONDS`），消费者为 `notify::INPROC_CONSUMER`；每次成功的 push 对账之后立即排空一次，所以 push 事件在一次 drain 间隔内投递，不必等下一个 cron tick。 +- 外部 `/worker` 与它共用同一个 Redis 消费者组 `stalwart-workers`，重叠触发是空操作而不是重复发送。`/worker` 降级为可选安全网，`scripts/cron-drain.sh` 相应降级。 +- 章程不变量 21（「无长轮询调度」）在同一批里重新表述：drain 是短 Redis 往返的循环，不是长连接。 + +确认：2026-10-16 真机用户测试，邮件→Telegram 全链路由运维在设备上确认通过（「现在收到邮件了」）。 + +经验：唯一投递路径不能依赖进程外调度器——cron 会被暂停、会随重启丢失。Redis 消费者组让双重触发幂等，所以进程内循环与外部触发可以安全共存。 + +## 两次事故沉淀的流程经验 + +- **一次性交接文档不在任何校验器的文档清单里。** `HANDOFF-push-401.md`（仓库根，本轮退役并在 `docs/retired.md`「已删除文档」表留了一行）里的「已知残留」实际已在上游修复，而没有任何门禁发现这个漂移。对策：这类文件退役掉，可复用事实搬进常驻文档，事故收口前用真机验证。 +- **修改变了一个假设时，文档必须同批改。** 进程内 drain 让 `README.md` §3.10、`docs/design.md`、`docs/reference.md`、本文件、`docs/charter.md`（技术选型「调度」行 + 不变量 21）里「外部 cron 是唯一触发器」的表述都变成了假的；这几处在 `ac0f291` 同一批内同步完毕。 +- **真流量是「到底通不通」的最终权威。** 两次事故都以运维在设备上的确认为收口依据，而不是单测或读文档。 + ## 本轮发现并修复的部署文档问题 `README.md` §3.10 原来只给了 `/reconcile` 的调度示例，而 `/worker`——全代码库唯一的队列消费入口——被描述成「运维手工触发」，且刻意不在 Worker 白名单里。**照文档照抄部署 = 邮件持续进队列、通知永远发不出去**，这正是联调环境积压了数天才被手工排空的成因。`README.md` §3.10 已补上 `/worker` 的调度步骤、顺序要求、批量上限与「204 不能当成功信号」的说明。后来进程内 stream drain（`notify::run_stream_drain`，一个被 spawn 的 10 秒循环 + 每次成功的 push 对账之后立即排空一次）把外部 `/worker` 触发从唯一消费路径降级成了**可选安全网**，`scripts/cron-drain.sh` 也相应降级。 diff --git a/docs/retired.md b/docs/retired.md index f93592c..08cfc51 100644 --- a/docs/retired.md +++ b/docs/retired.md @@ -110,6 +110,7 @@ These names never exist in the code. Some of them appear in the directory tree o |---|---|---|---| | `docs/todo.md` | Superseded (deleted) | Unclear structure, overlapping with design/deployment, and once carrying historical narratives such as "this round has been closed" | First replaced by `docs/roadmap.md`, which was renamed and narrowed to `docs/opengaps.md` on 2026-09-29 (only unfinished/unverified gaps, blockages and next-stage goals are included) | | Temporary handover files in the root directory (no such files are left) | Temporary handover files, **never entered the git history** | Handover content should be returned to the permanent document, no temporary files in the root directory are left | Contents are merged into `docs/design.md`, `docs/reference.md`, `docs/opengaps.md` | +| `HANDOFF-push-401.md` | Retired handover document (deleted 2026-10-16; committed as `09a108b`, still in git history) | One-off handover for the 2026-10-13 push-401 incident. All follow-ups closed: TTL of the push-verification key `push:subscription:{id}` 300s→0 (commits `d452f95` / `7331cb7`), verification reply / best-effort API call (`f9160c0`), verification-writeback vs. rate-limiter ordering (`3b36fe3`), and the full chain was confirmed by a real-machine user test on 2026-10-16. Its "two delivery paths" table is superseded by the in-process drain (`ac0f291`); the file was outside every validator's document list | Closing records and lessons in `docs/opengaps.md` §4 / §5; current facts in `docs/reference.md` | --- diff --git a/docs/retired.zh-CN.md b/docs/retired.zh-CN.md index 15b375f..7293508 100644 --- a/docs/retired.zh-CN.md +++ b/docs/retired.zh-CN.md @@ -110,6 +110,7 @@ |---|---|---|---| | `docs/todo.md` | 已取代（已删除） | 结构不清，与 design/deployment 重叠，且曾承载「本轮已收口」这类历史叙述 | 先由 `docs/roadmap.md` 取代，后者于 2026-09-29 更名并收窄为 `docs/opengaps.md`（只放仍未完成/未验证的缺口、阻塞与下一阶段目标） | | 根目录临时交接件（不留此类文件） | 临时交接件，**从未进入 git 历史** | 交接内容应归位到常驻文档，不留根目录临时件 | 内容并入 `docs/design.md`、`docs/reference.md`、`docs/opengaps.md` | +| `HANDOFF-push-401.md` | 一次性交接文档（2026-10-16 删除；由 `09a108b` 提交，git 历史可证） | 2026-10-13 push-401 事故的一次性交接件。后续项全部关闭：push 校验键 `push:subscription:{id}` 的 TTL 300s→0（提交 `d452f95` / `7331cb7`）、验证回执 / 尽力 API 调用（`f9160c0`）、验证写回与限流器的先后顺序（`3b36fe3`），且 2026-10-16 已由用户真机测试确认全链路可用。其「两条投递路径」表已被进程内 drain（`ac0f291`）取代；该文件不在任何校验器的文档清单里 | 关闭记录与经验见 `docs/opengaps.md` §4 / §5；现行事实见 `docs/reference.md` | ---