The P0 push-delivery problem is stuck on an unresolvable contradiction:
push:subscription:b is confirmed in Redis (TTL=-1) and a manual curl
{"pushSubscriptionId":"b"} returns 204, yet Stalwart's real event push
gets 401 from the same dedup_exists check.
Reconstructing the timeline from email-arrival timestamps is unreliable.
Instead, add diagnostic logging so the next real push produces evidence:
- Entry log: subscription_id, account_id, email_id, has_verification_code
- dedup_exists outcome: which key was queried and whether it hit
- verify-branch outcome: when a verification request rewrites the key
- enqueue success: when an event is accepted for delivery
A single request_id is minted at entry and threaded through push_401 so
the request_id in Stalwart's 401 response matches the entry/branch logs
(previously error_response minted a fresh random id per call, making
correlation impossible).
No delivery logic, TTL, key-writing, limiter ordering, or routes changed.
All info! lines are temporary diagnostics, removed after root cause found.
GATE-P0: 106 passed / 0 failed / 4 ignored
GATE-DOCS: 6/6 PASS
Co-Authored-By: Claude Code <noreply@anthropic.com>
The previous round (88f0923) threaded a `backend_id` from `BACKEND_ID`/`HOSTNAME`
env vars into `AppState`. That required the operator to manually align each
instance's `BACKEND_ID` with the Cloudflare Worker's `BACKEND_ORIGINS_JSON`, which
is exactly the kind of manual cross-config sync that drifts.
The request's own `Host` header already carries the answer: the Worker load
balancer (`ARCH-LB-WORKER`) preserves the inbound `Host` while rewriting the target
to a backend origin (`joinUrl` builds `origin + path + query`), so `Host` is the
public address the LB routed this instance as. `backend_label_from_headers` reads
it per-request in `setup_status`, trims the default https/http port
(`example.com:443` -> `example.com`) so the label mirrors a bare origin, and returns
`None` when the request carries no usable `Host`.
This removes the startup env entirely — `backend_id_from_env`, `AppState::backend_id`,
all five router-builder `backend_id` parameters, the `.env.example` `BACKEND_ID`
entry, and the `BACKEND_ID` env documentation are all gone. Handlers still never
touch the environment: `Host` is request-scoped, not process-global, so there is no
test race (the `debug_surface_requested` rationale was about env, not headers).
Tests: `setup_status_reports_backend_label` now sends `Host: edge-1.example.com:443`
and asserts the label is the bare origin `edge-1.example.com` (port trimmed). A new
`setup_status_reports_null_backend_without_host` asserts `null` when no Host is sent.
GATE-P0 106 passed / 0 failed / 4 ignored; GATE-DOCS 6/6 clean (49 line-refs).
`/api/status` now echoes a `backend` field alongside `version`. The baked
`BUILD_VERSION` is identical across every instance built from the same image bath,
so it answers "did this deploy land?" but not "which origin answered?" when a
probe goes through the Cloudflare Worker load balancer (`ARCH-LB-WORKER`) — and the
Worker's `x-lb-backend` header is unreliable (it is silently dropped when the
response headers are immutable, e.g. cross-origin without CORS). Having the backend
self-report in the body removes that dependency entirely.
The value is a per-instance observability label read once at startup from
`BACKEND_ID` (falling back to `HOSTNAME`, which container runtimes set their
selves). It is not a connection secret and is never persisted (`C-REDIS-ONLY-STATE`).
`null` means neither env var was set — two such instances are indistinguishable by
this field by design.
Wiring keeps handlers off the environment (matching the `debug_surface_requested`
rationale: env is process-global and reading it inside a handler races tests):
`main` reads it once and threads `backend_id` through the router builders into
`AppState::backend_id`.
Also replaces the drift-prone `src/main.rs:<line>` anchors in design.md /
design.zh-CN.md with symbol references (`std::env::var` key names,
`LlmClient::with_runtime`), per the symbol-not-line rule; the insert had already
shifted them.
Tests: `setup_status_reports_backend_label` asserts the label round-trips into the
body. GATE-P0 105 passed / 0 failed / 4 ignored; GATE-DOCS 6/6 clean.
Two issues preventing push verification from completing:
1. Response to Stalwart's verification request was 204 (empty body),
but RFC 8620 §7.2 requires a JSON body containing exactly
{"verificationCode": "..."}. Stalwart likely checks the body
and rejects empty responses → subscription stays unverified →
no event pushes ever sent.
2. verify_push_subscription (JMAP API PushSubscription/verify) was
hard-fail (503 on error). This non-standard method may not be
supported by Stalwart; a failure blocks verification entirely.
Made best-effort: warn on failure, continue storing the key.
Per RFC 8620, verification is done by the callback response,
not by a separate API call.
Two layers were masking the real cause of reconcile failures:
1. Backend: worker.rs reconciled all JMAP/Redis failures into Err(()),
so handle_reconcile returned 503 with {"error":"reconcile_retry"} but
no indication of WHICH step broke. Now reconcile() returns
Result<String, &'static str> with diagnostic codes:
- jmap_session_failed — session_state() failed (JMAP unreachable/auth)
- jmap_list_failed — list_emails_page() failed
- reconcile_budget_exhausted — budget/pages exhausted before any progress
- reconcile_no_progress — walk made no progress
- reconcile_lease_lost — background heartbeat lost the lock
- reconcile_state_write_failed — set_reconcile_state() Redis failure
- enqueue_failed — claim_dedup_and_enqueue() Redis failure
The 503 body now includes a "reason" field alongside the documented
"reconcile_retry" error code, so operators can see the cause without
server-side logs. The error code is unchanged (stable contract for
cron-drain.sh and other callers).
2. Cloudflare Worker LB: forwardWithFailover treated all 5xx as backend
failure and replaced the real response body with synthesized
"all backends failed: ..." text. Now the last 5xx HTTP response is
kept and passed through, so the backend's real error body (including
the new reason field) reaches the client. Pure network errors (no HTTP
response at all) still get the synthesized 503 fallback.
Registering with a new callback_url previously left the old Stalwart
subscription live alongside the new one, so Stalwart pushed the same mail
event to both origins and the user got duplicate notifications. The
per-URL mapping had no record of which subscription was current.
Add a push:registration:current pointer written alongside the per-URL
mapping. register_push now retires the previous subscription under the
existing registration lock before creating the new one: destroy first,
then delete the mapping, then mark the status disabled. A destroy failure
returns a retryable 503 (push_destroy_failed) rather than proceeding
silently - silently skipping is exactly the shape of the original bug.
The idempotency short-circuit still runs first, so re-registering the
same URL never touches the retire path. disable_push clears the pointer
only when the disabled URL is the current one.
Extracts destroy_push_registration, shared by disable_push and the new
retire path. Telegram needs no change: setWebhook replaces the previous
webhook atomically.
Adds 5 tests with a recording mock that captures the create/destroy
call order: idempotent no-recreate, retire-on-change, abort-without-
create when the old destroy fails, skip-destroy when the previous
mapping expired, and pointer untouched when disabling a non-current URL.
The persisted cursor was a bare (String, usize) tuple in which usize::MAX meant
"already in replay mode". A real walk position of usize::MAX would have decoded
as a bare state token and its walk would have been skipped silently.
ReconcileCursor now carries the two modes, and the pass is split along the same
boundary: reconcile -> walk_baseline (WalkOutcome: ReplayFrom / Persist /
NoProgress) -> replay_changes.
Behaviour is unchanged; the four walk/replay budget constants move out of the
function body to module level, and the /changes parameter is renamed
max_changes -> initial_changes because it only sizes the first call.
domain/jmap: current_state -> session_state (RFC 8620 Section 2.1, which is not
the Email-collection token that Email/changes wants as sinceState) and
has_more -> has_more_changes (hasMoreChanges). EmailChanges and both trait
methods now carry RFC-referenced docs that spell out why oldState and destroyed
are dropped at the boundary.
Bare TTL literals at call sites were invisible to the test suite:
MemoryState ignores the TTL argument entirely, so shrinking a 24h dedup
window to 60_480 compiled and stayed green. All 17 values now live in
pub(crate) mod ttl, the call sites reference them, and
ttl_contract_is_pinned asserts each value plus two ordering invariants
(heartbeat shorter than the lock it renews; consent tiers strictly
increasing).
The one value the earlier sweep could not reach is now reached: the EXPIRE
inside retry_or_dlq's Lua script is built with an interpolated
ttl::RETRY_COUNTER_SECONDS, so the script carries no second literal.
Gate: fmt/check/clippy clean; cargo test --locked 89 passed / 0 failed /
4 ignored (the ignored ones need REDIS_TEST_URL or JMAP_TEST_URL).
Not claimed: the two real-Redis PTTL assertions are wired but cannot run
today. Both Upstash tokens I have are rejected with -WRONGPASS, so
connect() fails before any PTTL is read.
revision becomes a request-body control field on BusinessConfigPatch. It is
never a setting: apply() and is_complete() ignore it, so the stored wire
never gains a revision key, and a PUT body that omits it keeps
last-write-wins for callers that never read the revision back.
The SPA records the revision it read, echoes it on every partial submit, and
on a 409 re-reads the stored values into the form and tells the operator to
recheck what they changed. Without that reload every retry would 409
forever; if the reload itself fails the SPA falls back to last-write-wins
rather than trapping the operator. The control field is excluded from the
empty-patch check so echoing it never counts as a change.
The compare-then-write is deliberately not atomic. Without a Lua script an
optimistic lock still has a residual window in which two requests racing
inside the same tick both pass and the later one wins; the window stays
small because a patch carries only the fields the operator actually changed.
notify.rs gained 18 lines before line 700, so its anchors in deployment.md,
opengaps.md and retired.md shifted by +18. Every one was checked against the
real line content, since the anchor audit only flags blank targets.
Gates: rust fmt/clippy clean, cargo test 87 passed 2 ignored, node --test
web/config.test.mjs 18 passed, docs_check/run_all.sh 6/6.
Adds the missing test coverage for the GET /api/business-config read-back
and the partial-patch PUT, then syncs the docs to the behaviour the code
actually has.
src/notify.rs
- PUT response body now carries "revision" alongside the
x-business-config-revision header, which the SPA surfaces in its status line.
- 4 test helpers (a router that returns (Router, MemoryState) so a config can be
seeded first, a full valid wire value, a direct state writer that skips
validation, and a wire value whose JMAP URL fails fast) plus 9 tests:
- GET 401 leaks no business field at all (body is exactly error + request_id)
- GET echoes the 10 non-secret values and returns secrets only as presence
booleans; the serialised body is asserted to contain none of the 6 secret
values (SAF-NO-SECRET-ECHO)
- GET with nothing stored returns 200 configured:false revision:0, not 404
- a first save with an incomplete patch is rejected and writes nothing
- a partial submit replaces only the fields it carries, verified by reading
the merged result back over the same path the SPA uses
- validation runs on the merged result, so changing one field cannot bypass
the rest, and a rejected submit neither persists nor bumps the revision
- telegram_chat_id is accepted as a number or a numeric string and a garbage
string is rejected without overwriting the stored value
- a typo in a field name is rejected rather than silently dropped
src/config.rs
- Pure deserialisation test for de_i64_from_number_or_string, with no router and
no store. This is the only place that separates "the custom deserializer
refuses the string" from "a later validation step refused it" - a real
distinction, since the SPA submits telegram_chat_id as a string on a partial
submit while the read-back model returns a number.
cloudflare-worker/test/index.test.js
- The forwarded-route test asserted a fixed upstream origin, which is not stable:
the LB rotates its start index between tests. Assert the origin is one of the
configured backends and check the path and method exactly instead.
docs
- 17 spots bring the docs in line with the shipped contract: GET is now
documented, PUT as a partial patch, the worker safelist at 16 routes, and the
preflight path listed as forwarded.
- Corrections the docs had backwards: the response field is "configured"
(revision is a u64, 0 when nothing is stored); Invalid returns 422, only
Unreachable returns 503; there is no chat_allowlist_empty error code.
- "Blank keeps the stored value" is now stated as a client-side contract. The
backend does not distinguish an omitted field from an empty string - apply()
uses unwrap_or_else, so an explicitly submitted "" really does clear the
secret (validation then rejects it). Saying it was server-side was wrong.
- POST /worker was listed as a forwarded route though it is not in SAFE_ROUTES,
and the per-category counts did not match the lists they introduced.
- POST /api/bootstrap does not share the patch-merge contract; it deserialises a
full BusinessConfigWire, which is the only path that can create a config from
nothing.
- Re-anchored 9 src/notify.rs line refs that had drifted blank and raised the
baseline to 86 passed / 0 failed / 2 ignored.
Gates: fmt, clippy -D warnings, test 86/0/2 ignored, worker 32/32, SPA 17/17,
GATE-DOCS 6/6 clean.
GET /api/business-config returns the stored values with presence-only secrets, and PUT now merges a patch against the stored config instead of replacing it, so the SPA can show what is already set and send back only the fields that changed.
The SPA marks secrets as set-but-unreadable, edits chat_allowlist as an unordered set, and omits fields it did not touch; blank required fields keep the stored value, blank optional fields clear it. The read-back is Serialize-only, so a client cannot inject secrets_present.
Forward both paths at the gateway: /api/business-config accepted PUT only, so the read-back returned 405, and the preflight path was missing from the safelist entirely, so it returned 404. The SPA is served from the Worker domain, so the whole edit experience was unreachable behind the production gateway. Route methods are now asserted per path, which the safelist test never checked.
Also: /debug/config reports timezone again, the timezone select is styled like the other fields, and the docs follow the new contract.
The Received line was sent as a raw unix timestamp. It is now a business
configuration field (`timezone`, IANA name, default `Asia/Shanghai`), edited
in the admin SPA and applied through the existing 1s hot-reload.
The offline build has no IANA tz database (chrono-tz is unavailable), so the
server keeps a static table of 16 zones that carry no daylight saving
transition and rejects anything else with the existing 422 configuration
error instead of guessing an offset.
`format_received_at` renders to `%Y-%m-%d %H:%M` at notification time; a
timestamp outside the representable range falls back to `unknown`.
Chrono 0.4 was already in Cargo.lock as a jmap-client transitive dependency,
so this only promotes it to a direct dependency.
PUT /api/business-config and POST /api/bootstrap now treat validation as the only
write gate: a valid wire is always persisted, and only then are the clients built
and the running worker swapped. A failed build returns 200 with persisted=true,
runtime_applied=false and a warnings array of {component, step, detail} instead of
503, so an unreachable JMAP/LLM no longer turns a valid configuration into a silent
data loss. The previous worker keeps serving.
POST /api/business-config/preflight runs the same validation and client build
against the submitted wire without writing anything and without touching the
running worker. It answers 200 even when the configuration is bad, because that
is the answer the caller asked for; components is null after a validation
failure, and llm is null rather than vacuously ok when LLM is disabled.
Authentication is the admin-session gate, not DEBUG_TOKEN.
The SPA drops the account_id field (the backend still accepts it; omitted falls
back to the session primary account), adds a Test connection button, and shows
the build fingerprint in the footer, the topbar and the setup card. The
503 message no longer claims JMAP/LLM build failure, which is not what produces
it.
BusinessConfigWire.account_id gains #[serde(default)]; without it, omitting the
field in the wire body made deserialization fail outright.
build.rs bakes BUILD_VERSION (git sha or nogit, plus a UTC build timestamp) which
GET /api/status exposes as version.
The setup router delegated to the full factory, so all 12 business and
admin routes were registered even though not one could ever work:
admin_token is an empty SecretString and constant_time_eq has a
!expected.is_empty() guard, so an empty expected value rejects every
candidate forever. MemoryState could not persist a bootstrap write either.
Those routes answered 401 {"error":"unauthorized"}, which is
indistinguishable from a mistyped credential. An operator or a probe
reads that as "try a different token" when the truth is "these
endpoints do not exist in this mode".
router_configuration_setup now builds its own AppState and mounts only
/api/status, /ready and /healthz. Verified live with no env vars:
healthz 200, ready 503, /api/status 200 with the missing list, and all
12 business/admin routes 404. The SPA is unaffected: /api/status is the
only endpoint it calls at startup, and web/config.js reveals the
admin-session card only when ready === true.
Added configuration_setup_mounts_only_status_and_probes to guard the
route table, and configuration_setup_call as its helper.
Docs: reference.md section 3 and 5.1 now describe the reduced surface
and cite the empty-expected-value guard; deployment.md's probe table had
/api/status listed as 503 {"status":...} when it returns 200 with a
"mode" field; retired.md section 4.1 records the removed route
registrations. Anchors refreshed for the +19 lines in notify.rs.
Every member's audit is now read and each real finding fixed, not just reported.
AGENTS.md §2.1
- The 500-line soft cap now has a machine-checkable exception: an over-limit
file must carry a one-line `SPLIT-EVAL:` marker stating the split decision.
- Six files were over the cap with no justification: notify.rs 2036, state.rs
1586, worker.rs 1045, config.js 1079, client.rs 601, config.rs 521. All now
carry the marker. client.rs was missed by my own earlier line count, which
did not descend into src/domain/jmap/.
- New validator scripts/docs_check/check_file_size.py; GATE-DOCS 4/4 -> 5/5.
It proves the marker exists, not that the written reason is sound.
- Re-based 144 line anchors across six docs for the +1 line each file gained;
every shifted anchor verified against the code, not just for non-blankness.
audit-arch.md
- design.md §6.1 module matrix omitted the `debug` module entirely; added.
- Seven numeric duplications in design.md now defer to reference.md §6.1 as
the single authority instead of restating the values.
- reference.md §5.1 was missing DEBUG_TOKEN; §5.3 listed six recognised names
and guessed `JMAP_*`. It now lists all 20 exactly as config.rs reads them.
- Corrected two wrong claims in the audit itself: debug.rs is compiled
unconditionally, not under #[cfg(debug_assertions)] (there are no cargo
features at all), and /debug/* exposes 7 routes, not the invented session
list / resend / rebuild surface.
audit-fileplanner.md
- reference.md cited a nonexistent `jmap_service.rs`; replaced with the real
adapter chain worker.rs -> domain/jmap.rs -> domain/jmap/client.rs.
- cloudflare-worker/README.md was outside every validator's scope; all four
doc validators now include it (208 section refs, 250 line refs).
- README.zh-CN.md stated only half of what README.md says about RUN_MODE; the
forward-placeholder sentence is now bilingual.
- charter.md §7 boundary table was missing rows for itself and for web/.
audit-rust.md / audit-frontend.md
- Error envelope corrected: `error` is a bare machine code, no `message`.
- Status table, AI-consent labels and ~15 stale or phantom facts fixed against
notify.rs; `TelegramClient::SendMessage` and `web_config` phantoms removed.
- GET|PUT /api/enabled forwarded by the worker so the SPA kill switch works.
- Ten UI strings that called the admin credential the Redis ACL password now
say CONFIG_ENCRYPTION_KEY.
- POST /healthz returns 405; dead HTTP_STATUS removed.
- audit_anchors.py now flags identifiers that do not exist in the file they
are anchored to.
charter.md
- `SAF-ENABLE-FLAG` registered (85 IDs); `SAF-ADMIN-SESSION` row corrected;
- `REQ-PUSH-TYPES` registered; the stale `admin_session_digest` removed.
Gates: GATE-DOCS PASS 5/5 (250 line-refs, 208 section refs, 0 errors);
GATE-P0 62 passed / 0 failed / 1 ignored, baseline unchanged; worker and SPA
36 tests, 0 failures.
Real Stalwart / Telegram / managed Redis verification stays blocked on missing
credentials and remains labelled as unverified.
- GET /ready: ready -> 200 + diagnostic report (unchanged success body);
not ready -> 503 + {"error":"service_unavailable","request_id":id}
+ Retry-After: 30, consistent with the other public entry points
- drop the legacy {"status":"not_ready"} diagnostic block from the 503 body
(backward compatibility not required)
- HTTP 200/503 semantics preserved, so Uptime Kuma HTTP(s) monitors
(status-code based) keep working
- sync docs: README.md, README.zh-CN.md, docs/design.md, docs/deployment.md,
AGENTS.md, docs/todo.md
- all public write routes now return {"error":code,"request_id":id};
retryable 503 responses add Retry-After: 30; 204 and success bodies unchanged
- GET /ready keeps its structured diagnostic JSON (Uptime Kuma runtime contract)
- add HTTP tests for the push register/disable routes
anonsaber
anonsaber