Two layers were masking the real cause of reconcile failures:
1. Backend: worker.rs reconciled all JMAP/Redis failures into Err(()),
so handle_reconcile returned 503 with {"error":"reconcile_retry"} but
no indication of WHICH step broke. Now reconcile() returns
Result<String, &'static str> with diagnostic codes:
- jmap_session_failed — session_state() failed (JMAP unreachable/auth)
- jmap_list_failed — list_emails_page() failed
- reconcile_budget_exhausted — budget/pages exhausted before any progress
- reconcile_no_progress — walk made no progress
- reconcile_lease_lost — background heartbeat lost the lock
- reconcile_state_write_failed — set_reconcile_state() Redis failure
- enqueue_failed — claim_dedup_and_enqueue() Redis failure
The 503 body now includes a "reason" field alongside the documented
"reconcile_retry" error code, so operators can see the cause without
server-side logs. The error code is unchanged (stable contract for
cron-drain.sh and other callers).
2. Cloudflare Worker LB: forwardWithFailover treated all 5xx as backend
failure and replaced the real response body with synthesized
"all backends failed: ..." text. Now the last 5xx HTTP response is
kept and passed through, so the backend's real error body (including
the new reason field) reaches the client. Pure network errors (no HTTP
response at all) still get the synthesized 503 fallback.
The persisted cursor was a bare (String, usize) tuple in which usize::MAX meant
"already in replay mode". A real walk position of usize::MAX would have decoded
as a bare state token and its walk would have been skipped silently.
ReconcileCursor now carries the two modes, and the pass is split along the same
boundary: reconcile -> walk_baseline (WalkOutcome: ReplayFrom / Persist /
NoProgress) -> replay_changes.
Behaviour is unchanged; the four walk/replay budget constants move out of the
function body to module level, and the /changes parameter is renamed
max_changes -> initial_changes because it only sizes the first call.
domain/jmap: current_state -> session_state (RFC 8620 Section 2.1, which is not
the Email-collection token that Email/changes wants as sinceState) and
has_more -> has_more_changes (hasMoreChanges). EmailChanges and both trait
methods now carry RFC-referenced docs that spell out why oldState and destroyed
are dropped at the boundary.
Bare TTL literals at call sites were invisible to the test suite:
MemoryState ignores the TTL argument entirely, so shrinking a 24h dedup
window to 60_480 compiled and stayed green. All 17 values now live in
pub(crate) mod ttl, the call sites reference them, and
ttl_contract_is_pinned asserts each value plus two ordering invariants
(heartbeat shorter than the lock it renews; consent tiers strictly
increasing).
The one value the earlier sweep could not reach is now reached: the EXPIRE
inside retry_or_dlq's Lua script is built with an interpolated
ttl::RETRY_COUNTER_SECONDS, so the script carries no second literal.
Gate: fmt/check/clippy clean; cargo test --locked 89 passed / 0 failed /
4 ignored (the ignored ones need REDIS_TEST_URL or JMAP_TEST_URL).
Not claimed: the two real-Redis PTTL assertions are wired but cannot run
today. Both Upstash tokens I have are rejected with -WRONGPASS, so
connect() fails before any PTTL is read.
It was populated at the client boundary but read nowhere in production -
the doc comment claimed "retained for cursor accounting", yet the cursor
is just new_state. Deletions are dropped at the boundary instead of being
plumbed through and then ignored one layer in.
The upstream crate still returns destroyed; we simply stop carrying it.
Re-add it only when something acts on deletions (e.g. pruning a dedup
key) - keeping an unread Vec costs a copy on every changes page for
nothing.
A stale sinceState made every later run retry the same dead state and
reconcile:state froze forever, so reconcile answered 503 with no forward
progress. Re-baseline to a freshly fetched state at position 0 instead of
failing: the position walk never depends on server-side changelog
retention, and the 24h dedup key keeps the re-read at-most-duplicate.
Closes opengaps.md item 1 without a downtime window.
The Received line was sent as a raw unix timestamp. It is now a business
configuration field (`timezone`, IANA name, default `Asia/Shanghai`), edited
in the admin SPA and applied through the existing 1s hot-reload.
The offline build has no IANA tz database (chrono-tz is unavailable), so the
server keeps a static table of 16 zones that carry no daylight saving
transition and rejects anything else with the existing 422 configuration
error instead of guessing an offset.
`format_received_at` renders to `%Y-%m-%d %H:%M` at notification time; a
timestamp outside the representable range falls back to `unknown`.
Chrono 0.4 was already in Cargo.lock as a jmap-client transitive dependency,
so this only promotes it to a direct dependency.
Every member's audit is now read and each real finding fixed, not just reported.
AGENTS.md §2.1
- The 500-line soft cap now has a machine-checkable exception: an over-limit
file must carry a one-line `SPLIT-EVAL:` marker stating the split decision.
- Six files were over the cap with no justification: notify.rs 2036, state.rs
1586, worker.rs 1045, config.js 1079, client.rs 601, config.rs 521. All now
carry the marker. client.rs was missed by my own earlier line count, which
did not descend into src/domain/jmap/.
- New validator scripts/docs_check/check_file_size.py; GATE-DOCS 4/4 -> 5/5.
It proves the marker exists, not that the written reason is sound.
- Re-based 144 line anchors across six docs for the +1 line each file gained;
every shifted anchor verified against the code, not just for non-blankness.
audit-arch.md
- design.md §6.1 module matrix omitted the `debug` module entirely; added.
- Seven numeric duplications in design.md now defer to reference.md §6.1 as
the single authority instead of restating the values.
- reference.md §5.1 was missing DEBUG_TOKEN; §5.3 listed six recognised names
and guessed `JMAP_*`. It now lists all 20 exactly as config.rs reads them.
- Corrected two wrong claims in the audit itself: debug.rs is compiled
unconditionally, not under #[cfg(debug_assertions)] (there are no cargo
features at all), and /debug/* exposes 7 routes, not the invented session
list / resend / rebuild surface.
audit-fileplanner.md
- reference.md cited a nonexistent `jmap_service.rs`; replaced with the real
adapter chain worker.rs -> domain/jmap.rs -> domain/jmap/client.rs.
- cloudflare-worker/README.md was outside every validator's scope; all four
doc validators now include it (208 section refs, 250 line refs).
- README.zh-CN.md stated only half of what README.md says about RUN_MODE; the
forward-placeholder sentence is now bilingual.
- charter.md §7 boundary table was missing rows for itself and for web/.
audit-rust.md / audit-frontend.md
- Error envelope corrected: `error` is a bare machine code, no `message`.
- Status table, AI-consent labels and ~15 stale or phantom facts fixed against
notify.rs; `TelegramClient::SendMessage` and `web_config` phantoms removed.
- GET|PUT /api/enabled forwarded by the worker so the SPA kill switch works.
- Ten UI strings that called the admin credential the Redis ACL password now
say CONFIG_ENCRYPTION_KEY.
- POST /healthz returns 405; dead HTTP_STATUS removed.
- audit_anchors.py now flags identifiers that do not exist in the file they
are anchored to.
charter.md
- `SAF-ENABLE-FLAG` registered (85 IDs); `SAF-ADMIN-SESSION` row corrected;
- `REQ-PUSH-TYPES` registered; the stale `admin_session_digest` removed.
Gates: GATE-DOCS PASS 5/5 (250 line-refs, 208 section refs, 0 errors);
GATE-P0 62 passed / 0 failed / 1 ignored, baseline unchanged; worker and SPA
36 tests, 0 failures.
Real Stalwart / Telegram / managed Redis verification stays blocked on missing
credentials and remains labelled as unverified.
- worker/state: paginate Email/changes by widening maxChanges on the same
sinceState (jmap-client 0.4.2 has no upToId) to avoid skipping a batch when
advancing per page; make dedup+enqueue atomic via claim_dedup_and_enqueue
(Lua SET NX EX then XADD); scale the XAUTOCLAIM idle threshold by batch size;
XACK payload-less malformed stream entries out of the PEL via ack_malformed;
fail closed on unknown streams (Err); tighten SET NX EX TTL to .max(1);
derive the CSPRNG fallback owner-token from time+PID+counter instead of a
constant; clear the push verification digest on push:disable
- worker: /help menu now lists only the consent triggers the parser accepts
- jmap/config: complete client and configuration validation
anonsaber
anonsaber