Two layers were masking the real cause of reconcile failures:
1. Backend: worker.rs reconciled all JMAP/Redis failures into Err(()),
so handle_reconcile returned 503 with {"error":"reconcile_retry"} but
no indication of WHICH step broke. Now reconcile() returns
Result<String, &'static str> with diagnostic codes:
- jmap_session_failed — session_state() failed (JMAP unreachable/auth)
- jmap_list_failed — list_emails_page() failed
- reconcile_budget_exhausted — budget/pages exhausted before any progress
- reconcile_no_progress — walk made no progress
- reconcile_lease_lost — background heartbeat lost the lock
- reconcile_state_write_failed — set_reconcile_state() Redis failure
- enqueue_failed — claim_dedup_and_enqueue() Redis failure
The 503 body now includes a "reason" field alongside the documented
"reconcile_retry" error code, so operators can see the cause without
server-side logs. The error code is unchanged (stable contract for
cron-drain.sh and other callers).
2. Cloudflare Worker LB: forwardWithFailover treated all 5xx as backend
failure and replaced the real response body with synthesized
"all backends failed: ..." text. Now the last 5xx HTTP response is
kept and passed through, so the backend's real error body (including
the new reason field) reaches the client. Pure network errors (no HTTP
response at all) still get the synthesized 503 fallback.
anonsaber