Registering with a new callback_url previously left the old Stalwart
subscription live alongside the new one, so Stalwart pushed the same mail
event to both origins and the user got duplicate notifications. The
per-URL mapping had no record of which subscription was current.
Add a push:registration:current pointer written alongside the per-URL
mapping. register_push now retires the previous subscription under the
existing registration lock before creating the new one: destroy first,
then delete the mapping, then mark the status disabled. A destroy failure
returns a retryable 503 (push_destroy_failed) rather than proceeding
silently - silently skipping is exactly the shape of the original bug.
The idempotency short-circuit still runs first, so re-registering the
same URL never touches the retire path. disable_push clears the pointer
only when the disabled URL is the current one.
Extracts destroy_push_registration, shared by disable_push and the new
retire path. Telegram needs no change: setWebhook replaces the previous
webhook atomically.
Adds 5 tests with a recording mock that captures the create/destroy
call order: idempotent no-recreate, retire-on-change, abort-without-
create when the old destroy fails, skip-destroy when the previous
mapping expired, and pointer untouched when disabling a non-current URL.
CI 实测装出 3.114.17 并打印
'The version of Wrangler you are using is now out-of-date ... to prevent
critical errors'。锁文件重新生成后 wrangler 4.144.0：
- node 24.21.0 / npm 11.19.0: npm ci 干净装出 35 个包
- npm test: 32/32 通过
- npx wrangler deploy --dry-run --outdir=.build-check: 32.30 KiB 上传，
无配置警告、无 out-of-date 提示
- opengaps: stage goal 2 is fully closed, the blocker list is empty, and the
remaining user action is only the multi-account product decision.
- opengaps: record the fourth self-inflicted test bug -- the flaky 1s TTL
window -- and what the corrected assertion actually proves, including the
B-class Lua EXPIRE value now verified on a real instance.
- design: the stage-0 paragraph still claimed 69 passed / 1 ignored, which
contradicted charter TEST_BASELINE; point it at the authoritative line.
- No credential values anywhere; the URL is written as a placeholder.
A remote TLS readback arrives after most of the 1s floor has been consumed,
so the +/-500 ms window could never be met (PTTL -2 / 269 / 118 ms across
three runs). PTTL -1 is what actually signals the regression this assertion
guards against -- EX silently dropped, key lives forever -- and a live value
can only be counting down, so bound it from above.
SET ... NX EX 1 itself is unchanged; the expired (-2) outcome is what proved
it took effect.
reference.md: new Section 6.5 explains the bimodal cursor, the two token
classes (Session.state is not an Email-collection token; newState is), why a
truncated walk persists as a walk, and why oldState and destroyed are dropped.
Section 6.2 budget table is renamed to the new constant names and gains
RECONCILE_INITIAL_CHANGES. state:jmap:since is listed as a pipeline key.
opengaps.md: the re-baseline fix is described with the new session_state()
name, and the position-walk note no longer claims a state argument is passed
as None - the None is folder_id, and the method has no state parameter at all.
retired.md: the dead-code removal no longer claims those were the last three
#[expect(dead_code)] attributes in src/; reload_async still has one.
deployment.md / design.md: the 54+14 anchors the code move had shifted by one
hunk are re-anchored, plus two manual corrections (the /reconcile route and the
unconditional 204 return). charter.md: baseline 89 -> 93 passed.
AGENTS.md: the two anchor counts it quotes (60, 44) were stale; they are 54
and 17.
check_sec_refs.py: two-part section refs attributed to an RFC (RFC 8620
Section 2.1) are external, not doc references - the trailing-dot test only
caught the three-part form.
The persisted cursor was a bare (String, usize) tuple in which usize::MAX meant
"already in replay mode". A real walk position of usize::MAX would have decoded
as a bare state token and its walk would have been skipped silently.
ReconcileCursor now carries the two modes, and the pass is split along the same
boundary: reconcile -> walk_baseline (WalkOutcome: ReplayFrom / Persist /
NoProgress) -> replay_changes.
Behaviour is unchanged; the four walk/replay budget constants move out of the
function body to module level, and the /changes parameter is renamed
max_changes -> initial_changes because it only sizes the first call.
domain/jmap: current_state -> session_state (RFC 8620 Section 2.1, which is not
the Email-collection token that Email/changes wants as sinceState) and
has_more -> has_more_changes (hasMoreChanges). EmailChanges and both trait
methods now carry RFC-referenced docs that spell out why oldState and destroyed
are dropped at the boundary.
opengaps.md: the TTL item is now closed on the code side. The 21-value
table is keyed by ttl constant rather than by line number, so it stops
drifting. The honest caveat is called out in the open instead of being
implied away: the two real-Redis assertions are wired but unrunnable with
the credentials on hand, so 'assertion ready' is not the same as 'verified
on real hardware'. The residual-risk note that was still sitting under the
closed item has been removed along with it.
charter.md: test baseline 88 -> 89 passed.
deployment.md, design.md, retired.md: 41 stale src/ line anchors. The
drift was real and the audit missed it every time, because
audit_anchors.py only asserts that a referenced line exists and is
non-blank. Three are off-by-one; the rest are wrong code --
state.rs:379 pointed at XREADGROUP XACK while the claim was about
read_batch, retired.md:99 pointed into Intent::Search::parse, the two
route-table anchors landed on state: Arc::new(state),, and one
register_push anchor sat 31 lines above the fn.
scripts/docs_check/reanchor.py: maps HEAD -> working-tree line numbers
straight off git diff -U0 hunks, applies the fix or reports only with
--check. scripts/docs_check/show_anchors.py: prints every doc anchor next
to the line it lands on, which is what caught the semantic errors the
audit structurally cannot.
Bare TTL literals at call sites were invisible to the test suite:
MemoryState ignores the TTL argument entirely, so shrinking a 24h dedup
window to 60_480 compiled and stayed green. All 17 values now live in
pub(crate) mod ttl, the call sites reference them, and
ttl_contract_is_pinned asserts each value plus two ordering invariants
(heartbeat shorter than the lock it renews; consent tiers strictly
increasing).
The one value the earlier sweep could not reach is now reached: the EXPIRE
inside retry_or_dlq's Lua script is built with an interpolated
ttl::RETRY_COUNTER_SECONDS, so the script carries no second literal.
Gate: fmt/check/clippy clean; cargo test --locked 89 passed / 0 failed /
4 ignored (the ignored ones need REDIS_TEST_URL or JMAP_TEST_URL).
Not claimed: the two real-Redis PTTL assertions are wired but cannot run
today. Both Upstash tokens I have are rejected with -WRONGPASS, so
connect() fails before any PTTL is read.
The two REDIS_TEST_URL-gated tests could never have passed as written: a
unit-test binary never runs main(), so no rustls crypto provider was
installed and the TLS handshake to a rediss:// URL panicked before any
command was sent. Install the provider in a helper inside the test module
and reuse the same call main.rs already makes.
Also fix a key-construction typo in the retry/DLQ test: the retry counter
key embeds the message id, not the run id.
Verified against the live Upstash instance:
REDIS_TEST_URL=rediss://... cargo test --locked -- --ignored ttl
ttl_claim_dedup_sets_the_exactly_requested_expiry ... ok
ttl_consent_and_retry_landing_on_real_redis ... ok
Two latent bugs surfaced while preparing to run these against Upstash:
1. The retry key was mis-constructed. builds its counter
as , so the test had to read
. It read
, a key the call never writes, so the
assertion would have failed on the first real run (PTTL of a missing
key is -2, and the test's own guard rejects it).
Derive the key from the same stream name and message id the call
receives instead of re-spelling it by hand.
2. The consent probe used the process id as a Telegram chat_id. A
private Telegram chat_id is a plain positive integer, so a small pid
can legitimately be a real user. If the test is pointed at the live
database it would have overwritten that user's AI consent flag with a
1h expiry. Anchor at -4e18 + pid instead: real Telegram ids stay
under ~1e11 in magnitude, so no live flag can be shadowed.
All test keys remain prefixed and are deleted at the end of each test.
real_redis_ttl_tests in src/state.rs closes the "no test asserts any TTL
value" gap for the three atomic write points that actually reach Redis:
- set_nx_ex SET NX EX, with .max(1) floor (Redis rejects a zero EX)
- retry_or_dlq EXPIRE KEYS[1] 86400, embedded in the Lua script
- set_ai_consent SET EX
Each is opt-in via REDIS_TEST_URL (#[ignore], mirrors the JMAP gating in
src/domain/jmap/client.rs:522); missing the env var prints 'skipped' and
returns, so CI stays green without credentials and the URL is never logged.
retry_or_dlq uses max_attempts=99 so the script returns before XADD/XACK and
no stream or group needs to be created.
opengaps.md item 2 is now blocked only on a reachable Redis URL. Two
defects in that section were fixed while touching it: the consent table
rendered 7 * 86_400 as 604_800 and 365 * 86_400 as 31_536_000 (arithmetic
right, literal wrong) and listed 300s/7200s values that do not exist in
the code - the sixth real consent TTL is 0 (revoke). 16 line anchors had
also drifted onto comments and neighbours; all are re-pointed. The line-
range audit passes 138 refs but only checks that a line is non-blank, not
that it holds the claimed content, which is how these slipped through.
GATE-P0 baseline moves to 88 passed / 0 failed / 4 ignored.
It was populated at the client boundary but read nowhere in production -
the doc comment claimed "retained for cursor accounting", yet the cursor
is just new_state. Deletions are dropped at the boundary instead of being
plumbed through and then ignored one layer in.
The upstream crate still returns destroyed; we simply stop carrying it.
Re-add it only when something acts on deletions (e.g. pruning a dedup
key) - keeping an unread Vec costs a copy on every changes page for
nothing.
debug.rs grew past 500 lines with no SPLIT-EVAL marker, and the seven
/src/debug/* anchors in deployment.md pointed at the blank line just
above each function - so GATE-DOCS was red on two unrelated pre-existing
counts (SIZE CHECK, ANCHOR AUDIT) while the other four checks were green.
Re-anchor all seven to the function bodies they describe, add the marker
with the reason we are keeping the module whole, and document the
redis_failure_detail classification that deployment.md still called
"redis_probe_failed". Bump the GATE-P0 baseline to 88.
Item 1 asked for a downtime window to observe the newState semantics, but
jmap-client 0.4.2 exposes no reset/re-baseline signal field, so the only
observable outcome is a plain Err - the window would have proven nothing.
The real behavior was worse than the item described: Err propagated as
Err(()) with reconcile:state left untouched, so every cron retried the
same dead sinceState and reconcile answered 503 forever with no forward
progress. Fix landed in 269c8f6.
Only item 2 (TTL measurement) remains, and it needs nothing from the user.
A stale sinceState made every later run retry the same dead state and
reconcile:state froze forever, so reconcile answered 503 with no forward
progress. Re-baseline to a freshly fetched state at position 0 instead of
failing: the position walk never depends on server-side changelog
retention, and the 24h dedup key keeps the re-read at-most-duplicate.
Closes opengaps.md item 1 without a downtime window.
The previous commit switched to ConnectionManager but took its config
defaults. Two of them are unusable.
Default number_of_retries is 6, which is not a bounded retry in redis
0.27.6. Measured with a throwaway binary against a refusing host
(127.0.0.1:1, ConnectionManager::new_with_config):
retries = 0 -> returns in ~0.5 ms with IoError
retries = 1 -> returns in ~1.5 s with IoError
retries = 6 -> never returns (still hung at 300 s)
So the crate default is an infinite loop, not "six tries". With it in
place a Redis host that is down at boot makes startup hang forever: no
error, no log, and a container that reports Up while serving nothing.
The code path this replaced (get_multiplexed_async_connection) returned
in about half a millisecond, so the default quietly turned a fail-fast
boot into a silent hang.
Default connection_timeout is None, which leaves each attempt at the OS
connect timeout: roughly two minutes for a host that drops packets
instead of refusing them.
Setting retries to 1 restores fail-fast boot -- an unreachable host now
exits in under two seconds with `State(Redis(Connection refused))` --
while still absorbing a single transient refusal. Anything more is
redundant here: the manager re-enters the connect path on every
subsequent command once the connection drops, so recovery never depended
on one operation retrying many times.
Measured end-to-end against a local redis container, killing the Redis
and restarting it while the app stayed up:
before (retries 6): 61.6 s to recover, and during the outage
/debug/redis hung past a 25 s curl timeout
after (retries 1): 3.0 s to recover, and /debug/redis answers
immediately with detail=io_error
Auto-recovery still works with zero restarts; this only makes it faster
and stops the health endpoint from wedging while Redis is down.
Both values are named constants with the measurements in their docs so
the numbers can be checked and the "why only one retry" question does
not have to be answered again.
Gates: fmt clean, clippy -D warnings clean, cargo test 87 passed,
2 ignored, unchanged from the previous baseline.
RedisState built one MultiplexedConnection at startup and kept it for the
life of the process; every call cloned the handle, which clones the handle
and not the socket. When the underlying TCP connection dropped -- an Upstash
idle timeout, a network blip, an idle-connection cutoff -- the handle was
permanently dead. The process kept running, every Redis read failed, and
only a restart recovered it. That is what had to be restarted.
The crate says so in its own docs: redis 0.27.6 lib.rs:71 recommends
ConnectionManager "for automatic reconnections" behind the
connection-manager feature, which we had never enabled.
RedisState now holds a ConnectionManager built from the Client. Both types
derive Clone and implement ConnectionLike, so the five query_async call
sites are untouched; the three helper signatures that took a cloned
connection by value or by mutable reference change type only.
MultiplexedConnection no longer appears anywhere in src/.
Start-up semantics are unchanged: ConnectionManager::new awaits
new_connection before returning, so an unreachable Redis still fails closed
at boot rather than succeeding and erroring on first use. Reconnect is gated
on is_io_error(), which is exactly kind() == IoError. A dropped socket is an
IoError, so it reconnects with exponential backoff and jitter (base 2,
100ms factor, 6 retries); an authentication failure is AuthenticationFailed
and is deliberately not retried, because that error does not fix itself.
No Drop or cancel is needed: the live connection sits behind
Arc<ArcSwap<SharedRedisFuture>>, the Client is retained for rebuilding, and
no detached control task is spawned, so nothing leaks when RedisState is
dropped and the existing #[derive(Clone)] still holds.
/debug/redis collapsed every failure into the single string
"redis_probe_failed", which is why the real error was invisible and the
outage took a session to trace. StateError::Redis carries the underlying
RedisError, so the class is readable here: authentication_failed,
io_error, busy_loading, client_error, response_error, plus
state_lock_poisoned and encryption_failed. ErrorKind is #[non_exhaustive],
so the match keeps a fallback arm for future variants.
No NotReachable variant exists in redis 0.27.6 -- a network outage
surfaces there as IoError or ClientError -- so the classifier does not
mention one.
Four new transitive dependencies come in with the feature: backon 1.6.0,
fastrand 2.4.1, futures 0.3.32, futures-executor 0.3.32.
Not covered: there is no test for the automatic reconnect path, which would
require tearing down a real TCP connection mid-flight. /debug/redis still
probes through is_enabled() rather than issuing a bare PING.
Gates: rust fmt clean, clippy --all-targets -D warnings clean,
cargo test 87 passed 2 ignored, unchanged from the previous baseline.
Same work, two instruction versions in the queue: the executing agent follows
the stale one while the lead has already finished and committed the revised
one. Nothing in the queue records which instruction is current, so which
version runs is effectively random. That is what the mutual-overwrite round
was, not a communication failure.
AGENTS.md gains a 9th section with the five rules, including the one that
actually had to be enforced: withdrawing an instruction requires telling the
executor by name, because replacing it in the queue is not withdrawal.
Mechanical follow-up work that cannot collide with someone else (anchor
recalculation, wording, missing tests) should be done while waiting rather
than queued. Committing stays with the lead.
GATE-DOCS 6/6, ANCHOR AUDIT 129 refs / 0, PATH AUDIT 454 / 0. AGENTS.md
248 -> 267 lines, outside the 500-line soft cap and the size checker only
covers src/, web/ and cloudflare-worker/.
revision becomes a request-body control field on BusinessConfigPatch. It is
never a setting: apply() and is_complete() ignore it, so the stored wire
never gains a revision key, and a PUT body that omits it keeps
last-write-wins for callers that never read the revision back.
The SPA records the revision it read, echoes it on every partial submit, and
on a 409 re-reads the stored values into the form and tells the operator to
recheck what they changed. Without that reload every retry would 409
forever; if the reload itself fails the SPA falls back to last-write-wins
rather than trapping the operator. The control field is excluded from the
empty-patch check so echoing it never counts as a change.
The compare-then-write is deliberately not atomic. Without a Lua script an
optimistic lock still has a residual window in which two requests racing
inside the same tick both pass and the later one wins; the window stays
small because a patch carries only the fields the operator actually changed.
notify.rs gained 18 lines before line 700, so its anchors in deployment.md,
opengaps.md and retired.md shifted by +18. Every one was checked against the
real line content, since the anchor audit only flags blank targets.
Gates: rust fmt/clippy clean, cargo test 87 passed 2 ignored, node --test
web/config.test.mjs 18 passed, docs_check/run_all.sh 6/6.
docs/retired.md:90 cited notify.rs:860 for register_push, which is
'if let Ok(mut current) = app.business_runtime.write()' inside the worker -
the anchor audit passes it because that line exists and is non-blank. The
handler is at notify.rs:1261.
Found while reconciling the last open task against the tree: this was the only
one of the five audit_anchor_semantics.py candidates that was a real drift
rather than a nearest-name false positive.
The convention added in 166b77e said deployment.md was the only exception, but
opengaps.md, retired.md and design.md still carry 44 anchors and reference.md is
the file that actually lost its 56. State the policy as it is applied: symbol
names for anything new, deployment.md's 60 anchors kept because it is an
operator runbook, reference.md stays line-number-free, and the 44 legacy anchors
are neither added to nor cleaned up because those files rarely change.
`docs/reference.md` is the single source of truth and gets touched on every API
change, yet 40 of its 56 `notify.rs` anchors had drifted onto a closing brace,
a doc comment, or the wrong symbol - including one whole line pointing at
`revoke_admin_session` where a jmap dedup key is claimed - while
`scripts/docs_check/audit_anchors.py` reported every one of them clean. Its own
docstring says it only checks that the target line exists and is non-blank.
Remove the anchors rather than correcting them: fixing 40 numbers creates a
treadmill that re-drifts on the next unrelated edit to the same file, and the
file's own convention already called the line number "not an identity". Symbol
names are kept everywhere, which is what the gates and the readers need.
`docs/deployment.md` keeps its line numbers - it is an operator runbook read
mid-incident - and the five anchors that were wrong there are corrected.
docs/reference.md
- Drop the false "Verified against commit b2dbe7c" header claim; the anchors
never matched that commit. Replace it with the design decision and the
reason, and drop the "Only this file carries line numbers" convention that
`deployment.md` was already violating.
- Remove all `*.rs:NNN` and bare `:NNN` anchors, keeping symbol names.
- Fix a wrong fact the anchors were hiding: the session issued by
`POST /api/admin/session` is a fresh CSPRNG token generated per call, not a
"fixed 32-byte hex secret derived from admin_token". The anchor pointed at
`bootstrap`'s body.
docs/deployment.md
- `retry_or_dlq` call site 481 -> 479; `lock:reconcile` acquire 289 -> 290
(twice); heartbeat `renew_lock` 302 -> 303 (twice).
AGENTS.md
- Record the convention: symbol names in docs, line numbers only in
`docs/deployment.md`, plus the audit script's limits so nobody reads its
output as "the anchors are correct".
scripts/docs_check/audit_anchor_semantics.py (new)
- Independent script, not wired into `run_all.sh`, always `exit 0`, prints
candidates only. It flags an anchor whose target line has no relation to the
symbol the doc claims. Candidates go 20 -> 5 and `reference.md` disappears
from the list entirely. It cannot catch a one- or two-line off-by-one - that
still needs a human - and a bare `:NNN` shorthand is not a matchable anchor.
Gates: GATE-DOCS 6/6 clean (ANCHOR AUDIT 324 -> 129 refs, 0 errors).
`src/` is untouched, so the test baseline stands at 86 passed / 0 failed /
2 ignored.
anonsaber
anonsaber