docs(deploy): document shared-Redis multi-origin topology and config convergence
10.2 now spells out that every Redis key is globally named with no instance dimension, tabulates the shared state (config blob, global revision, enable gate, cursor, streams, reconcile lock) and records two consequences: the business configuration exists once so an SPA edit from any origin writes the same copy (intended, no per-instance slot exists), and CONFIG_ENCRYPTION_KEY must be byte-identical across origins since it is the AES-256-GCM key and the SPA admin credential. Extra origins buy failover, not throughput, because /reconcile is serialized by lock:reconcile and the cursor is singular. New 10.8 documents lazy convergence at the request boundary via refresh_business_config, and the known behavior that a build_worker failure advances the local revision so the instance does not retry until a new revision is written. Includes detection via the live field values in GET /debug/config, the warning that its revision field is the global Redis value and cannot show convergence, and recovery by bumping the revision again. Appended as 10.8 rather than renumbering 10.3-10.7, which charter.md NFR-HA-MULTI-INSTANCE cites. No code changes; GATE-DOCS 6/6.
This commit is contained in:
messageweave
parent
42128433ed
commit
09d4e01ec5
|
|
@ -192,6 +192,38 @@ Multiple backends can share traffic because request state is stored in Redis, du
|
|||
deliveries are deduplicated, and reconciliation repairs missed push events. No sticky session
|
||||
or application code change is required.
|
||||
|
||||
All Redis keys are named globally and carry no instance, hostname, or origin suffix. Sharing one
|
||||
Redis is not a compromise here: several origins hold a single copy of each item below.
|
||||
|
||||
| Data | Key | Semantics |
|
||||
|---|---|---|
|
||||
| Business configuration | `config:business` | one global ciphertext blob (AES-256-GCM), with no per-instance slot |
|
||||
| Configuration revision | `config:business:revision` | one global monotonic counter, bumped in the same Lua transaction as the write (`CONFIG_STORE_REV_SCRIPT`) |
|
||||
| Enable gate | `config:enabled` | one global boolean |
|
||||
| Reconcile cursor | `state:jmap:since` | one global cursor, with no account dimension |
|
||||
| Delivery streams | `stalwart:jmap:events` / `stalwart:telegram:events` | shared through a consumer group; the group is created lazily and `XAUTOCLAIM` reclaims idle messages |
|
||||
| Reconcile lock | `lock:reconcile` | one global single-flight lock, mutual exclusion across instances |
|
||||
|
||||
Two consequences matter in practice.
|
||||
|
||||
First, **the business configuration exists once, and an SPA edit from any origin writes the same
|
||||
copy.** There is no way to change only one instance: no key carries an instance dimension and
|
||||
`refresh_business_config` takes no instance argument. This is intended behavior. If two genuinely
|
||||
independent configurations are needed, for example production alongside a pre-production twin,
|
||||
deploy a second instance pointed at a second Redis rather than distinguishing inside one Redis.
|
||||
|
||||
Second, **`CONFIG_ENCRYPTION_KEY` must be byte-identical on every origin.** It is 64 hex
|
||||
characters (32 bytes) and is used directly as the AES-256-GCM key, with the AAD fixed at
|
||||
`message-weave/config:business/v1`. It also doubles as the SPA admin credential. An origin holding
|
||||
a different value can neither decrypt configuration written by another origin nor log in to the
|
||||
SPA, and both paths fail closed.
|
||||
|
||||
Extra origins buy failover rather than throughput. Delivery is spread across the shared consumer
|
||||
group, but `/reconcile` is serialized globally by `lock:reconcile` and the cursor is singular. If
|
||||
one instance dies mid-batch, the next instance picks the messages up once they pass the
|
||||
`XAUTOCLAIM` idle window; the cost is possible duplicate delivery, guarded by the `dedup:*` keys,
|
||||
never message loss.
|
||||
|
||||
### 10.3 Trust model
|
||||
|
||||
The Worker passes request headers and bodies through. Backend authentication remains required
|
||||
|
|
@ -223,3 +255,31 @@ HTTP 200 when no valid origin is configured.
|
|||
The Worker is not a Redis proxy and does not make the backend stateless; shared external Redis
|
||||
remains required. It does not add a database, a long-lived connection, or a second layer of
|
||||
business authentication.
|
||||
|
||||
### 10.8 Configuration convergence across origins
|
||||
|
||||
One SPA edit writes once to Redis. The origins converge **lazily, at the request boundary**: every
|
||||
request-bearing entry point (`/worker`, `/reconcile`, `/healthz`, `/push/jmap`, `/api/*`) calls
|
||||
`refresh_business_config`, which reads the global revision from Redis and returns immediately when
|
||||
`remote_revision <= local_revision`. There is no polling loop and no watch, so an origin that
|
||||
receives no traffic keeps the configuration it already had.
|
||||
|
||||
**Known behavior: a failed rebuild is not retried until the next revision.** The revision check
|
||||
above is an early exit, not a retry guard. When `build_worker` fails for one origin — a JMAP
|
||||
session URL that is unreachable, or an `llm_api_key` / `llm_base_url` that is missing or invalid —
|
||||
that origin advances its in-memory revision to the remote value and keeps the runtime it already
|
||||
had. The early-exit condition then holds forever for that origin, so it does not try again until a
|
||||
**new** revision is written. Every other failure inside the same function (Redis unreachable, blob
|
||||
missing, JSON parse failure, validation failure) leaves the local revision untouched and therefore
|
||||
does retry on the next request; those paths fail before a runtime is built, so no worker is
|
||||
rebuilt.
|
||||
|
||||
The symptom is one origin running the new configuration and another still on the old one, with no
|
||||
error and no log line: the failed path returns silently. To detect it, compare the field values
|
||||
reported by `GET /debug/config` on each origin — those come from the live in-memory runtime. Do
|
||||
not use its `revision` field for this: that value is read from Redis and shows the latest global
|
||||
revision even when the instance has not converged.
|
||||
|
||||
To recover, touch the configuration again from the SPA so that the revision moves; the stuck
|
||||
origin will attempt the rebuild once. If the origin is still in the Worker's origin list, remove
|
||||
it in the meantime so requests are not split between two configurations.
|
||||
|
|
|
|||
|
|
@ -172,6 +172,33 @@ Telegram webhooks、Stalwart 推送回调和外部调度可以使用稳定的 Wo
|
|||
交付已进行重复数据删除，协调修复了错过的推送事件。无粘性会话
|
||||
或需要更改应用程序代码。
|
||||
|
||||
所有 Redis 键名都是全局命名的，**不带实例、主机名或源站后缀**。共享同一个 Redis 在这里不是
|
||||
折中：下列每一项在多源站之间都只有一份。
|
||||
|
||||
| 数据 | 键 | 语义 |
|
||||
|---|---|---|
|
||||
| 业务配置 | `config:business` | 单一全局密文块（AES-256-GCM），没有按实例划分的槽位 |
|
||||
| 配置版本 | `config:business:revision` | 单一全局单调计数器，与写入在同一次 Lua 事务中提交（`CONFIG_STORE_REV_SCRIPT`） |
|
||||
| 启停门 | `config:enabled` | 一个全局布尔值 |
|
||||
| 对账游标 | `state:jmap:since` | 单一全局游标，无账户维度 |
|
||||
| 投递流 | `stalwart:jmap:events` / `stalwart:telegram:events` | 由消费者组共享，组惰性创建，`XAUTOCLAIM` 回收空闲消息 |
|
||||
| 对账锁 | `lock:reconcile` | 单一全局单飞锁，跨实例互斥 |
|
||||
|
||||
由此有两条必须知道的推论。
|
||||
|
||||
第一，**业务配置只有一份，从任何一台源站的 SPA 编辑写进去的都是同一份**。不存在“只改其中一台”
|
||||
的表达方式：没有任何键带实例维度，`refresh_business_config` 也不接收实例参数。这是预期行为。
|
||||
如果确实需要两套互不影响的配置（例如生产与预发并行），正确做法是部署第二个实例并指向另一个
|
||||
Redis，而不是在同一个 Redis 内部做区分。
|
||||
|
||||
第二，**`CONFIG_ENCRYPTION_KEY` 必须逐字一致**。它是 64 个十六进制字符（32 字节），直接作为
|
||||
AES-256-GCM 密钥，AAD 固定为 `message-weave/config:business/v1`；它同时也是 SPA 的管理凭据。
|
||||
任一台源站持有不同的值，既解不出别的源站写入的配置，也登录不了 SPA，两条路都是 fail closed。
|
||||
|
||||
加源站买到的是故障转移，不是吞吐。投递靠共享消费者组分摊，但 `/reconcile` 受 `lock:reconcile`
|
||||
全局串行，游标也只有一份。一台实例在批次中间挂掉，消息会在 `XAUTOCLAIM` 空闲窗口过后被另一台
|
||||
接手；代价是可能重复投递（由 `dedup:*` 键去重），不会丢消息。
|
||||
|
||||
### 10.3 信任模型
|
||||
|
||||
Worker 传递请求标头和正文。仍需要后端身份验证
|
||||
|
|
@ -200,6 +227,27 @@ HTTPS URL，之后才进行路由校验或转发。Worker 使用“404”拒绝
|
|||
|
||||
### 10.7 功能边界
|
||||
|
||||
Worker不是Redis代理，不会使后端无状态；共享外部Redis
|
||||
仍然需要。它不添加数据库、长期连接或第二层
|
||||
商业认证。
|
||||
Worker 不是 Redis 代理，也不会使后端无状态；共享外部
|
||||
Redis 仍然需要。它不添加数据库、长期连接或第二层
|
||||
业务认证。
|
||||
|
||||
### 10.8 多源站之间的配置收敛
|
||||
|
||||
SPA 的一次编辑只写入 Redis 一次。各源站**惰性收敛，发生在请求边界上**：每个带请求的入口
|
||||
（`/worker`、`/reconcile`、`/healthz`、`/push/jmap`、`/api/*`）都会调用
|
||||
`refresh_business_config`，它先从 Redis 读全局版本，当 `remote_revision <= local_revision` 时
|
||||
立即返回。没有轮询循环，也没有 watch，所以一台完全收不到流量的源站会一直保留它原有的配置。
|
||||
|
||||
**已知行为：重建失败后不会重试，直到下一次版本更新。** 上面的版本判断只是一个提前返回，它不是
|
||||
重试保护。当某台源站的 `build_worker` 失败——JMAP session URL 不可达，或 `llm_api_key` /
|
||||
`llm_base_url` 缺失或非法——该源站会把内存中的版本推进到远端值，并保留原有运行时。此后提前返回
|
||||
条件对该源站永久成立，它不会再尝试，直到写入了一个**新的**版本。同一函数里的其他失败（Redis
|
||||
不可达、密文块缺失、JSON 解析失败、校验失败）都不推进本地版本，因此会在下一次请求时重试；这些
|
||||
路径都在构建运行时之前就返回，所以不会重建 worker。
|
||||
|
||||
症状是一台源站已运行新配置、另一台还停在旧配置上，而且**没有报错也没有日志**：失败路径是静默
|
||||
返回的。排查方法是逐台比对 `GET /debug/config` 返回的字段值——那些值来自内存里的实时运行时。
|
||||
不要用它的 `revision` 字段判断：那个值从 Redis 读取，即使该实例尚未收敛也会显示最新的全局版本。
|
||||
|
||||
恢复办法：在 SPA 里再动一次配置让版本前进，卡住的源站就会重新尝试一次。如果这台源站仍在
|
||||
Worker 的源站列表里，这期间把它从列表中移除，避免请求被分裂到两套配置上。
|
||||
|
|
|
|||
Loading…
Reference in New Issue