23 KiB

Raw Blame History

MessageWeave

中文文档：README.zh-CN.md · Architecture & design: docs/design.md · Deployment & operations: docs/deployment.md

A personal email assistant that connects your Stalwart mailbox to Telegram: you get notified in Telegram when a new email arrives, and you can read or summarise emails on demand.

⚠️ This is an early-stage project — please complete a small-scale integration test first. Real mailboxes and Telegram still need to be verified by the operator. JMAP Push subscription registration and verification write-back are implemented, but the callback URL is never guessed automatically: an administrator must call the registration API explicitly.

What this project is

You run your own Stalwart mail server.
This tool connects your mailbox to a Telegram bot: when a new email arrives, it pushes a notification (sender / subject / time only — no message body).
In Telegram you can ask to view the raw email, or to have it summarised or translated by an AI.

Privacy model

Viewing the raw email never touches any AI. It is read straight from your mailbox.
The body is sent to an AI only when you explicitly ask for it and confirm.
Analysis results are never stored.
The AI is optional and can be left unconfigured or fully disabled.

What you need before deploying

A Stalwart mailbox account, with an App Password generated for it — not your account's master password.
A Telegram bot (create one via @BotFather), and its Bot Token.
A Redis database that you operate (AOF persistence recommended). All of the bot's state lives there. Use rediss:// for TLS Redis and redis:// for plain connections; if your password (Upstash etc.) contains @, :, / or #, URL-encode it first.
A public HTTPS address. The platform points this address at the bot container (HTTPS is provided by the platform).
An external cron / scheduler that calls /reconcile every 5–10 minutes so the bot periodically reconciles against your mailbox. The bot does not run any timers of its own.

Addresses that must be registered

Tell Telegram to send its webhook to your-public-address/webhook/tg.

If you use the optional "multiple instances + Cloudflare Worker load balancer" setup (see docs/deployment.md §10), point the Telegram webhook and the reconcile URL at the Worker's stable domain (https://<your-worker>.workers.dev/webhook/tg etc.) instead of the per-instance platform addresses.

Creating the container in your Docker manager

Create a new container, fill in the usual image / port / memory fields, and inject only the Redis credentials:

Environment variable	Required?	Value
`REDIS_URL`	✔	Redis connection string (including password), e.g. `redis://:password@redis.example.com:6379/0`
`CONFIG_ENCRYPTION_KEY`	✔	A 32-byte hex key; generate with `openssl rand -hex 32`

After the first start, submit the Telegram / JMAP / LLM and business-auth configuration through the protected SPA page. Configuration secrets are encrypted into Redis and are never returned by the API; after a successful admin PUT the service validates the new configuration and hot-rebuilds the clients atomically, so subsequent requests use it immediately, and on failure the previous instances are kept.

AI summarisation is off by default. A Telegram user must first send one of the recognised consent phrases, then /summary <email_id>: /ai on or /ai yes (1 hour); any message containing 临时 or 一次, e.g. 临时一次 (one-off, 1 hour); 今天 (24 hours); 7天 (7 days); or 直到我撤销 / 长期 (until revoked, capped at 365 days). These phrases are matched literally, so type them in Chinese exactly as shown — English aliases are not recognised. To revoke: /ai off (or 撤销授权). Authorisation never auto-renews once it expires. Without a valid authorisation only metadata is returned, and if the LLM is unset or fails the bot falls back to a locally truncated summary.

If REDIS_URL or CONFIG_ENCRYPTION_KEY is missing, the service enters configuration-setup mode: the SPA, /api/status and the probes still work. The SPA hides the admin-session section and shows only which variables are missing, together with the note that keys are never stored or echoed. Once the configuration is restored, the authorisation UI appears only after /api/status reports ready=true.

HostStack native deployment

HostStack builds and runs the Rust service using the repository-root hoststack.yaml: it runs cargo build --release --locked, executes ./target/release/message-weave, and uses /healthz as a health check every 30 seconds with a 5-second timeout. Provide REDIS_URL and CONFIG_ENCRYPTION_KEY as HostStack Secrets; the keys are never written into the YAML, the image, or any repository file.

The "enable business processing" switch in the admin session is persisted in Redis at config:enabled; a missing value or a read failure is treated as disabled. While disabled, the webhook, reconcile and worker routes return HTTP 503 and neither ack nor drop upstream events; the SPA, status and configuration APIs stay available.

How to check that it is healthy after startup

Open your-public-address/healthz in a browser or with curl: a normal response means the bot process is alive.
/ready checks configuration completeness and Redis reachability. When ready it returns 200 with a JSON readiness report ({"status":"ready","configured":...,...}). When not ready it returns 503 with the standard error envelope {"error":"service_unavailable","request_id":"<id>"} and Retry-After: 30. It never calls JMAP or Telegram and never triggers a business side effect.
For monitoring, Uptime Kuma is recommended: an HTTP liveness check on /healthz and an HTTP readiness check on /ready (expect 200, 30-second interval, 5-second timeout). The project ships no Prometheus exporter.
/reconcile performs authentication, takes a Redis single-flight lock, and runs a bounded JMAP Email/changes reconciliation. Events are written to Redis Streams idempotently, and the Redis sinceState cursor is only advanced after all of them are queued; a temporary dependency failure returns 503 so cron can retry. Push events still go through the authenticated, de-duplicated enqueue path, so this must not be described as complete real-time synchronisation.
The cold-start baseline also has page, message and time budgets. When a budget is exhausted the checkpoint is saved and the next /reconcile continues from it, so a large mailbox is never re-scanned from the beginning.

Configuration management

Open the root address your-public-address/ and enter the password of the Redis ACL user from REDIS_URL to create an admin session valid for 15 minutes. The session and the keys typed into the form live only in the current page's memory; after a refresh or closing the page you must authorise again. The page never stores sessions, keys or configuration in browser storage or cookies.

The page keeps two kinds of configuration separate: runtime parameters (JMAP, Telegram, LLM timeouts and retry limits) can be read and saved through the API; business configuration (JMAP, Telegram, LLM, chat allowlist and the various auth tokens) is fully replaced through PUT /api/business-config. To avoid echoing secrets, the service exposes no read endpoint for business configuration, so every full replacement requires re-entering the mandatory fields. After a 204 response the page reports that the new configuration has been hot-loaded and clears the key inputs.

While Redis is not connected or initialised, the admin-session and configuration APIs return 503; the page explains the unavailable state and keeps runtime-parameter saving disabled until the read succeeds. When reached through a Cloudflare Worker, use the Worker's root URL; the admin API paths /api/config, /api/business-config, /api/admin/session and /api/admin/session/revoke are on the restricted route allowlist (POST /api/push/register is not — see docs/deployment.md §10.5).

End-to-end first integration

This is a repeatable path from an empty deployment to a first message in Telegram. Every <...> below is a placeholder; do not copy the angle brackets, and never paste real secrets into chats, tickets or the repository.

1. Verify the deployment and Redis

curl -fsS https://<message-weave-domain>/api/status
curl -fsS https://<message-weave-domain>/healthz

With correct REDIS_URL and CONFIG_ENCRYPTION_KEY, /api/status should contain:

{"ready":true,"mode":"configured","missing":[]}

If a variable is missing, the container does not exit; it enters configuration-setup, and the SPA shows only the missing variables without the admin-session section. The Redis URL must use the right scheme: rediss:// for TLS, redis:// for plain connections. A TLS Redis must not be verified only with redis-cli --tls — the application's own REDIS_URL must use rediss:// too.

2. Create a Telegram bot and a test group

Open @BotFather in Telegram, send /newbot, follow the prompts and save the bot token. Then create a group (not a channel), e.g. MessageWeave, add the bot and make it an administrator.

Send /start in the group, then read the Chat ID from the Bot API's getUpdates endpoint:

curl "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/getUpdates"

In the response:

"chat":{"id":-5260770881,"title":"MessageWeave","type":"group"}

-5260770881 is the Chat ID. Group IDs are usually negative — do not drop the minus sign. getUpdates is not a group-list endpoint; the group only appears once the bot has actually received a message in it. If it returns an empty array, send /start in the group first. If a webhook is already set, delete it first, otherwise messages are consumed by the webhook:

curl -X POST "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/deleteWebhook"

If the bot cannot see ordinary messages, run /setprivacy in @BotFather, choose the bot and select Disable; or make sure the bot is an administrator. Switch the webhook back on once debugging is done.

3. Generate the application-side auth secrets

Generate these three values separately. Do not reuse the Redis password, the bot token, or the SPA admin session:

openssl rand -hex 32   # Telegram Webhook Secret
openssl rand -hex 32   # Reconcile Token
openssl rand -hex 32   # Worker Token

Telegram Webhook Secret: you generate it, submit it through the SPA, and pass it as secret_token to setWebhook. MessageWeave validates the X-Telegram-Bot-Api-Secret-Token request header.
Reconcile Token: you generate it; the external cron uses it for /reconcile.
Worker Token: you generate it; it protects /worker and the compatibility admin API.

4. Create the SPA admin session

Open https://<message-weave-domain>/, enter the password of the Redis ACL user from REDIS_URL, and click to create the admin session. It is valid for 900 seconds, lives only in the current page's memory, and must be created again after a refresh.

5. Fill in the Telegram business configuration

Example:

Target Chat ID:          -5260770881
Chat allowlist:          -5260770881
Telegram Webhook Secret: <first openssl random value>
Reconcile Token:         <second openssl random value>
Worker Token:            <third openssl random value>

Target Chat ID and Chat allowlist are not the same field: the former is the default target, the latter is the whitelist of chats allowed to drive the bot. For a single test group they can hold the same ID. The allowlist accepts one entry per line or a comma-separated list; duplicates are rejected.

6. Fill in the JMAP business configuration

The JMAP Session URL may be either https://mail.example.com or https://mail.example.com/.well-known/jmap; both are normalised.
HTTPS only. Do not put a username, password, query string or fragment into the URL.
JMAP Username is the mailbox account.
JMAP Password is the Stalwart App Password.
Account ID may be left empty for the first integration test; the session's primary account is used.
You do not need to create a PushSubscription by hand in the Stalwart WebUI. The built-in user role normally already includes reading, creating, modifying and deleting PushSubscriptions; only if registration actually returns forbidden should an administrator check the role under /admin → Management → Directory → Accounts/Roles. Stalwart's Push global settings under Settings → Network → JMAP → Push are not a per-subscription configuration.
Keep the LLM disabled for the first integration test, so you do not mix mailbox problems with AI network configuration.

7. Register the JMAP PushSubscription

MessageWeave does not require you to create a PushSubscription by hand in the Stalwart WebUI. After saving the JMAP business configuration, call the registration API with the current SPA admin session or WORKER_TOKEN, supplying a public HTTPS callback URL explicitly:

curl -X POST "https://<messageweave-domain>/api/push/register" \
  -H "Authorization: Bearer <admin-session-or-worker-token>" \
  -H "Content-Type: application/json" \
  -d '{"callback_url":"https://<messageweave-domain>/push/jmap"}'

On success the response contains push_subscription_id. The URL must be HTTPS and must not contain a username, password or any other embedded credential. Re-registering the same callback URL reuses the existing subscription instead of creating a duplicate. Stalwart then sends its generated PushVerification to /push/jmap; MessageWeave writes the verification code back through JMAP PushSubscription/set, and only accepts real Push events once verification has succeeded. The registration API never guesses the platform domain and never accepts a user-entered verification code.

The regular user role normally already has read/create/modify/delete permissions on PushSubscriptions, so no manual configuration is needed; only check /admin → Management → Directory → Accounts/Roles if you actually get forbidden. Stalwart's Settings → Network → JMAP → Push page holds global Push parameters, not per-subscription creation.

References: Stalwart Push notifications, Stalwart Permissions, Stalwart Roles, RFC 8620 §7.2 PushSubscription.

When the business configuration is saved the service really instantiates the JMAP client; if that fails it returns 503, stores no broken configuration, and you can fix and re-save. There is no GET endpoint for business configuration, so after a successful save the keys are not echoed back; every full replacement requires re-entering the mandatory fields.

8. Set the Telegram webhook and enable business processing

After the business configuration is saved:

curl -X POST "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/setWebhook" \
  --data-urlencode "url=https://<message-weave-domain>/webhook/tg" \
  --data-urlencode "secret_token=<TELEGRAM_WEBHOOK_SECRET>"

curl "https://api.telegram.org/bot<TELEGRAM_BOT_TOKEN>/getWebhookInfo"

Make sure the returned url is correct and that last_error_message does not keep growing. Then go back to the SPA and switch on "enable business processing". While the global switch is off, the webhook, reconcile and worker routes return 503 service disabled and neither enqueue nor ack upstream messages.

Finally send a test message in the allowlisted group. When it arrives, watch Telegram getWebhookInfo, the platform logs and the bot's reply. If there is no reply, check the Chat ID's minus sign, the allowlist, the Webhook Secret, the global switch, and the bot's admin rights first.

9. Test reconcile authentication

curl -i -X POST "https://<message-weave-domain>/reconcile" \
  -H "Authorization: Bearer <RECONCILE_TOKEN>"

A wrong token must return 401. The external cron is recommended to call once every 5–10 minutes, as a safety net for email reconciliation when Push events are lost.

How email synchronisation works today

Do not treat PushSubscription as the only reliable source in the current release. Push callbacks still have to be authenticated and enqueued, so run the external cron calling /reconcile at the same time. Reconciliation uses a persistent Redis sinceState and returns 503 on enqueue failure instead of advancing the cursor early.

If your platform lets the service sleep, the cron's HTTP request will usually wake the container, but the first request pays the cold-start cost and may exceed the cron's or the platform's request timeout. Therefore:

set the cron timeout long enough and allow retries on failure;
do not call more often than the platform's wake-up / request limits allow;
do not use /healthz as email synchronisation — it only checks that the process is alive;
if the service cannot be woken while sleeping, reconciliation cannot run either; in that case disable sleeping or use an always-on instance.

The reconciliation lock is renewed by its holder while the task runs; if the renewal fails the cursor is not advanced. Push registration and reconciliation both use Redis owner-token single-flight locks, so concurrent requests cannot create duplicate subscriptions or advance the same JMAP cursor twice.

Troubleshooting

Redis: `can't connect with TLS, the feature is not enabled`

Usually the Redis TLS feature was not enabled when the image was built, or redis:// / rediss:// is wrong. Use the latest image with TLS support and make sure REDIS_URL=rediss://.... Special characters in an Upstash password must be URL-encoded, e.g. @ becomes %40.

Redis: `Multiplexed connection driver unexpectedly terminated`

First validate the network and credentials with the same URL from outside. Do not validate only redis-cli --tls while the application still uses redis://. Check that HostStack / the platform allows outbound 6379/TLS, and check the ACL user, password and port in the URL.

rustls provider panic

This is the old image loading more than one TLS provider. You must use the latest image, which installs the explicit rustls ring provider; do not keep using cached old image tags.

`getUpdates` returns an empty array

The bot has received no new messages, or the message was already consumed by the webhook. Delete the webhook and send /start in the group again; confirm the bot is in the group, is an administrator, and — if needed — that privacy mode is disabled through BotFather.

Telegram says it is not allowed to read messages

Confirm that you created a group and not a channel; make the bot an administrator; send /start again. With privacy mode enabled, the bot normally only sees commands, replies and mentions.

SPA returns `503` when saving business configuration

This usually means the JMAP URL is unreachable from the deployment platform, the certificate/DNS is broken, the app password is wrong, or the LLM configuration fails HTTPS validation. A failed save keeps the old worker and the old configuration, and does not lock the initialisation slot; fix and retry.

SPA shows configuration-setup

Check that the HostStack Secrets or container environment contain both:

REDIS_URL
CONFIG_ENCRYPTION_KEY

CONFIG_ENCRYPTION_KEY must be a 32-byte hex value, e.g.:

openssl rand -hex 32

It must never be written into the public hoststack.yaml, the Dockerfile, the README or the image.

Privacy and operational red lines (transparent to users)

No database, no local files: all of the bot's state lives in your Redis. The container writes no local files and mounts no local volumes; after a restart it recovers from Redis plus mailbox reconciliation.
Logs go to stdout only: the container prints logs to stdout/stderr, and your platform or collector is responsible for collection and storage. The bot does not write log files.
Sensitive content never lands in logs: mailbox bodies, keys, AI requests/responses and attachment contents are never written to logs or Redis. Only structured events, counters, timestamps and redacted summaries are recorded.
See the checklists in docs/deployment.md §0 and §8.2.

What it can do today / what is still missing

Already in place

The entry-point framework, three layers of auth protection, and configuration loading.
A configuration SPA backed by short-lived Redis admin sessions: the full business configuration can be edited and hot-loaded, and /api/config runtime parameters can be read and saved.
Basic code for reading your Stalwart mailbox (list folders, list emails, read raw messages).
Telegram webhook, Chat ID allowlist, Reconcile / Worker bearer tokens, and a Redis-persisted global enable switch.
HostStack native Rust deployment configuration and a Docker build.
Optional: a free Cloudflare Worker front-load-balancer for multi-instance HA — see docs/deployment.md §10 and the cloudflare-worker/ directory.

Still missing / not yet verified

⚠️ Real mailbox integration needs to be verified by the operator: the JMAP client code and configuration validation are provided, but different Stalwart networks, permissions and app passwords still differ.
JMAP Push registration is implemented: after an administrator submits a public HTTPS callback URL through POST /api/push/register, the backend calls PushSubscription/set create and /push/jmap writes the verification code back automatically. A real Stalwart environment still needs to be verified, and the external cron calling /reconcile remains the reliable compensation channel.
Production-grade multi-instance coordination and sending email are not finished yet; AI summarisation is optional and needs separate configuration and verification.

Both a Docker image and the HostStack native Rust deployment are provided, but we still recommend running the integration steps above on a small scale before using it in production.

Quality gates for contributors

Everything must be green before a change is released (run inside the Debian rust:1-slim-bookworm container — see AGENTS.md §5 and docs/deployment.md §8):

cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test

Real-device verification tests are marked #[ignore] and driven by environment variables (cargo test -- --ignored jmap::); CI does not run them by default. Do not claim "verified on real hardware" before actually running them.
Any change that touches a security boundary must be reflected consistently in all three of docs/design.md, docs/deployment.md and AGENTS.md.

For developers

Architecture, design and deployment details:

License

(TBD)

23 KiB Raw Blame History Unescape Escape