# Operations

Audience: the operator running a deployed bot. Purpose: know what to watch, what each signal means, and what to do when something breaks. Prerequisites: a running deployment (DEPLOYMENT ); the glossary .

# The three health surfaces

1. The chat surfaces — /status in Discourse server reachability and version, active actions, held locks, and pending gates. Announcements (server down, stream down) arrive as posts, once each.
2. The metrics endpoint — curl http://127.0.0.1:9464/metrics (loopback only, gate G18).
3. The logs — one JSON line per event via journalctl -u ocbot -o cat.

# Metrics reference

Every family exposed on 127.0.0.1:9464 (names are load-bearing — see crates/ocbot/src/app/metrics.rs):

# Surface health

Each configured chat surface is wrapped in an instrumented port (SPEC-004 PH-2). Three consecutive failed sends declare the surface down; the first successful send after an outage declares it recovered. Both transitions are edge-triggered and announced exactly once — a downed surface posts one notice on every other configured surface (with one surface configured, the notice goes nowhere and only the gauge moves). Sustained ocbot_surface_send_failures_total growth with ocbot_surface_up{surface}=0 means the transport (bot token, daemon, socket) is the problem, not the bot: restart the surface’s daemon, and the recovery notice confirms the fix.

# The signal-cli surface (SPEC-004 PH-3 contract layer)

Selecting signal in channel gates the daemon at boot, before the Discord credentials are read, so a signal-only box can be diagnosed without any token. The whole check is budgeted at 2 s (an absent daemon never stalls boot).

Exit codes: 1 config/secrets/version/permission-baseline faults, 2 permission baseline, 3 no chat surface survived the doctor (e.g. channel = "signal" with a dead daemon). A signal daemon problem never takes a healthy Discord surface down — that is the design (D-10), and the log line signal surface refused by the boot doctor names the reason.

The pinned wire contract — the exact JSON-RPC subset, the reply shapes, and what a live capture must still confirm — is recorded in crates/ocbot/src/adapters/signal/KNOWN_CAPS.md. tests/common/signal_fake.rs is that contract as an executable double: every client method is tested against it, so a signal-cli upgrade that drifts shows up as a contract-test failure, not as a mystery in production.

# Talking to the bot over Signal (SPEC-004 PH-4)

A healthy daemon now carries the full surface. Send plain text on the 1:1 chat; the bot answers as ordinary messages.

• Trust boundary. Only [signal].operator_number is served. Any other sender is dropped with no reply (a reply would confirm a live number), audited as denied_surface, and counted in ocbot_surface_denied_total{surface="signal"}. Group conversations are dropped the same way.
• Rate limit. A per-sender token bucket ([signal] rate_limit) drops bursts with ocbot_surface_rate_dropped_total — never a busy-loop reply.
• Dedupe. A redelivered envelope (same sender + timestamp) is counted in ocbot_signal_envelope_duplicates_total and ignored, so a side-effecting command never runs twice.
• Commands. run [dir] asks for the prompt (reply with it, or run <dir> | <prompt> to skip), prompt [text], status, abort, release, mute, reset, ls [dir] [page], cd <dir>, history [page], send/edit (the latest staged action), pick <n>, accept|park|follow|discard [label], gate <label> <once|always|reject>, help, and cancel (closes an open flow). An unknown verb replies with the command table.
• Decisions. Send a bare verb to decide the latest reported action in that chat, or qualify it with the action’s label. Messages that carry decision buttons also render a legend (✅ accept · 🔁 follow up · 🅿️ park · 🗑️ discard, keycaps for gate options) — react with the matching emoji to decide. A reaction on a disabled or unmapped verb replies with a one-line correction.
• Multiline flows. A modal on Discord (new run, edit prompt, cd, accept-with-note) becomes a short text conversation. One flow is open per chat; a new one replaces it. Flows expire after 10 minutes (ocbot_pending_flows_open tracks live ones) or with cancel.
• Attachments. Files are downloaded by the daemon, ingested through the same inbox path as Discord uploads, and become the draft for the next prompt.
• Quoting. Quote a delivered report and reply to continue that session (the same adopt path as Discord’s reply-to-report).
• Quiet edits. edit on Signal is a fallback post prefixed ↻ update: (native edit is version-dependent), folded to at most one per ops tick.

A down Signal surface moves ocbot_surface_up{surface="signal"} to 0 and, when Discord is also configured, posts one notice there. Restart the daemon; the machine reconnects with backoff and the recovery notice confirms it.

Degraded-surface playbook.

1. systemctl status signal-cli — a crash loop, an old version, or a bad socket.
2. journalctl -u ocbot -b | grep signal — the boot doctor’s reason (FR-114/FR-116) or a reconnect loop.
3. curl -s http://127.0.0.1:9464/metrics | grep surface_up — which surfaces are up.
4. Restart the daemon (systemctl restart signal-cli); the gauge flips back and one recovery notice is posted.
5. If the socket hygiene is the fault: ls -l /run/signal-cli/socket must show the service user and mode 0600. The shipped unit guarantees both; a hand-rolled one usually does not.

Rate limits, both directions. Outbound, the surface sends at most one message per second and preserves ordering (a report never lands after its own confirmation). Inbound, a per-sender token bucket ([signal] rate_limit, default 10 with a 1/s refill) drops bursts into ocbot_surface_rate_dropped_total — never a reply, so a flapping sender cannot make the bot a busy-loop. A Signal redelivery (the same envelope resent by the server) is absorbed by the dedupe cache and counted, never re-executed.

# Logs

Structured JSON from tracing, one line per event. Spans carry action_id, session_id, and dir, so you can follow one action through the log. The filter comes from RUST_LOG and defaults to info.

journalctl -u ocbot -o cat -n 100
RUST_LOG=debug systemctl restart ocbot   # temporary, not for production

Every line passes through the scrubbing writer: secret-shaped strings (tokens, passwords, API keys) are redacted before they hit disk. The built-in patterns ship in config.rs; add more with redact_patterns in the config.

# Alerts worth adding

The bot ships metrics, not alert rules. Against a stock Prometheus, these match the design’s own thresholds:

# Degraded behaviors (all by design)

• opencode unreachable: exactly one “server down” post, one recovery post (FR-38). Actions in flight resume via reconcile when the server returns.
• Event stream down: reconnect with backoff; after announce_after_secs (default 60), exactly one announcement and one recovery post. Fallback ticks keep correctness; worst-case gate latency regresses to gate_poll_interval_ms… or, if you tuned the ticks for doorbell-era latency, to that cadence.
• Doorbell disabled ([events] enabled = false): pure polling mode; ocbot_detection_total{source="fallback"} tells you the doorbell is off duty.
• Stall watchdog: an active action silent for stall_minutes (default 45) gets exactly one plain ping — it surfaces silence, it does not diagnose.

# Incident runbook

# Database operations

The store actor owns the single SQLite connection (WAL mode) and applies schema migrations at boot, tracked in a one-row schema_version table. You never need to touch the file while the bot runs. For snapshots, use the backup script’s VACUUM INTO — never copy the WAL sidecars by hand while the process is live. Restore is RESTORE ’s job.

# Capacity

Single operator, single host: that is the design envelope, not a limitation to grow out of — see scaling notes . Memory steady state is ~20 MB against a 200 MB ceiling; the backup set is single-digit MB; request volume is bounded by one SSE stream and roughly one Discord message per event.