Corrective actions runbook

At-a-glance list of all warnings + thresholds: alert-definitions.md

One section per alert. Every alert's runbook_url annotation links here, so each Telegram notification carries a direct link to its fix. Thresholds are tuned by monitoring/threshold_tuning/tune_thresholds.py (see alert-relationships.md).

Access: AWS hosts are reached over SSH through the bastion (ProxyJump); the Vultr sui node directly via its operator SSH alias. Exact hostnames, IPs, ports and keys live in the operator's ~/.ssh/config and password manager — not here (hosts are identified by role in this doc).

PgDumpDurationDrift

Means: the 7-day-median nightly pg_dump runtime exceeds the threshold (4h). This is a capacity-drift / trend warning, not an outage — backups are still succeeding. It rises as the DB grows. Severity: warning. Not urgent unless approaching the 14h TimeoutStartSec. Check: birdai_pg_backup_last_duration_seconds trend in Grafana. Fix (in order):

  1. Confirm backups still succeed (see BackupStale). If yes, this is informational.
  2. Reduce dump volume: ensure cold mev_sc_tx_* partitions are excluded
  3. (--exclude-table-data) in /usr/local/bin/pg-dump-to-s3.sh, and that the
  4. partition-archive offload is running (see PartitionArchiveStale).
  5. Longer term: move backups to EBS snapshots (decouples from the ~80 MB/s
  6. instance EBS ceiling that makes the dump I/O-bound).

SuiNodeFallingBehind

Means: the Sui node's checkpoint execution lag exceeded 30 min — the indexer is ingesting stale or no chain data. Severity: critical. Most common cause: the WireGuard tunnel (indexer ↔ Vultr node) is down after an egress-IP change. Check: on the indexer, sudo wg show → 0 B received? Fix:

  1. Tunnel down → on the Vultr node allow the indexer's current egress IP:
  2. sudo ufw allow from <INDEXER_EIP> to any port 51820 proto udp. Handshake
  3. recovers in ~25s; indexer catches up at ~30× realtime.
  4. Node itself behind → check sudo systemctl status sui-node on the Vultr node;
  5. review journalctl -u sui-node. The node auto-upgrades nightly (rollback on
  6. failure) — a failed upgrade rolls back automatically.
  7. Confirm recovery: max(timestamp_ms) in ticks resumes advancing.

SuiIndexerDown

Means: the sui-indexer systemd service is not active (its host is still reachable — otherwise InstanceDown fires instead). Chain ingestion has stopped; MEV data will go stale. Severity: critical. Check: systemctl status sui-indexer on the indexer host; journalctl -u sui-indexer -n 100. Fix:

  1. If crash-looping, read the journal for the cause (often a pipeline/binary
  2. mismatch — e.g., an enabled-but-unregistered pipeline → `Tried to enable
  3. pipelines that this indexer does not know about`). Fix config / rebuild.
  4. If the node feed is the problem, check SuiNodeFallingBehind
  5. (WireGuard tunnel) first — the indexer can wedge when its RPC source is gone.
  6. systemctl restart sui-indexer; confirm max(created_at) in ticks resumes
  7. advancing.

BackupStale

Means: no successful pg_dump in >24h (one daily run missed). Severity: critical. Check: journalctl -u pg-dump-to-s3 (or the backup timer) on the DB host; birdai_pg_backup_last_success_timestamp. Fix:

  1. Inspect the last run's logs for the failure (timeout? S3 auth? disk?).
  2. Run a manual backup: trigger pg-dump-to-s3.service (User=postgres).
  3. If it timed out, see PgDumpDurationDrift — the dump may
  4. be exceeding TimeoutStartSec.

PartitionArchiveStale

Means: no successful partition-archive (cold-partition offload to S3) in >12d. This is the real DB-size governor; if it stalls the DB and dumps both grow. Severity: critical. Check: the partition-archive timer/logs; birdai_partition_archive_last_success_timestamp. Note: the archive offload was scaffolded but historically under-implemented — if the metric is absent the textfile-collector path may be broken. Verify the job actually exists before assuming a transient failure.

DiskAlmostFull

Means: a db-host filesystem is <10% free. Severity: warning (→ becomes an outage at 0%). Fix:

  1. df -h on the DB host; find the growing mount (du -xhd1 /var/lib/postgresql | sort -h).
  2. Usual culprits: WAL buildup, runaway logs, an un-rotated dump. Truncate/rotate.
  3. If it's genuine DB growth, run partition archival / drop cold partitions per
  4. the MEV retention policy, or grow the gp3 volume.

HostMemoryAlmostExhausted

Means: MemAvailable < 500 MB — OOM risk to Postgres/uvicorn. Severity: critical. Fix: free -h, ps aux --sort=-%mem | head. Identify the consumer; restart the offending service if leaking; check for a runaway query (pg_stat_activity). Confirm swap isn't thrashing.

InstanceDown / PostgresDown

Means: a scrape target (up==0) or Postgres (pg_up==0) is unreachable for 2 min. Severity: critical. Binary — always actionable. Fix: SSH to the instance; systemctl status the service (birdai-api, postgresql@16-main, prometheus, node_exporter, postgres_exporter, sui-node); review journal; restart. If the host itself is down, check the EC2/Vultr console.

AlertmanagerDown

Means: Prometheus cannot scrape the local Alertmanager (up{job="alertmanager"}==0) for >5 min — rule evaluation still works, but alert delivery (Telegram) is broken on this box. Only fires on validator stamps that scrape localhost:9093; the AWS box has no such scrape job, so it can never fire there. Severity: critical. Inherent limit: a dead Alertmanager can't deliver its own death notice — this alert surfaces in the Prometheus/Grafana UI and during stamp verification drills, not via Telegram. Fix: SSH to the box; systemctl status alertmanager, journalctl -u alertmanager -n 50. Common causes: bad render of /etc/alertmanager/alertmanager.yml (re-run monitoring/render.sh), a missing /etc/alertmanager/bot_token, or the service failing to bind :9093. Restart and confirm up{job="alertmanager"} returns to 1.

Thresholds last tuned by the monthly alert-threshold-tuner job. To re-run on demand (on the app host, where Prometheus is local): PROM_URL=http://localhost:9090 python3 monitoring/threshold_tuning/tune_thresholds.py. Operator UI access is via the HTTPS URLs — see access.md.