Plain-English reference for every monitoring warning that is live in production (Prometheus tier0.yml). Each row links to its fix.
(the failure model) and the report at monitoring/threshold_tuning/threshold_report.md.
| Alert | Sev | Watches | Fires when | What it means | Fix |
|---|---|---|---|---|---|
| InstanceDown | critical | a scrape target (prometheus, node_exporter, postgres_exporter, sui_node) | unreachable for 2 min | a host or exporter is down → we're flying blind on it | ↗ |
| PostgresDown | critical | PostgreSQL (pg_up) | down for 2 min | the database is unreachable → the API can't serve | ↗ |
| SuiIndexerDown | critical | the sui-indexer service | stopped >2 min (host still up) | chain ingestion has halted → MEV data goes stale | ↗ |
| SuiNodeFallingBehind | critical | Sui node checkpoint lag | >30 min behind (tuned from 10 min) | the node isn't executing checkpoints → indexer gets stale data. Usual cause: WireGuard tunnel down | ↗ |
| HostMemoryAlmostExhausted | critical | host free memory (MemAvailable) | < 500 MB for 2 min | OOM-kill risk to Postgres / API / sshd | ↗ |
| BackupStale | critical | age of last successful pg_dump→S3 | > 24 h (tuned from 26 h) | a daily backup run was missed → no recent restore point | ↗ |
| PartitionArchiveStale | critical | age of last partition archive | > 9 days | cold-partition offload stalled → DB + dumps grow unbounded | ↗ |
| DiskAlmostFull | warning | any DB-host filesystem | < 10 % free for 5 min | disk filling → Postgres write failure at 0 % | ↗ |
| PgDumpDurationDrift | warning | 7-day median pg_dump runtime | > 4 h (tuned from 90 min) | capacity drift — backups still succeed, but the dump is growing toward the 14 h timeout; schedule partitioning | ↗ |
(the old values fired on normal operation — chiefly PgDumpDurationDrift, which accounted for ~99 % of alert noise).
sui-monitoring.yml alerts (replication-to-Neon, /mt/sp1, indexer crons) wereretired — they referenced metrics no longer scraped after the Vultr migration and could never fire. See alert-relationships.md.