The model behind estimate set #2 of the threshold tuner (monitoring/threshold_tuning/tune_thresholds.py). For each monitored data point: what failure it predicts, where the failure point actually is, and the safety margin that backs the warning off from it. The tuner encodes these as failure_point / safety per alert.
Evidence grade: D = direct (we've seen the failure), I = inferred from the system's design, S = standard SRE convention.
tier0.yml)| Alert | Data point | Relationship to failure | Failure point | Margin | Grade |
|---|---|---|---|---|---|
| PgDumpDurationDrift | 7d-median pg_dump duration | A dump that can't finish inside its TimeoutStartSec (14h) or collides with the next nightly window → no backup that night. Slow-moving capacity signal, not an acute failure. | ~14h (50400s) — the systemd timeout | warn at 50% of it (~7h); but the data-extreme (≈228min) is far tighter, so that wins | I |
| SuiNodeFallingBehind | checkpoint execution lag (s) | Node not executing checkpoints → indexer ingests stale/no data. Brief lag (<~min) is normal catch-up; sustained lag = real (tunnel down, node wedged). | unrecoverable ≈ 1h behind | warn at 30min (50% of failure) | D (WireGuard breaks caused 645min lag this month) |
| BackupStale | age of last successful backup | No recent restore point. Backups are daily → >24h means a run was missed. | 48h (2 missed days) | warn at 24h (1 missed day) | I |
| PartitionArchiveStale | age of last partition archive | Cold-partition offload (the real DB-size governor) stalled. Weekly cadence. | 14d | warn at ~12d (7d cadence + grace) | I |
| DiskAlmostFull (db) | min free fraction | 0% free → Postgres write failure / crash. | 0% free | warn at 10% free (standard runway) | S |
| HostMemoryAlmostExhausted | min MemAvailable | 0 → OOM-killer reaps Postgres/uvicorn. | 0 bytes | warn at ~500MB | S |
| InstanceDown / PostgresDown | up / pg_up | Binary. A down target/DB is always an incident. | n/a | n/a — not tunable | D |
sui-monitoring.yml, NOT loaded in prod)These reference metrics that this Prometheus does not scrape (pre-migration indexer setup: replication-to-Neon, /mt/sp1, sui_indexer_cron_*). They are a disabled reference file — they can't fire. Recommendation: retire them or re-wire their exporters before relying on them. The tuner reports them as metric_absent / dead.
duration is a slope/forecast alert: fire when the 7d median's growth rate projects crossing the 14h timeout within N days. Worth adding.
for: duration (sustained-lag) already filtersblips; the threshold mostly governs sensitivity to a wedged node.
far from danger); their standard fixed margins are the right relationship and the tuner deliberately leaves them unchanged.