Oleksandr Kozachuk
33 lines
1.3 KiB
Markdown
33 lines
1.3 KiB
Markdown
# SPEC-012 — Operations hardening
|
|
|
|
Runtime + host operability (FDB-012). Backups and host wiring are
|
|
`manual` coverage; the in-process alerting is `test`.
|
|
|
|
### OPS-13 — Consistent DB backups (coverage: test)
|
|
|
|
`deploy/backup_db.py` writes a gzipped snapshot of `bot.db` using the
|
|
sqlite3 online-backup API — consistent even while the bot writes
|
|
(WAL-safe) — with 0600 permissions. Restoring a snapshot yields a
|
|
readable database with the same rows.
|
|
|
|
### OPS-14 — Backups are rotated (coverage: test)
|
|
|
|
The newest `backup-keep` (default 14) snapshots are kept; older ones
|
|
are deleted. Timestamped names sort chronologically so rotation is a
|
|
pure list operation.
|
|
|
|
### OPS-15 — Backup cron on each host (coverage: manual)
|
|
|
|
Each host runs `backup_db.py` daily via cron, writing to
|
|
`~/backups/<bot>/` (outside `~/fjerkroa_bot`, so deploys and service
|
|
restarts never touch it). Verified by presence of the cron line and a
|
|
fresh snapshot.
|
|
|
|
### OPS-16 — Repeated API errors alert staff (coverage: test)
|
|
|
|
The responder counts consecutive OpenAI request failures; at
|
|
`api-error-alert-threshold` (default 5) in a row it fires one staff
|
|
alert (rate-limited like all staff alerts) so a silently-broken bot
|
|
(cf. the gpt-5.6 tools/reasoning incident) surfaces within minutes
|
|
instead of hours. A success resets the counter.
|