feat(resilience): operational hardening (NEXT phase of the audit)
Deploy to VPS / deploy (push) Has been cancelled
Deploy to VPS / deploy (push) Has been cancelled
Acts on the audit's NEXT block — operational resilience. Backups (N1): - New `backup` compose service (postgres:16-alpine) runs scripts/backup-loop.sh: immediate pg_dump on start, then nightly, gzip, 14-day rotation into ./backups on the host. Configurable via BACKUP_RETENTION_DAYS / BACKUP_INTERVAL_SECONDS. (Offsite copy is the documented next step.) Resource limits + healthchecks (N2): - deploy.resources.limits.memory on postgres (2g), app (1500m), nginx (256m), backup (256m) so no container can starve the others (the Nginx outage was a reminder). - Nginx now has a healthcheck hitting a new self-served `/nginx-health` endpoint on the default_server (no upstream dependency). Chat resilience (N3): - buildSystemPrompt() wraps its 4 Prisma queries in try/catch with safe defaults — if Postgres is down the assistant degrades instead of 500-ing. - Result is cached for 60s (only on healthy builds) so we don't run 4 queries per message; CMS edits still appear within the TTL. - POST fails fast with 503 if OPENAI_API_KEY is missing (instead of breaking mid-stream after headers are sent). - streamText gets an onError handler that logs + persists an `error` AiEvent. Idempotent submissions (N4): - consultation/route.ts and operations.ts now wrap the email-tracking UPDATE in try/catch — the lead/signal is already saved, so a telemetry hiccup can't 500 the request and trigger a duplicate retry. operations.ts also returns emailError. Performance (N5): - Index GlobalNode(application, isActive) — backs the case-study join on every application page. Migration 20260609130000_index_globalnode_application. Verified: next build compiles (Docker parity, SESSION_SECRET unset), TypeScript clean, prisma schema valid, golden tests 17/17, `docker compose config` valid. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -17,6 +17,12 @@ services:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
networks:
|
||||
- flux-net
|
||||
# Resource caps so no single container can starve the others (the Nginx
|
||||
# outage earlier was a reminder). VPS has ~11 GB; these leave headroom.
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 2g
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U ${DB_USER} -d ${DB_NAME}"]
|
||||
interval: 5s
|
||||
@@ -81,6 +87,10 @@ services:
|
||||
- flux-net
|
||||
expose:
|
||||
- "3000"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 1500m
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
@@ -114,6 +124,46 @@ services:
|
||||
- app
|
||||
networks:
|
||||
- flux-net
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 256m
|
||||
healthcheck:
|
||||
# Nginx self-health (served directly by the default_server, no upstream).
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1/nginx-health || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
# ── Automated Postgres backups ──
|
||||
# Nightly pg_dump -> gzip into ./backups on the host, 14-day rotation.
|
||||
# NOTE: this is LOCAL to the VPS. Offsite copy (S3/rsync) is the recommended
|
||||
# next step once the client provides storage credentials.
|
||||
backup:
|
||||
image: postgres:16-alpine
|
||||
restart: always
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DB_USER: ${DB_USER}
|
||||
DB_PASSWORD: ${DB_PASSWORD}
|
||||
DB_NAME: ${DB_NAME}
|
||||
BACKUP_DIR: /backups
|
||||
RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-14}
|
||||
BACKUP_INTERVAL_SECONDS: ${BACKUP_INTERVAL_SECONDS:-86400}
|
||||
volumes:
|
||||
- ./backups:/backups
|
||||
- ./scripts/db-backup.sh:/usr/local/bin/db-backup.sh:ro
|
||||
- ./scripts/backup-loop.sh:/usr/local/bin/backup-loop.sh:ro
|
||||
entrypoint: ["/bin/sh", "/usr/local/bin/backup-loop.sh"]
|
||||
networks:
|
||||
- flux-net
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 256m
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
|
||||
Reference in New Issue
Block a user