# OneagencyOS Operations and Recovery

## Monitoring

OneagencyOS runs a Netlify scheduled monitor every five minutes.

- `GET /api/health` verifies the production function runtime and Supabase connectivity.
- The scheduled monitor checks the public website and health endpoint.
- High and critical incidents are emailed to `OPERATIONS_ALERT_EMAIL`.
- Repeated incidents are grouped by fingerprint and counted instead of flooding the inbox.
- Failed email deliveries and failed webhooks are written to the centralized tracker.
- Browser exceptions from authenticated sessions are recorded without request bodies, credentials, or tokens.
- AI usage is calculated over the prior 24 hours. Alerts use `AI_DAILY_COST_ALERT_USD` and `AI_DAILY_TOKEN_ALERT`.
- An optional external dead-man monitor should call `GET /api/health` and use `UPTIME_HEARTBEAT_URL`. The external monitor is necessary because a monitor running inside Netlify cannot report a total Netlify outage.

The employee-only operations console is at `/netlify-health.html`.

## Required Netlify Variables

| Variable | Purpose | Recommended value |
| --- | --- | --- |
| `OPERATIONS_ALERT_EMAIL` | Operational alerts | `darrell@oneagencyos.com` |
| `AI_DAILY_COST_ALERT_USD` | 24-hour estimated AI cost threshold | `25` |
| `AI_DAILY_TOKEN_ALERT` | 24-hour token threshold | `2000000` |
| `AI_INPUT_COST_PER_1M` | Current model input cost per 1M tokens | Set from the provider price sheet |
| `AI_OUTPUT_COST_PER_1M` | Current model output cost per 1M tokens | Set from the provider price sheet |
| `UPTIME_HEARTBEAT_URL` | Optional external monitor heartbeat | Secret URL supplied by the uptime service |

Review model pricing whenever `OPENAI_MODEL` changes.

## Recovery Targets

- Target RPO: 24 hours with daily backups; lower only when point-in-time recovery is enabled.
- Target RTO: 4 hours for database restore and application verification.
- Quarterly: run the synthetic logical restore drill in `scripts/supabase-backup-restore-drill.sql`.
- Annually and before major schema releases: perform a full managed-backup restore into a temporary isolated Supabase project.

## Restore Drill Completed by This Repository

The synthetic drill:

1. Creates three isolated fixture records.
2. Serializes the records as a logical JSON backup.
3. Calculates a deterministic checksum and row count.
4. Deletes the fixture records.
5. Restores them from the captured backup.
6. Recalculates and compares the checksum and count.
7. Stores durable evidence in `backup_restore_drills`.
8. Removes the synthetic fixture records.

This verifies the restore logic and integrity checks without modifying agency data. It does not replace the annual full managed-backup restoration test.

## Full Supabase Managed-Backup Restore Runbook

1. Announce a maintenance window and stop writes from Netlify.
2. Record the backup timestamp, expected RPO, table counts, and Storage object inventory.
3. Restore into a separate temporary Supabase project whenever possible. Do not test first against production.
4. If using CLI logical backups, dump roles, schema, and data separately with the Supabase CLI, then restore them with `psql` in a single transaction.
5. Restore Storage objects separately. Database backups contain Storage metadata, not the actual objects.
6. Reapply passwords for custom database roles; backup contents do not preserve those passwords.
7. Verify:
   - authentication for an employee and one agency user;
   - agency isolation and RLS;
   - row counts for agencies, members, clients, policies, documents, email connections, and signature packets;
   - random record checksums;
   - signed document and client portal downloads;
   - email, signature, Stripe, AI, and webhook functions in test mode.
8. Point a preview deployment at the restored project and run `npm run verify`.
9. Record start/end times, restored backup identifier, checks, failures, and approver.
10. Destroy the temporary project only after evidence has been exported and retained.

## Incident Response

1. Acknowledge the alert and identify the first failing check.
2. Preserve the operational event, Netlify function log, provider event ID, and affected agency ID.
3. Contain the issue by disabling the failing worker or provider integration if data integrity is at risk.
4. Recover from the last verified backup or replay idempotent webhook/email jobs.
5. Confirm tenant isolation, record counts, and customer-facing behavior.
6. Resolve the event in the operations console and document the cause and corrective action.