# WYG Operational Reliability Runbook — Pilot

**Owner:** [NAME / ROLE]
**Backup owner:** [NAME / ROLE]
**Updated:** August 23, 2026
**Review cadence:** Monthly during a pilot and after every material incident

This runbook is the operating baseline for a school pilot. It distinguishes controls present in the application from provider controls and launch actions that require named people or account configuration.

## Control register

| Area | Implemented control | Evidence | Activation gate |
|---|---|---|---|
| Database backup | Cloudflare D1 Time Travel supplies automatic point-in-time restore history; a daily scheduled check is included | `daily-backup-check.yml`; Backup and Recovery Procedure | Confirm the production D1 backend, plan recovery window, Cloudflare credentials, and daily alert recipient |
| Recovery | Isolated export/restore drill applies every migration and verifies a synthetic integrity marker | `pnpm ops:recovery-test` output | Run before student launch, quarterly, and after schema/backup changes; retain the dated result |
| Uptime | Public `/api/health` probes the application and database without exposing student or infrastructure details | HTTP 200 JSON response; non-200 alert | Configure an independent HTTPS monitor, two alert recipients, and a five-minute interval |
| Error monitoring | Cloudflare Workers Logs are explicitly enabled; the edge logs structured request IDs for 5xx responses and uncaught exceptions | Workers Observability; `x-request-id` response header | Confirm retention, 5xx alert, access list, and weekly review owner |
| Audit logs | Program actions, access events, play/test changes, privacy requests, and grading actions are immutable rows visible only to coaches | Workspace **Audit trail** and CSV export | Approve retention window and review cadence |
| Abuse prevention | D1-backed rate buckets protect login, enrollment, recovery, workspace, file, and export routes; account lockout remains active | HTTP 429 plus hashed operational event | Load-test expected team traffic; document any limit adjustment |
| Production access | Operations endpoint is unavailable until a 32+ character secret exists; application access is role- and program-scoped | `/api/ops/readiness`; access-denial events | Add individual maintainers, MFA, secret rotation, quarterly review, and emergency-access owner |
| Dependencies | Exact versions are locked; scheduled quality/security workflow and Dependabot configuration are included | CI run, audit result, update pull requests | Mirror the source to the approved GitHub repository and enable dependency graph, alerts, and security updates |
| Incident response | Written severity, containment, school notice, recovery, and learning procedure | Incident Response Procedure and school Breach Notification Procedure | Name incident commander, privacy lead, technical lead, counsel, insurer, and school contacts; run a tabletop |

## Daily operating checks

1. Confirm the uptime monitor is green and the most recent database restore-point check passed.
2. Review new 5xx errors, uncaught exceptions, health failures, rate-limit spikes, and denied operations access.
3. Triage any student-impacting or security-impacting event immediately under the incident procedure.
4. Confirm no emergency production access remains active.

## Weekly checks

1. Review unresolved dependency alerts and security-update pull requests.
2. Sample program audit events for unexpected privilege, roster, deletion, export, or scoring changes.
3. Review capacity, response time, storage, and repeated rate-limit signals.
4. Confirm the primary and backup operational owners can receive alerts.

## Monthly and release checks

1. Review production access and remove stale maintainers.
2. Verify secrets are stored only in the platform secret manager and rotate any exposed or shared value.
3. Review this runbook, the subprocessor register, and incident contacts.
4. Before each release, require clean type, lint, product, build, recovery, and dependency checks; retain release evidence.

## Pilot service objectives

- Health monitor: every five minutes.
- Availability target: 99.5% during the defined pilot measurement window, excluding agreed maintenance.
- Critical alert acknowledgment: 15 minutes during named coverage hours.
- School-facing critical incident acknowledgment: according to the signed agreement and incident procedure.
- Recovery objectives proposed for counsel and district review: RPO no greater than 24 hours and RTO no greater than four hours. Provider Time Travel may support a smaller RPO, but WYG does not promise it until the production account and a recovery drill are verified.

## Evidence folder

Retain dated health-alert tests, backup checks, recovery-drill output, access reviews, dependency reports, release approvals, incident/tabletop records, and corrective actions in a restricted company-controlled location. Do not put student records in CI logs or public issue trackers.