# Recovery and Business Continuity

## Status and principles

The browser runtime implements explicit profile/session recovery, durable checkpoints, control
leases with fencing, pause/takeover, workspace emergency stop, unknown-write reconciliation, and a
production maintenance sweep for expired leases/approvals/artifacts/idempotency. Unit/integration
tests cover this behavior. Encrypted backup and isolated restore tooling exists, but no target-host
container restart, backup/restore, or production-like failure campaign is proven.

Recovery prioritizes human safety and prevention of duplicate external actions
over automatic completion. An action with an uncertain result is not retried until
its external state is reconciled.

## Recovery objectives

Recovery time and recovery point objectives must be selected from measured system
behavior and recorded before production. The database is the source of truth for
workflows; queues are delivery mechanisms. Durable assets include the database,
encrypted browser profiles, uploads, audit records, and critical configuration.

## Incident sequence

1. Stop or fence unsafe execution using application emergency stop.
2. Preserve logs, execution IDs, timestamps, and affected resource identifiers.
3. Determine scope and whether credentials or personal data may be exposed.
4. Revoke sessions, tokens, or keys when compromise is plausible.
5. Restore the minimum required component from a known-good source.
6. Reconcile all in-flight external actions before resuming workers.
7. Verify health, authorization, audit continuity, and integration state.
8. Resume gradually, monitor, notify affected owners, and document the incident.

## Workflow and queue recovery

Each job needs a stable idempotency key, execution checkpoint, attempt history,
lease, and fencing token where shared state is controlled. Use a transactional
outbox for committed side effects. On worker or Redis failure, expired work may be
redelivered only after checking the authoritative execution record.

States must distinguish `planned`, `awaiting_approval`, `ready`, `running`,
`succeeded`, `failed`, `cancelled`, and `outcome_uncertain`. `outcome_uncertain`
requires reconciliation or human review; it must never be treated as success.

## Browser failure and takeover

- Current operator path: use Computer **Pause** when the worker is reachable; use **Recover** only
  for a paused/failed session. Recovery reopens the same randomized persistent profile, issues a new
  AI fence, and records an observation checkpoint before returning control.
- Current safety path: use workspace **Emergency stop** to persist the stop state, fence controllers,
  and attempt to stop every active worker. Treat sessions listed as unverified as an incident; verify
  worker/container termination independently. **Resume workspace** only clears the global block and
  never restarts a session.
- Fence the failed controller before starting a replacement.
- Preserve task goal, URL, completed steps, last verified state, screenshots,
  attempt history, and external idempotency data.
- Reopen the isolated persistent profile and observe current state before acting.
- For a send, submit, delete, upload, or purchase-like action, verify external state
  rather than repeating the click.
- Human takeover immediately invalidates the AI controller lease. AI resumes only
  after explicit return, a fresh observation, and a new lease.
- If a profile may be stolen, disconnect integrations, revoke provider sessions,
  rotate associated credentials, and rebuild the profile.

The worker writes a small runtime manifest beside each persistent profile so a replacement process
can validate session/profile identity and the latest fence. Browser actions are synchronous today;
there is no durable Redis browser-job queue. If a dispatched write loses its response, Core marks it
`unknown`, pauses and releases control, and requires an authorized human evidence summary through
the reconciliation endpoint. Downloads/uploads are disabled, so file-transfer recovery is not yet
applicable.

## Provider and network outages

Use bounded exponential backoff with jitter and circuit breakers. Do not let an
optional provider make core application health fail. Queue only operations whose
authorization and expiry allow delayed execution. Expired approvals and stale
calendar/message preconditions require new evaluation. Surface degraded status and
never claim an unverified action succeeded.

## Database and Redis

For database failure, stop mutating workers, restore to an isolated environment,
validate migrations and integrity, then reconcile outbox and external action state
before promotion. Redis loss must not erase authoritative workflow state. Rebuild
derived cache from the database; restore queue persistence only after duplicate
delivery controls are confirmed.

## Backups

Back up the database, browser profiles, uploaded files, audit data, and critical
configuration. Backups are encrypted, versioned, access-controlled, monitored,
and stored separately from the VPS. The encryption key is not stored only beside
the backup. Retention and deletion policies apply to backup copies.

`scripts/backup_production.sh` creates a PostgreSQL custom dump plus browser-profile and artifact
archives, packages them with a versioned manifest, encrypts the payload to a separately controlled
Age recipient, and writes a SHA-256 checksum. `scripts/restore_drill.sh` verifies the checksum,
decrypts, validates both file archives, restores PostgreSQL into a temporary networkless container,
and runs basic schema/data queries. The scripts have not been executed on this Docker-less host.

A complete restoration drill must additionally prove:

- backup authenticity and decryption;
- restoration into an isolated network;
- database integrity and migrations;
- workspace and authorization isolation;
- browser profile readability without exposing it;
- audit sequence continuity;
- application health and owner login;
- reconciliation without duplicate external actions.

Record drill date, backup identifier, duration, data loss window, operator,
failures, and corrective work. A backup that has not been restored is unverified.

## Compromise recovery

Preserve forensic evidence and avoid destroying affected systems prematurely.
Rotate application, database, signing, webhook, OAuth, backup, and encryption
credentials according to scope. Invalidate sessions and API keys. If an encryption
key is compromised, re-encrypt affected material under a new version. Review audit
events for unauthorized reads and actions and notify affected parties as required.

## Disaster recovery verification

Before production, exercise server reboot, container restart, Redis restart,
database restoration, browser crash during a send, network interruption, provider
outage, disk-full behavior, key rotation, emergency stop, and human takeover race.
Link evidence and unresolved findings from `PRODUCTION_CHECKLIST.md`.