# Testing

Hayva.ai uses automated checks to keep security-sensitive behavior explicit and fail closed.
The current suite covers the Phase 1 security foundation plus the first Phase 2 typed AI-planning
and durable-execution boundary. It uses provider-compatible fakes and never requires a real model
credential.

## Prerequisites

- Python 3.12
- Node.js 24 and pnpm 11
- A clean virtual environment is recommended

## Install development dependencies

From the repository root:

```bash
python -m pip install --upgrade pip
python -m pip install -e ".[dev,computer]"
python -m playwright install chromium
```

The editable install provides the application dependencies plus pytest, Ruff, and HTTPX.
The `computer` extra and Chromium installation are required for the real persistent-browser runtime
tests. They do not use a remote browser or production profile.

Install the locked frontend dependencies from `web/` with `pnpm install --frozen-lockfile`.

## Required checks

Run the same checks as CI from the repository root:

```bash
ruff check .
pytest
cd web
pnpm typecheck
pnpm test
pnpm build
```

A change is not ready to merge unless all commands pass. Do not weaken or skip a failing check
to obtain a green build.

## Current coverage

The current suite verifies these core invariants:

- missing permissions deny an action;
- draft-only agents cannot execute writes;
- high-risk actions always require approval; and
- controlled autonomy may perform an explicitly permitted medium-risk action.
- approvals cannot cross workspaces, be decided twice, outlive expiry, bypass the approval
  permission, or authorize a modified payload;
- passwords use one-way Argon2 hashes and reject short values;
- owned tables carry workspace scope; and
- liveness is minimal and propagates request IDs.
- owner setup is one-time, persists workspace roles/permissions, and sets a protected session;
- login failures use a generic response; and
- logout requires matching CSRF and revokes the session.
- permission resolution cannot cross workspace membership boundaries;
- logout-all revokes the user's active sessions and session responses exclude hashes;
- dependency health reports configured/unavailable states without exception details; and
- Compose does not publish PostgreSQL/Redis and orders migration before API startup.
- owner setup rejects callers without the bootstrap capability;
- deactivated users and suspended memberships invalidate existing sessions;
- malformed cross-workspace RBAC mappings fail relational integrity;
- oversized request IDs are replaced and production cookies carry `Secure`;
- login rate limits are enforced without storing raw account/source keys;
- attacker-generated account failures cannot block a subsequent correct password;
- audit verification detects content changes and forks; and
- Alembic upgrade, drift check, downgrade, append-only audit/login triggers, seeded installation
  state, and re-upgrade execute in tests;
- infrastructure checks keep Redis credentials out of process arguments and reject forwarded-IP
  spoofing at the gateway; and
- CI includes a live PostgreSQL probe for the one-time seed and append-only trigger behavior.
- the React shell routes setup/login from server state, keeps safe API errors, and builds with a
  locked dependency graph; and
- desktop and mobile setup/dashboard states were smoke-tested in a real browser.
- the AI service reports provider configuration honestly and rejects missing/invalid service tokens;
- proposed plans are schema-validated, tool-allowlisted, and rejected when dependencies contain a cycle;
- the OpenAI adapter requests structured output with provider storage disabled and does not send raw
  actor/workspace identifiers in the provider prompt;
- Core persists validated plans/steps and token use, reauthorizes trusted tools, records audit events,
  and supports CSRF-protected cancellation;
- an unconfigured provider leaves a durable `blocked` execution rather than simulated success; and
- execution list/detail queries and foreign keys reject cross-workspace access.
- execution creation/run require CSRF, runtime permission revocation is enforced after planning,
  verified read results are durable, and completed read-run replay is idempotent;
- migration data updates add runtime permissions to existing system roles while fresh setup safely
  reuses the seeded global permission catalogue; and
- the AI and Core copies of security-critical tool metadata are checked for contract drift.
- owner setup and migration upgrades create/backfill the default Executive Assistant safely;
- agent creation, configuration versions, schedules, lifecycle, duplication, and current detail are
  workspace scoped and CSRF protected;
- unknown/escalated grants, hierarchy cycles, and cross-workspace agent reads fail closed; and
- delegation scope is the actor/source/target permission intersection and produces audit evidence.
- model-profile reads preserve honest provider state; mutations require authorization and CSRF;
  disabled, missing, referenced, and cross-workspace profiles fail closed; and
- the Agent Studio browser pass covers model creation, least-privilege agent creation, routing,
  lifecycle versioning, duplication, the provider-disabled command surface, and a 390px breakpoint
  without horizontal overflow or console errors.
- asynchronous owner setup retains its form reference and transitions after a successful response;
  this regression was found in the real-browser pass and is covered by Vitest.
- computer profiles, sessions, actions, approvals, control leases, checkpoints, artifacts, and
  emergency-stop records are created by reversible migrations with workspace ownership constraints;
- Core computer APIs reject cross-workspace access, stale/missing leases, changed approval payloads,
  idempotency conflicts, unauthorized human input, unavailable workers, and artifact integrity
  failures;
- browser action replay returns the original Core response without executing the worker twice;
- human takeover fences AI, return-to-AI snapshots and re-observes with distinct request IDs, and
  emergency resume never restarts stopped sessions;
- URL guards reject file URLs, embedded credentials, local/metadata destinations, and simulated DNS
  rebinding;
- real headless Chromium tests exercise DOM observe/navigate/type/click/extract, screenshot metadata,
  profile persistence across a runtime restart, stale-fence rejection, takeover, and human keyboard;
- the premium Computer control room was exercised in the real browser at desktop and 390px widths,
  including a live session, navigation, observation, takeover, return-to-AI, and responsive layout.
- dedicated-computer migrations create first-class templates, VM ownership, network policy, limits,
  operations, snapshots, and metrics with reversible schema and model-drift checks;
- missing compute credentials remain honestly `unconfigured`, block activation, and never simulate
  a successful VM;
- provider-compatible fakes verify VM create/start/stop/snapshot flows, durable idempotent replay,
  exact one-to-one Windows VM ownership, license confirmation, quotas, and host-policy rejection.

## Verification backlog

As Phase 1 components land, add tests for:

- standardized API error responses beyond liveness;
- production-like authentication abuse, expiry, reset delivery, and MFA recovery drills;
- every role/permission combination and cross-workspace access denial;
- least-privilege database roles plus PostgreSQL and Redis integration beyond the current CI probe;
- Docker Compose configuration and service health checks;
- expanded frontend accessibility and interaction coverage; and
- automated browser E2E for the complete Computer UI and failure/race scenarios (the current UI pass
  is manual), plus logout and later module routes as they become executable.
- production-like KVM/libvirt or Proxmox provisioning, reboot persistence, provider timeouts,
  ambiguous-outcome reconciliation, snapshot restore, resize, quota races, encrypted deletion,
  guest/guest denial, and guest/host escape resistance;
- authenticated in-guest terminal, application, file, screenshot, browser, desktop-streaming, and
  daemon restart tests for every supported Linux and licensed Windows template.

Tests must not use real provider credentials or claim an unconfigured integration succeeded.
Provider-compatible fakes are permitted only inside automated tests.

## CI

GitHub Actions runs on pushes and pull requests. It checks Python 3.12 with Ruff/pytest and pinned
Chromium, exercises migrations on PostgreSQL, and checks Node 24 with TypeScript, Vitest, and a
production build. A separate security workflow runs Python/web dependency audits, Gitleaks, Trivy
vulnerability/secret/IaC scanning, CodeQL, and an SPDX SBOM on changes and weekly. A local pass is
useful but does not replace successful remote CI for the exact release commit.