diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md new file mode 100644 index 0000000..f52dcae --- /dev/null +++ b/docs/implementation-plan.md @@ -0,0 +1,268 @@ +# Implementation plan + +This plan turns the roadmap into independently testable increments. A stage is +complete only after formatting, linting, strict type checking, relevant tests, +documentation, and a focused commit succeed. Unsupported behavior remains +explicit throughout development. + +## Quality gate used by every code stage + +1. Run `make format`. +2. Run `make lint`. +3. Run `make typecheck`. +4. Run the narrow test target while developing, then `make test`. +5. Run `make ci` before committing a stage. +6. Update user-facing and architectural documentation. +7. Commit only the coherent stage changes. + +Tests that need PostgreSQL use a dedicated database and are marked +`integration`. Network-dependent research and differential tests are never part +of the default offline unit suite. + +## Milestone A: runnable foundation + +### A1 — Repository and documentation + +- Add architecture, contribution conventions, license, honest README, and this + implementation plan. +- Define package layout without placeholder functions. +- Verify Markdown links, Mermaid syntax by inspection, and `git diff --check`. + +Exit: design boundaries and the staged delivery policy are documented. + +### A2 — Python project and test toolchain + +- Add Python 3.13 metadata and bounded runtime/development dependency ranges. +- Configure Ruff formatting/linting, strict mypy, pytest-asyncio, coverage, and + test markers. +- Add the complete Makefile command surface; commands for future features must + fail with a clear message until implemented rather than silently succeed. +- Add package skeleton only where immediately used. + +Exit: dependency installation is reproducible and a minimal unit test passes all +local static checks. + +### A3 — Application foundation + +- Implement typed Pydantic settings and an application factory. +- Add lifespan-owned resources and explicit dependency injection. +- Implement JSON logs, request-ID middleware, safe error boundary, `/health/live`, + and `/health/ready`. +- Implement an asyncpg pool adapter with startup timeout, query timeout, readiness + probe, and graceful close. +- Unit-test configuration and middleware; integration-test readiness against + PostgreSQL. + +Exit: the app starts, readiness reflects database state, shutdown closes all +resources, and standard errors do not leak internal details. + +### A4 — Local containers + +- Add multi-stage Dockerfile with locked-down non-root runtime and healthcheck. +- Add Docker Compose simulator/PostgreSQL services, named healthchecks, and + persistent development volume. +- Add `.env.example`, `.dockerignore`, and documented HTTP quick start. +- Test image build, Compose startup, migrations, and both health endpoints. + +Exit: `make ci` and the Stage 0 Docker acceptance path both pass. + +## Milestone B: authoritative API contracts + +### B1 — API Viewer research + +- Inspect official HTML and its loaded static resources. +- Identify the real machine-readable artifact and version source without assuming + an address. +- Record retrieval date, format, limitations, format-change risks, and offline + fallback in `docs/api-viewer-research.md`. +- Store a small unmodified raw sample plus provenance and checksum. + +Exit: the source is documented and the fixture can be parsed without network. + +### B2 — Source parser + +- Define an importer protocol and adapters for the discovered artifact and local + files. +- Parse the raw fixture while retaining unknown fields and emitting structured + warnings for recoverable source variations. +- Add malformed, truncated, and unexpected-field tests. + +Exit: one real saved sample parses deterministically and offline. + +### B3 — Normalized contract model + +- Implement immutable Pydantic models for snapshots, paths, methods, parameters, + schemas, permissions, formats, constraints, versions, and manifests. +- Implement canonical JSON serialization and SHA-256 for raw artifact, + normalized snapshot, and each method. +- Preserve source metadata in `extra`; validate counts and references. +- Add determinism, round-trip, unknown-field, and property-based tests. + +Exit: repeated normalization produces byte-identical canonical JSON and checksums. + +### B4 — Secure asynchronous import CLI + +- Add local and remote import commands using async `httpx`. +- Enforce HTTPS, official-domain allowlist, public address resolution, redirect and + size limits, timeouts, and bounded retries. +- Store immutable raw revisions, normalized snapshots, and manifests atomically. +- Add `import`, `validate`, `list`, and `show`; optionally persist to PostgreSQL. +- Test SSRF defenses and idempotency without internet. + +Exit: local and controlled remote imports produce verified, revision-safe assets. + +### B5 — Semantic diff + +- Compare paths, methods, parameters, schemas, permissions, defaults, constraints, + and documentation independently. +- Classify changes and render stable text, JSON, Markdown, and HTML reports. +- Add CI exit policy for breaking changes and golden/property tests. + +Exit: two snapshots produce deterministic, actionable reports in every format. + +## Milestone C: contract-driven HTTP surface + +### C1 — Dynamic route registry + +- Load and verify the selected snapshot and compatibility profile at startup. +- Register `/api2/json` and `/api2/extjs` methods dynamically with collision + detection and contract-derived schemas. +- Dispatch through a typed semantic-handler registry. +- Implement explicit `error`, `schema-default`, and `fixture` fallback modes; + reserve guarded proxy/record modes for a later milestone. + +Exit: OpenAPI builds without collisions and a fixture contract drives routes, +parameters, and methods without generated per-endpoint files. + +### C2 — Compatible input and output + +- Parse path, query, form, and supported JSON inputs according to contract rules. +- Render the `/api2/json` data envelope without coercing scalar or null data. +- Map validation, routing, authorization, domain, and database exceptions through + centralized versioned Proxmox error templates. +- Add golden tests for the required validation and routing cases. + +Exit: native FastAPI validation bodies never escape and unsupported semantics are +unambiguous. + +### C3 — Compatibility accounting + +- Track declared, schema-only, implemented, observed, and verified methods. +- Score each required compatibility level separately and by API group/version. +- Expose initial JSON/Markdown reports and simulator admin read endpoint. + +Exit: claims in documentation are generated from test evidence. + +## Milestone D: persistent simulation core + +### D1 — Migrations and database primitives + +- Implement an asynchronous, checksummed migration runner without an ORM. +- Create contract, cluster/resource, identity/ACL, task, scenario, and audit + tables with constraints and indexes. +- Implement typed pool, transaction/savepoint context, error mapping, query + timeout, affected-row checks, and safe transient retry policy. +- Test clean migration, repeat invocation, rollback, and constraint behavior. + +Exit: a clean PostgreSQL database reaches the expected schema deterministically. + +### D2 — Deterministic seed + +- Implement idempotent `small` profile first using a seeded data generator and + batch operations. +- Add logical-state snapshot assertions independent of generated UUID/timestamps. +- Add medium, large, HA, and failure profiles only after the vertical slice. + +Exit: identical seeds yield identical logical small clusters. + +### D3 — Authentication and authorization + +- Implement password hashing, signed expiring tickets, cookies, CSRF tokens, and + API-token hashing/parsing with log redaction. +- Implement realms, principals, roles, privileges, ACL propagation, and token + privilege separation. +- Map contract permissions to centralized capability-driven evaluation. + +Exit: authentication/CSRF/permission matrices pass without storing plaintext +secrets. + +### D4 — Durable task engine + +- Implement and property-test UPID formatting and parsing. +- Atomically create tasks and resource locks; claim with `SKIP LOCKED` leases. +- Implement progress, append-only logs/events, cancellation, recovery, bounded + lifespan workers, and idempotency rules. +- Test two-worker exclusion, process restart recovery, lease expiry, and shutdown. + +Exit: no acknowledged task is lost or executed concurrently by two workers. + +### D5 — State machines and clocks + +- Implement explicit VM states/transitions and PostgreSQL resource locks. +- Inject real, accelerated, and manual clocks into simulation services; reserve + monotonic real time for worker leases and document the distinction. +- Add deterministic fault/scenario evaluation and concurrency tests. + +Exit: valid asynchronous transitions finish predictably and incompatible +operations conflict without corrupting state. + +## Milestone E: release 0.1.0 vertical slice + +### E1 — Read endpoints and login + +- Implement `/version`, `/access/ticket`, `/nodes`, `/nodes/{node}/status`, and + `/cluster/resources` through semantic services. +- Verify headers, cookies, envelope, required fields, permissions, and errors + against the selected contract/observations. + +Exit: authenticated HTTPX and curl smoke flows pass from a seeded database. + +### E2 — Basic QEMU and tasks + +- Implement QEMU list, config, current status, start, and stop. +- Return persistent UPIDs for mutations and expose task list/status/log. +- Drive VM intermediate/final states through worker-executed transitions. +- Test duplicate and conflicting operations plus restart recovery. + +Exit: a stopped seeded VM becomes running only after its successful task. + +### E3 — Client and packaging acceptance + +- Add proxmoxer smoke test and document supported client settings. +- Run the complete container acceptance sequence from a clean volume. +- Generate the first evidence-based compatibility report and limitation matrix. +- Tag the supported surface as release `0.1.0` after all gates pass. + +Exit: every command and request in the first-result Definition of Done succeeds. + +## Subsequent releases + +- **0.2.0:** QEMU create/update/delete, tokens/complete ACLs, snapshots, clone, and + migration. +- **0.3.0:** LXC, storage metadata/content operations, pools, and backups. +- **0.4.0:** administrative scenarios, fault injection, virtual-time controls, and + large-cluster profiles. +- **0.5.0:** sanitized recorder, differential lab tests, HTML reports, and multiple + verified PVE profiles. +- **1.0.0:** stable compatibility contract, published matrix, Kubernetes/Helm, + client certification, migration policy, and security review. + +Each later endpoint is delivered as a narrow vertical increment: imported +contract, handler, domain semantics, persistence, permissions, task behavior when +applicable, golden/compatibility evidence, and documentation are completed +together. + +## Known delivery risks and mitigations + +- **API Viewer format changes:** keep raw immutable artifacts, multiple adapters, + parser warnings, and fixture-based offline tests. +- **Documentation differs from reality:** store declared and observed contracts + separately and bind observations to exact versions. +- **Error text varies by release:** assert structural compatibility first and use + exact golden text only when observed. +- **PostgreSQL concurrency complexity:** establish leases, locks, and transaction + tests before exposing mutation endpoints. +- **False compatibility confidence:** default unsupported routes to errors and + generate claims only from recorded tests. +- **Large scope:** never advance a partially passing stage; prefer a complete + vertical slice over breadth.