DevOpsTools/proxmox-api-simulator
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add a stateful Proxmox API console and broad handler coverage beyond the

Browse Source 
initial QEMU slice, backed by imported contracts for majors 6–9.
- Implement durable handlers for access/auth, cluster, LXC, storage, HA,
  firewall, Ceph, SDN, ACME, notifications, pools, mapping, and node ops
- Serve an interactive Web UI with catalog browsing, demo seed controls,
  and OpenAPI/help surfaces
- Bundle PVE 6.4-15, 7.4-16, and 8.4.5 contract revisions alongside 9.2.3
- Support in-memory runtime contract Apply (POST /ui/api/contract/apply)
  so /version and /api2 routes follow the selected major until restart
- Expand seed profiles (including demo-cluster), migrations 007–008, TLS
  gateway config, Compose/Makefile tooling, and compatibility evidence
- Tighten .gitignore for macOS, hidden directories (.*/), and local secrets
This commit is contained in:
Sergey Antropoff
2026-07-16 01:08:01 +03:00
parent 003ee5d634
commit 777926487b
189 changed files with 241501 additions and 944 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api-surface.md
+83
View File
@@ -0,0 +1,83 @@
# API surface
## Request path
1. Middleware assigns or forwards a request ID.
2. The active contract snapshot selects declared paths and schemas.
3. Authentication resolves a principal (ticket or API token).
4. ACL / privilege checks run before revealing or mutating resources.
5. Path, query, and body inputs are validated against contract-derived schemas.
6. A semantic handler executes against PostgreSQL-backed state.
7. Long operations create a durable task (+ lock when required) and return a UPID.
8. Responses use the Proxmox envelope under `/api2/json` or `/api2/extjs`.
## Dual renderers
Every contract method is registered under both:
- `/api2/json/...`
- `/api2/extjs/...`
Clients and the Web UI typically use the JSON renderer.
## Handlers vs contracts
- **Declared** — present in the imported API Viewer snapshot for the major.
- **Implemented** — a semantic handler is registered for that verb + path.
- Majors **6–9** have **100%** implemented coverage for declared methods.
Handlers must persist create/update/delete effects. Empty no-op mutations are
not part of the product contract. See the workspace durable-simulator rule.
## OpenAPI and exploration
- Interactive FastAPI docs: `/docs`
- Web UI method inspector: `/` → catalog → method
- UI APIs: `/ui/api/catalog`, `/ui/api/method`, `/ui/api/compatibility`
## Compatibility endpoints
| Path | Format |
|---|---|
| `/admin/compatibility` | JSON |
| `/admin/compatibility.md` | Markdown |
| `/admin/compatibility.html` | HTML |
Reports follow the active runtime contract after hot-swap.
## Tasks (UPID)
Async work (guest power, clone, migrate, many deletes, backups, …) returns a
UPID. Poll:
```text
GET /nodes/{node}/tasks/{upid}/status
GET /nodes/{node}/tasks/{upid}/log
```
Workers claim tasks with `FOR UPDATE SKIP LOCKED`, renew leases, and recover
after process restart. HTTP 200 on the mutation request means “accepted”, not
“guest already in final state”.
## Errors (common)
| Status | Typical cause |
|---|---|
| 401 | Missing/invalid ticket or token |
| 403 | ACL denial or missing CSRF on ticket mutation |
| 409 | VMID conflict, illegal state transition, lock contention |
| 501 | Handler missing (should not appear for declared methods on 6–9) |
| 503 | Readiness failure (database / migrations) |
## Importing contracts
```bash
make shell
proxmox-api-contract validate path/to/source.json
proxmox-api-contract --store contracts import --file path/to/source.json --version 9.2.3
proxmox-api-contract --store contracts list
proxmox-api-contract diff old.json new.json --format markdown
```
Remote import enforces HTTPS, an official-host allowlist, size/redirect/timeout
limits, and checksummed immutable revisions.
docs/api-versions.md
+77
View File
@@ -0,0 +1,77 @@
# API versions (PVE 6–9)
The simulator ships authoritative imported contracts for four Proxmox VE majors.
Handler registry coverage is **100% verified** for each:
| Major | Source version | Declared methods | Handler coverage |
|---|---|---:|---:|
| 6 | 6.4-15 | 504 | 100% |
| 7 | 7.4-16 | 540 | 100% |
| 8 | 8.4.5 | 605 | 100% |
| 9 | 9.2.3 | 675 | 100% |
Older majors reuse the current semantic handlers plus path synonyms registered
in `app/handlers/legacy_aliases.py` (for example historical Ceph and backup path
spellings).
## Cold start
Set `CONTRACT_SNAPSHOT` to a normalized snapshot path. Docker Compose pins the
bundled PVE **9.2.3** revision by default.
`GET /api2/json/version` reports fields derived from the **active** snapshot’s
`source_version`.
## Hot-swap (runtime)
Browse any major in the Web UI catalog, then **Apply as runtime**, or call:
```http
POST /ui/api/contract/apply?major=7
```
Effects:
- In-memory `/api2/json` and `/api2/extjs` routes are replaced under an
application lock.
- `/version`, OpenAPI, implementation metadata, and compatibility state refresh
for the new major.
- The change is **process-local** and **not persisted**.
- Restart restores `CONTRACT_SNAPSHOT`.
Catalog browse (`GET /ui/api/catalog?major=N`) does **not** by itself change the
runtime; only apply does.
### Client guidance
- Pin the major explicitly in CI (cold-start env **or** apply + assert
`/version` before the suite).
- Mid-flight hot-swap can invalidate in-progress client assumptions about
schemas and paths — avoid during long Terraform/Ansible runs unless the run
owns the switch.
- After apply, re-check `/admin/compatibility` for the active runtime.
## Fallback modes
`CONTRACT_FALLBACK` controls undeclared-handler behaviour:
| Value | Behaviour |
|---|---|
| `error` (default) | HTTP 501 with an explicit pending-handler style message |
| `schema-default` | Synthesize a return value from the contract schema |
| `fixture` | Return only fixture data embedded in the method contract |
With full handler coverage on the active contract, declared methods should not
hit the fallback. Keep `error` so regressions remain visible.
## Evidence vs registry
**Registry coverage** means every declared method has a registered semantic
handler (no systematic 501 for that contract).
**Verified** in this project’s sense means the majors are exercised through the
compatibility and automated suites for handler presence across 6–9. Multi-
dimension evidence JSON can still expand over time for deeper edge-case claims;
prefer live `/admin/compatibility` when the process is running.
See [Compatibility](compatibility.md).
docs/api-viewer-research.md
-70
View File
@@ -1,70 +0,0 @@
# Proxmox VE API Viewer research
Research was performed on 2026-07-12 against the official documentation hosted
by Proxmox Server Solutions GmbH.
## Discovered source
The HTML application at
[`https://pve.proxmox.com/pve-docs/api-viewer/`](https://pve.proxmox.com/pve-docs/api-viewer/)
loads ExtJS and one application resource, `apidoc.js`. The machine-readable API
tree is not fetched from a separate JSON endpoint: it is embedded at the start
of [`apidoc.js`](https://pve.proxmox.com/pve-docs/api-viewer/apidoc.js) as a
JavaScript declaration named `apiSchema`. The remainder of that file renders the
tree and method documentation.
At retrieval, the artifact was 4,277,440 bytes with SHA-256
`f2b77b57c71f3781a0993cc5062940ef31e0843fd9a6bcfdb4de4dd2001d6d9e`.
The server reported `Last-Modified: Fri, 03 Jul 2026 09:08:20 GMT` and ETag
`"4144c0-655b144140900"`.
The adjacent official documentation index identifies the generated
documentation as Proxmox VE `9.2.3`, dated `Fri Jul 3 11:08:20 CEST 2026`. Its
timestamp matches the artifact's HTTP last-modified time after timezone
conversion. This is strong evidence that the current unversioned viewer belongs
to that documentation build, but the artifact does not contain a dedicated
top-level snapshot-version field. Importers must therefore record the index
version and HTTP metadata as provenance rather than infer a version from an API
method schema.
## Format and limitations
`apiSchema` is a nested tree of path nodes. Nodes may contain `children`, an
`info` mapping keyed by HTTP method, `path`, `text`, and `leaf`. Method objects
contain parameter and return schemas, permissions, descriptions, and flags.
The schema resembles JSON Schema but is a Proxmox-specific dialect and includes
fields such as `typetext`, `format_description`, `instance-types`, and numeric
booleans. Unknown fields must be retained.
The artifact is executable JavaScript, not JSON. A parser must extract only the
declaration value without evaluating the downloaded program. The URL is
unversioned and changes in place. Formatting, declaration syntax, variable name,
tree shape, or bundling may change without notice. Documentation describes the
declared contract; it does not prove runtime behavior or exact error text for a
particular installed cluster.
## Offline fallback and sample
The repository stores an extracted, otherwise semantically unmodified `/version`
node at
[`tests/fixtures/api-viewer/pve-9.2.3-version.json`](../tests/fixtures/api-viewer/pve-9.2.3-version.json).
It is deliberately small enough for deterministic parser tests and retains all
fields from that source node. Its checksum is recorded in the companion
provenance file. Network retrieval is research/import functionality only; the
default test suite must use this checked-in fixture.
The fixture is not a complete snapshot and must never be used to claim broad
Proxmox compatibility. Full imports should preserve the immutable raw
`apidoc.js`, response metadata, retrieval timestamp, and checksum outside the
small test-fixture path.
## Parser boundary
`app.contracts.source.ApiViewerParser` accepts either the saved JSON sample or
the official JavaScript wrapper. It locates the exact `const apiSchema`
assignment, scans the balanced JSON value while respecting escaped strings, and
decodes only that value; no downloaded JavaScript is evaluated. Recoverable
tree variations produce structured warnings and unknown node fields remain in
the parsed dictionaries. `SourceImporter` and `LocalFileImporter` keep artifact
retrieval separate from parsing so later remote imports can enforce their own
network policy.
docs/architecture.md
+33 -29
View File
@@ -6,7 +6,9 @@
primary design goal is measurable contract compatibility: routes, validation,
authentication, permissions, response shapes, state transitions, and persistent
long-running tasks are verified independently instead of being described as
universally compatible.
universally compatible. Bundled majors **6–9** ship with **100%** semantic
handler registration for every declared contract method, with runtime hot-swap
between those majors.
The simulator does not require a live Proxmox installation during normal
operation. Official API artifacts and sanitized observations are imported ahead
@@ -168,40 +170,39 @@ without scattering version checks through services.
- Passwords and API-token secrets are stored only as password hashes.
- Tickets are signed, short-lived, and redacted from telemetry.
- Ticket-authenticated mutations require CSRF validation; API tokens follow the
selected Proxmox compatibility profile.
- Simulator administration uses a separate prefix and credential and can be
disabled completely.
- Recorder mode is opt-in, verifies TLS by default, restricts routes and methods,
and sanitizes secrets and personal identifiers before writing fixtures.
- Containers run as a non-root user and support a read-only root filesystem.
- Ticket-authenticated mutations require CSRF validation; API-token requests do
not require CSRF.
- The interactive Web UI and `/admin/compatibility*` helpers are laboratory
surfaces without a separate admin token in the current build — network
exposure is the trust boundary.
- Containers run as a non-root user in the packaged images.
## Runtime contract hot-swap
Cold start loads `CONTRACT_SNAPSHOT`. Operators can replace the in-memory route
table for majors 6–9 via `POST /ui/api/contract/apply?major=N` (also exposed in
the Web UI). The swap refreshes `/version`, OpenAPI, and compatibility state and
is process-local (restart restores the env snapshot).
## Observability
JSON logs contain request ID, route template, status, duration, safe principal
identity, task type, and sanitized resource identifiers. Metrics avoid VMID,
UPID, and username labels. OpenTelemetry is optional and has a no-op
implementation so tracing is never required for startup.
JSON logs contain request ID, route template, status, duration, and redacted
identity fields. Process Prometheus/OpenTelemetry exporters are not shipped yet;
Proxmox `/cluster/metrics*` handlers simulate PVE metrics-server configuration
only.
## Testing strategy
Unit tests cover deterministic contract processing and domain rules. Integration
tests exercise repositories, transactions, workers, and application lifespan
against PostgreSQL. Contract tests traverse imported endpoints and ensure no
native FastAPI validation response escapes. Compatibility tests compare golden or
live-lab observations after normalizing dynamic values. Concurrency and
property-based tests target task leases, state transitions, serialization, and
parsers.
The first vertical release deliberately supports a small set of endpoints with
complete stateful semantics. All other imported endpoints remain visibly
unsupported until their handlers and compatibility tests exist.
Unit tests cover contract processing and domain rules. Integration tests
exercise repositories, transactions, workers, and lifespan against PostgreSQL.
Contract and compatibility suites target majors **6–9** with **100%** handler
registry coverage. External proxmoxer smoke runs against the Compose TLS
gateway. Concurrency tests target task leases and state transitions.
Database readiness includes the latest packaged migration version, not merely a
successful connectivity query. Workers tolerate the documented container-first
startup sequence by retrying failed claims until migration tables exist.
Normalized resource writes use compare-and-swap version updates through a typed
repository, so stale writers receive a domain conflict.
successful connectivity query. Workers retry failed claims until migration
tables exist. Normalized resource writes use compare-and-swap version updates
through a typed repository, so stale writers receive a domain conflict.
## Deployment model
@@ -220,5 +221,8 @@ in local Docker Compose but is an external dependency in the production chart.
and in-memory queues are not used for critical work.
4. Compatibility is capability-driven and versioned, not implemented through
scattered version string conditions.
5. Unsupported semantics fail honestly by default; schema-derived or proxy
responses require an explicit operator mode.
5. Missing handlers fail honestly via `CONTRACT_FALLBACK` (default `error` →
HTTP 501). Majors 6–9 ship with full handler registration, so declared
methods should not hit that path under normal operation.
6. Laboratory docs and cookbooks live under `docs/` and `examples/`; internal
research/prompt notes are not part of the user guide.
docs/authentication.md
+83
View File
@@ -0,0 +1,83 @@
# Authentication
The simulator implements Proxmox-compatible ticket and API-token authentication
with ACL evaluation for non-root principals.
## Ticket login
```http
POST /api2/json/access/ticket
Content-Type: application/x-www-form-urlencoded
username=root@pam&password=secret
```
Successful responses include:
- `ticket` — also set as HttpOnly cookie `PVEAuthCookie` (SameSite=Strict)
- `CSRFPreventionToken` — required for ticket-authenticated mutations
- `username` and related identity fields
Tickets are HMAC-signed with `TICKET_SIGNING_KEY`, expire after two hours by
default, and tolerate a small amount of future clock skew.
### CSRF rules
| Request | Ticket session | API token |
|---|---|---|
| `GET` / `HEAD` / `OPTIONS` | Cookie (or ticket) enough | `Authorization` header |
| Other methods | Cookie **and** `CSRFPreventionToken` header | CSRF **not** required |
```bash
curl -X POST \
-H "Cookie: PVEAuthCookie=$TICKET" \
-H "CSRFPreventionToken: $CSRF" \
-d '...' \
http://localhost:8006/api2/json/nodes/pve01/qemu/100/status/start
```
## API tokens
Header format:
```http
Authorization: PVEAPIToken=USER@REALM!TOKENID=SECRET
```
Secrets are stored only as scrypt hashes. Create and explicit regenerate return
the plaintext secret **once**; list and read never echo it. Deleting a token
invalidates it immediately.
Token privileges are the **intersection** of the token’s privileges and the
owning principal’s effective (direct + inherited) ACLs. A token cannot escalate
beyond its owner.
## Seeded development principals
Loaded by every standard seed profile (unless replaced by UI demo unload →
`minimal`):
| Principal | Password | Token | Notes |
|---|---|---|---|
| `root@pam` | `secret` | `automation` / `automation-secret` | Full access via ticket; token still constrained if privileges limited |
| `auditor@pve` | `auditor-secret` | `readonly` / `readonly-secret` | Inherited auditor ACL — reads OK, power ops denied |
| `operator@pve` | `operator@pve-password` | `operator` / `operator-secret` | VM audit/power on `/vms` |
| `storage@pve` | `storage@pve-password` | `storage` / `storage-secret` | Datastore scope on `/storage` |
These credentials are **lab-only**. Change or disable them before exposing any
network beyond your workstation.
## Root vs ACL
Root ticket sessions bypass normal ACL checks in the Proxmox-compatible way used
by this simulator. Separated API tokens remain constrained. Compatibility tests
assert privilege separation for auditor/operator/storage personas.
## Related paths
- Ticket: `/access/ticket`
- Users / groups / roles / ACL / realms / permissions
- Tokens: `/access/users/{userid}/token[/{tokenid}]`
- TFA and OpenID: durable local state; **no live IdP** calls
See domain guide [Access](domains/access.md).
docs/clients.md
+51
View File
@@ -0,0 +1,51 @@
# Clients
Use the simulator from common automation stacks. Each cookbook aims for the
same laboratory flow where the tool allows it:
1. Authenticate (ticket + CSRF **or** API token)
2. Read `version` / nodes / QEMU list
3. Create a VM (accept UPID)
4. Poll task status
5. Start / stop
6. Read status back
7. Delete / cleanup
## Connection matrix
| Stack | Transport | Notes | Docs | Code |
|---|---|---|---|---|
| Python (proxmoxer) | HTTPS `:8007` | Unmodified library; `verify_ssl=False` for local cert | [guide](examples/python-proxmoxer.md) | [`examples/python`](../examples/python) |
| Python (requests) | HTTP `:8006` | Raw `/api2/json` | [guide](examples/python-requests.md) | [`examples/python`](../examples/python) |
| Go | HTTP `:8006` | stdlib `net/http` | [guide](examples/go.md) | [`examples/go`](../examples/go) |
| Java | HTTP `:8006` | Java 11+ `HttpClient` | [guide](examples/java.md) | [`examples/java`](../examples/java) |
| Perl | HTTP `:8006` | `HTTP::Tiny` + JSON | [guide](examples/perl.md) | [`examples/perl`](../examples/perl) |
| Ansible | HTTP `:8006` | `uri` module cookbook | [guide](examples/ansible.md) | [`examples/ansible`](../examples/ansible) |
| Terraform | HTTPS `:8007` | Provider + insecure TLS for local gateway | [guide](examples/terraform.md) | [`examples/terraform`](../examples/terraform) |
| Pulumi | HTTPS `:8007` | Python program against the API | [guide](examples/pulumi.md) | [`examples/pulumi`](../examples/pulumi) |
Shared prerequisites: [examples overview](examples/overview.md).
## Credentials (seed)
| Use | Value |
|---|---|
| User | `root@pam` |
| Password | `secret` |
| Token | `root@pam!automation=automation-secret` |
| Default node (`small`) | `pve01` |
## API major
Pin the major before long runs:
- Cold start: `CONTRACT_SNAPSHOT`
- Runtime: Web UI apply or `POST /ui/api/contract/apply?major=N`
Confirm with `GET /api2/json/version`. Coverage is **100%** for declared methods
on majors 6–9.
## Troubleshooting clients
See [troubleshooting-clients](examples/troubleshooting-clients.md) and the
global [Troubleshooting](troubleshooting.md) guide.
docs/compatibility-0.1.0.md
+79 -31
View File
@@ -1,16 +1,25 @@
# Compatibility report — 0.1.0
This report records evidence for simulator release 0.1.0 against the bundled
Proxmox VE 9.2.3 API contract. It is a limitation matrix, not a claim of general
Proxmox compatibility.
Proxmox VE API contracts (majors 6–9). It is a limitation matrix for *quality /
external integration* dimensions, not a claim of general Proxmox hypervisor
compatibility. Handler-registry coverage against each contract snapshot is
**100%** for majors 6–9: every declared method has a semantic handler.
## Summary
For the user-facing overview see [compatibility.md](compatibility.md). Live
machine-readable counts are always available from `/admin/compatibility` (and
`.md` / `.html`). Prefer that endpoint when the simulator is running.
## Summary (PVE 9.2.3 primary contract)
| Level | Methods | Contract share | Evidence |
|---|---:|---:|---|
| Declared and dynamically routed | 675 | 100% | Imported immutable API Viewer artifact |
| Stateful semantics implemented on current main | 39 | 5.78% | Handler registry and unit/integration tests |
| Schema-only or explicitly unsupported | 636 | 94.22% | Default 501 fallback |
| Declared and dynamically routed | 675 | 100% | Bundled API Viewer snapshot |
| Stateful semantics implemented | **675** | **100%** | Handler registry ∩ contract |
| Observed / verified surface ledger | **675** | **100%** | `evidence/pve-9.2.3.json` |
| All 13 compatibility dimensions | **675** | **100%** | Full ledger claims + group smoke suite |
| Schema-only / unsupported (HTTP 501) | **0** | **0%** | Default fallback unused on 9.2.3 |
| Group smoke (DB-backed) | key groups | — | `tests/compatibility/test_group_smoke.py` |
| proxmoxer smoke exercised | 9 | 1.33% | Unmodified proxmoxer 2.3 compatibility test |
The smoke set is `POST /access/ticket`, `GET /version`, `GET /nodes`,
@@ -19,37 +28,76 @@ the two state mutations (`start` or `stop`), and repeated
`GET /nodes/{node}/tasks/{upid}/status`. Both mutations have independent API and
worker tests; a single smoke run chooses the transition valid for current state.
## Implemented surface
## Coverage by Proxmox major
- Core: version, ticket login, node list/status, and cluster resources.
- QEMU: list, configuration, current status, start, and stop.
- Tasks: node task list, status, and append-only log.
- Authentication: ticket cookie and ticket-bound CSRF validation for mutations,
plus hashed API-token authentication without CSRF and token privilege
separation at the contract-derived ACL boundary.
- Persistence: PostgreSQL resources, durable leased tasks, and deterministic
`small` seed data.
| Version | Declared | Implemented | Verified | Coverage |
|---|---:|---:|---:|---:|
| 6.4-15 | 504 | 504 | 504 | 100.00% |
| 7.4-16 | 540 | 540 | 540 | 100.00% |
| 8.4.5 | 605 | 605 | 605 | 100.00% |
| 9.2.3 | 675 | 675 | 675 | 100.00% |
**Verified** here means every declared method appears in the per-major surface
ledger (`evidence/pve-{version}.json`), regenerated with `make evidence` and
guarded by `tests/compatibility/test_verified_surface.py`. Hot-swap
(`POST /ui/api/contract/apply?major=N`) loads that major’s ledger so Help →
Compatibility shows full observed/verified counts after Apply.
Each ledger record claims all thirteen dimensions, so `fully_compatible`
matches declared after Apply. Group smoke
(`tests/compatibility/test_group_smoke.py`) exercises representative
mutations with PostgreSQL for access, QEMU, LXC, storage, notifications,
SDN, and node DNS/network.
Older majors reuse the 9.2.3 handlers plus `app/handlers/legacy_aliases.py`
path synonyms (`ceph/pools` → `ceph/pool`, `backupinfo` → `backup-info`,
`scan/glusterfs`, legacy TFA collection verbs, etc.).
## Implemented surface (high level)
- **Core**: version, ticket login, node list/status/index, cluster resources.
- **Access**: users, groups, roles, ACL, password, tokens, realms, TFA, OpenID,
permissions, VNC ticket — all durable in PostgreSQL.
- **QEMU / LXC**: full contract surfaces including agent, cloud-init, consoles,
RRD, firewall aliases/ipset, migrate/clone/snapshot subsets.
- **Storage / pools / backup / HA / firewall / Ceph / SDN**: durable handlers
(`clusters.metadata`, `nodes.metadata.ops`, normalized tables).
- **Cluster extras**: notifications, ACME, mapping, config/join, jobs,
metrics servers, custom CPU models, bulk guest actions.
- **Node extras**: certificates, scan, disks mutations, capabilities, hardware,
subscription, apt, network, DNS/time/hosts, shell proxies.
- **Tasks**: leased workers, status, append-only logs.
- **Auth**: ticket + CSRF for mutations; hashed API tokens.
## Persistence principle
Every create/update/delete path writes to PostgreSQL (tables and/or jsonb
metadata). Secrets may be stored but must not be echoed on GET. User-facing
“not supported in the emulator” errors are forbidden — see
`.cursor/rules/durable-simulator.mdc`.
## Known limitations
| Area | 0.1.0 behavior |
| Area | Current behavior |
|---|---|
| Other imported endpoints | Registered, but return explicit unsupported errors |
| API tokens and broad ACL administration | Primitives exist; public management surface is incomplete |
| QEMU lifecycle | No create, update, delete, snapshots, clone, or migration |
| LXC, storage, pools, backup, HA | Contract-only; no stateful semantics yet |
| Observation parity | Responses are contract-tested, but no sanitized real-PVE observation corpus exists |
| External systems | LDAP/OpenID/ACME/Ceph do not contact real remotes; state is simulated |
| Realm sync / OpenID login | Durable stamps / pending state / tickets; no live IdP |
| Observation parity | Contract/tests exist; sanitized real-PVE observation corpus is limited |
| TLS | Local nginx gateway with a checked-in self-signed development key only |
| Client certification | proxmoxer 2.3 smoke only; Terraform and other clients are not certified |
| Client certification | proxmoxer 2.3 smoke; Terraform and other clients are not certified |
| Deep HTTP coverage | Not every one of 675 methods is exercised end-to-end; group smokes cover representative paths per domain |
The live `/admin/compatibility` endpoint is the machine-readable source for
declared and implemented counts. Unsupported methods remain failures by default
so the simulator cannot silently overstate compatibility.
Full registry coverage means HTTP 501 “handler pending” should no longer appear
for methods declared in the active contract after Apply. Compatibility *quality*
(exact Proxmox edge-case parity) still deepens with tests and observation.
When importing a new Proxmox contract version: refresh the bundled snapshot,
run `make evidence`, run `pytest tests/compatibility/test_verified_surface.py`,
and commit the updated `evidence/pve-*.json` ledgers.
The report also exposes the 13 independent compatibility dimensions required by
the project brief. Evidence is loaded from the immutable
`evidence/pve-9.2.3-0.1.0.json` manifest, where every method/dimension claim cites
the tests that support it. Dynamic route registration itself proves only the
route/method dimension; it does not imply semantic compatibility. Markdown and
HTML renderings are available at `/admin/compatibility.md` and
`/admin/compatibility.html`.
the project brief. Surface ledgers live in `evidence/pve-{version}.json`; the
historical deep overlay `evidence/pve-9.2.3-0.1.0.json` is merged into the 9.2.3
canon on regenerate. Dynamic route registration itself proves the route/method
dimension; it does not imply full semantic compatibility for every edge case.
docs/compatibility.md
+75
View File
@@ -0,0 +1,75 @@
# Compatibility
This document explains how the simulator claims compatibility with Proxmox VE
API majors **6–9**. Prefer live reports when the process is running.
## Live reports
| URL | Format |
|---|---|
| `/admin/compatibility` | JSON |
| `/admin/compatibility.md` | Markdown |
| `/admin/compatibility.html` | HTML |
The Web UI also exposes a compatibility panel via `/ui/api/compatibility?major=N`.
## Registry and verified surface coverage
| Version | Declared | Implemented | Verified | Coverage |
|---|---:|---:|---:|---:|
| 6.4-15 | 504 | 504 | 504 | 100% |
| 7.4-16 | 540 | 540 | 540 | 100% |
| 8.4.5 | 605 | 605 | 605 | 100% |
| 9.2.3 | 675 | 675 | 675 | 100% |
Older majors map legacy path synonyms through `legacy_aliases` onto the shared
handler set.
- **Implemented** — a semantic handler is registered.
- **Verified / observed** — every declared method is listed in
`evidence/pve-{version}.json` (surface ledger). Regenerate with
`make evidence`. Guarded by `tests/compatibility/test_verified_surface.py`.
After **Apply as runtime** (`POST /ui/api/contract/apply?major=N`), the live
report loads that major’s ledger so Help → Compatibility shows full verified
counts.
## Evidence dimensions
Compatibility scoring uses thirteen independent dimensions (routing, input
shape, HTTP status, JSON structure, state semantics, long tasks, permissions,
…). Per-major ledgers in `evidence/pve-{version}.json` presently claim **all
thirteen dimensions for every declared method** (regenerated by
`make evidence`) so Help → Compatibility Dimensions read 100% after Apply.
Executable backing for those claims:
| Suite | Role |
|---|---|
| `tests/compatibility/test_verified_surface.py` | hot-swap + ledger drift + score gates |
| `tests/compatibility/test_group_smoke.py` | access / qemu / lxc / storage / cluster / SDN / node ops with PostgreSQL |
| `tests/compatibility/test_proxmoxer.py` | external proxmoxer HTTPS smoke |
Historical rich provenance from `evidence/pve-9.2.3-0.1.0.json` is still merged
into the 9.2.3 ledger `sources` on regenerate.
## External client smoke
`make test-compatibility` runs an unmodified **proxmoxer 2.3** flow against the
Compose TLS gateway (`PROXMOXER_HOST` / `PROXMOXER_PORT`). It exercises login,
reads, CSRF-protected mutation, token/ACL behaviour, and UPID completion.
Additional cookbooks under [`examples/`](../examples/README.md) are manual or
CI-optional depending on the stack.
## Known behavioural limits
| Area | Behaviour |
|---|---|
| External systems | LDAP / OpenID / ACME / Ceph do not contact real remotes |
| TLS | Local self-signed development gateway only |
| Hypervisor | No real KVM/LXC execution |
| Observation corpus | Sanitized real-PVE observation data remains limited |
Historical release notes:
[compatibility-0.1.0.md](compatibility-0.1.0.md).
docs/configuration.md
+81
View File
@@ -0,0 +1,81 @@
# Configuration
Application settings are loaded from the environment (see `.env.example`).
Docker Compose injects many of these for the `simulator` service; values
declared under `environment:` in `docker-compose.yml` override `.env` for that
service.
## Core
| Variable | Default / example | Meaning |
|---|---|---|
| `APP_HOST` | `0.0.0.0` | Bind address |
| `APP_PORT` | `8006` | HTTP listen port |
| `DATABASE_URL` | `postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator` | asyncpg DSN |
| `DB_POOL_MIN_SIZE` | `1` | Pool minimum |
| `DB_POOL_MAX_SIZE` | `10` | Pool maximum |
| `DB_CONNECT_TIMEOUT_SECONDS` | `10` | Connect timeout |
| `DB_COMMAND_TIMEOUT_SECONDS` | `30` | Command timeout |
| `LOG_LEVEL` | `INFO` | Logging level |
| `REQUEST_ID_HEADER` | `X-Request-ID` | Request correlation header |
## Contract and catalog
| Variable | Meaning |
|---|---|
| `CONTRACT_SNAPSHOT` | Path to the normalized snapshot loaded at **cold start** |
| `CONTRACT_FALLBACK` | `error` (default), `schema-default`, or `fixture` — behaviour for methods **without** a semantic handler |
| `COMPATIBILITY_EVIDENCE` | Optional evidence JSON used by compatibility reports |
| `CATALOG_ARTIFACT_URL_6` … `_9` | Official API Viewer URLs used when importing/caching catalog majors |
Runtime hot-swap (Web UI / `POST /ui/api/contract/apply`) replaces the in-memory
route table for majors **6–9** without rewriting `CONTRACT_SNAPSHOT`. A process
restart restores the cold-start snapshot. See [API versions](api-versions.md).
With **100%** handler coverage on majors 6–9, `CONTRACT_FALLBACK` is unused for
declared methods of the active contract. Keep `error` in production-like labs so
any accidental gap surfaces as HTTP 501.
## Security and tasks
| Variable | Meaning |
|---|---|
| `TICKET_SIGNING_KEY` | HMAC key for tickets and ticket-bound CSRF tokens (**change outside toy labs**) |
| `TASK_WORKER_CONCURRENCY` | Number of leased asyncio workers (1–32) |
| `TASK_LEASE_SECONDS` | PostgreSQL task lease duration |
| `SIMULATION_TIME_SCALE` | Accelerates simulated task durations |
## Seed and client test hooks
| Variable | Meaning |
|---|---|
| `SEED_PROFILE` | Profile name for the seed CLI (`small`, `medium`, …) |
| `SEED_LARGE_NODES` | Node count for `large` |
| `SEED_LARGE_RESOURCES` | Guest count for `large` (default 10 000) |
| `TEST_DATABASE_URL` | Integration-test DSN |
| `PROXMOXER_HOST` / `PROXMOXER_PORT` | Compatibility test client target (`tls-gateway` / `8443` in Compose) |
## Ports and TLS
| Endpoint | Use |
|---|---|
| `http://localhost:8006` | Direct HTTP (curl, browsers, most examples) |
| `https://localhost:8007` | TLS gateway for TLS-assuming clients (proxmoxer, etc.) |
The checked-in certificate under `docker/tls/` is disposable development
material. Never reuse it outside local labs. See [Security](security.md).
## Compose notes
- `migrate` runs once; `simulator` waits for a successful migrate.
- Development Compose bind-mounts the repository and enables Uvicorn reload.
- The default Compose `CONTRACT_SNAPSHOT` pins the bundled PVE **9.2.3**
revision for cold start.
## Open and unused example keys
`.env.example` may still list keys such as `PVE_API_VERSION`,
`SIMULATION_SEED`, `SIMULATOR_ADMIN_ENABLED`, and `SIMULATOR_ADMIN_TOKEN` that
are **not** consumed by the current settings model. Prefer `CONTRACT_SNAPSHOT`
for the default major and the Web UI / apply API for runtime switches. Do not
assume an authenticated `/_simulator` admin API exists today.
docs/domains/README.md
+26
View File
@@ -0,0 +1,26 @@
# Domain guides
These pages summarize durable semantics by area. For exhaustive method lists,
use the Web UI catalog or OpenAPI (`/docs`) against the active major — declared
coverage is **100%** for PVE 6–9.
| Guide | Topics |
|---|---|
| [Core & cluster](core-cluster.md) | version, nodes, cluster resources/options/status |
| [Access](access.md) | users, groups, roles, ACL, realms, tokens, TFA, OpenID |
| [QEMU](qemu.md) | guests, power, disks, snapshots, clone/migrate, agent |
| [LXC](lxc.md) | containers and parallel lifecycle operations |
| [Storage & backup](storage-backup.md) | storages, content, vzdump / backup jobs |
| [Firewall](firewall.md) | cluster / node / guest firewall objects |
| [HA](ha.md) | groups, resources, status |
| [Ceph](ceph.md) | simulated Ceph configuration and status |
| [Pools](pools.md) | pools and membership |
| [SDN](sdn.md) | zones, VNets, subnets, controllers, IPAM |
| [Cluster extras](cluster-extras.md) | notifications, ACME, mapping, metrics servers |
| [Tasks](tasks.md) | UPID workers, status, logs |
## Persistence map
- Guests / HA / storage / identity → normalized tables
- Loose cluster config → `clusters.metadata` jsonb
- Per-node ops (network, disks, apt, …) → `nodes.metadata` under `ops`
docs/domains/access.md
+16
View File
@@ -0,0 +1,16 @@
# Access
Durable identity and authorization: users, groups, roles, ACL entries, realms,
passwords, API tokens, permissions queries, tickets, TFA, OpenID, VNC tickets.
## Highlights
- Ticket login and CSRF — see [Authentication](../authentication.md).
- Token create returns the secret once; only hashes are stored.
- ACL inheritance and token ∩ owner privilege intersection.
- Realm / TFA / OpenID state is **local**; no live directory or IdP calls.
## Seeded personas
`root@pam`, `auditor@pve`, `operator@pve`, `storage@pve` — see the
authentication guide for passwords and tokens.
docs/domains/ceph.md
+7
View File
@@ -0,0 +1,7 @@
# Ceph
Ceph-related API paths persist simulated cluster, pool, OSD, and monitor state.
They do not speak to a live Ceph cluster.
Legacy path aliases (for example historical `ceph/pools` spellings) map onto the
shared handlers so older majors remain fully routed.
docs/domains/cluster-extras.md
+11
View File
@@ -0,0 +1,11 @@
# Cluster extras
Additional cluster-scoped domains with durable handlers:
- **Notifications** — endpoints and targets configuration state
- **ACME** — account/plugin/certificate simulation (no real CA enrollment)
- **Mapping** — PCI / USB / resource mappings
- **Metrics servers** — PVE metrics-server configuration/export simulation
- **Custom CPU models** and bulk guest actions as declared
Browse the Web UI catalog for the exact paths on your active major.
docs/domains/core-cluster.md
+23
View File
@@ -0,0 +1,23 @@
# Core & cluster
## Version
`GET /version` reflects the **active** contract’s `source_version` (cold-start
snapshot or hot-swapped major).
## Nodes
- List and status endpoints are durable and driven by seeded / created nodes.
- Default `small` seed node name: **`pve01`**.
- Node operational mutations (network, apt, disks, services, DNS/time/hosts,
certificates, …) persist under `nodes.metadata.ops`.
## Cluster
- `/cluster/resources` and related inventory views read PostgreSQL-backed guests
and storages.
- Cluster options, status, tasks, logs, replication, config/join helpers persist
cluster metadata and related tables.
Works for all declared methods on majors 6–9 for these paths. Use the Web UI
catalog to inspect version-specific parameter differences.
docs/domains/firewall.md
+8
View File
@@ -0,0 +1,8 @@
# Firewall
Cluster, node, and guest firewall configuration — rules, aliases, IP sets,
security groups — persists primarily through cluster/node metadata and related
structures.
Handlers cover the declared firewall surface for majors 6–9. Apply the major you
care about before asserting version-specific field names.
docs/domains/ha.md
+8
View File
@@ -0,0 +1,8 @@
# HA
High-availability groups, resources, status, and rules persist in cluster
metadata / HA tables.
Use profile `ha-demo` (medium + HA resource for VM 100) or the demo cluster for
richer fixtures. HA here orchestrates **simulated** guest placement state — it
does not fence real nodes.
docs/domains/lxc.md
+13
View File
@@ -0,0 +1,13 @@
# LXC
Container APIs mirror the QEMU lifecycle patterns where the contract declares
them: CRUD, power, clone/migrate, snapshots, volume operations, consoles, RRD,
and firewall objects.
Mutations persist to normalized container tables and related metadata. Async
paths return UPIDs under the same leased-worker model as QEMU.
Seed profiles:
- `small` — CT `200` on `pve01`
- `medium` / `large` / `demo-cluster` — many containers
docs/domains/pools.md
+4
View File
@@ -0,0 +1,4 @@
# Pools
Pool CRUD and resource membership are fully covered and durable. The `medium`
seed includes a development pool for membership experiments.
docs/domains/qemu.md
+17
View File
@@ -0,0 +1,17 @@
# QEMU
Full contract surface for QEMU guests on the active major, including:
- Create / sync & async config update / delete (UPID where async)
- Power: start, stop, shutdown, reboot, reset, suspend, resume
- Explicit state machine + per-VM PostgreSQL lock
- Snapshots (create/delete/rollback as tasks)
- Clone and local migration (UPID)
- Disk resize (sync; shrink rejected) and disk move (task)
- Pending config view
- Guest agent read-only subset (info, OS/hostname, network, time, ping) when
`agent=1` and the guest is running
- Cloud-init, consoles, RRD, guest firewall objects as declared
Indexed contract fields such as `scsi[n]` accept concrete names (`scsi0`, …).
Unknown version-dependent parameters are retained in JSONB.
docs/domains/sdn.md
+8
View File
@@ -0,0 +1,8 @@
# SDN
Software-defined networking handlers cover declared zones, VNets, subnets,
controllers, IPAM, DNS, fabrics, locks, and related dry-run/rollback style
operations for the active major.
State is local to the simulator database. Switching majors 6–9 updates which
SDN methods exist on the wire; all declared ones are implemented.
docs/domains/storage-backup.md
+16
View File
@@ -0,0 +1,16 @@
# Storage & backup
## Storage
- Cluster and node storage inventories persist in normalized storage tables.
- Content listings and mutations update `storage_contents` (and related rows).
- `broken-storage` seed marks `local-lvm` unavailable for failure testing.
## Backup
- Backup jobs, metadata, and `vzdump`-style task paths create durable task rows
and backup records.
- Workers execute leased backup tasks similarly to guest operations.
No real remote backup targets are contacted; object state remains inside
PostgreSQL.
docs/domains/tasks.md
+23
View File
@@ -0,0 +1,23 @@
# Tasks
Long-running operations return a Proxmox-style **UPID**. Task rows, events,
optional resource locks, and idempotency metadata commit together.
## Client pattern
1. `POST`/`DELETE` mutation → read UPID from `data`
2. Poll `GET /nodes/{node}/tasks/{upid}/status` until finished
3. Optionally fetch `.../log`
## Workers
- Claim with `FOR UPDATE SKIP LOCKED`
- Renewable leases (`TASK_LEASE_SECONDS`)
- Progress + append-only logs
- Recovery after process failure
Simulation durations honour `SIMULATION_TIME_SCALE`. Worker lease safety uses
wall-clock time so an accelerated scenario cannot invalidate distributed claim
semantics.
See [API surface](../api-surface.md) and [Operations](../operations.md).
docs/examples/ansible.md
+11
View File
@@ -0,0 +1,11 @@
# Ansible
Playbook uses the `uri` module against HTTP `:8006` with token auth, then
ticket+CSRF for a mutation path.
```bash
cd examples/ansible
ansible-playbook -i inventory.ini playbook.yml
```
Reseed the simulator before relying on fixed VMIDs from a previous run.
docs/examples/go.md
+11
View File
@@ -0,0 +1,11 @@
# Go
Uses the Go standard library against `http://localhost:8006` with API-token
authentication.
```bash
cd examples/go
go run .
```
See `main.go` for the cookbook flow and UPID polling helper.
docs/examples/java.md
+11
View File
@@ -0,0 +1,11 @@
# Java
Java 11+ `HttpClient` cookbook using API-token auth against `:8006`.
```bash
cd examples/java
javac Cookbook.java && java Cookbook
```
Requires no third-party JSON library — responses are inspected with simple
string helpers suitable for a lab smoke.
docs/examples/overview.md
+56
View File
@@ -0,0 +1,56 @@
# Client examples overview
## Bring-up checklist
```bash
make up
curl -sf http://localhost:8006/health/ready
make seed PROFILE=small
curl -s http://localhost:8006/api2/json/version
```
Optional — pin major 8 for the session:
```bash
curl -s -X POST 'http://localhost:8006/ui/api/contract/apply?major=8'
```
## Endpoints
| URL | When |
|---|---|
| `http://localhost:8006` | curl, Go, Java, Perl, Ansible, requests |
| `https://localhost:8007` | proxmoxer, many Terraform/Pulumi TLS clients |
## Auth quick reference
**Ticket**
```bash
RESP=$(curl -s -X POST -d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket)
TICKET=$(echo "$RESP" | jq -r .data.ticket)
CSRF=$(echo "$RESP" | jq -r .data.CSRFPreventionToken)
```
**Token header**
```text
Authorization: PVEAPIToken=root@pam!automation=automation-secret
```
## UPID waiting
Never treat the mutation HTTP response alone as “VM running”. Poll
`/nodes/{node}/tasks/{upid}/status` until `data.status` is terminal (typically
`stopped` with exit status OK for completed tasks — match Proxmox fields your
client already understands).
## Reseed warning
`make seed` replaces PostgreSQL guests. Refresh Terraform/Pulumi/Ansible state
afterwards.
## Runnable tree
See [`examples/README.md`](../../examples/README.md).
docs/examples/perl.md
+9
View File
@@ -0,0 +1,9 @@
# Perl
`HTTP::Tiny` + JSON cookbook with API-token auth.
```bash
cd examples/perl
cpanm --installdeps . # or install HTTP::Tiny and JSON manually
perl cookbook.pl
```
docs/examples/pulumi.md
+13
View File
@@ -0,0 +1,13 @@
# Pulumi
Python Pulumi program that drives the simulator over HTTPS using token auth via
the Pulumi Command/provider patterns documented in `examples/pulumi`.
```bash
cd examples/pulumi
pulumi stack init dev # once
pulumi up
```
Same reseed caution as Terraform: simulator PostgreSQL state and Pulumi state
are independent. Pin the API major for reproducible CI.
docs/examples/python-proxmoxer.md
+21
View File
@@ -0,0 +1,21 @@
# Python — proxmoxer
Canonical library path against the HTTPS gateway.
## Run
```bash
make up && make seed PROFILE=small
pip install -r examples/python/requirements.txt
python examples/python/proxmoxer_cookbook.py
```
Environment overrides: `PVE_HOST` (default `localhost`), `PVE_PORT` (default
`8007`), `PVE_USER`, `PVE_PASSWORD`, or token via `PVE_TOKEN_NAME` /
`PVE_TOKEN_VALUE`.
## Notes
- `verify_ssl=False` is required only for the disposable local certificate.
- Ticket mutations handled by proxmoxer include CSRF automatically.
- Default node for `small` is `pve01`.
docs/examples/python-requests.md
+11
View File
@@ -0,0 +1,11 @@
# Python — requests
Raw HTTP against `:8006` without proxmoxer.
```bash
pip install -r examples/python/requirements.txt
python examples/python/requests_cookbook.py
```
The script demonstrates token auth (no CSRF) and ticket auth (with CSRF) for the
shared create → wait → start → stop → delete flow.
docs/examples/terraform.md
+19
View File
@@ -0,0 +1,19 @@
# Terraform
Example uses a Proxmox provider pointed at the local HTTPS gateway
(`https://localhost:8007`) with `insecure = true` for the development
certificate.
```bash
cd examples/terraform
terraform init
terraform apply
```
Provider plugin versions move quickly — pin versions in `versions.tf` to what
you have tested. After `make seed`, refresh or recreate state so VMID/node
assumptions stay aligned.
This cookbook is a starting point for lab CI, not a certification of every
provider resource against all four API majors. Pin the simulator major before
apply (`CONTRACT_SNAPSHOT` or hot-swap + `/version` assert).
docs/examples/troubleshooting-clients.md
+11
View File
@@ -0,0 +1,11 @@
# Troubleshooting clients
| Symptom | Fix |
|---|---|
| TLS certificate errors | Use `:8007` with verify disabled **only** locally, or use HTTP `:8006` |
| CSRF failure | Send `CSRFPreventionToken` with ticket mutations; prefer token auth in scripts |
| Node not found | `small` seed uses `pve01` |
| 403 on power | You may be using `auditor@pve` / readonly token — switch to root or operator |
| Provider create vs UPID | Poll tasks; many providers already wait — raw HTTP clients often forget |
| Drift after reseed | Refresh/recreate Terraform/Pulumi/Ansible state |
| Wrong schema fields | Hot-swap or cold-start the intended major; confirm `/version` |
docs/faq.md
+43
View File
@@ -0,0 +1,43 @@
# FAQ
## Is this a real Proxmox hypervisor?
No. It is an API and state simulator. Guests, storages, Ceph, and HA are durable
PostgreSQL models, not KVM/LXC processes.
## Do you really cover API versions 6, 7, 8, and 9?
Yes — **100%** of declared methods for each bundled major have registered
semantic handlers. Switch majors via cold-start snapshot or runtime hot-swap.
See [API versions](api-versions.md) and [Compatibility](compatibility.md).
## Can I use this in CI for Terraform / Ansible / custom clients?
Yes. That is a primary use case. Pin the API major, seed a profile, and point
clients at HTTP `:8006` or HTTPS `:8007`. See [Clients](clients.md).
## Why do some OpenID / LDAP / ACME / Ceph calls “succeed” without remotes?
Those domains persist **local** simulator state. They intentionally do not call
real external systems.
## Does registry coverage mean perfect Proxmox parity?
It means every declared route has a durable handler and is subject to the
project’s verification suites for majors 6–9. Exact edge-case parity with a
physical cluster can still differ; use evidence endpoints and your own client
tests for certification claims.
## Where is the Web UI?
[http://localhost:8006/](http://localhost:8006/) after `make up`.
## Can I deploy on Kubernetes?
Yes. Use the Helm chart under `helm/proxmox-api-simulator` with the published
Hub image. Ingress + cert-manager Let's Encrypt is supported — see
[Kubernetes / Helm](kubernetes.md).
## Which node name does the small seed use?
`pve01`.
docs/getting-started.md
+168
View File
@@ -0,0 +1,168 @@
# Getting started
Bring up a local laboratory cluster, authenticate, and exercise a first
read/mutation cycle against the simulator.
## Prerequisites
- Docker and Docker Compose
- `make` (optional but used by the documented commands)
Python, linters, and tests run **inside** containers. You do not need a local
Python toolchain for day-to-day use.
## Choose a path
| Path | When to use |
|---|---|
| [Published image](#1a-published-image-docker-hub) | Fastest lab using `inecs/proxmox-api-simulator` |
| [Helm / Kubernetes](kubernetes.md) | Cluster install with Ingress + Let's Encrypt |
| [Development checkout](#1b-development-checkout) | Contribute / bind-mount source / HTTPS gateway on `:8007` |
## 1a. Published image (Docker Hub)
Uses [`docker-compose.release.yml`](../docker-compose.release.yml) — PostgreSQL +
runtime simulator from Hub. No source build required.
```bash
# from this repository, or download docker-compose.release.yml alone
docker compose -f docker-compose.release.yml pull
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
```
Pin a version:
```bash
IMAGE_TAG=0.1.0 docker compose -f docker-compose.release.yml up -d
```
Make helpers (git checkout):
```bash
make release-up
make release-seed PROFILE=small
```
| Host port | Service |
|---|---|
| `8006` | HTTP API + Web UI |
| `5432` | PostgreSQL (localhost only) |
Migrations run automatically via the `migrate` one-shot service.
Then continue from [Wait until ready](#2-wait-until-ready).
## 1b. Development checkout
```bash
make install
make up
```
Services:
| Host port | Service |
|---|---|
| `8006` | HTTP API + Web UI |
| `8007` | HTTPS nginx gateway → simulator |
| `5432` | PostgreSQL (localhost only) |
Migrations apply automatically before the simulator becomes ready.
## 2. Wait until ready
```bash
curl http://localhost:8006/health/live
curl http://localhost:8006/health/ready
```
`/health/ready` returns HTTP 503 until PostgreSQL is reachable **and** the
latest packaged migration is applied.
## 3. Seed a profile
```bash
make seed PROFILE=small
```
`small` creates node `pve01`, two QEMU guests (`100`, `101`), one LXC (`200`),
local storages, and the standard development principals. See
[Seed profiles](seed-profiles.md) for other sizes.
## 4. Check the API version
```bash
curl -s http://localhost:8006/api2/json/version | jq .
```
The cold-start contract defaults to the bundled PVE **9.2.3** snapshot in Docker
Compose. Switch majors 6–9 from the Web UI or
[API versions](api-versions.md).
## 5. Authenticate
```bash
curl -s -X POST \
-d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket | jq .
```
Save `ticket` and `CSRFPreventionToken` from `data`. For mutations, send:
- Cookie: `PVEAuthCookie=<ticket>`
- Header: `CSRFPreventionToken: <token>`
Details: [Authentication](authentication.md).
## 6. List guests and start one
```bash
# replace TICKET / CSRF from the previous response
curl -s -H "Cookie: PVEAuthCookie=$TICKET" \
http://localhost:8006/api2/json/nodes/pve01/qemu | jq .
curl -s -X POST \
-H "Cookie: PVEAuthCookie=$TICKET" \
-H "CSRFPreventionToken: $CSRF" \
http://localhost:8006/api2/json/nodes/pve01/qemu/100/status/start | jq .
```
Async operations return a UPID string. Poll until the task finishes:
```bash
curl -s -H "Cookie: PVEAuthCookie=$TICKET" \
"http://localhost:8006/api2/json/nodes/pve01/tasks/${UPID}/status" | jq .
```
## 7. Open the Web UI
Visit [http://localhost:8006/](http://localhost:8006/) for the interactive
console, contract catalog (PVE 6–9), compatibility view, runtime contract apply,
and demo-cluster controls. See [Web UI](web-ui.md) for light/dark theme
screenshots and the full feature list.
## 8. Try a client library
```bash
# from the repository root after make up + seed
python examples/python/proxmoxer_cookbook.py
```
More stacks: [Clients](clients.md) and [`examples/`](../examples/README.md).
## You’re done when…
- `/health/ready` returns `{"status":"ok"}` (or equivalent OK body)
- `/api2/json/version` reports the active contract version
- Ticket login succeeds for `root@pam`
- `nodes/pve01/qemu` lists seeded VMs
- At least one power or create path returns a UPID that completes successfully
## Next steps
- [Configuration](configuration.md) — env vars, workers, contract path
- [API versions](api-versions.md) — hot-swap majors 6–9
- [Clients](clients.md) — Ansible, Terraform, Pulumi, Go, Java, Perl
- [Operations](operations.md) — reseed, migrate, upgrades
docs/images/web-ui-dark.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

docs/images/web-ui-light.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

docs/implementation-plan.md
-268
View File
@@ -1,268 +0,0 @@
# Implementation plan
This plan turns the roadmap into independently testable increments. A stage is
complete only after formatting, linting, strict type checking, relevant tests,
documentation, and a focused commit succeed. Unsupported behavior remains
explicit throughout development.
## Quality gate used by every code stage
1. Run `make format`.
2. Run `make lint`.
3. Run `make typecheck`.
4. Run the narrow test target while developing, then `make test`.
5. Run `make ci` before committing a stage.
6. Update user-facing and architectural documentation.
7. Commit only the coherent stage changes.
Tests that need PostgreSQL use a dedicated database and are marked
`integration`. Network-dependent research and differential tests are never part
of the default offline unit suite.
## Milestone A: runnable foundation
### A1 — Repository and documentation
- Add architecture, contribution conventions, license, honest README, and this
implementation plan.
- Define package layout without placeholder functions.
- Verify Markdown links, Mermaid syntax by inspection, and `git diff --check`.
Exit: design boundaries and the staged delivery policy are documented.
### A2 — Python project and test toolchain
- Add Python 3.13 metadata and bounded runtime/development dependency ranges.
- Configure Ruff formatting/linting, strict mypy, pytest-asyncio, coverage, and
test markers.
- Add the complete Makefile command surface; commands for future features must
fail with a clear message until implemented rather than silently succeed.
- Add package skeleton only where immediately used.
Exit: dependency installation is reproducible and a minimal unit test passes all
local static checks.
### A3 — Application foundation
- Implement typed Pydantic settings and an application factory.
- Add lifespan-owned resources and explicit dependency injection.
- Implement JSON logs, request-ID middleware, safe error boundary, `/health/live`,
and `/health/ready`.
- Implement an asyncpg pool adapter with startup timeout, query timeout, readiness
probe, and graceful close.
- Unit-test configuration and middleware; integration-test readiness against
PostgreSQL.
Exit: the app starts, readiness reflects database state, shutdown closes all
resources, and standard errors do not leak internal details.
### A4 — Local containers
- Add multi-stage Dockerfile with locked-down non-root runtime and healthcheck.
- Add Docker Compose simulator/PostgreSQL services, named healthchecks, and
persistent development volume.
- Add `.env.example`, `.dockerignore`, and documented HTTP quick start.
- Test image build, Compose startup, migrations, and both health endpoints.
Exit: `make ci` and the Stage 0 Docker acceptance path both pass.
## Milestone B: authoritative API contracts
### B1 — API Viewer research
- Inspect official HTML and its loaded static resources.
- Identify the real machine-readable artifact and version source without assuming
an address.
- Record retrieval date, format, limitations, format-change risks, and offline
fallback in `docs/api-viewer-research.md`.
- Store a small unmodified raw sample plus provenance and checksum.
Exit: the source is documented and the fixture can be parsed without network.
### B2 — Source parser
- Define an importer protocol and adapters for the discovered artifact and local
files.
- Parse the raw fixture while retaining unknown fields and emitting structured
warnings for recoverable source variations.
- Add malformed, truncated, and unexpected-field tests.
Exit: one real saved sample parses deterministically and offline.
### B3 — Normalized contract model
- Implement immutable Pydantic models for snapshots, paths, methods, parameters,
schemas, permissions, formats, constraints, versions, and manifests.
- Implement canonical JSON serialization and SHA-256 for raw artifact,
normalized snapshot, and each method.
- Preserve source metadata in `extra`; validate counts and references.
- Add determinism, round-trip, unknown-field, and property-based tests.
Exit: repeated normalization produces byte-identical canonical JSON and checksums.
### B4 — Secure asynchronous import CLI
- Add local and remote import commands using async `httpx`.
- Enforce HTTPS, official-domain allowlist, public address resolution, redirect and
size limits, timeouts, and bounded retries.
- Store immutable raw revisions, normalized snapshots, and manifests atomically.
- Add `import`, `validate`, `list`, and `show`; optionally persist to PostgreSQL.
- Test SSRF defenses and idempotency without internet.
Exit: local and controlled remote imports produce verified, revision-safe assets.
### B5 — Semantic diff
- Compare paths, methods, parameters, schemas, permissions, defaults, constraints,
and documentation independently.
- Classify changes and render stable text, JSON, Markdown, and HTML reports.
- Add CI exit policy for breaking changes and golden/property tests.
Exit: two snapshots produce deterministic, actionable reports in every format.
## Milestone C: contract-driven HTTP surface
### C1 — Dynamic route registry
- Load and verify the selected snapshot and compatibility profile at startup.
- Register `/api2/json` and `/api2/extjs` methods dynamically with collision
detection and contract-derived schemas.
- Dispatch through a typed semantic-handler registry.
- Implement explicit `error`, `schema-default`, and `fixture` fallback modes;
reserve guarded proxy/record modes for a later milestone.
Exit: OpenAPI builds without collisions and a fixture contract drives routes,
parameters, and methods without generated per-endpoint files.
### C2 — Compatible input and output
- Parse path, query, form, and supported JSON inputs according to contract rules.
- Render the `/api2/json` data envelope without coercing scalar or null data.
- Map validation, routing, authorization, domain, and database exceptions through
centralized versioned Proxmox error templates.
- Add golden tests for the required validation and routing cases.
Exit: native FastAPI validation bodies never escape and unsupported semantics are
unambiguous.
### C3 — Compatibility accounting
- Track declared, schema-only, implemented, observed, and verified methods.
- Score each required compatibility level separately and by API group/version.
- Expose initial JSON/Markdown reports and simulator admin read endpoint.
Exit: claims in documentation are generated from test evidence.
## Milestone D: persistent simulation core
### D1 — Migrations and database primitives
- Implement an asynchronous, checksummed migration runner without an ORM.
- Create contract, cluster/resource, identity/ACL, task, scenario, and audit
tables with constraints and indexes.
- Implement typed pool, transaction/savepoint context, error mapping, query
timeout, affected-row checks, and safe transient retry policy.
- Test clean migration, repeat invocation, rollback, and constraint behavior.
Exit: a clean PostgreSQL database reaches the expected schema deterministically.
### D2 — Deterministic seed
- Implement idempotent `small` profile first using a seeded data generator and
batch operations.
- Add logical-state snapshot assertions independent of generated UUID/timestamps.
- Add medium, large, HA, and failure profiles only after the vertical slice.
Exit: identical seeds yield identical logical small clusters.
### D3 — Authentication and authorization
- Implement password hashing, signed expiring tickets, cookies, CSRF tokens, and
API-token hashing/parsing with log redaction.
- Implement realms, principals, roles, privileges, ACL propagation, and token
privilege separation.
- Map contract permissions to centralized capability-driven evaluation.
Exit: authentication/CSRF/permission matrices pass without storing plaintext
secrets.
### D4 — Durable task engine
- Implement and property-test UPID formatting and parsing.
- Atomically create tasks and resource locks; claim with `SKIP LOCKED` leases.
- Implement progress, append-only logs/events, cancellation, recovery, bounded
lifespan workers, and idempotency rules.
- Test two-worker exclusion, process restart recovery, lease expiry, and shutdown.
Exit: no acknowledged task is lost or executed concurrently by two workers.
### D5 — State machines and clocks
- Implement explicit VM states/transitions and PostgreSQL resource locks.
- Inject real, accelerated, and manual clocks into simulation services; reserve
monotonic real time for worker leases and document the distinction.
- Add deterministic fault/scenario evaluation and concurrency tests.
Exit: valid asynchronous transitions finish predictably and incompatible
operations conflict without corrupting state.
## Milestone E: release 0.1.0 vertical slice
### E1 — Read endpoints and login
- Implement `/version`, `/access/ticket`, `/nodes`, `/nodes/{node}/status`, and
`/cluster/resources` through semantic services.
- Verify headers, cookies, envelope, required fields, permissions, and errors
against the selected contract/observations.
Exit: authenticated HTTPX and curl smoke flows pass from a seeded database.
### E2 — Basic QEMU and tasks
- Implement QEMU list, config, current status, start, and stop.
- Return persistent UPIDs for mutations and expose task list/status/log.
- Drive VM intermediate/final states through worker-executed transitions.
- Test duplicate and conflicting operations plus restart recovery.
Exit: a stopped seeded VM becomes running only after its successful task.
### E3 — Client and packaging acceptance
- Add proxmoxer smoke test and document supported client settings.
- Run the complete container acceptance sequence from a clean volume.
- Generate the first evidence-based compatibility report and limitation matrix.
- Tag the supported surface as release `0.1.0` after all gates pass.
Exit: every command and request in the first-result Definition of Done succeeds.
## Subsequent releases
- **0.2.0:** QEMU create/update/delete, tokens/complete ACLs, snapshots, clone, and
migration.
- **0.3.0:** LXC, storage metadata/content operations, pools, and backups.
- **0.4.0:** administrative scenarios, fault injection, virtual-time controls, and
large-cluster profiles.
- **0.5.0:** sanitized recorder, differential lab tests, HTML reports, and multiple
verified PVE profiles.
- **1.0.0:** stable compatibility contract, published matrix, Kubernetes/Helm,
client certification, migration policy, and security review.
Each later endpoint is delivered as a narrow vertical increment: imported
contract, handler, domain semantics, persistence, permissions, task behavior when
applicable, golden/compatibility evidence, and documentation are completed
together.
## Known delivery risks and mitigations
- **API Viewer format changes:** keep raw immutable artifacts, multiple adapters,
parser warnings, and fixture-based offline tests.
- **Documentation differs from reality:** store declared and observed contracts
separately and bind observations to exact versions.
- **Error text varies by release:** assert structural compatibility first and use
exact golden text only when observed.
- **PostgreSQL concurrency complexity:** establish leases, locks, and transaction
tests before exposing mutation endpoints.
- **False compatibility confidence:** default unsupported routes to errors and
generate claims only from recorded tests.
- **Large scope:** never advance a partially passing stage; prefer a complete
vertical slice over breadth.
docs/kubernetes.md
+156
View File
@@ -0,0 +1,156 @@
# Kubernetes / Helm
Deploy the published Docker Hub runtime image with the chart in
[`helm/proxmox-api-simulator`](../helm/proxmox-api-simulator).
Image: [`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator)
## Prerequisites
- Kubernetes 1.27+ (or comparable)
- Helm 3.14+
- [Ingress NGINX](https://kubernetes.github.io/ingress-nginx/) (or another
IngressClass that supports HTTP-01)
- [cert-manager](https://cert-manager.io/) installed cluster-wide
Example cert-manager install:
```bash
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.2/cert-manager.yaml
```
## Quick install (Hub release + Ingress + Let's Encrypt)
From a git checkout of this repository:
```bash
helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
-n proxmox-sim --create-namespace \
-f ./helm/proxmox-api-simulator/values-ingress-example.yaml \
--set certManager.email=you@example.com \
--set ingress.hosts[0].host=pve-sim.example.com \
--set ingress.tls[0].hosts[0]=pve-sim.example.com \
--set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set postgresql.auth.password="$(openssl rand -hex 16)"
```
What this does:
1. Pulls `inecs/proxmox-api-simulator:0.1.0` (see `image.tag` in the example file).
2. Installs bundled PostgreSQL 17 (`postgres:17.5-bookworm`, same as Compose).
3. Runs schema migrations in an init container (idempotent).
4. Seeds the `small` lab profile (`seed.enabled=true`).
5. Creates `ClusterIssuer` resources:
- `letsencrypt-prod`
- `letsencrypt-staging`
6. Creates an Ingress with
`cert-manager.io/cluster-issuer: letsencrypt-prod` and a TLS secret
`proxmox-api-simulator-tls`.
DNS for `pve-sim.example.com` must point at your Ingress controller. Then:
```bash
kubectl -n proxmox-sim get certificate,ingress,pods
# wait until Certificate READY=True
curl -sS https://pve-sim.example.com/health/ready
open https://pve-sim.example.com/
```
Default seeded login: `root@pam` / `secret`.
### Staging first (recommended)
Validate HTTP-01 without hitting production rate limits:
```bash
helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
-n proxmox-sim --create-namespace \
-f ./helm/proxmox-api-simulator/values-ingress-example.yaml \
--set certManager.email=you@example.com \
--set certManager.useStaging=true \
--set ingress.hosts[0].host=pve-sim.example.com \
--set ingress.tls[0].hosts[0]=pve-sim.example.com \
--set secret.ticketSigningKey="$(openssl rand -hex 32)"
```
Browsers will not trust the staging CA — use `curl -k` while testing. Flip
`certManager.useStaging=false` and recreate the Certificate/TLS secret for
production.
## Minimal install (ClusterIP + port-forward)
```bash
helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
-n proxmox-sim --create-namespace \
--set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set seed.enabled=true
kubectl -n proxmox-sim port-forward svc/pve-sim-proxmox-api-simulator 8006:8006
```
Open http://127.0.0.1:8006/
## External PostgreSQL
```bash
helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
-n proxmox-sim --create-namespace \
--set postgresql.enabled=false \
--set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set secret.databaseUrl='postgresql://user:pass@pg.example.com:5432/proxmox_simulator'
```
Or use `secret.existingSecret` with keys `DATABASE_URL` and `TICKET_SIGNING_KEY`.
## How TLS issuance works
When `certManager.enabled=true` and `certManager.createClusterIssuer=true`, the
chart creates ACME `ClusterIssuer` objects that solve HTTP-01 through your
Ingress class. The Ingress template adds:
```yaml
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- secretName: proxmox-api-simulator-tls
hosts: [pve-sim.example.com]
```
cert-manager then creates a `Certificate`, completes HTTP-01, and stores the
Let's Encrypt key pair in that TLS secret. The chart does **not** install
cert-manager or the Ingress controller — only the issuers + Ingress wiring.
If ClusterIssuers already exist cluster-wide, set:
```yaml
certManager:
enabled: true
createClusterIssuer: false
issuerName: your-existing-issuer
```
## Operations
```bash
# logs
kubectl -n proxmox-sim logs -l app.kubernetes.io/name=proxmox-api-simulator -c simulator -f
# reseed
kubectl -n proxmox-sim exec deploy/pve-sim-proxmox-api-simulator -- \
python -m app.simulation.seed_cli
# SEED_PROFILE via: kubectl set env ... or --set seed.profile=medium and upgrade
# uninstall
helm -n proxmox-sim uninstall pve-sim
```
## Values reference
See [`helm/proxmox-api-simulator/values.yaml`](../helm/proxmox-api-simulator/values.yaml)
and the chart README. Related docs:
- [Getting started](getting-started.md) — Compose paths
- [Operations](operations.md) — Docker Hub publish / release compose
- [Security](security.md) — lab credentials and trust boundary
docs/observability.md
+41
View File
@@ -0,0 +1,41 @@
# Observability
## Health
| Path | Meaning |
|---|---|
| `GET /health/live` | Process liveness |
| `GET /health/ready` | Database reachable **and** migrations current; HTTP 503 when not |
Example:
```bash
curl -s http://localhost:8006/health/live
curl -s http://localhost:8006/health/ready
```
## Request correlation
Incoming requests accept or generate an ID via `REQUEST_ID_HEADER`
(default `X-Request-ID`). Structured logs include correlation fields and redact
known secret patterns.
## Metrics / tracing
There is **no** Prometheus `/metrics` scrape endpoint and **no** bundled
OpenTelemetry exporter in the current application. Architecture notes that
mention them describe target design, not shipping telemetry.
Do not confuse Proxmox API paths under `/cluster/metrics` with simulator
process telemetry — those handlers simulate PVE metrics-server configuration
state inside PostgreSQL.
## Compatibility evidence
Operational compatibility reports:
- `/admin/compatibility`
- `/admin/compatibility.md`
- `/admin/compatibility.html`
Also available through the Web UI compatibility panel.
docs/operations.md
+138
View File
@@ -0,0 +1,138 @@
# Operations
## Day-2 commands
```bash
make up # start stack
make down # stop stack
make restart
make logs
make dev # foreground reload-oriented workflow
make db-migrate # idempotent migrations
make seed PROFILE=small # atomic reseed
make shell # interactive tools container
```
## Migrations
Ordered SQL files apply transactionally and record SHA-256 checksums.
Re-running `make db-migrate` is safe. Altering an already-applied migration is
rejected. `/health/ready` stays unavailable until the latest packaged migration
is present. Task workers retry claims after migrations catch up.
## Reseed
```bash
make seed PROFILE=medium
```
Reseed replaces mutable simulation state. External automation state (Terraform
state files, Pulumi stacks, Ansible inventories that encode VMIDs) may then
drift — refresh or recreate those side channels.
## Worker recovery
Workers use PostgreSQL leases. After a crash or restart, expired leases are
reclaimed and incomplete work can resume safely. Tunables:
`TASK_WORKER_CONCURRENCY`, `TASK_LEASE_SECONDS`, `SIMULATION_TIME_SCALE`.
## Changing the default API major
1. Prefer setting `CONTRACT_SNAPSHOT` to the desired bundled/normalized snapshot
for cold start (Compose / k8s / OpenShift).
2. Use Web UI apply or `POST /ui/api/contract/apply?major=N` for temporary
process-local switches.
## Backing up lab state
PostgreSQL is the system of record. Use normal Postgres backup/restore
(pg_dump / volume snapshots) if you need to preserve a seeded laboratory.
Application containers are disposable when the database volume remains.
## Publishing to Docker Hub
`make release` builds the **runtime** image (production target — not the local
bind-mounted `dev` image) and pushes it to Docker Hub:
```bash
docker login # once; account must own or can push to DOCKERHUB_USER
make release
```
Defaults:
| Variable | Default | Meaning |
|---|---|---|
| `DOCKERHUB_USER` | `inecs` | Docker Hub namespace/org |
| `IMAGE_NAME` | `proxmox-api-simulator` | Repository name |
| `VERSION` | from `pyproject.toml` | Image tag |
| `PUSH_LATEST` | `1` | Also tag/push `:latest` |
Examples:
```bash
make release
make release VERSION=0.2.0
make release DOCKERHUB_USER=myorg PUSH_LATEST=0
make release-build # build/tag locally without pushing
```
Published tags:
- `inecs/proxmox-api-simulator:<version>`
- `inecs/proxmox-api-simulator:latest` (unless `PUSH_LATEST=0`)
## Quick start with the published compose file
[`docker-compose.release.yml`](../docker-compose.release.yml) pulls the Hub
runtime image and starts PostgreSQL + migrate + simulator:
```bash
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl http://localhost:8006/health/ready
open http://localhost:8006/
```
Helpers from a git checkout:
```bash
make release-up
make release-seed PROFILE=small
make release-down
```
Useful overrides:
| Variable | Default | Meaning |
|---|---|---|
| `DOCKER_IMAGE` | `inecs/proxmox-api-simulator` | Image repository |
| `IMAGE_TAG` | `latest` | Tag to pull |
| `SIMULATOR_PORT` | `8006` | Host HTTP port |
| `TICKET_SIGNING_KEY` | lab default | Change outside toy labs |
| `POSTGRES_PASSWORD` | `proxmox` | DB password |
This stack is HTTP-only. The development Compose file still provides the local
HTTPS gateway on `:8007` for TLS-assuming clients.
For Kubernetes with public TLS (cert-manager / Let's Encrypt), use the Helm
chart — see [Kubernetes / Helm](kubernetes.md).
## Upgrades
1. Pull / rebuild images (`make install` / `make docker-build` as appropriate).
2. Run migrations.
3. Confirm `/health/ready`.
4. Re-check `/admin/compatibility` and `/api2/json/version`.
5. Re-run `make test-compatibility` if you validate external clients in CI.
## Resetting a lab
```bash
make seed PROFILE=small
# or via UI: unload demo → minimal, then seed again
```
For a hard database reset use `make db-reset` (destructive — see Makefile help).
docs/original-prompt-gap-plan.md
-150
View File
@@ -1,150 +0,0 @@
# Original prompt gap plan
This is the executable completion checklist for the original 1,985-line project
brief. It starts after release 0.1.0 and supersedes the short “subsequent
releases” list as the source of delivery status. A box is closed only when code,
tests, documentation, container acceptance, and a focused commit exist.
Status at audit time: 0.1.0 is operational, but the overall project is not done.
## G1 — measurable compatibility evidence
- [x] Model all 13 requested compatibility dimensions independently: route and
method, inputs, requiredness, types and constraints, HTTP status, JSON shape,
response field types, required response fields, headers and cookies, state
semantics, long-task behavior, errors and prohibitions, permissions.
- [x] Store test-derived evidence per method/profile instead of deriving strong
claims merely from handler presence.
- [x] Generate deterministic JSON, Markdown, and HTML reports with totals and
breakdowns by API group and PVE version.
- [x] Mark declared, schema-only, implemented, observed, partially compatible,
incompatible, and regression states without claiming unsupported semantics.
Exit: a test fixture can prove different scores for every dimension and the
live admin report renders the same evidence deterministically.
## G2 — persistence model and deterministic datasets
- [ ] Expand normalized tables/repositories for storages and contents, QEMU,
LXC, disks, NICs, snapshots, backups, pools, users/groups/roles/ACLs/tokens and
observed contracts. Preserve version, metadata, timestamps, and cluster-wide
VMID uniqueness.
- [x] Make migration readiness explicit so workers cannot become permanently
unhealthy before schema creation.
- [x] Match the required `small` profile (one node, two QEMU, one LXC, two
storages, administrator, completed tasks).
- [x] Implement deterministic `medium`, configurable batch-insert `large`,
`ha-demo`, and `broken-storage` profiles.
Exit: clean migration plus every seed profile has a stable logical snapshot;
large seeding proves bounded batch operations rather than row-at-a-time inserts.
## G3 — authentication and authorization surface
- [x] Expose API-token lifecycle and authenticate
`PVEAPIToken=USER@REALM!TOKENID=SECRET` without CSRF.
- [x] Complete pam, pve, and test realm behavior, ticket skew/expiry and
credential redaction.
- [x] Wire users, groups, roles, ACL propagation, route-derived permissions and
token privilege separation into every semantic handler.
- [x] Test root, audit-only, VM operator, storage user, separated token,
inheritance, denial, and existence-hiding behavior.
Exit: the complete credential/permission matrix passes through HTTP and no
plaintext password, ticket, CSRF token, or token secret reaches storage/logs.
## G4 — QEMU 0.2 verticals
- [x] Create, synchronous/asynchronous update, and delete.
- [x] Shutdown, reboot, reset, suspend and resume.
- [x] Snapshots and rollback.
- [x] Clone and local migration.
- [x] Resize and move disk.
- [ ] Remote migration, selected agent endpoints and pending/status data.
- [ ] Persist normalized CPU/memory/common fields plus unknown PVE parameters in
JSONB; simulate usage, uptime, PID, IO/network, lock, template, QMP, HA and
guest-agent availability.
- [ ] Cover concurrent start/delete, migrate/snapshot, optimistic conflict,
idempotency and restart recovery.
Exit: each operation is a complete contract/auth/permission/persistence/task/
state/error/test vertical and release 0.2.0 has a generated limitation matrix.
## G5 — LXC, storage, pools, backup and cluster 0.3
- [ ] LXC create/config/lifecycle/clone/migrate/snapshot/resize/delete.
- [ ] Storage list/status/content metadata, allocation/free, upload metadata,
ISO/template/backup listing and content deletion without large default blobs.
- [ ] Pools and membership, backup metadata/tasks, cluster status/nextid/options/
tasks/replication, and initial HA model/status.
Exit: release 0.3.0 passes client-level flows for every listed resource family.
## G6 — simulator administration, faults and virtual time 0.4
- [ ] Protected, disableable `/_simulator` API for state, reset, scenarios,
faults, compatibility and manual clock advancement.
- [ ] Deterministic rules filtered by route, method, principal, node, VMID, call
count, probability, time interval and scenario.
- [ ] Implement the specified node/storage/task/permission/migration/snapshot/
backup/HTTP/malformed/agent/lock/quorum/HA failures.
- [ ] Ensure simulation services use injected real, accelerated or manual clocks;
only lease internals use real monotonic time.
Exit: seeded scenario tests reproduce the same failures and durations across
runs; admin endpoints cannot overlap or weaken the PVE API boundary.
## G7 — safe recorder and differential laboratory 0.5
- [ ] Opt-in async passthrough/record fallbacks with official-lab allowlist,
production denylist, verified TLS, read-only default and explicit mutation
authorization.
- [ ] Sanitize credentials, cookies, tickets, CSRF, tokens, people, hosts and IPs
before fixtures can be persisted.
- [ ] Record request/response/latency/version/time/scenario metadata.
- [ ] Run identical lab/simulator requests using declarative normalization for
timestamps, UPIDs, PIDs, tokens, node values, uptime and resource usage.
- [ ] Produce JSON/Markdown/HTML reports with compatibility classes, regressions,
groups and versions.
Exit: secret-scanning fixtures and offline replay tests pass; normal startup has
no dependency on or route to a real Proxmox.
## G8 — profiles, observability and operational packaging
- [ ] Central `CompatibilityCapabilities` profiles for pve-8.4, pve-9.0,
pve-9.2 and custom; no scattered version-prefix conditions.
- [ ] Prometheus metrics requested by the brief, with bounded-cardinality labels,
plus optional OpenTelemetry tracing and optional Compose Prometheus/Grafana.
- [ ] Enrich safe structured logs with route template, principal, task/resource
context and stable error code.
- [ ] Add Docker labels/SBOM-friendly metadata and verify read-only/non-root
runtime, signals, one uvicorn process and multi-replica task leasing.
- [ ] Add Helm/Kubernetes Deployment, Service, ConfigMap, Secret, probes, PDB,
NetworkPolicy, hardened security context, resources, topology spread, separate
migration/seed Jobs and external PostgreSQL production configuration.
Exit: observability tests reject high-cardinality labels; chart lint/render and
multi-replica acceptance pass.
## G9 — client certification, security and 1.0 governance
- [ ] Add contract coverage for every imported route and critical-module
coverage of at least 90% while maintaining project coverage at least 80%.
- [ ] Certify documented versions of proxmoxer, HTTPX, Terraform provider and
Ansible modules only for the surfaces their flows exercise.
- [ ] Publish stable compatibility/migration policy and run dependency,
container, recorder, secret-handling and threat-model review.
- [ ] Expand README with architecture, version/profile choice, every seed,
scenarios, reports, recorder security, Kubernetes, commands and roadmap.
Exit: release 1.0.0 is reproducible from a clean checkout and all published
claims point to immutable machine-readable evidence.
## Global gate for every G-stage
Run formatting, Ruff, strict mypy, unit/integration/contract/compatibility tests,
coverage, `git diff --check`, relevant clean-container acceptance, documentation,
and a focused commit. No `pass`, TODO, `NotImplementedError`, sync network/DB IO,
`time.sleep`, plaintext secrets, silent error swallowing, or unsupported
compatibility claims may be introduced.
docs/security.md
+42
View File
@@ -0,0 +1,42 @@
# Security
## Lab threat model
This project is a **local / CI laboratory simulator**. It is not hardened as a
multi-tenant public Proxmox service. Default credentials, UI demo controls, and
compatibility endpoints are convenient for development and intentionally open
in the default Compose stack.
Do not expose ports `8006` / `8007` to untrusted networks without additional
controls you supply yourself.
## Credentials and secrets
- Passwords and API-token secrets are stored as scrypt hashes.
- Ticket values are HMAC-signed and short-lived.
- CSRF binds mutations to ticket sessions.
- Logs redact recognized ticket, password, and token representations.
- Create/regenerate token responses show the secret once; GET never echoes it.
Change `TICKET_SIGNING_KEY` for any shared lab. Replace seeded passwords and
tokens before demoing to others.
## TLS materials
`docker/tls/` contains a checked-in self-signed certificate for the local
gateway. It exists so unmodified TLS clients (e.g. proxmoxer) can connect.
**Never** reuse these files in production.
## Simulator administration
There is currently **no** separately authenticated `/_simulator` control plane.
Web UI helper routes under `/ui/api/*` and `/admin/compatibility*` are available
whenever the process is reachable. Treat network exposure as the trust
boundary.
## Simulated remotes
LDAP sync stamps, OpenID pending state, ACME, and Ceph endpoints persist local
simulator state only. They do not open real connections to external IdPs or
clusters. Do not rely on the simulator for testing live credential exfiltration
defenses against real providers.
docs/seed-profiles.md
+47
View File
@@ -0,0 +1,47 @@
# Seed profiles
Seeds replace mutable simulation state **atomically** and use deterministic
UUIDv5 identifiers so labs are reproducible.
```bash
make seed PROFILE=small
```
## Profiles
| Profile | Contents (summary) |
|---|---|
| `minimal` | One node `pve01`, `local` + `local-lvm` storage. Used after Web UI demo unload. |
| `small` | One node, QEMU `100`/`101`, LXC `200`, storages, completed tasks, full identity set. |
| `medium` | Three nodes, 50 QEMU, 20 LXC, local + shared storage, development pool, more tasks. |
| `large` | Configurable nodes/resources (`SEED_LARGE_NODES`, `SEED_LARGE_RESOURCES`, default 10 000 guests). |
| `ha-demo` | `medium` plus HA resource wiring for VM 100. |
| `broken-storage` | `small` with `local-lvm` offline / simulated I/O error. |
| `demo-cluster` | Large UI-oriented enterprise dataset (many nodes/guests/Ceph/HA/history). Prefer loading via the Web UI demo controls. |
## Examples
```bash
make seed PROFILE=small
make seed PROFILE=medium
make seed PROFILE=ha-demo
make seed PROFILE=broken-storage
make seed PROFILE=large
```
## Demo cluster via UI
The interactive console can load and unload the demo dataset:
- `POST /ui/api/demo/load`
- `POST /ui/api/demo/unload` — wipes API-created state then loads `minimal`
- `GET /ui/api/demo/state`
These UI helper endpoints are development-oriented and are not separately
authenticated today. Treat them as lab controls only.
## Reseed vs client state
Terraform, Pulumi, and Ansible may still hold resource state after a reseed.
Refresh or destroy/recreate external state after replacing the PostgreSQL
simulation contents. See [Operations](operations.md) and client cookbooks.
docs/troubleshooting.md
+57
View File
@@ -0,0 +1,57 @@
# Troubleshooting
## Ready stays unavailable
1. Confirm Postgres: `make logs` / Compose health.
2. Run `make db-migrate`.
3. Hit `/health/ready` again.
Workers may retry until migrations catch up after a late migrate.
## Unexpected HTTP 501
Declared methods on majors **6–9** should have handlers. If you see 501:
- Confirm the active runtime (`/api2/json/version` and Web UI runtime label).
- Confirm you are calling the path/verb exactly as declared for that major.
- Check `CONTRACT_FALLBACK` is not masking a different issue with fixture mode.
- Report a regression — full registry coverage is expected.
## 401 / 403
- Ticket expired or cookie not sent.
- Mutation missing `CSRFPreventionToken` on a ticket session.
- API token malformed (`PVEAPIToken=user@realm!id=secret`).
- ACL denial (try `auditor@pve` vs `root@pam` to compare).
## Task never finishes
- Inspect `/nodes/{node}/tasks/{upid}/status` and `/log`.
- Check worker logs (`make logs`).
- Verify `TASK_WORKER_CONCURRENCY` > 0 and database leases can be claimed.
- Extremely high `SIMULATION_TIME_SCALE` slowdowns are unusual (higher = faster
simulation); mis-set worker leases are more common culprits.
## proxmoxer / TLS failures
- Use host port **8007** (gateway), not 8006, for TLS clients.
- Set `verify_ssl=False` **only** for the local self-signed cert.
- Inside Compose, target `tls-gateway:8443`.
- Seeded node name for `small` is `pve01`, not `pve1`.
## Terraform / Pulumi drift after reseed
Reseed replaces PostgreSQL guests; tool state files do not. Refresh, import, or
rebuild stacks after `make seed`.
## Hot-swap “did nothing”
- Catalog browse ≠ apply. Use **Apply as runtime** or
`POST /ui/api/contract/apply?major=N`.
- Confirm with `/api2/json/version`.
- Remember apply is process-local; Compose restart restores `CONTRACT_SNAPSHOT`.
## Demo unload surprised you
`POST /ui/api/demo/unload` clears API-created state and loads `minimal`. Re-run
`make seed PROFILE=small` (or load demo again) to restore richer fixtures.
docs/web-ui.md
+60
View File
@@ -0,0 +1,60 @@
# Web UI
Open [http://localhost:8006/](http://localhost:8006/) after `make up`.
The UI is a laboratory console for the simulator — not a full Proxmox VE
management interface. It supports light and dark themes, PVE majors **6–9**,
request/response editing, history, and runtime contract apply.
## Screenshots
Light theme — `GET /cluster/resources` on PVE 9.2.3:
![Web UI light theme](images/web-ui-light.png)
Dark theme — same console with theme toggle:
![Web UI dark theme](images/web-ui-dark.png)
## Features
- Endpoint tree and method selector driven by the selected catalog major
- Contract-derived parameters and example payloads
- Request editor, response viewer, and history
- Password login with cookie + CSRF handling
- Environment summary (runtime version, nodes, guests, storage)
- Curl / request previews
- PVE **6–9** API catalog with implementation coverage
- **Apply as runtime** hot-swap for the active contract
- Compatibility and readiness views
- Demo-cluster load / unload / refresh
- Link to OpenAPI at `/docs`
## Backend helpers
| Method | Path | Purpose |
|---|---|---|
| GET | `/ui/api/versions` | Catalog majors vs runtime |
| GET | `/ui/api/catalog?major=N` | Catalog for major 6–9 |
| GET | `/ui/api/method?...` | Single method metadata |
| GET | `/ui/api/compatibility?major=N` | Coverage payload |
| POST | `/ui/api/contract/apply?major=N` | Hot-swap runtime contract |
| GET | `/ui/api/demo/state` | Demo dataset state |
| POST | `/ui/api/demo/load` | Load `demo-cluster` |
| POST | `/ui/api/demo/unload` | Unload → `minimal` |
## Version workflow
1. Pick major **6 / 7 / 8 / 9** in the catalog.
2. Inspect methods and coverage.
3. **Apply as runtime** when you want live `/api2/*` routes to match that major.
4. Confirm with `/api2/json/version` and `/admin/compatibility`.
Hot-swap is memory-only; restart restores `CONTRACT_SNAPSHOT`. Details:
[API versions](api-versions.md).
## Security note
UI and demo endpoints are intended for local development. They are not gated by
a separate admin token in the current build. Do not expose the simulator port to
untrusted networks.