inecs
131e2e63d2
Record offline 161 passed, green test-vsphere probes, and full pulumi-tests 7/7 with REST matrix 1987/1987 (critical=0); note the Terraform cookbook binary mismatch separately.
133 lines
5.2 KiB
Markdown
133 lines
5.2 KiB
Markdown
# pulumi hybrid lab suite for vmware-api-simulator
|
||
|
||
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
|
||
|
||
Hybrid suite under `pulumi-tests/`:
|
||
|
||
1. **Layer A (required for “100%”)** — HTTP contract matrix: all `IMPLEMENTED`
|
||
REST `verb+path` × catalog majors **6–9**, plus synthetic **HEAD** on every GET
|
||
2. **Layer B (smoke / lifecycle)** — official
|
||
[`pulumi-vsphere`](https://www.pulumi.com/registry/packages/vsphere/) programs
|
||
3. Deep REST CRUD (session / folder / tag / content library / VM)
|
||
4. SOAP WSDL ops already advertised on `/sdk/vimService.wsdl` (not a new SOAP universe)
|
||
|
||
`pulumi-vsphere` alone cannot hit ~1092 REST routes — **100% means the HTTP
|
||
matrix** (`coverage probed/declared`, `critical=0`), not provider resource count.
|
||
|
||
## Latest lab run (2026-07-18)
|
||
|
||
Full hybrid (`make pulumi-tests`, seed profile `small`):
|
||
|
||
| Check | Result |
|
||
|-------|--------|
|
||
| Cases | **7 / 7 passed** (`PU-INV`, `PU-FOLDER`, `PU-VM`, `PU-TAG`, `PU-REST`, `PU-CRUD`, `PU-SOAP`) |
|
||
| Layer A REST matrix | **1987 / 1987** (`critical=0`) across majors 6–9 |
|
||
| Verb histogram | GET 682 · HEAD 682 · POST 301 · PUT 96 · PATCH 91 · DELETE 135 |
|
||
|
||
Artifacts (local copy from the `lab-reports` volume; directory is gitignored):
|
||
`pulumi-tests/reports/pulumi-report.html`,
|
||
`pulumi-tests/reports/pulumi-summary.json`.
|
||
|
||
Related offline / gateway probes from the same day:
|
||
|
||
| Suite | Result |
|
||
|-------|--------|
|
||
| `make test` (offline unit + contract) | **161 passed** |
|
||
| `make test-vsphere` pytest + surface/matrix/dump/nonempty | **73** pytest passed; matrix/surface/nonempty **0 failures** |
|
||
| Client cookbooks (part of `test-vsphere`) | Python / Ansible URI / Pulumi-style OK; **Terraform cookbook** failed (`Exec format error` on `.tools/terraform` binary — host/arch mismatch, not an API regression) |
|
||
|
||
## Pass rules (Layer A)
|
||
|
||
| Check | Rule |
|
||
|-------|------|
|
||
| Declared | `methods_for_major(M)` minus session DELETE, plus `HEAD` per GET |
|
||
| Probed | Exactly one auth’d probe per declared route (`probed == declared`) |
|
||
| Critical | `0` — no 5xx/501; no `"stub": true` / empty body on inventory-critical GETs (major 9) |
|
||
| Coverage line | `{probed - critical}/{declared}` (aggregate + per major) |
|
||
| Verb histogram | Includes `HEAD` |
|
||
| Stubs | Spec-shaped durable JSON OK; reported as `stub`, not durable CRUD |
|
||
| Auth | Fresh `vmware-api-session-id` session for each major pass |
|
||
|
||
## Pass rules (Layer B + extras)
|
||
|
||
| Class | Pass rule |
|
||
|-------|-----------|
|
||
| pulumi-vsphere | Existing cases + nonempty exports (lifecycle smoke, not full API) |
|
||
| REST deep CRUD | create → GET nonempty → update (where supported) → delete → GET missing |
|
||
| SOAP WSDL ops | POST `/sdk` without 5xx; Create/Power/Clone/Destroy checked via inventory |
|
||
|
||
## What runs
|
||
|
||
| Case | Layer | Notes |
|
||
|------|-------|-------|
|
||
| `PU-INV` | B · pulumi-vsphere | Inventory data sources |
|
||
| `PU-FOLDER` | B · pulumi-vsphere | Folder create/destroy |
|
||
| `PU-VM` | B · pulumi-vsphere | VirtualMachine create/destroy |
|
||
| `PU-TAG` | B · pulumi-vsphere | TagCategory + Tag |
|
||
| `PU-REST` | **A · HTTP matrix** | `IMPLEMENTED` × majors 6–9 + HEAD (smoke: major 9) |
|
||
| `PU-CRUD` | Deep REST | Session, folder, tagging, content library, VM |
|
||
| `PU-SOAP` | SOAP | Existing WSDL ops only |
|
||
|
||
Artifacts: HTML + JSON + JUnit under the `lab-reports` volume (and optionally
|
||
`pulumi-tests/reports/` on the host). Look for `rest.coverage_line`,
|
||
`rest.critical`, `rest.by_verb` (incl. HEAD). Open the HTML report after a run:
|
||
|
||
```bash
|
||
mkdir -p pulumi-tests/reports
|
||
cd pulumi-tests && docker compose --profile test run --rm --no-deps \
|
||
-v "$(pwd)/reports:/out" pulumi-runner \
|
||
cp /reports/pulumi-report.html /reports/pulumi-summary.json /out/
|
||
open reports/pulumi-report.html
|
||
```
|
||
|
||
## Quick start
|
||
|
||
From the **repo root**:
|
||
|
||
```bash
|
||
make pulumi-tests # full hybrid suite
|
||
make pulumi-tests-smoke # PU-INV + Layer A major-9 smoke
|
||
```
|
||
|
||
Or from this directory:
|
||
|
||
```bash
|
||
cd pulumi-tests
|
||
make up
|
||
make test-pulumi # or: make test-pulumi-smoke
|
||
```
|
||
|
||
Gateway (in-compose): `https://api-gateway` (runner → service name).
|
||
Optional host map: `127.0.0.1:28443` (does **not** replace main lab
|
||
`https://localhost` / `:18443` — stop this stack with `make down` when done).
|
||
Seed profile: `small` → `Datacenter` / `Cluster` / `datastore1` / `VM Network` /
|
||
`web-01` / `esxi01.lab.local` / `/Datacenter/vm/production`.
|
||
|
||
## Make targets
|
||
|
||
| Target | Meaning |
|
||
|--------|---------|
|
||
| `make test-pulumi` / `pulumi-tests` | Full hybrid: Layer A ×6–9 + Layer B + CRUD + SOAP |
|
||
| `make test-pulumi-smoke` | `PU-INV` + Layer A major-9 smoke (no VM/tags/CRUD/SOAP) |
|
||
| `make up` / `down` / `seed` | Lab stack lifecycle |
|
||
|
||
## Layout
|
||
|
||
```
|
||
pulumi-tests/
|
||
run_suite.py # Automation API + REST/SOAP probes
|
||
report_html.py # coverage line + verb histogram
|
||
lib/assert_nonempty.py
|
||
lib/rest_matrix.py # Layer A: IMPLEMENTED × majors + HEAD
|
||
lib/rest_crud.py # deep create/read/update/delete
|
||
lib/soap_ops.py # WSDL SOAP ops
|
||
programs/inventory/
|
||
programs/folders/
|
||
programs/vm_lifecycle/
|
||
programs/tags/
|
||
docker/Dockerfile.pulumi-runner
|
||
docker-compose.yml
|
||
```
|
||
|
||
`PYTHONPATH` mounts `/workspace` so probes import `app.vsphere.*` coverage/matrix.
|