vmware-api-simulator/pulumi-tests

pulumi hybrid lab suite for vmware-api-simulator

Language / Язык: English | Русский

Hybrid suite under pulumi-tests/:

1. Layer A (required for “100%”) — HTTP contract matrix: all IMPLEMENTED REST verb+path × catalog majors 6–9, plus synthetic HEAD on every GET
2. Layer B (smoke / lifecycle) — official pulumi-vsphere programs
3. Deep REST CRUD (session / folder / tag / content library / VM)
4. SOAP WSDL ops already advertised on /sdk/vimService.wsdl (not a new SOAP universe)

pulumi-vsphere alone cannot hit ~1092 REST routes — 100% means the HTTP matrix (coverage probed/declared, critical=0), not provider resource count.

Latest lab run (2026-07-18)

Full hybrid (make pulumi-tests, seed profile small):

Check	Result
Cases	7 / 7 passed (PU-INV, PU-FOLDER, PU-VM, PU-TAG, PU-REST, PU-CRUD, PU-SOAP)
Layer A REST matrix	1987 / 1987 (critical=0) across majors 6–9
Verb histogram	GET 682 · HEAD 682 · POST 301 · PUT 96 · PATCH 91 · DELETE 135

Artifacts (local copy from the lab-reports volume; directory is gitignored): pulumi-tests/reports/pulumi-report.html, pulumi-tests/reports/pulumi-summary.json.

Related offline / gateway probes from the same day:

Suite	Result
make test (offline unit + contract)	161 passed
make test-vsphere pytest + surface/matrix/dump/nonempty	73 pytest passed; matrix/surface/nonempty 0 failures
Client cookbooks (part of test-vsphere)	Python / Ansible URI / Pulumi-style OK; Terraform cookbook failed (Exec format error on .tools/terraform binary — host/arch mismatch, not an API regression)

Pass rules (Layer A)

Check	Rule
Declared	methods_for_major(M) minus session DELETE, plus HEAD per GET
Probed	Exactly one auth’d probe per declared route (probed == declared)
Critical	0 — no 5xx/501; no "stub": true / empty body on inventory-critical GETs (major 9)
Coverage line	{probed - critical}/{declared} (aggregate + per major)
Verb histogram	Includes HEAD
Stubs	Spec-shaped durable JSON OK; reported as stub, not durable CRUD
Auth	Fresh vmware-api-session-id session for each major pass

Pass rules (Layer B + extras)

Class	Pass rule
pulumi-vsphere	Existing cases + nonempty exports (lifecycle smoke, not full API)
REST deep CRUD	create → GET nonempty → update (where supported) → delete → GET missing
SOAP WSDL ops	POST /sdk without 5xx; Create/Power/Clone/Destroy checked via inventory

What runs

Case	Layer	Notes
PU-INV	B · pulumi-vsphere	Inventory data sources
PU-FOLDER	B · pulumi-vsphere	Folder create/destroy
PU-VM	B · pulumi-vsphere	VirtualMachine create/destroy
PU-TAG	B · pulumi-vsphere	TagCategory + Tag
PU-REST	A · HTTP matrix	IMPLEMENTED × majors 6–9 + HEAD (smoke: major 9)
PU-CRUD	Deep REST	Session, folder, tagging, content library, VM
PU-SOAP	SOAP	Existing WSDL ops only

Artifacts: HTML + JSON + JUnit under the lab-reports volume (and optionally pulumi-tests/reports/ on the host). Look for rest.coverage_line, rest.critical, rest.by_verb (incl. HEAD). Open the HTML report after a run:

mkdir -p pulumi-tests/reports
cd pulumi-tests && docker compose --profile test run --rm --no-deps \
  -v "$(pwd)/reports:/out" pulumi-runner \
  cp /reports/pulumi-report.html /reports/pulumi-summary.json /out/
open reports/pulumi-report.html

Quick start

From the repo root:

make pulumi-tests          # full hybrid suite
make pulumi-tests-smoke    # PU-INV + Layer A major-9 smoke

Or from this directory:

cd pulumi-tests
make up
make test-pulumi           # or: make test-pulumi-smoke

Gateway (in-compose): https://api-gateway (runner → service name). Optional host map: 127.0.0.1:28443 (does not replace main lab https://localhost / :18443 — stop this stack with make down when done). Seed profile: small → Datacenter / Cluster / datastore1 / VM Network / web-01 / esxi01.lab.local / /Datacenter/vm/production.

Make targets

Target	Meaning
make test-pulumi / pulumi-tests	Full hybrid: Layer A ×6–9 + Layer B + CRUD + SOAP
make test-pulumi-smoke	PU-INV + Layer A major-9 smoke (no VM/tags/CRUD/SOAP)
make up / down / seed	Lab stack lifecycle

Layout

pulumi-tests/
  run_suite.py                 # Automation API + REST/SOAP probes
  report_html.py               # coverage line + verb histogram
  lib/assert_nonempty.py
  lib/rest_matrix.py           # Layer A: IMPLEMENTED × majors + HEAD
  lib/rest_crud.py             # deep create/read/update/delete
  lib/soap_ops.py              # WSDL SOAP ops
  programs/inventory/
  programs/folders/
  programs/vm_lifecycle/
  programs/tags/
  docker/Dockerfile.pulumi-runner
  docker-compose.yml

PYTHONPATH mounts /workspace so probes import app.vsphere.* coverage/matrix.