DevOpsTools/vmware-api-simulator
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Expand lab seed tiers, OpenAPI PARAMS, and console DATA/Params UX.

Browse Source
This commit is contained in:
Sergey Antropoff
2026-07-18 08:46:39 +03:00
parent f8d3cbdd59
commit 63cc409424
71 changed files with 38380 additions and 796 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pulumi-tests/README.md
+39 -24
View File
@@ -2,37 +2,50 @@
Hybrid suite under `pulumi-tests/`:
1. Official [`pulumi-vsphere`](https://www.pulumi.com/registry/packages/vsphere/) (SOAP/VIM via govmomi)
2. Full REST `IMPLEMENTED` × majors **6–9** matrix (deep + stub response checks)
1. **Layer A (required for “100%”)** — HTTP contract matrix: all `IMPLEMENTED`
REST `verb+path` × catalog majors **6–9**, plus synthetic **HEAD** on every GET
2. **Layer B (smoke / lifecycle)** — official
[`pulumi-vsphere`](https://www.pulumi.com/registry/packages/vsphere/) programs
3. Deep REST CRUD (session / folder / tag / content library / VM)
4. All SOAP WSDL ops advertised on `/sdk/vimService.wsdl`
4. SOAP WSDL ops already advertised on `/sdk/vimService.wsdl` (not a new SOAP universe)
`pulumi-vsphere` alone cannot hit ~1092 REST routes — the HTTP matrix is required for full-surface confidence.
`pulumi-vsphere` alone cannot hit ~1092 REST routes — **100% means the HTTP
matrix** (`coverage probed/declared`, `critical=0`), not provider resource count.
## Pass rules
## Pass rules (Layer A)
| Check | Rule |
|-------|------|
| Declared | `methods_for_major(M)` minus session DELETE, plus `HEAD` per GET |
| Probed | Exactly one auth’d probe per declared route (`probed == declared`) |
| Critical | `0` — no 5xx/501; no `"stub": true` / empty body on inventory-critical GETs (major 9) |
| Coverage line | `{probed - critical}/{declared}` (aggregate + per major) |
| Verb histogram | Includes `HEAD` |
| Stubs | Spec-shaped durable JSON OK; reported as `stub`, not durable CRUD |
| Auth | Fresh `vmware-api-session-id` session for each major pass |
## Pass rules (Layer B + extras)
| Class | Pass rule |
|-------|-----------|
| REST GET (inventory-critical / deep) | 2xx, body non-empty, not `"stub": true` on major 9 critical paths |
| REST stubs (universe) | no 5xx/501; response present; reported as `stub` (not claimed durable CRUD) |
| pulumi-vsphere | Existing cases + nonempty exports (lifecycle smoke, not full API) |
| REST deep CRUD | create → GET nonempty → update (where supported) → delete → GET missing |
| SOAP WSDL ops (~45) | POST `/sdk` without 5xx; Create/Power/Clone/Destroy checked via inventory |
| pulumi-vsphere | existing cases + nonempty exports |
| SOAP WSDL ops | POST `/sdk` without 5xx; Create/Power/Clone/Destroy checked via inventory |
## What runs
| Case | Layer | Notes |
|------|-------|-------|
| `PU-INV` | pulumi-vsphere | Inventory data sources |
| `PU-FOLDER` | pulumi-vsphere | Folder create/destroy |
| `PU-VM` | pulumi-vsphere | VirtualMachine create/destroy |
| `PU-TAG` | pulumi-vsphere | TagCategory + Tag |
| `PU-REST` | REST matrix | `IMPLEMENTED` × majors 6–9 (smoke: major 9 only) |
| `PU-CRUD` | REST CRUD | Session, folder, tagging, content library, VM |
| `PU-SOAP` | SOAP | All WSDL ops |
| `PU-INV` | B · pulumi-vsphere | Inventory data sources |
| `PU-FOLDER` | B · pulumi-vsphere | Folder create/destroy |
| `PU-VM` | B · pulumi-vsphere | VirtualMachine create/destroy |
| `PU-TAG` | B · pulumi-vsphere | TagCategory + Tag |
| `PU-REST` | **A · HTTP matrix** | `IMPLEMENTED` × majors 6–9 + HEAD (smoke: major 9) |
| `PU-CRUD` | Deep REST | Session, folder, tagging, content library, VM |
| `PU-SOAP` | SOAP | Existing WSDL ops only |
Artifacts: HTML + JSON + JUnit under the `lab-reports` volume. JSON includes
`rest.total` / `rest.failed`, `crud.failed`, `soap.failed`.
Artifacts: HTML + JSON + JUnit under the `lab-reports` volume. Look for
`rest.coverage_line`, `rest.critical`, `rest.by_verb` (incl. HEAD).
## Quick start
@@ -40,7 +53,7 @@ From the **repo root**:
```bash
make pulumi-tests # full hybrid suite
make pulumi-tests-smoke # PU-INV + one-major REST smoke
make pulumi-tests-smoke # PU-INV + Layer A major-9 smoke
```
Or from this directory:
@@ -51,7 +64,9 @@ make up
make test-pulumi # or: make test-pulumi-smoke
```
Gateway (in-compose): `https://api-gateway` (host map `127.0.0.1:18443`).
Gateway (in-compose): `https://api-gateway` (runner → service name).
Optional host map: `127.0.0.1:28443` (does **not** replace main lab
`https://localhost` / `:18443` — stop this stack with `make down` when done).
Seed profile: `small` → `Datacenter` / `Cluster` / `datastore1` / `VM Network` /
`web-01` / `esxi01.lab.local` / `/Datacenter/vm/production`.
@@ -59,8 +74,8 @@ Seed profile: `small` → `Datacenter` / `Cluster` / `datastore1` / `VM Network`
| Target | Meaning |
|--------|---------|
| `make test-pulumi` / `pulumi-tests` | Full hybrid: pulumi-vsphere + REST×6–9 + CRUD + SOAP |
| `make test-pulumi-smoke` | `PU-INV` + REST major-9 smoke (no VM/tags/CRUD/SOAP) |
| `make test-pulumi` / `pulumi-tests` | Full hybrid: Layer A ×6–9 + Layer B + CRUD + SOAP |
| `make test-pulumi-smoke` | `PU-INV` + Layer A major-9 smoke (no VM/tags/CRUD/SOAP) |
| `make up` / `down` / `seed` | Lab stack lifecycle |
## Layout
@@ -68,9 +83,9 @@ Seed profile: `small` → `Datacenter` / `Cluster` / `datastore1` / `VM Network`
```
pulumi-tests/
run_suite.py # Automation API + REST/SOAP probes
report_html.py
report_html.py # coverage line + verb histogram
lib/assert_nonempty.py
lib/rest_matrix.py # IMPLEMENTED × majors
lib/rest_matrix.py # Layer A: IMPLEMENTED × majors + HEAD
lib/rest_crud.py # deep create/read/update/delete
lib/soap_ops.py # WSDL SOAP ops
programs/inventory/
pulumi-tests/README.ru.md
+23 -58
View File
@@ -2,77 +2,42 @@
Гибрид под `pulumi-tests/`:
1. Официальный [`pulumi-vsphere`](https://www.pulumi.com/registry/packages/vsphere/) (SOAP/VIM через govmomi)
2. Полная REST-матрица `IMPLEMENTED` × majors **6–9** (deep + stub)
1. **Layer A (обязателен для «100%»)** — HTTP contract matrix: все `IMPLEMENTED`
REST `verb+path` × majors **6–9**, плюс синтетический **HEAD** на каждый GET
2. **Layer B (smoke / lifecycle)** — официальный
[`pulumi-vsphere`](https://www.pulumi.com/registry/packages/vsphere/)
3. Deep REST CRUD (session / folder / tag / content library / VM)
4. Все SOAP-операции из `/sdk/vimService.wsdl`
4. SOAP-операции из `/sdk/vimService.wsdl` (без нового SOAP universe)
Одним `pulumi-vsphere` закрыть ~1092 REST `verb×path` нельзя — HTTP-матрица обязательна для полной уверенности.
Одним `pulumi-vsphere` закрыть ~1092 REST `verb×path` нельзя — **100% = HTTP-матрица**
(`coverage probed/declared`, `critical=0`), а не число provider-ресурсов.
## Критерии pass
## Критерии pass (Layer A)
| Класс | Правило |
|-------|---------|
| REST GET (inventory / deep) | 2xx, непустое тело, без `"stub": true` на critical-путях major 9 |
| REST stubs (universe) | нет 5xx/501; ответ есть; в отчёте как `stub` (не durable CRUD) |
| REST deep CRUD | create → GET nonempty → update (если есть) → delete → GET missing |
| SOAP WSDL (~45) | POST `/sdk` без 5xx; Create/Power/Clone/Destroy — проверка inventory |
| pulumi-vsphere | существующие кейсы + nonempty export’ы |
| Проверка | Правило |
|----------|---------|
| Declared | `methods_for_major(M)` минус session DELETE, плюс `HEAD` на каждый GET |
| Probed | Ровно один auth’d probe на каждый declared (`probed == declared`) |
| Critical | `0` — нет 5xx/501; нет `"stub": true` / пустого тела на inventory-critical GET (major 9) |
| Coverage | `{probed - critical}/{declared}` (aggregate + per major) |
| Verb histogram | Включая `HEAD` |
## Что запускается
| Кейс | Слой | Заметки |
|------|------|---------|
| `PU-INV` | pulumi-vsphere | Inventory data sources |
| `PU-FOLDER` | pulumi-vsphere | Folder create/destroy |
| `PU-VM` | pulumi-vsphere | VirtualMachine create/destroy |
| `PU-TAG` | pulumi-vsphere | TagCategory + Tag |
| `PU-REST` | REST matrix | `IMPLEMENTED` × majors 6–9 (smoke: только major 9) |
| `PU-CRUD` | REST CRUD | Session, folder, tagging, content library, VM |
| `PU-SOAP` | SOAP | Все WSDL ops |
Артефакты: HTML + JSON + JUnit в volume `lab-reports`. В JSON:
`rest.total` / `rest.failed`, `crud.failed`, `soap.failed`.
| `PU-INV` … `PU-TAG` | B · pulumi-vsphere | Lifecycle / inventory smoke |
| `PU-REST` | **A · HTTP matrix** | `IMPLEMENTED` × 6–9 + HEAD (smoke: major 9) |
| `PU-CRUD` / `PU-SOAP` | extras | Deep CRUD + существующий SOAP |
## Быстрый старт
Из **корня репозитория**:
```bash
make pulumi-tests # полный гибрид
make pulumi-tests-smoke # PU-INV + REST smoke на одном major
make pulumi-tests-smoke # PU-INV + Layer A major-9
```
Или из этой директории:
```bash
cd pulumi-tests
make up
make test-pulumi # или: make test-pulumi-smoke
```
Шлюз (в compose): `https://api-gateway` (с хоста `127.0.0.1:18443`).
Seed-профиль: `small` → `Datacenter` / `Cluster` / `datastore1` / `VM Network` /
`web-01` / `esxi01.lab.local` / `/Datacenter/vm/production`.
## Цели Make
| Цель | Смысл |
|------|--------|
| `make test-pulumi` / `pulumi-tests` | Полный гибрид: pulumi-vsphere + REST×6–9 + CRUD + SOAP |
| `make test-pulumi-smoke` | `PU-INV` + REST major-9 smoke (без VM/tags/CRUD/SOAP) |
| `make up` / `down` / `seed` | Жизненный цикл lab-стека |
## Структура
```
pulumi-tests/
run_suite.py
report_html.py
lib/rest_matrix.py
lib/rest_crud.py
lib/soap_ops.py
programs/...
```
`PYTHONPATH` монтирует `/workspace` для импорта `app.vsphere.*`.
Шлюз (в compose): `https://api-gateway`. Хост-алиас: `127.0.0.1:28443`
(не занимает `https://localhost` / `:18443` основного lab — после тестов
`make down`).
Seed: `small`. Артефакты: `rest.coverage_line`, `rest.critical`, `rest.by_verb`.
pulumi-tests/docker-compose.yml
+3 -1
View File
@@ -146,7 +146,9 @@ services:
- ../docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
- ../docker/tls/server.key:/etc/nginx/tls/server.key:ro
ports:
- "127.0.0.1:18443:443"
# Host-only alias; keep off 443/18443 so this stack can coexist with the
# main repo compose (https://localhost) without stealing the lab gateway.
- "127.0.0.1:28443:443"
pulumi-runner:
build:
pulumi-tests/lib/rest_matrix.py
+168 -131
View File
@@ -1,7 +1,12 @@
"""Full REST verb×path×majors matrix for pulumi-tests (hybrid suite).
"""Full REST verb×path×majors matrix for pulumi-tests (Layer A HTTP contract).
Reuses path substitution / session patterns from scripts/vsphere_full_matrix_probe.py.
Pass rules match that probe: no 5xx/501; inventory GETs nonempty + non-stub on major 9.
Probes ``methods_for_major(M)`` for majors 6–9 with session auth.
Verbs: whatever is in IMPLEMENTED (GET/PUT/PATCH/POST/DELETE) plus a synthetic
HEAD for every GET path. Pass requires ``critical == 0`` and
``probed == declared`` per major and in aggregate.
100% coverage here means the HTTP contract matrix — not pulumi-vsphere resource
count (Layer B).
"""
from __future__ import annotations
@@ -18,7 +23,7 @@ from collections import Counter
from typing import Any
from urllib.parse import urlencode
from app.vsphere.contracts.matrix import VERSIONS, catalog_entries_for_major, methods_for_major
from app.vsphere.contracts.matrix import VERSIONS, methods_for_major
from app.vsphere.rest.coverage import CORE_IMPLEMENTED, IMPLEMENTED
_PATH_SUBS = {
@@ -30,6 +35,7 @@ _PATH_SUBS = {
"{category_id}": "cat-lab-1",
"{tag_id}": "tag-lab-1",
"{item_id}": "item-ubuntu",
"{library_item_id}": "item-ubuntu",
"{library_id}": "lib-local-1",
"{folder}": "group-v23",
"{datacenter}": "datacenter-21",
@@ -75,20 +81,47 @@ _PATH_SUBS = {
_ACCEPT_CLIENT = {400, 401, 403, 404, 405, 409, 412, 422}
_SKIP_DELETE = frozenset(
{
("DELETE", "/api/session"),
("DELETE", "/rest/com/vmware/cis/session"),
}
)
# Collection / entity GETs that must be non-empty after small seed (major 9).
_INVENTORY_CRITICAL = {
"/api/vcenter/vm",
"/api/vcenter/host",
"/api/vcenter/datastore",
"/api/vcenter/network",
"/api/vcenter/cluster",
"/api/vcenter/datacenter",
"/api/vcenter/folder",
"/api/vcenter/resource-pool",
"/api/cis/tagging/category",
"/api/cis/tagging/tag",
"/api/content/library",
"/api/content/local-library",
"/api/vcenter/network/dvs",
"/api/vcenter/storage/policies",
"/api/vcenter/privilege",
"/api/vcenter/authorization/roles",
"/api/vcenter/authorization/permissions",
"/api/cis/tasks",
"/api/esx/settings/clusters/{cluster}/software",
"/api/vcenter/namespace-management/supervisors/{supervisor}/summary",
"/api/appliance/access/ssh",
"/api/appliance/services",
}
# GET paths that may legally return empty/null bodies.
_EMPTY_OK_GET = frozenset(
{
"/api/session",
"/rest/com/vmware/cis/session",
}
)
def _base() -> str:
explicit = os.environ.get("VSPHERE_BASE")
@@ -127,7 +160,6 @@ def request(
headers: dict[str, str],
data: bytes | None = None,
) -> tuple[int, str]:
# Paths may already include query strings from _payload_for.
if "?" in path:
base_path, query = path.split("?", 1)
url = f"{_base()}{concrete_path(base_path)}?{query}"
@@ -152,6 +184,16 @@ def login() -> str:
return json.loads(body)
def declared_routes(major: int) -> list[tuple[str, str]]:
"""Registry routes for a major, plus synthetic HEAD for each GET."""
methods = methods_for_major(major)
base = [(verb, path) for (verb, path) in sorted(methods) if (verb, path) not in _SKIP_DELETE]
heads = [("HEAD", path) for verb, path in base if verb == "GET"]
# Keep HEAD adjacent after its GET in verb order during probe via sort key.
return base + heads
def _payload_for(verb: str, path: str) -> tuple[str, bytes | None]:
if verb not in {"POST", "PUT", "PATCH"}:
return path, None
@@ -165,7 +207,14 @@ def _payload_for(verb: str, path: str) -> tuple[str, bytes | None]:
return f"{path}?action=enter", b"{}"
if path.endswith("/folder/{folder}") and verb == "POST":
return f"{path}?action=rename", json.dumps({"name": "folder-renamed-probe"}).encode()
# Do not rename seed folder MOIDs (breaks /Datacenter/vm/... inventory paths).
return (
"/api/vcenter/folder/folder-missing-matrix?action=rename",
json.dumps({"name": "folder-renamed-probe"}).encode(),
)
if path == "/api/cis/tasks" and verb == "POST":
return f"{path}?action=list", json.dumps({"filter_spec": {}}).encode()
suffix = secrets.token_hex(4)
bodies: dict[str, dict[str, Any]] = {
@@ -175,10 +224,15 @@ def _payload_for(verb: str, path: str) -> tuple[str, bytes | None]:
"cpu_count": 1,
"memory_size_MiB": 512,
},
"/api/vcenter/datacenter": {"name": f"probe-dc-{suffix}"},
"/api/vcenter/cluster": {"name": f"probe-cluster-{suffix}"},
# Missing parent → 404 (client_4xx). Avoid creating extra Datacenter/Cluster
# trees that leave a second ResourcePool named "Resources" for govmomi.
"/api/vcenter/datacenter": {"name": f"probe-dc-{suffix}", "folder": "folder-missing-matrix"},
"/api/vcenter/cluster": {"name": f"probe-cluster-{suffix}", "folder": "folder-missing-matrix"},
"/api/vcenter/folder": {"name": f"probe-folder-{suffix}", "parent": "group-v23"},
"/api/vcenter/resource-pool": {"name": f"probe-rp-{suffix}", "parent": "resgroup-22"},
"/api/vcenter/resource-pool": {
"name": f"probe-rp-{suffix}",
"parent": "resgroup-missing-matrix",
},
"/api/vcenter/network/dvs": {"name": f"probe-dvs-{suffix}"},
"/api/vcenter/network/dvpg": {
"name": f"probe-dvpg-{suffix}",
@@ -263,14 +317,25 @@ def apply_major(major: int, headers: dict[str, str]) -> dict[str, Any]:
def _classify(verb: str, path: str) -> str:
status = CORE_IMPLEMENTED.get((verb, path)) or IMPLEMENTED.get((verb, path))
if (verb, path) in CORE_IMPLEMENTED:
lookup = verb if verb != "HEAD" else "GET"
if (lookup, path) in CORE_IMPLEMENTED:
return "deep"
status = IMPLEMENTED.get((lookup, path))
if status == "stub":
return "stub"
return "deep" if status == "implemented" else "unknown"
def _is_empty_payload(body: str) -> bool:
if not body or not body.strip():
return True
try:
parsed = json.loads(body)
except json.JSONDecodeError:
return False
return parsed in ([], {}, None, "")
def _record_result(
*,
major: int,
@@ -284,82 +349,55 @@ def _record_result(
) -> None:
kind = _classify(verb, path)
deep_stub[kind] += 1
def _fail(expected: str, *, bucket: str) -> None:
buckets[bucket] += 1
failures.append(
{
"major": major,
"verb": verb,
"path": path,
"kind": kind,
"status": code,
"critical": True,
"body": body[:200],
"expected": expected,
}
)
if 200 <= code < 300:
buckets["success_2xx"] += 1
if kind == "stub":
buckets["stub_ok"] += 1
if major == 9 and verb == "GET" and body:
# HEAD bodies are always empty by design.
if verb == "HEAD":
return
if major == 9 and verb == "GET" and path not in _EMPTY_OK_GET:
if '"stub": true' in body or '"stub":true' in body:
if path in _INVENTORY_CRITICAL or kind == "deep":
buckets["stub_marker"] += 1
failures.append(
{
"major": major,
"verb": verb,
"path": path,
"kind": kind,
"status": code,
"body": body[:200],
"expected": "non-stub JSON",
}
)
elif path in _INVENTORY_CRITICAL:
try:
parsed = json.loads(body)
except json.JSONDecodeError:
parsed = None
empty = parsed in ([], {}, None, "")
if empty:
buckets["empty_inventory"] += 1
failures.append(
{
"major": major,
"verb": verb,
"path": path,
"kind": kind,
"status": code,
"body": body[:200],
"expected": "non-empty seeded data",
}
)
elif code in _ACCEPT_CLIENT:
_fail("non-stub JSON", bucket="stub_marker")
elif path in _INVENTORY_CRITICAL and _is_empty_payload(body):
_fail("non-empty seeded data", bucket="empty_inventory")
return
# Synthetic HEAD must be served (middleware); 405 is critical for HEAD only.
if verb == "HEAD" and code == 405:
_fail("HEAD supported via GET route", bucket="head_405")
return
if code in _ACCEPT_CLIENT:
buckets["client_4xx"] += 1
elif code == 501:
buckets["unexpected_501"] += 1
failures.append(
{
"major": major,
"verb": verb,
"path": path,
"kind": kind,
"status": code,
"body": body[:200],
}
)
elif code >= 500:
buckets["server_5xx"] += 1
failures.append(
{
"major": major,
"verb": verb,
"path": path,
"kind": kind,
"status": code,
"body": body[:200],
}
)
else:
buckets[f"other_{code}"] += 1
failures.append(
{
"major": major,
"verb": verb,
"path": path,
"kind": kind,
"status": code,
"body": body[:200],
}
)
return
if code == 501:
_fail("no 501 on registered route", bucket="unexpected_501")
return
if code >= 500:
_fail("no 5xx", bucket="server_5xx")
return
_fail(f"unexpected status {code}", bucket=f"other_{code}")
def probe_major(major: int, session: str) -> dict[str, Any]:
@@ -369,12 +407,11 @@ def probe_major(major: int, session: str) -> dict[str, Any]:
"Accept": "application/json",
}
applied = apply_major(major, headers)
active = methods_for_major(major)
verb_order = {"GET": 0, "PUT": 1, "PATCH": 2, "POST": 3, "DELETE": 4}
entries = sorted(
catalog_entries_for_major(major),
key=lambda e: (verb_order.get(e["verb"], 9), e["path"]),
)
routes = declared_routes(major)
declared = len(routes)
verb_order = {"GET": 0, "HEAD": 1, "PUT": 2, "PATCH": 3, "POST": 4, "DELETE": 5}
routes = sorted(routes, key=lambda item: (verb_order.get(item[0], 9), item[1]))
buckets: Counter[str] = Counter()
deep_stub: Counter[str] = Counter()
@@ -382,30 +419,35 @@ def probe_major(major: int, session: str) -> dict[str, Any]:
probed = 0
by_verb: Counter[str] = Counter()
for entry in entries:
verb = entry["verb"]
path = entry["path"]
for verb, path in routes:
by_verb[verb] += 1
if verb == "DELETE" and path in {"/api/session", "/rest/com/vmware/cis/session"}:
continue
if verb == "DELETE" and path in {
"/api/vcenter/datacenter/{datacenter}",
"/api/vcenter/cluster/{cluster}",
"/api/vcenter/folder/{folder}",
"/api/vcenter/resource-pool/{resource_pool}",
"/api/vcenter/vm/{vm}",
"/api/content/local-library/{library_id}",
"/api/content/library/item/{library_item_id}",
}:
if path.endswith("{vm}"):
url_path = path.replace("{vm}", "vm-missing-matrix")
elif path.endswith("{datacenter}"):
url_path = path.replace("{datacenter}", "dc-missing")
elif path.endswith("{cluster}"):
url_path = path.replace("{cluster}", "cluster-missing")
elif path.endswith("{folder}"):
url_path = path.replace("{folder}", "folder-missing")
else:
url_path = path.replace("{resource_pool}", "rp-missing")
# Avoid destroying seed MOIDs — probe missing ids (expect 404).
url_path = path
for token, missing in (
("{vm}", "vm-missing-matrix"),
("{datacenter}", "dc-missing"),
("{cluster}", "cluster-missing"),
("{folder}", "folder-missing"),
("{resource_pool}", "rp-missing"),
("{library_id}", "lib-missing-matrix"),
("{library_item_id}", "item-missing-matrix"),
):
url_path = url_path.replace(token, missing)
code, body = request(verb, url_path, headers=headers)
elif verb == "HEAD":
url_path = path
if path == "/api/content/library/item":
url_path = f"{path}?library_id=lib-local-1"
code, body = request("HEAD", url_path, headers=headers)
else:
url_path, data = _payload_for(verb, path)
if verb == "GET" and path == "/api/content/library/item":
@@ -424,44 +466,28 @@ def probe_major(major: int, session: str) -> dict[str, Any]:
deep_stub=deep_stub,
)
above_floor = 0
for (verb, path), _status in sorted(IMPLEMENTED.items()):
if (verb, path) in active:
continue
if verb == "DELETE" and path in {"/api/session", "/rest/com/vmware/cis/session"}:
continue
url_path, data = _payload_for(verb, path)
code, body = request(verb, url_path, headers=headers, data=data)
above_floor += 1
probed += 1
by_verb[verb] += 1
_record_result(
major=major,
verb=verb,
path=path,
code=code,
body=body,
buckets=buckets,
failures=failures,
deep_stub=deep_stub,
)
critical = sum(1 for f in failures if f.get("critical"))
coverage_ok = probed == declared and critical == 0
return {
"major": major,
"version": applied.get("runtime_version"),
"method_count": len(entries),
"declared": declared,
"method_count": len(methods_for_major(major)),
"by_verb": dict(by_verb),
"probed": probed,
"above_floor_checked": above_floor,
"probed_eq_declared": probed == declared,
"buckets": dict(buckets),
"deep_vs_stub": dict(deep_stub),
"failures": failures,
"failed": len(failures),
"critical": critical,
"coverage_line": f"{probed - critical}/{declared}",
"ok": coverage_ok,
}
def run_rest_matrix(*, majors: list[int] | None = None) -> dict[str, Any]:
"""Probe IMPLEMENTED × majors. Returns summary suitable for suite JSON/HTML."""
"""Probe IMPLEMENTED × majors (+ HEAD). Returns summary for suite JSON/HTML."""
if majors is None:
majors = [6, 7, 8, 9]
@@ -473,25 +499,36 @@ def run_rest_matrix(*, majors: list[int] | None = None) -> dict[str, Any]:
reports: list[dict[str, Any]] = []
all_failures: list[dict[str, Any]] = []
verb_totals: Counter[str] = Counter()
declared_total = 0
probed_total = 0
critical_total = 0
for major in majors:
report = probe_major(major, session)
reports.append(report)
all_failures.extend(report["failures"])
declared_total += int(report["declared"])
probed_total += int(report["probed"])
critical_total += int(report["critical"])
for verb, count in report["by_verb"].items():
verb_totals[verb] += count
session = login()
apply_major(9, {"vmware-api-session-id": session, "Content-Type": "application/json"})
total = sum(r["probed"] for r in reports)
failed = len(all_failures)
coverage_line = f"{probed_total - critical_total}/{declared_total}"
ok = critical_total == 0 and probed_total == declared_total and all(r["ok"] for r in reports)
return {
"base": _base(),
"majors": reports,
"by_verb": dict(verb_totals),
"total": total,
"failed": failed,
"declared": declared_total,
"probed": probed_total,
"total": probed_total,
"critical": critical_total,
"failed": critical_total,
"failures": all_failures[:120],
"ok": failed == 0,
"coverage_line": coverage_line,
"probed_eq_declared": probed_total == declared_total,
"ok": ok,
}
pulumi-tests/programs/inventory/Pulumi.dbg-inv.yaml
+1
View File
@@ -0,0 +1 @@
encryptionsalt: v1:plwEBConp18=:v1:ZsvOhY9oF/fv3+9T:mfrqPTkwm7ehCrIB2RyY7nM1YyPelg==
pulumi-tests/programs/inventory/Pulumi.dbg-inv2.yaml
+1
View File
@@ -0,0 +1 @@
encryptionsalt: v1:YdNhRHu/0Mg=:v1:3cfGg0xOH+zbxrg5:cMV7UqHRgphaG0bTa39P+MC2eZi+8w==
pulumi-tests/programs/inventory/Pulumi.dbg-inv3.yaml
+1
View File
@@ -0,0 +1 @@
encryptionsalt: v1:xkpCARg0inY=:v1:+gu7XsBoXJf1x1R8:UewZrU1hON5hMYxbmVd24oXzxQDZXA==
pulumi-tests/programs/inventory/__main__.py
+3 -1
View File
@@ -56,8 +56,10 @@ host = vsphere.get_host_output(
opts=invoke_opts,
)
folder = vsphere.get_folder_output(path=folder_path, opts=invoke_opts)
# Prefer cluster-scoped path so a polluted lab with multiple "Resources" pools
# (e.g. leftover probe datacenters) still resolves uniquely.
pool = vsphere.get_resource_pool_output(
name=pool_name,
name=f"{cluster_name}/{pool_name}",
datacenter_id=dc.id,
opts=invoke_opts,
)
pulumi-tests/report_html.py
+16 -9
View File
@@ -17,6 +17,8 @@ def _section_rest(summary: dict[str, Any]) -> str:
majors = rest.get("majors") or []
by_verb = rest.get("by_verb") or {}
failures = rest.get("failures") or []
coverage = rest.get("coverage_line") or f"{(rest.get('probed') or 0) - (rest.get('critical') or 0)}/{rest.get('declared') or rest.get('total') or 0}"
critical = rest.get("critical", rest.get("failed", 0))
major_rows = []
for m in majors:
@@ -26,8 +28,10 @@ def _section_rest(summary: dict[str, Any]) -> str:
"<tr>"
f"<td>{_esc(m.get('major'))}</td>"
f"<td>{_esc(m.get('version'))}</td>"
f"<td>{_esc(m.get('declared'))}</td>"
f"<td>{_esc(m.get('probed'))}</td>"
f"<td>{_esc(m.get('failed'))}</td>"
f"<td>{_esc(m.get('coverage_line') or '—')}</td>"
f"<td>{_esc(m.get('critical', m.get('failed')))}</td>"
f"<td>{_esc(buckets.get('success_2xx', 0))}</td>"
f"<td>{_esc(buckets.get('client_4xx', 0))}</td>"
f"<td>{_esc(buckets.get('server_5xx', 0))}</td>"
@@ -50,22 +54,25 @@ def _section_rest(summary: dict[str, Any]) -> str:
)
return f"""
<h2>REST matrix</h2>
<p class="meta">total={_esc(rest.get("total"))} · failed={_esc(rest.get("failed"))}
· verbs: {verb_bits or "—"}</p>
<h2>REST HTTP contract matrix (Layer A)</h2>
<p class="meta"><strong>coverage {_esc(coverage)}</strong>
· critical={_esc(critical)}
· probed={_esc(rest.get("probed") or rest.get("total"))}
· declared={_esc(rest.get("declared"))}
· verbs: {verb_bits or "—"}
<br/>100% = HTTP matrix (IMPLEMENTED × majors + HEAD), not pulumi-vsphere resource count.</p>
<table>
<thead><tr><th>Major</th><th>Version</th><th>Probed</th><th>Failed</th>
<th>2xx</th><th>4xx</th><th>5xx</th><th>deep/stub</th></tr></thead>
<tbody>{"".join(major_rows) or '<tr><td colspan="8">—</td></tr>'}</tbody>
<thead><tr><th>Major</th><th>Version</th><th>Declared</th><th>Probed</th><th>Coverage</th>
<th>Critical</th><th>2xx</th><th>4xx</th><th>5xx</th><th>deep/stub</th></tr></thead>
<tbody>{"".join(major_rows) or '<tr><td colspan="10">—</td></tr>'}</tbody>
</table>
<h3>REST failures (sample)</h3>
<h3>REST critical failures (sample)</h3>
<table>
<thead><tr><th>Major</th><th>Verb</th><th>Path</th><th>Kind</th><th>Status</th><th>Detail</th></tr></thead>
<tbody>{"".join(fail_rows) or '<tr><td colspan="6">none</td></tr>'}</tbody>
</table>
"""
def _section_crud(summary: dict[str, Any]) -> str:
crud = summary.get("crud") or {}
flows = crud.get("flows") or []
pulumi-tests/run_suite.py
+17 -5
View File
@@ -188,18 +188,24 @@ def _run_rest_matrix() -> dict:
}
status = "passed" if summary.get("ok") else "failed"
err = ""
coverage = summary.get("coverage_line") or "?"
if not summary.get("ok"):
sample = summary.get("failures") or []
err = f"REST matrix failures={summary.get('failed')} total={summary.get('total')}; "
err = (
f"REST matrix coverage={coverage} critical={summary.get('critical')} "
f"probed={summary.get('probed')} declared={summary.get('declared')}; "
)
err += "; ".join(f"{f.get('verb')} {f.get('path')} → {f.get('status')}" for f in sample[:8])
return {
"id": "PU-REST",
"title": f"REST IMPLEMENTED×majors {majors} (deep+stub response check)",
"title": f"REST HTTP matrix majors={majors} (IMPLEMENTED + HEAD; Layer A)",
"status": status,
"error": err,
"outputs": {
"total": summary.get("total"),
"failed": summary.get("failed"),
"coverage": coverage,
"critical": summary.get("critical"),
"probed": summary.get("probed"),
"declared": summary.get("declared"),
"by_verb": summary.get("by_verb"),
"majors": [m.get("major") for m in summary.get("majors") or []],
},
@@ -341,7 +347,12 @@ def main() -> int:
"total_failed": failed,
"rest": {
"total": rest_summary.get("total", 0),
"probed": rest_summary.get("probed", rest_summary.get("total", 0)),
"declared": rest_summary.get("declared", 0),
"critical": rest_summary.get("critical", rest_summary.get("failed", 0)),
"failed": rest_summary.get("failed", 0),
"coverage_line": rest_summary.get("coverage_line"),
"probed_eq_declared": rest_summary.get("probed_eq_declared"),
"by_verb": rest_summary.get("by_verb"),
"majors": rest_summary.get("majors"),
"failures": rest_summary.get("failures"),
@@ -386,7 +397,8 @@ def main() -> int:
print(f"Wrote {JUNIT_PATH}", flush=True)
print(
f"SUMMARY failed={failed} total={len(results)} "
f"rest.failed={summary['rest']['failed']} "
f"rest.coverage={summary['rest'].get('coverage_line')} "
f"rest.critical={summary['rest']['critical']} "
f"crud.failed={summary['crud']['failed']} "
f"soap.failed={summary['soap']['failed']}",
flush=True,