DevOpsTools/ovirt-api-simulator
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add OpenStack request-body schemas and nested console PARAM sync.

Browse Source
This commit is contained in:
Sergey Antropoff
2026-07-18 08:47:38 +03:00
parent cbd0adca91
commit ae297258b1
46 changed files with 42717 additions and 40135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pulumi-tests/README.md
+24 -9
View File
@@ -2,10 +2,10 @@
# oVirt Pulumi contract-coverage lab
Pulumi Automation API suite that exercises **every operation** declared in
`contracts/ovirt/<series>/api.json` across **all Engine series packs**, plus a
**synthetic HEAD** request for each GET path (contracts omit HEAD; the Engine
accepts it).
**100% coverage** here means the **HTTP contract matrix**: every operation
declared in `contracts/ovirt/<series>/api.json` for **all Engine series packs**,
plus a **synthetic HEAD** for each GET path (contracts omit HEAD; the Engine
accepts it). It is **not** a count of Pulumi provider resources.
| Series | Ops (approx.) |
|--------|--------------:|
@@ -16,10 +16,22 @@ make test-pulumi-smoke # 3.6 + 4.5 sample (fast)
make pulumi-tests # full matrix (alias: make test-pulumi)
```
Layer B (optional): provider lifecycle smoke only — do not treat it as API
parity.
Reports (written under `reports/`):
- `pulumi-contract-coverage.html` — human-readable summary (includes methods histogram)
- `pulumi-contract-coverage.json` — machine-readable results
- `pulumi-contract-coverage.html` — human-readable summary (method histogram
includes GET/PUT/POST/DELETE/HEAD)
- `pulumi-contract-coverage.json` — machine-readable results + `coverage`
(`probed/declared`, `critical`)
Pass line example:
```text
COVERAGE 9150/9150 (critical=0)
METHODS {"DELETE":1146,"GET":2314,"HEAD":2314,"POST":2230,"PUT":1146}
```
Optional filters:
@@ -29,11 +41,14 @@ OVIRT_METHODS_FILTER=GET make pulumi-tests
SMOKE_ONLY=1 make test-pulumi-smoke
```
All suites run **only in Docker**. Pass criteria:
All suites run **only in Docker** (lab compose seeds **minimal** inventory).
Pass criteria:
- The Engine route is reachable and returns a handled status (`200`/`201`/`202`/`204`,
`400`/`403`/`404`/`405`/`409`/`415`/`422`/`501`) — **not** `401` (the suite
re-authenticates after each series unload) and not a transport/`5xx` failure.
- For `200`/`201`/`202` the response body must be non-empty (HEAD exempt).
- Full runs must exercise **GET, POST, PUT, DELETE, and HEAD**; any failures or
missing methods fail the suite.
- Successful **collection** GETs must return a **non-empty** list with `id`
(empty `[]` is a seed/`ov_api_objects` gap — fix data, do not skip).
- Full runs must exercise **GET, POST, PUT, DELETE, and HEAD**; any failures,
missing methods, or `probed != declared` fail the suite (`critical=0` required).
pulumi-tests/README.ru.md
+22 -7
View File
@@ -2,10 +2,10 @@
# Лаборатория Pulumi: покрытие контрактов oVirt
Suite на Pulumi Automation API, который вызывает **каждую операцию** из
**100% coverage** здесь — это **HTTP contract matrix**: каждая операция из
`contracts/ovirt/<series>/api.json` для **всех series packs** Engine, плюс
**синтетический HEAD** для каждого GET-пути (в contracts нет HEAD; Engine его
принимает).
принимает). Это **не** число ресурсов Pulumi provider.
| Series | Операций (примерно) |
|--------|--------------------:|
@@ -16,10 +16,22 @@ make test-pulumi-smoke # выборка 3.6 + 4.5 (быстро)
make pulumi-tests # полная матрица (alias: make test-pulumi)
```
Layer B (опционально): только smoke lifecycle провайдера — не выдавать за
полноту API.
Отчёты (в `reports/`):
- `pulumi-contract-coverage.html` — сводка для человека (включая гистограмму методов)
- `pulumi-contract-coverage.json` — машиночитаемый результат
- `pulumi-contract-coverage.html` — сводка (гистограмма методов:
GET/PUT/POST/DELETE/HEAD)
- `pulumi-contract-coverage.json` — результат + `coverage`
(`probed/declared`, `critical`)
Пример pass-строки:
```text
COVERAGE 9150/9150 (critical=0)
METHODS {"DELETE":1146,"GET":2314,"HEAD":2314,"POST":2230,"PUT":1146}
```
Фильтры:
@@ -29,11 +41,14 @@ OVIRT_METHODS_FILTER=GET make pulumi-tests
SMOKE_ONLY=1 make test-pulumi-smoke
```
Все suites — **только в Docker**. Критерии pass:
Все suites — **только в Docker** (lab compose сидит **minimal** seed).
Критерии pass:
- Маршрут Engine достижим и возвращает обработанный статус (`200`/`201`/`202`/`204`,
`400`/`403`/`404`/`405`/`409`/`415`/`422`/`501`) — **не** `401` (после unload
каждой series suite заново логинится) и не транспортную / `5xx` ошибку.
- Для `200`/`201`/`202` тело ответа должно быть непустым (HEAD исключён).
- Полный прогон должен покрыть **GET, POST, PUT, DELETE и HEAD**; любые failures
или отсутствующие методы валят suite.
- Успешные **collection** GET должны возвращать **непустой** список с `id`
(пустой `[]` — дыра seed/`ov_api_objects`, чинить данные, не skip).
- Полный прогон должен покрыть **GET, POST, PUT, DELETE и HEAD**; любые failures,
отсутствующие методы или `probed != declared` валят suite (`critical=0`).
pulumi-tests/docker-compose.yml
+1 -1
View File
@@ -51,7 +51,7 @@ services:
depends_on:
simulator:
condition: service_healthy
entrypoint: ["python", "-m", "app.ovirt.seed_cli", "--profile", "demo"]
entrypoint: ["python", "-m", "app.ovirt.seed_cli", "--profile", "minimal"]
api-gateway:
image: nginx:1.28.0-alpine
pulumi-tests/pulumi/coverage/executor.py
+231 -12
View File
@@ -42,7 +42,7 @@ def _value_empty(value: Any) -> bool:
return value is None or value == "" or value == [] or value == {}
def _payload_nonempty(response: Any, *, method: str) -> tuple[bool, str]:
def _payload_nonempty(response: Any, *, method: str, kind: str = "") -> tuple[bool, str]:
"""Require non-empty response data for successful body-bearing statuses."""
# HEAD has no body; DELETE often returns 200 with an empty body (Engine-style).
if method in {"HEAD", "DELETE"}:
@@ -61,10 +61,17 @@ def _payload_nonempty(response: Any, *, method: str) -> tuple[bool, str]:
return False, "null JSON body"
if isinstance(data, (list, dict)) and len(data) == 0:
return False, "empty JSON body"
# Declared collection GETs must return durable lab samples (not []).
if method == "GET" and kind == "collection" and isinstance(data, dict):
for value in data.values():
if isinstance(value, list):
if len(value) == 0:
return False, "empty collection list"
first = value[0]
if isinstance(first, dict) and not first.get("id"):
return False, "collection item missing id"
break
if isinstance(data, dict) and all(_value_empty(v) for v in data.values()):
# Allow Engine empty collections: {"vms": []} has structure but no rows.
if len(data) == 1 and isinstance(next(iter(data.values())), list):
return True, ""
return False, "JSON body has only empty fields"
return True, ""
@@ -146,7 +153,47 @@ def synthesize_head_ops(ops: list[dict[str, Any]]) -> list[dict[str, Any]]:
class Inventory:
"""Cache of collection → first entity id for path placeholder expansion."""
"""Cache of collection → entity ids for path placeholder expansion.
PUT/DELETE resolve to *disposable* entities created for the op so the
minimal seed (lab-vm-01, Default DC/cluster, …) is not wiped before later
collection GETs run in contract order.
"""
_ELEMENT = {
"vms": "vm",
"hosts": "host",
"clusters": "cluster",
"datacenters": "data_center",
"networks": "network",
"disks": "disk",
"templates": "template",
"storagedomains": "storage_domain",
"storageconnections": "storage_connection",
"vnicprofiles": "vnic_profile",
"users": "user",
"groups": "group",
"roles": "role",
"tags": "tag",
"bookmarks": "bookmark",
"affinitylabels": "affinity_label",
"instancetypes": "instance_type",
"macpools": "mac_pool",
"schedulingpolicies": "scheduling_policy",
"vmpools": "vm_pool",
"permissions": "permission",
"domains": "domain",
"icons": "icon",
"jobs": "job",
"events": "event",
"nics": "nic",
"snapshots": "snapshot",
"diskattachments": "disk_attachment",
"cdroms": "cdrom",
"graphicsconsoles": "graphics_console",
"quotas": "quota",
"affinitygroups": "affinity_group",
}
def __init__(self, client: OVirtClient, version: str) -> None:
self.client = client
@@ -154,8 +201,12 @@ class Inventory:
self._ids: dict[str, str] = {}
self._listed: set[str] = set()
def id_for(self, collection: str) -> str | None:
def id_for(self, collection: str, *, method: str = "GET") -> str | None:
collection = collection.strip("/")
if method in {"DELETE", "PUT"}:
created = self.create_disposable(collection)
if created:
return created
if collection in self._ids:
return self._ids[collection]
if collection in self._listed:
@@ -172,7 +223,6 @@ class Inventory:
body = r.json()
except Exception:
return None
# Engine collections are usually { "<singular_or_plural>": [ {...}, ... ] }
for value in body.values() if isinstance(body, dict) else []:
if isinstance(value, list) and value:
first = value[0]
@@ -184,6 +234,129 @@ class Inventory:
return self._ids[collection]
return None
def create_disposable(self, collection: str) -> str | None:
"""POST a throwaway entity and return its id (best-effort)."""
element = self._ELEMENT.get(collection) or collection.rstrip("s") or "object"
name = f"pulumi-{uuid4().hex[:8]}"
payload: dict[str, Any] = {
"name": name,
"description": "pulumi disposable",
}
if collection == "clusters":
dc = self.id_for("datacenters", method="GET")
if dc:
payload["data_center"] = {"id": dc}
elif collection == "hosts":
cl = self.id_for("clusters", method="GET")
if cl:
payload["cluster"] = {"id": cl}
payload["address"] = "127.0.0.1"
elif collection == "networks":
dc = self.id_for("datacenters", method="GET")
if dc:
payload["data_center"] = {"id": dc}
elif collection == "vms":
cl = self.id_for("clusters", method="GET")
if cl:
payload["cluster"] = {"id": cl}
tpl = self.id_for("templates", method="GET")
if tpl:
payload["template"] = {"id": tpl}
elif collection == "disks":
sd = self.id_for("storagedomains", method="GET")
if sd:
payload["storage_domains"] = {"storage_domain": [{"id": sd}]}
payload["provisioned_size"] = 1073741824
elif collection == "vnicprofiles":
net = self.id_for("networks", method="GET")
if net:
payload["network"] = {"id": net}
elif collection == "templates":
cl = self.id_for("clusters", method="GET")
if cl:
payload["cluster"] = {"id": cl}
elif collection == "storageconnections":
payload = {
"type": "nfs",
"address": "nfs.pulumi.local",
"path": f"/export/{name}",
}
elif collection == "storagedomains":
payload["type"] = "data"
payload["storage"] = {
"type": "nfs",
"address": "nfs.pulumi.local",
"path": f"/export/{name}",
}
elif collection == "users":
payload = {
"user_name": f"{name}@internal",
"name": name,
"password": "secret",
}
domain = self.id_for("domains", method="GET")
if domain:
payload["domain"] = {"id": domain}
elif collection == "bookmarks":
payload["value"] = "Vms:"
path = f"/ovirt-engine/api/{collection}"
return self._post_for_id(path, element, payload)
def create_disposable_at(self, collection_path: str, collection: str) -> str | None:
"""POST under an already-resolved collection path (nested resources)."""
element = self._ELEMENT.get(collection) or collection.rstrip("s") or "object"
name = f"pulumi-{uuid4().hex[:8]}"
payload: dict[str, Any] = {"name": name, "description": "pulumi disposable"}
if collection == "snapshots":
payload = {"description": name}
elif collection == "diskattachments":
payload = {
"interface": "virtio_scsi",
"bootable": False,
"active": True,
"disk": {
"name": f"{name}-disk",
"provisioned_size": 1073741824,
"format": "cow",
},
}
elif collection == "cdroms":
payload = {"file": {"id": ""}}
elif collection == "graphicsconsoles":
payload = {"protocol": "spice"}
elif collection == "nics":
payload = {"name": name, "interface": "virtio"}
return self._post_for_id(collection_path, element, payload)
def _post_for_id(
self, path: str, element: str, payload: dict[str, Any]
) -> str | None:
try:
r = self.client.request(
"POST",
path,
headers=self.client.headers(version=self.version),
json={element: payload},
)
except Exception:
return None
if r.status_code not in {200, 201, 202}:
return None
try:
body = r.json()
except Exception:
return None
entity = body.get(element) if isinstance(body, dict) else None
if isinstance(entity, dict) and entity.get("id"):
return str(entity["id"])
for value in body.values() if isinstance(body, dict) else []:
if isinstance(value, dict) and value.get("id"):
return str(value["id"])
return None
def _collection_before_param(parts: list[str], index: int) -> str | None:
# /ovirt-engine/api/vms/{id}/nics/{id} → for first {id} use vms, for second use nics
@@ -195,10 +368,11 @@ def _collection_before_param(parts: list[str], index: int) -> str | None:
return prev
def resolve_path(template: str, inventory: Inventory) -> tuple[str, bool]:
def resolve_path(template: str, inventory: Inventory, *, method: str = "GET") -> tuple[str, bool]:
"""Return resolved path and whether every placeholder was satisfied from inventory."""
parts = template.strip("/").split("/")
param_indices = [i for i, part in enumerate(parts) if _PATH_PARAM.fullmatch(part)]
resolved: list[str] = []
complete = True
for i, part in enumerate(parts):
@@ -207,7 +381,29 @@ def resolve_path(template: str, inventory: Inventory) -> tuple[str, bool]:
resolved.append(part)
continue
collection = _collection_before_param(parts, i)
entity_id = inventory.id_for(collection) if collection else None
is_leaf = bool(param_indices) and i == param_indices[-1]
entity_id = None
if collection:
if method in {"DELETE", "PUT"} and is_leaf:
# Only the leaf id is disposable — parents keep seed inventory.
# Never fall back to seed ids for DELETE (would wipe minimal lab).
if collection in Inventory._ELEMENT and collection not in {
"nics",
"snapshots",
"diskattachments",
"cdroms",
"graphicsconsoles",
"quotas",
"affinitygroups",
}:
entity_id = inventory.create_disposable(collection)
else:
parent_path = "/" + "/".join(resolved)
entity_id = inventory.create_disposable_at(parent_path, collection)
if not entity_id and method == "PUT":
entity_id = inventory.id_for(collection, method="GET")
else:
entity_id = inventory.id_for(collection, method="GET")
if entity_id:
resolved.append(entity_id)
else:
@@ -242,7 +438,7 @@ def execute_operation(
expected = int(op.get("create_status") or op.get("status_code") or 200) if method == "POST" else int(
op.get("status_code") or 200
)
path, _complete = resolve_path(template, inventory)
path, _complete = resolve_path(template, inventory, method=method)
body = _minimal_body(op)
started = time.perf_counter()
try:
@@ -254,7 +450,9 @@ def execute_operation(
ok = response.status_code in _PASS_STATUSES
detail = ""
if ok:
body_ok, body_detail = _payload_nonempty(response, method=method)
body_ok, body_detail = _payload_nonempty(
response, method=method, kind=kind
)
if not body_ok:
ok = False
detail = body_detail
@@ -400,11 +598,32 @@ def run_coverage(cfg: SuiteConfig) -> CoverageReport:
def report_to_dict(report: CoverageReport) -> dict[str, Any]:
totals = report.totals
declared = totals["total"]
probed = totals["passed"] + totals["failed"] + totals["skipped"]
critical = totals["failed"]
series_coverage = []
for s in report.series:
series_coverage.append(
{
"series": s.series,
"declared": s.total,
"probed": s.passed + s.failed + s.skipped,
"critical": s.failed,
}
)
return {
"generated_at": report.generated_at,
"engine_url": report.engine_url,
"totals": report.totals,
"totals": totals,
"methods": report.methods,
"coverage": {
"declared": declared,
"probed": probed,
"critical": critical,
"line": f"{probed}/{declared}",
"series": series_coverage,
},
"series": [asdict(s) for s in report.series],
"results": [asdict(r) for r in report.results],
}
pulumi-tests/pulumi/coverage/report.py
+3 -3
View File
@@ -121,10 +121,10 @@ def render_html(payload: dict[str, Any]) -> str:
· Engine {html.escape(str(payload.get('engine_url')))}
</div>
<div class="cards">
<div class="card"><div class="label">Total</div><div class="value">{totals.get('total', 0)}</div></div>
<div class="card"><div class="label">Total / Declared</div><div class="value">{totals.get('total', 0)}</div></div>
<div class="card ok"><div class="label">Passed</div><div class="value">{totals.get('passed', 0)}</div></div>
<div class="card bad"><div class="label">Failed</div><div class="value">{totals.get('failed', 0)}</div></div>
<div class="card"><div class="label">Skipped</div><div class="value">{totals.get('skipped', 0)}</div></div>
<div class="card bad"><div class="label">Critical</div><div class="value">{(payload.get('coverage') or {}).get('critical', totals.get('failed', 0))}</div></div>
<div class="card"><div class="label">Coverage</div><div class="value">{html.escape(str((payload.get('coverage') or {}).get('line', f"{totals.get('total', 0)}/{totals.get('total', 0)}")))}</div></div>
</div>
<h2>By HTTP method</h2>
pulumi-tests/pulumi/run_suite.py
+9 -1
View File
@@ -52,6 +52,7 @@ def main() -> int:
"passed": totals["passed"],
"failed": totals["failed"],
"skipped": totals["skipped"],
"coverage": payload.get("coverage"),
"methods": methods,
"series": series_names,
"report_json": str(json_path),
@@ -86,7 +87,12 @@ def main() -> int:
failed = int(outputs.get("failed") or totals["failed"])
total = int(outputs.get("total") or totals["total"])
passed = int(outputs.get("passed") or totals["passed"])
coverage = payload.get("coverage") or {}
declared = int(coverage.get("declared") or total)
probed = int(coverage.get("probed") or (passed + failed + int(totals.get("skipped") or 0)))
critical = int(coverage.get("critical") or failed)
print(f"SUMMARY total={total} passed={passed} failed={failed}", flush=True)
print(f"COVERAGE {probed}/{declared} (critical={critical})", flush=True)
print(f"METHODS {json.dumps(methods, sort_keys=True)}", flush=True)
print(f"HTML report: {html_path}", flush=True)
@@ -94,7 +100,9 @@ def main() -> int:
missing_methods = sorted(_FULL_RUN_METHODS - set(methods)) if full_run else []
if missing_methods:
print(f"MISSING METHODS on full run: {', '.join(missing_methods)}", flush=True)
if failed or missing_methods:
if probed != declared:
print(f"COVERAGE MISMATCH probed={probed} declared={declared}", flush=True)
if failed or missing_methods or critical or probed != declared:
return 1
return 0
pulumi-tests/reports/pulumi-contract-coverage.html
+6 -6
View File
File diff suppressed because one or more lines are too long
pulumi-tests/reports/pulumi-contract-coverage.json
+39705 -39631
View File
File diff suppressed because it is too large Load Diff
pulumi-tests/reports/pulumi-stack-summary.json
+74
View File
@@ -3,6 +3,80 @@
"passed": 9150,
"failed": 0,
"skipped": 0,
"coverage": {
"declared": 9150,
"probed": 9150,
"critical": 0,
"line": "9150/9150",
"series": [
{
"series": "3.0",
"declared": 622,
"probed": 622,
"critical": 0
},
{
"series": "3.1",
"declared": 664,
"probed": 664,
"critical": 0
},
{
"series": "3.2",
"declared": 672,
"probed": 672,
"critical": 0
},
{
"series": "3.3",
"declared": 770,
"probed": 770,
"critical": 0
},
{
"series": "3.4",
"declared": 800,
"probed": 800,
"critical": 0
},
{
"series": "3.5",
"declared": 858,
"probed": 858,
"critical": 0
},
{
"series": "3.6",
"declared": 918,
"probed": 918,
"critical": 0
},
{
"series": "4.3",
"declared": 948,
"probed": 948,
"critical": 0
},
{
"series": "4.4",
"declared": 966,
"probed": 966,
"critical": 0
},
{
"series": "4.5",
"declared": 966,
"probed": 966,
"critical": 0
},
{
"series": "master",
"declared": 966,
"probed": 966,
"critical": 0
}
]
},
"methods": {
"DELETE": 1146,
"GET": 2314,