DevOpsTools/openstack-api-simulator
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add OpenStack request-body schemas and nested console PARAM sync.

Browse Source
This commit is contained in:
Sergey Antropoff
2026-07-18 08:47:16 +03:00
parent 6033967e6a
commit 2a8ce3a287
101 changed files with 150428 additions and 612 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pulumi-tests/README.md
+29 -16
View File
@@ -2,8 +2,16 @@
# Pulumi OpenStack tests (`pulumi-tests`)
Coverage lab that **maximises `pulumi_openstack`**, then probes remaining pack
operations over HTTP with **non-empty body checks** and **full method coverage**.
**100% coverage = HTTP contract matrix** (every series-pack operation + synthetic
`HEAD` on each GET path), not the number of `pulumi_openstack` resources.
Layer A (gate): pack ops × yoga→dalmatian on **real service ports**
([docs/ports.md](../docs/ports.md)), microversions when the pack declares them,
non-empty success bodies for entity/collection envelopes, `probed == declared + HEAD`,
`critical=0`.
Layer B (smoke): `pulumi_openstack` lifecycle — maximises provider surface; does
**not** define 100%.
## Quick start
@@ -14,8 +22,8 @@ make pulumi-tests
# or
cd pulumi-tests
make up && make build
make test-pulumi-smoke # fast: collection GET only
make test-pulumi # full: all pack ops × all HTTP methods
make test-pulumi-smoke # fast: collection GET + HEAD
make test-pulumi # full Layer A matrix × all series
open reports/pulumi-report.html
```
@@ -23,28 +31,33 @@ open reports/pulumi-report.html
| Target | Mode | What is exercised |
|---|---|---|
| `make test-pulumi-smoke` | Smoke (`TEST_SMOKE=1`) | `pulumi_openstack` + **collection GET** nonempty checks (fast) |
| `make pulumi-tests` / `make test-pulumi` | Full lifecycle | `pulumi_openstack` + **every pack operation × GET/POST/PUT/PATCH/DELETE**, completeness assert (`total == pack size`), nonempty bodies on succeeded responses (DELETE/204 may be empty) |
| `make test-pulumi-smoke` | Smoke (`TEST_SMOKE=1`) | Layer B + **collection GET + HEAD** nonempty checks (fast) |
| `make pulumi-tests` / `make test-pulumi` | Full matrix | Layer B + **every pack op × GET/POST/PUT/PATCH/DELETE** + **synthetic HEAD per GET**, completeness (`total == declared + HEAD`), nonempty bodies on succeeded responses (DELETE/204/HEAD may be empty) |
Pack sizes (ops): yoga ~1060 → antelope ~1108 → caracal ~1196 → **dalmatian ~1357**.
Pack sizes (ops, without HEAD): yoga ~1060 → antelope ~1108 → caracal ~1196 → **dalmatian ~1357**.
## What runs (per series: yoga → dalmatian)
1. Activate OpenStack series pack
2. **`pulumi up`** `programs/os_coverage` via Automation API — creates/looks up
resources with `pulumi_openstack` (identity, images, compute, networking,
blockstorage, objectstorage, dns, orchestration)
3. Assert **every stack export is non-empty**
4. HTTP-probe pack operations (smoke: collection GET; full: all methods lifecycle)
5. Assert coverage completeness + nonempty JSON on successful body responses
2. **`pulumi up`** `programs/os_coverage` (Layer B — provider smoke)
3. Assert stack exports are non-empty (Layer B)
4. HTTP contract matrix on **catalog ports** (Keystone token → Nova `:8774`, Neutron `:9696`, …)
5. Assert `probed == declared + HEAD`, `critical=0`, nonempty JSON on success bodies
6. `pulumi destroy`
7. Write `pulumi-report.html` + `pulumi-junit.xml`
7. Write `pulumi-report.html` + `pulumi-junit.xml` + verb histogram (incl. HEAD)
## Reports
| File | Contents |
|---|---|
| `reports/pulumi-report.html` | HTML summary (expected vs actual + method breakdown) |
| `reports/pulumi-report.html` | HTML summary (expected vs actual + method breakdown incl. HEAD) |
| `reports/pulumi-junit.xml` | JUnit |
| `reports/series-<name>.json` | Per-series pulumi + HTTP details |
| `reports/summary.json` | Aggregates |
| `reports/summary.json` | Aggregates (`http_total` / `http_expected`, `http_critical`) |
## Approximate (not blockers)
Nova create may skip a long `BUILD` window; many Nova actions only flip server
status; Placement mutations and Octavia listeners/pools are largely schema /
`os_api_objects`; Neutron `router:external` is derived from the shared `public`
network name.
pulumi-tests/README.ru.md
+28 -16
View File
@@ -2,8 +2,16 @@
# Тесты Pulumi OpenStack (`pulumi-tests`)
Лаборатория покрытия: **максимально `pulumi_openstack`**, затем HTTP-probe
остальных pack-операций с проверкой **непустых тел** и **полным покрытием методов**.
**100% покрытия = HTTP contract matrix** (каждая операция series-pack +
синтетический `HEAD` на каждый GET path), а не число ресурсов `pulumi_openstack`.
Layer A (гейт): pack ops × yoga→dalmatian на **реальных портах сервисов**
([docs/ports.md](../docs/ports.md) / [docs/ru/ports.md](../docs/ru/ports.md)),
microversions где pack их задаёт, nonempty тела успешных ответов,
`probed == declared + HEAD`, `critical=0`.
Layer B (smoke): lifecycle `pulumi_openstack` — расширяет provider surface; **не**
определяет 100%.
## Быстрый старт
@@ -14,8 +22,8 @@ make pulumi-tests
# или
cd pulumi-tests
make up && make build
make test-pulumi-smoke # быстро: только collection GET
make test-pulumi # полный: все ручки пака × все HTTP-методы
make test-pulumi-smoke # быстро: collection GET + HEAD
make test-pulumi # полная Layer A matrix × все серии
open reports/pulumi-report.html
```
@@ -23,28 +31,32 @@ open reports/pulumi-report.html
| Цель | Режим | Что проверяется |
|---|---|---|
| `make test-pulumi-smoke` | Smoke (`TEST_SMOKE=1`) | `pulumi_openstack` + **collection GET** с nonempty (быстро) |
| `make pulumi-tests` / `make test-pulumi` | Полный lifecycle | `pulumi_openstack` + **все операции пака × GET/POST/PUT/PATCH/DELETE**, assert полноты (`total == размер пака`), nonempty тел успешных ответов (DELETE/204 могут быть пустыми) |
| `make test-pulumi-smoke` | Smoke (`TEST_SMOKE=1`) | Layer B + **collection GET + HEAD** с nonempty (быстро) |
| `make pulumi-tests` / `make test-pulumi` | Полная matrix | Layer B + **все ops пака × GET/POST/PUT/PATCH/DELETE** + **синтетический HEAD на каждый GET**, полнота (`total == declared + HEAD`), nonempty тел (DELETE/204/HEAD могут быть пустыми) |
Размеры паков (ops): yoga ~1060 → antelope ~1108 → caracal ~1196 → **dalmatian ~1357**.
Размеры паков (ops без HEAD): yoga ~1060 → antelope ~1108 → caracal ~1196 → **dalmatian ~1357**.
## Что выполняется (на каждую серию yoga → dalmatian)
1. Активация pack серии OpenStack
2. **`pulumi up`** программы `programs/os_coverage` через Automation API —
ресурсы через `pulumi_openstack` (identity, images, compute, networking,
blockstorage, objectstorage, dns, orchestration)
3. Проверка: **каждый export стека непустой**
4. HTTP-probe pack-операций (smoke: collection GET; полный: lifecycle всех методов)
5. Assert полноты покрытия + nonempty JSON у успешных ответов с телом
2. **`pulumi up`** `programs/os_coverage` (Layer B — provider smoke)
3. Проверка непустых export стека (Layer B)
4. HTTP contract matrix на **портах каталога** (токен Keystone → Nova `:8774`, Neutron `:9696`, …)
5. Assert `probed == declared + HEAD`, `critical=0`, nonempty JSON
6. `pulumi destroy`
7. Отчёты `pulumi-report.html` + `pulumi-junit.xml`
7. Отчёты HTML/JUnit + histogram глаголов (включая HEAD)
## Отчёты
| Файл | Содержимое |
|---|---|
| `reports/pulumi-report.html` | HTML-сводка (expected vs actual + breakdown методов) |
| `reports/pulumi-report.html` | HTML-сводка (expected vs actual + breakdown методов вкл. HEAD) |
| `reports/pulumi-junit.xml` | JUnit |
| `reports/series-<name>.json` | Детали pulumi + HTTP по серии |
| `reports/summary.json` | Агрегаты |
| `reports/summary.json` | Агрегаты (`http_total` / `http_expected`, `http_critical`) |
## Approximate (не блокеры)
Nova create может пропускать длинное окно `BUILD`; многие Nova actions только
меняют status; Placement mutations и Octavia listeners/pools в основном schema /
`os_api_objects`; Neutron `router:external` выводится из shared сети `public`.
pulumi-tests/docker-compose.yml
+2 -2
View File
@@ -132,8 +132,8 @@ services:
- ../docker/gateway/openstack-ports.conf:/etc/nginx/conf.d/default.conf:ro
- ../docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
- ../docker/tls/server.key:/etc/nginx/tls/server.key:ro
ports:
- "127.0.0.1:15000:5000"
# No host port publish — pulumi-runner reaches api-gateway on the lab
# network (avoids colliding with `make up-local` on :15000).
pulumi-runner:
build:
pulumi-tests/pulumi/_lib/http_coverage.py
+38 -13
View File
@@ -9,19 +9,28 @@ from typing import Any
from _lib.validate import payload_nonempty
# Methods that normally return a JSON body on success (DELETE / 204 may be empty).
# Methods that normally return a JSON body on success (DELETE / 204 / HEAD may be empty).
_BODY_METHODS = frozenset({"GET", "POST", "PUT", "PATCH"})
_VERB_ORDER = ("GET", "POST", "PUT", "PATCH", "DELETE", "HEAD")
def _expected_ops(packs: dict[str, Any], *, collections_only: bool) -> int:
def _expected_ops(packs: dict[str, Any], *, collections_only: bool) -> tuple[int, int, int]:
"""Return (expected_total, declared_pack_ops, synthetic_head_ops)."""
from app.openstack.surface_probe import count_declared_ops
if not collections_only:
return sum(len(p.operations) for p in packs.values())
total = 0
declared, heads = count_declared_ops(packs)
return declared + heads, declared, heads
declared = 0
heads = 0
for pack in packs.values():
for op in pack.operations:
if op.method == "GET" and "{" not in op.path:
total += 1
return total
declared += 1
heads += 1
return declared + heads, declared, heads
def _methods_breakdown(results: list[Any]) -> dict[str, int]:
@@ -30,16 +39,16 @@ def _methods_breakdown(results: list[Any]) -> dict[str, int]:
method = getattr(r, "method", None) or (r.get("method") if isinstance(r, dict) else None)
if method:
counts[str(method).upper()] += 1
return {m: counts.get(m, 0) for m in ("GET", "POST", "PUT", "PATCH", "DELETE")}
return {m: counts.get(m, 0) for m in _VERB_ORDER}
def _nonempty_from_lifecycle(report: Any) -> list[dict[str, Any]]:
"""Check succeeded lifecycle bodies (skip DELETE / 204 / 202 / no-body)."""
"""Check succeeded lifecycle bodies (skip DELETE / HEAD / 204 / 202 / no-body)."""
failures: list[dict[str, Any]] = []
for r in report.results:
if not r.succeeded:
continue
if r.method == "DELETE" or r.status in {202, 204}:
if r.method in {"DELETE", "HEAD"} or r.status in {202, 204}:
continue
if r.method not in _BODY_METHODS:
continue
@@ -74,6 +83,8 @@ def _nonempty_smoke_collections(
fill_path,
http_request,
issue_token,
microversion_headers,
service_base_url,
_seed_context,
)
@@ -96,8 +107,14 @@ def _nonempty_smoke_collections(
"account": project_id,
},
)
url = f"{host.rstrip('/')}{path}"
status, payload = http_request(op.method, url, token=token, service=pack.name)
url = f"{service_base_url(host, pack.name, port=pack.port)}{path}"
status, payload = http_request(
op.method,
url,
token=token,
service=pack.name,
headers=microversion_headers(pack, op) or None,
)
if status not in SUCCESS or status == 204:
continue
if not payload_nonempty(payload, collection_key=op.collection_key, method=op.method):
@@ -134,7 +151,7 @@ def probe_pack_operations(
)
packs = load_series_pack(series)
expected_ops = _expected_ops(packs, collections_only=collections_only)
expected_ops, declared_ops, head_ops = _expected_ops(packs, collections_only=collections_only)
methods = _methods_breakdown(report.results)
coverage_incomplete = len(report.results) != expected_ops
@@ -168,23 +185,31 @@ def probe_pack_operations(
"method": "*",
"path": "*",
"status": 0,
"detail": f"coverage_incomplete: total={len(report.results)} expected_ops={expected_ops}",
"detail": (
f"coverage_incomplete: total={len(report.results)} "
f"expected_ops={expected_ops} "
f"(declared={declared_ops} + HEAD={head_ops})"
),
"ok": False,
"nonempty": True,
}
)
critical = len(coverage_failures) + len(probe_failures) + len(nonempty_failures)
return {
"series": series,
"host": host,
"mode": report.mode,
"total": len(report.results),
"expected_ops": expected_ops,
"declared_ops": declared_ops,
"head_ops": head_ops,
"coverage_incomplete": coverage_incomplete,
"methods": methods,
"ok_count": len(report.results) - len(report.failures),
"fail_count": len(report.failures) + (1 if coverage_incomplete else 0),
"nonempty_fail_count": len(nonempty_failures),
"critical": critical,
"results": [
{
"service": r.service,
pulumi-tests/pulumi/_lib/report_html.py
+6 -4
View File
@@ -12,7 +12,9 @@ SERIES_ORDER = ("yoga", "antelope", "caracal", "dalmatian")
def _methods_line(methods: dict[str, Any]) -> str:
return " ".join(f"{m}={methods.get(m, 0)}" for m in ("GET", "POST", "PUT", "PATCH", "DELETE"))
return " ".join(
f"{m}={methods.get(m, 0)}" for m in ("GET", "POST", "PUT", "PATCH", "DELETE", "HEAD")
)
def render_html(summary: dict[str, Any], series_reports: list[dict[str, Any]]) -> str:
@@ -33,8 +35,8 @@ def render_html(summary: dict[str, Any], series_reports: list[dict[str, Any]]) -
</div>
<div class="stats">
<span class="ok">http ok={http.get("ok_count", 0)}</span>
<span class="fail">http fail={http.get("fail_count", 0)} nonempty_fail={http.get("nonempty_fail_count", 0)}</span>
<span>http total={http.get("total", 0)}/{http.get("expected_ops", "?")}</span>
<span class="fail">http fail={http.get("fail_count", 0)} nonempty_fail={http.get("nonempty_fail_count", 0)} critical={http.get("critical", 0)}</span>
<span>http total={http.get("total", 0)}/{http.get("expected_ops", "?")} (declared={http.get("declared_ops", "?")}+HEAD={http.get("head_ops", "?")})</span>
</div>
<div class="stats muted">
methods: {html.escape(_methods_line(http.get("methods") or {}))}
@@ -92,7 +94,7 @@ def render_html(summary: dict[str, Any], series_reports: list[dict[str, Any]]) -
<body>
<header>
<h1>Pulumi OpenStack API coverage</h1>
<p class="muted">Generated {html.escape(generated)} · pulumi_openstack primary + HTTP pack probe with non-empty checks</p>
<p class="muted">Generated {html.escape(generated)} · <strong>100% = HTTP contract matrix</strong> (pack ops + synthetic HEAD), not pulumi_openstack resource count. Layer B provider lifecycle is smoke only.</p>
</header>
<main>
<div class="summary">
pulumi-tests/pulumi/run_suite.py
+25 -3
View File
@@ -233,10 +233,20 @@ def main() -> int:
"total": 0,
"expected_ops": 0,
"coverage_incomplete": False,
"methods": {"GET": 0, "POST": 0, "PUT": 0, "PATCH": 0, "DELETE": 0},
"methods": {
"GET": 0,
"POST": 0,
"PUT": 0,
"PATCH": 0,
"DELETE": 0,
"HEAD": 0,
},
"ok_count": 0,
"fail_count": 0,
"nonempty_fail_count": 0,
"critical": 0,
"declared_ops": 0,
"head_ops": 0,
"results": [],
"failures": [],
}
@@ -245,12 +255,15 @@ def main() -> int:
http = run_http_coverage(series, collections_only=collections_only)
methods = http.get("methods") or {}
methods_s = " ".join(
f"{m}={methods.get(m, 0)}" for m in ("GET", "POST", "PUT", "PATCH", "DELETE")
f"{m}={methods.get(m, 0)}"
for m in ("GET", "POST", "PUT", "PATCH", "DELETE", "HEAD")
)
print(
f"{series}: http ok={http.get('ok_count')} fail={http.get('fail_count')} "
f"nonempty_fail={http.get('nonempty_fail_count')} "
f"critical={http.get('critical')} "
f"total={http.get('total')}/{http.get('expected_ops')} "
f"(declared={http.get('declared_ops')}+HEAD={http.get('head_ops')}) "
f"coverage_incomplete={http.get('coverage_incomplete')} "
f"methods[{methods_s}]"
)
@@ -273,10 +286,16 @@ def main() -> int:
int(r["http"].get("fail_count", 0)) + int(r["http"].get("nonempty_fail_count", 0))
for r in series_reports
),
"http_critical": sum(int(r["http"].get("critical", 0)) for r in series_reports),
"http_declared": sum(int(r["http"].get("declared_ops", 0)) for r in series_reports),
"http_head": sum(int(r["http"].get("head_ops", 0)) for r in series_reports),
"http_total": sum(int(r["http"].get("total", 0)) for r in series_reports),
"http_expected": sum(int(r["http"].get("expected_ops", 0)) for r in series_reports),
"coverage_incomplete": sum(
1 for r in series_reports if r["http"].get("coverage_incomplete")
),
"collections_only": collections_only,
"definition": "100% = HTTP contract matrix (pack ops + synthetic HEAD), not pulumi_openstack resource count",
}
html_path = write_html(REPORT_DIR, summary, series_reports)
(REPORT_DIR / "summary.json").write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
@@ -285,7 +304,10 @@ def main() -> int:
print(json.dumps(summary, indent=2))
failed = (
summary["pulumi_fail"] > 0 or summary["http_fail"] > 0 or summary["coverage_incomplete"] > 0
summary["pulumi_fail"] > 0
or summary["http_fail"] > 0
or summary["coverage_incomplete"] > 0
or summary["http_critical"] > 0
)
return 1 if failed else 0
pulumi-tests/reports/pulumi-junit.xml
+1 -1
View File
File diff suppressed because one or more lines are too long
pulumi-tests/reports/pulumi-report.html
+18 -18
View File
@@ -21,14 +21,14 @@
<body>
<header>
<h1>Pulumi OpenStack API coverage</h1>
<p class="muted">Generated 2026-07-17 13:12:06 UTC · pulumi_openstack primary + HTTP pack probe with non-empty checks</p>
<p class="muted">Generated 2026-07-18 03:50:29 UTC · <strong>100% = HTTP contract matrix</strong> (pack ops + synthetic HEAD), not pulumi_openstack resource count. Layer B provider lifecycle is smoke only.</p>
</header>
<main>
<div class="summary">
<div><strong>4</strong><span class="muted"> series</span></div>
<div><strong class="ok">4</strong><span class="muted"> pulumi stacks ok</span></div>
<div><strong class="fail">0</strong><span class="muted"> pulumi failures</span></div>
<div><strong class="ok">4721</strong><span class="muted"> http ops ok</span></div>
<div><strong class="ok">6514</strong><span class="muted"> http ops ok</span></div>
<div><strong class="fail">0</strong><span class="muted"> http / nonempty fails</span></div>
</div>
<h2>Series</h2>
@@ -41,12 +41,12 @@
<span class="fail">empty exports=0</span>
</div>
<div class="stats">
<span class="ok">http ok=1060</span>
<span class="fail">http fail=0 nonempty_fail=0</span>
<span>http total=1060/1060</span>
<span class="ok">http ok=1464</span>
<span class="fail">http fail=0 nonempty_fail=0 critical=0</span>
<span>http total=1464/1464 (declared=1060+HEAD=404)</span>
</div>
<div class="stats muted">
methods: GET=404 POST=168 PUT=171 PATCH=155 DELETE=162
methods: GET=404 POST=168 PUT=171 PATCH=155 DELETE=162 HEAD=404
</div>
</div>
@@ -58,12 +58,12 @@
<span class="fail">empty exports=0</span>
</div>
<div class="stats">
<span class="ok">http ok=1108</span>
<span class="fail">http fail=0 nonempty_fail=0</span>
<span>http total=1108/1108</span>
<span class="ok">http ok=1530</span>
<span class="fail">http fail=0 nonempty_fail=0 critical=0</span>
<span>http total=1530/1530 (declared=1108+HEAD=422)</span>
</div>
<div class="stats muted">
methods: GET=422 POST=177 PUT=178 PATCH=162 DELETE=169
methods: GET=422 POST=177 PUT=178 PATCH=162 DELETE=169 HEAD=422
</div>
</div>
@@ -75,12 +75,12 @@
<span class="fail">empty exports=0</span>
</div>
<div class="stats">
<span class="ok">http ok=1196</span>
<span class="fail">http fail=0 nonempty_fail=0</span>
<span>http total=1196/1196</span>
<span class="ok">http ok=1649</span>
<span class="fail">http fail=0 nonempty_fail=0 critical=0</span>
<span>http total=1649/1649 (declared=1196+HEAD=453)</span>
</div>
<div class="stats muted">
methods: GET=453 POST=192 PUT=192 PATCH=176 DELETE=183
methods: GET=453 POST=192 PUT=192 PATCH=176 DELETE=183 HEAD=453
</div>
</div>
@@ -92,12 +92,12 @@
<span class="fail">empty exports=0</span>
</div>
<div class="stats">
<span class="ok">http ok=1357</span>
<span class="fail">http fail=0 nonempty_fail=0</span>
<span>http total=1357/1357</span>
<span class="ok">http ok=1871</span>
<span class="fail">http fail=0 nonempty_fail=0 critical=0</span>
<span>http total=1871/1871 (declared=1357+HEAD=514)</span>
</div>
<div class="stats muted">
methods: GET=514 POST=217 PUT=217 PATCH=201 DELETE=208
methods: GET=514 POST=217 PUT=217 PATCH=201 DELETE=208 HEAD=514
</div>
</div></div>
pulumi-tests/reports/series-antelope.json
+4669 -23
View File
File diff suppressed because it is too large Load Diff
pulumi-tests/reports/series-caracal.json
+5010 -23
View File
File diff suppressed because it is too large Load Diff
pulumi-tests/reports/series-dalmatian.json
+5681 -23
View File
File diff suppressed because it is too large Load Diff
pulumi-tests/reports/series-yoga.json
+4471 -23
View File
File diff suppressed because it is too large Load Diff
pulumi-tests/reports/summary.json
+8 -2
View File
@@ -2,8 +2,14 @@
"series_count": 4,
"pulumi_ok": 4,
"pulumi_fail": 0,
"http_ok": 4721,
"http_ok": 6514,
"http_fail": 0,
"http_critical": 0,
"http_declared": 4721,
"http_head": 1793,
"http_total": 6514,
"http_expected": 6514,
"coverage_incomplete": 0,
"collections_only": false
"collections_only": false,
"definition": "100% = HTTP contract matrix (pack ops + synthetic HEAD), not pulumi_openstack resource count"
}