Initial commit: stateful OpenStack API laboratory simulator.
Ship Keystone auth, multi-service handlers (Yoga→Dalmatian), Compose/Helm packaging, API contract packs, and pytest/Pulumi coverage labs.
This commit is contained in:
@@ -0,0 +1,45 @@
|
||||
# Pulumi OpenStack coverage lab (pulumi_openstack + HTTP nonempty checks).
|
||||
COMPOSE ?= docker compose
|
||||
COMPOSE_FILE ?= docker-compose.yml
|
||||
COMPOSE_CMD = $(COMPOSE) -f $(COMPOSE_FILE)
|
||||
PROFILE = --profile test
|
||||
|
||||
.PHONY: help up down build seed test-pulumi-smoke test-pulumi pulumi-tests clean-test-resources report
|
||||
|
||||
help: ## Show targets
|
||||
@awk 'BEGIN {FS = ":.*## "}; /^[a-zA-Z0-9_-]+:.*## / {printf "%-26s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
|
||||
|
||||
up: ## Start simulator + gateway + seed
|
||||
$(COMPOSE_CMD) up -d --build postgres
|
||||
$(COMPOSE_CMD) up --build migrate
|
||||
$(COMPOSE_CMD) up -d --build simulator api-gateway
|
||||
$(COMPOSE_CMD) up --build seed
|
||||
|
||||
down: ## Stop lab stack
|
||||
$(COMPOSE_CMD) --profile test down -v
|
||||
|
||||
build: ## Build Pulumi runner image
|
||||
$(COMPOSE_CMD) $(PROFILE) build pulumi-runner
|
||||
|
||||
seed: ## Re-run demo seed
|
||||
$(COMPOSE_CMD) up --build seed
|
||||
|
||||
test-pulumi-smoke: up ## Fast: pulumi_openstack + collection GET nonempty × all series
|
||||
mkdir -p reports
|
||||
$(COMPOSE_CMD) $(PROFILE) run --rm -e TEST_SMOKE=1 pulumi-runner \
|
||||
python3 /suite/pulumi/run_suite.py
|
||||
|
||||
test-pulumi: up ## Full: pulumi_openstack + lifecycle HTTP nonempty × all series
|
||||
mkdir -p reports
|
||||
$(COMPOSE_CMD) $(PROFILE) run --rm pulumi-runner \
|
||||
python3 /suite/pulumi/run_suite.py
|
||||
|
||||
pulumi-tests: test-pulumi ## Alias for full suite
|
||||
|
||||
report: ## Rebuild HTML from series-*.json on host
|
||||
PYTHONPATH=pulumi python3 -c "from pathlib import Path; import json; from _lib.report_html import write_html, load_series_files; \
|
||||
d=Path('reports'); reps=load_series_files(d); s=json.loads((d/'summary.json').read_text()) if (d/'summary.json').exists() else {}; \
|
||||
print(write_html(d, s, reps))"
|
||||
|
||||
clean-test-resources: ## Best-effort cleanup via seed reload
|
||||
$(COMPOSE_CMD) up --build seed
|
||||
@@ -0,0 +1,50 @@
|
||||
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
|
||||
|
||||
# Pulumi OpenStack tests (`pulumi-tests`)
|
||||
|
||||
Coverage lab that **maximises `pulumi_openstack`**, then probes remaining pack
|
||||
operations over HTTP with **non-empty body checks** and **full method coverage**.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# from repo root
|
||||
make pulumi-tests
|
||||
|
||||
# or
|
||||
cd pulumi-tests
|
||||
make up && make build
|
||||
make test-pulumi-smoke # fast: collection GET only
|
||||
make test-pulumi # full: all pack ops × all HTTP methods
|
||||
open reports/pulumi-report.html
|
||||
```
|
||||
|
||||
## Smoke vs full suite
|
||||
|
||||
| Target | Mode | What is exercised |
|
||||
|---|---|---|
|
||||
| `make test-pulumi-smoke` | Smoke (`TEST_SMOKE=1`) | `pulumi_openstack` + **collection GET** nonempty checks (fast) |
|
||||
| `make pulumi-tests` / `make test-pulumi` | Full lifecycle | `pulumi_openstack` + **every pack operation × GET/POST/PUT/PATCH/DELETE**, completeness assert (`total == pack size`), nonempty bodies on succeeded responses (DELETE/204 may be empty) |
|
||||
|
||||
Pack sizes (ops): yoga ~1060 → antelope ~1108 → caracal ~1196 → **dalmatian ~1357**.
|
||||
|
||||
## What runs (per series: yoga → dalmatian)
|
||||
|
||||
1. Activate OpenStack series pack
|
||||
2. **`pulumi up`** `programs/os_coverage` via Automation API — creates/looks up
|
||||
resources with `pulumi_openstack` (identity, images, compute, networking,
|
||||
blockstorage, objectstorage, dns, orchestration)
|
||||
3. Assert **every stack export is non-empty**
|
||||
4. HTTP-probe pack operations (smoke: collection GET; full: all methods lifecycle)
|
||||
5. Assert coverage completeness + nonempty JSON on successful body responses
|
||||
6. `pulumi destroy`
|
||||
7. Write `pulumi-report.html` + `pulumi-junit.xml`
|
||||
|
||||
## Reports
|
||||
|
||||
| File | Contents |
|
||||
|---|---|
|
||||
| `reports/pulumi-report.html` | HTML summary (expected vs actual + method breakdown) |
|
||||
| `reports/pulumi-junit.xml` | JUnit |
|
||||
| `reports/series-<name>.json` | Per-series pulumi + HTTP details |
|
||||
| `reports/summary.json` | Aggregates |
|
||||
@@ -0,0 +1,50 @@
|
||||
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
|
||||
|
||||
# Тесты Pulumi OpenStack (`pulumi-tests`)
|
||||
|
||||
Лаборатория покрытия: **максимально `pulumi_openstack`**, затем HTTP-probe
|
||||
остальных pack-операций с проверкой **непустых тел** и **полным покрытием методов**.
|
||||
|
||||
## Быстрый старт
|
||||
|
||||
```bash
|
||||
# из корня репозитория
|
||||
make pulumi-tests
|
||||
|
||||
# или
|
||||
cd pulumi-tests
|
||||
make up && make build
|
||||
make test-pulumi-smoke # быстро: только collection GET
|
||||
make test-pulumi # полный: все ручки пака × все HTTP-методы
|
||||
open reports/pulumi-report.html
|
||||
```
|
||||
|
||||
## Smoke vs полный suite
|
||||
|
||||
| Цель | Режим | Что проверяется |
|
||||
|---|---|---|
|
||||
| `make test-pulumi-smoke` | Smoke (`TEST_SMOKE=1`) | `pulumi_openstack` + **collection GET** с nonempty (быстро) |
|
||||
| `make pulumi-tests` / `make test-pulumi` | Полный lifecycle | `pulumi_openstack` + **все операции пака × GET/POST/PUT/PATCH/DELETE**, assert полноты (`total == размер пака`), nonempty тел успешных ответов (DELETE/204 могут быть пустыми) |
|
||||
|
||||
Размеры паков (ops): yoga ~1060 → antelope ~1108 → caracal ~1196 → **dalmatian ~1357**.
|
||||
|
||||
## Что выполняется (на каждую серию yoga → dalmatian)
|
||||
|
||||
1. Активация pack серии OpenStack
|
||||
2. **`pulumi up`** программы `programs/os_coverage` через Automation API —
|
||||
ресурсы через `pulumi_openstack` (identity, images, compute, networking,
|
||||
blockstorage, objectstorage, dns, orchestration)
|
||||
3. Проверка: **каждый export стека непустой**
|
||||
4. HTTP-probe pack-операций (smoke: collection GET; полный: lifecycle всех методов)
|
||||
5. Assert полноты покрытия + nonempty JSON у успешных ответов с телом
|
||||
6. `pulumi destroy`
|
||||
7. Отчёты `pulumi-report.html` + `pulumi-junit.xml`
|
||||
|
||||
## Отчёты
|
||||
|
||||
| Файл | Содержимое |
|
||||
|---|---|
|
||||
| `reports/pulumi-report.html` | HTML-сводка (expected vs actual + breakdown методов) |
|
||||
| `reports/pulumi-junit.xml` | JUnit |
|
||||
| `reports/series-<name>.json` | Детали pulumi + HTTP по серии |
|
||||
| `reports/summary.json` | Агрегаты |
|
||||
@@ -0,0 +1,162 @@
|
||||
# Pulumi OpenStack coverage lab.
|
||||
# Usage from repo root:
|
||||
# make pulumi-tests
|
||||
# docker compose -f pulumi-tests/docker-compose.yml --profile test run --rm pulumi-runner
|
||||
name: openstack-pulumi-tests
|
||||
|
||||
networks:
|
||||
lab:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
lab-postgres-data:
|
||||
lab-reports:
|
||||
|
||||
x-os-env: &os-env
|
||||
OS_AUTH_URL: http://api-gateway:5000/v3
|
||||
OS_USERNAME: admin
|
||||
OS_PASSWORD: secret
|
||||
OS_PROJECT_NAME: demo
|
||||
OS_USER_DOMAIN_NAME: Default
|
||||
OS_PROJECT_DOMAIN_NAME: Default
|
||||
OS_REGION_NAME: RegionOne
|
||||
OS_IDENTITY_API_VERSION: "3"
|
||||
OS_GATEWAY_URL: http://api-gateway:5000
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17.5-bookworm
|
||||
networks: [lab]
|
||||
environment:
|
||||
POSTGRES_DB: openstack_simulator
|
||||
POSTGRES_USER: openstack
|
||||
POSTGRES_PASSWORD: openstack
|
||||
volumes:
|
||||
- lab-postgres-data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U openstack -d openstack_simulator"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
|
||||
migrate:
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: Dockerfile
|
||||
target: runtime
|
||||
networks: [lab]
|
||||
working_dir: /workspace
|
||||
volumes:
|
||||
- ..:/workspace
|
||||
environment:
|
||||
DATABASE_URL: postgresql://openstack:openstack@postgres:5432/openstack_simulator
|
||||
PYTHONPATH: /workspace
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
entrypoint: ["python"]
|
||||
command: ["-m", "app.db.migrate_cli"]
|
||||
restart: "no"
|
||||
|
||||
simulator:
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: Dockerfile
|
||||
target: dev
|
||||
networks: [lab]
|
||||
working_dir: /workspace
|
||||
volumes:
|
||||
- ..:/workspace
|
||||
environment:
|
||||
DATABASE_URL: postgresql://openstack:openstack@postgres:5432/openstack_simulator
|
||||
PYTHONPATH: /workspace
|
||||
APP_PORT: "8080"
|
||||
LOG_LEVEL: INFO
|
||||
TICKET_SIGNING_KEY: lab-signing-key
|
||||
OPENSTACK_SERIES: dalmatian
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
entrypoint: []
|
||||
command:
|
||||
[
|
||||
"uvicorn",
|
||||
"app.main:app",
|
||||
"--host",
|
||||
"0.0.0.0",
|
||||
"--port",
|
||||
"8080",
|
||||
"--reload",
|
||||
"--reload-dir",
|
||||
"/workspace/app",
|
||||
]
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"python",
|
||||
"-c",
|
||||
"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=2)",
|
||||
]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
start_period: 15s
|
||||
|
||||
seed:
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: Dockerfile
|
||||
target: runtime
|
||||
networks: [lab]
|
||||
working_dir: /workspace
|
||||
volumes:
|
||||
- ..:/workspace
|
||||
environment:
|
||||
DATABASE_URL: postgresql://openstack:openstack@postgres:5432/openstack_simulator
|
||||
PYTHONPATH: /workspace
|
||||
depends_on:
|
||||
simulator:
|
||||
condition: service_healthy
|
||||
entrypoint: ["python"]
|
||||
command: ["-m", "app.openstack.seed_cli", "--profile", "demo"]
|
||||
restart: "no"
|
||||
|
||||
api-gateway:
|
||||
image: nginx:1.28-alpine
|
||||
networks: [lab]
|
||||
depends_on:
|
||||
simulator:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- ../docker/gateway/openstack-ports.conf:/etc/nginx/conf.d/default.conf:ro
|
||||
- ../docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
|
||||
- ../docker/tls/server.key:/etc/nginx/tls/server.key:ro
|
||||
ports:
|
||||
- "127.0.0.1:15000:5000"
|
||||
|
||||
pulumi-runner:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: docker/Dockerfile.pulumi-runner
|
||||
networks: [lab]
|
||||
working_dir: /suite
|
||||
volumes:
|
||||
- ./:/suite
|
||||
- ..:/workspace
|
||||
- ./reports:/reports
|
||||
- lab-reports:/reports-volume
|
||||
environment:
|
||||
<<: *os-env
|
||||
PYTHONPATH: /workspace:/suite/pulumi
|
||||
WORKSPACE: /workspace
|
||||
REPORT_DIR: /reports
|
||||
REPORT_PATH: /reports/pulumi-junit.xml
|
||||
PULUMI_CONFIG_PASSPHRASE: lab
|
||||
PULUMI_BACKEND_URL: file:///tmp/pulumi-state
|
||||
depends_on:
|
||||
seed:
|
||||
condition: service_completed_successfully
|
||||
api-gateway:
|
||||
condition: service_started
|
||||
profiles: ["test"]
|
||||
@@ -0,0 +1,24 @@
|
||||
FROM python:3.13-slim
|
||||
RUN pip install --no-cache-dir \
|
||||
"pulumi>=3.140,<4" \
|
||||
"pulumi-openstack>=5.0,<6" \
|
||||
&& apt-get update && apt-get install -y --no-install-recommends curl ca-certificates \
|
||||
&& curl -fsSL https://get.pulumi.com | sh \
|
||||
&& ln -sf /root/.pulumi/bin/pulumi /usr/local/bin/pulumi \
|
||||
&& /root/.pulumi/bin/pulumi plugin install resource openstack v5.3.1 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
# App deps for HTTP pack probe (app.openstack.surface_probe).
|
||||
RUN pip install --no-cache-dir \
|
||||
"asyncpg>=0.30,<0.31" \
|
||||
"fastapi>=0.116,<0.117" \
|
||||
"httpx>=0.28,<0.29" \
|
||||
"pydantic>=2.11,<3" \
|
||||
"pydantic-settings>=2.10,<3" \
|
||||
"uvicorn[standard]>=0.35,<0.36"
|
||||
WORKDIR /suite
|
||||
ENV PYTHONPATH=/workspace:/suite/pulumi \
|
||||
PULUMI_CONFIG_PASSPHRASE=lab \
|
||||
PULUMI_BACKEND_URL=file:///tmp/pulumi-state \
|
||||
WORKSPACE=/workspace \
|
||||
REPORT_DIR=/reports
|
||||
CMD ["python3", "/suite/pulumi/run_suite.py"]
|
||||
@@ -0,0 +1,11 @@
|
||||
# Example env for local debugging (runners inject these in compose).
|
||||
OS_AUTH_URL=http://api-gateway:5000/v3
|
||||
OS_USERNAME=admin
|
||||
OS_PASSWORD=secret
|
||||
OS_PROJECT_NAME=demo
|
||||
OS_USER_DOMAIN_NAME=Default
|
||||
OS_PROJECT_DOMAIN_NAME=Default
|
||||
OS_REGION_NAME=RegionOne
|
||||
OS_HTTP_TIMEOUT=30
|
||||
OS_POLL_TIMEOUT=90
|
||||
OS_POLL_INTERVAL=0.4
|
||||
@@ -0,0 +1 @@
|
||||
# Pulumi OpenStack coverage helpers.
|
||||
@@ -0,0 +1,208 @@
|
||||
"""HTTP pack coverage with completeness and non-empty body checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from _lib.validate import payload_nonempty
|
||||
|
||||
# Methods that normally return a JSON body on success (DELETE / 204 may be empty).
|
||||
_BODY_METHODS = frozenset({"GET", "POST", "PUT", "PATCH"})
|
||||
|
||||
|
||||
def _expected_ops(packs: dict[str, Any], *, collections_only: bool) -> int:
|
||||
if not collections_only:
|
||||
return sum(len(p.operations) for p in packs.values())
|
||||
total = 0
|
||||
for pack in packs.values():
|
||||
for op in pack.operations:
|
||||
if op.method == "GET" and "{" not in op.path:
|
||||
total += 1
|
||||
return total
|
||||
|
||||
|
||||
def _methods_breakdown(results: list[Any]) -> dict[str, int]:
|
||||
counts: Counter[str] = Counter()
|
||||
for r in results:
|
||||
method = getattr(r, "method", None) or (r.get("method") if isinstance(r, dict) else None)
|
||||
if method:
|
||||
counts[str(method).upper()] += 1
|
||||
return {m: counts.get(m, 0) for m in ("GET", "POST", "PUT", "PATCH", "DELETE")}
|
||||
|
||||
|
||||
def _nonempty_from_lifecycle(report: Any) -> list[dict[str, Any]]:
|
||||
"""Check succeeded lifecycle bodies (skip DELETE / 204 / 202 / no-body)."""
|
||||
failures: list[dict[str, Any]] = []
|
||||
for r in report.results:
|
||||
if not r.succeeded:
|
||||
continue
|
||||
if r.method == "DELETE" or r.status in {202, 204}:
|
||||
continue
|
||||
if r.method not in _BODY_METHODS:
|
||||
continue
|
||||
# OpenStack often returns 200/201 with an empty body (Swift PUT, tag put).
|
||||
if r.payload is None:
|
||||
continue
|
||||
if not payload_nonempty(r.payload, collection_key=r.collection_key, method=r.method):
|
||||
failures.append(
|
||||
{
|
||||
"service": r.service,
|
||||
"operation_id": r.operation_id,
|
||||
"method": r.method,
|
||||
"path": r.path,
|
||||
"status": r.status,
|
||||
"detail": "empty response body",
|
||||
"ok": False,
|
||||
"nonempty": False,
|
||||
}
|
||||
)
|
||||
return failures
|
||||
|
||||
|
||||
def _nonempty_smoke_collections(
|
||||
series: str,
|
||||
*,
|
||||
host: str,
|
||||
packs: dict[str, Any],
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Re-check collection GET bodies for smoke mode (stable seed data)."""
|
||||
from app.openstack.surface_probe import (
|
||||
SUCCESS,
|
||||
fill_path,
|
||||
http_request,
|
||||
issue_token,
|
||||
_seed_context,
|
||||
)
|
||||
|
||||
token, auth_body = issue_token(host)
|
||||
project_id = str(((auth_body.get("token") or {}).get("project") or {}).get("id") or "")
|
||||
ctx = _seed_context(host, token, project_id)
|
||||
failures: list[dict[str, Any]] = []
|
||||
for name in sorted(packs):
|
||||
pack = packs[name]
|
||||
for op in pack.operations:
|
||||
if "{" in op.path or op.method != "GET":
|
||||
continue
|
||||
path = fill_path(
|
||||
op.path,
|
||||
{
|
||||
**ctx,
|
||||
"project_id": project_id,
|
||||
"project": project_id,
|
||||
"tenant_id": project_id,
|
||||
"account": project_id,
|
||||
},
|
||||
)
|
||||
url = f"{host.rstrip('/')}{path}"
|
||||
status, payload = http_request(op.method, url, token=token, service=pack.name)
|
||||
if status not in SUCCESS or status == 204:
|
||||
continue
|
||||
if not payload_nonempty(payload, collection_key=op.collection_key, method=op.method):
|
||||
failures.append(
|
||||
{
|
||||
"service": pack.name,
|
||||
"operation_id": op.operation_id,
|
||||
"method": op.method,
|
||||
"path": op.path,
|
||||
"status": status,
|
||||
"detail": "empty response body",
|
||||
"ok": False,
|
||||
"nonempty": False,
|
||||
}
|
||||
)
|
||||
return failures
|
||||
|
||||
|
||||
def probe_pack_operations(
|
||||
series: str,
|
||||
*,
|
||||
host: str,
|
||||
collections_only: bool = False,
|
||||
require_nonempty: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
from app.openstack.contract_loader import load_series_pack
|
||||
from app.openstack.surface_probe import probe_series
|
||||
|
||||
report = probe_series(
|
||||
series,
|
||||
host=host,
|
||||
collections_only=collections_only,
|
||||
lifecycle=not collections_only,
|
||||
)
|
||||
|
||||
packs = load_series_pack(series)
|
||||
expected_ops = _expected_ops(packs, collections_only=collections_only)
|
||||
methods = _methods_breakdown(report.results)
|
||||
coverage_incomplete = len(report.results) != expected_ops
|
||||
|
||||
nonempty_failures: list[dict[str, Any]] = []
|
||||
if require_nonempty:
|
||||
if collections_only:
|
||||
nonempty_failures = _nonempty_smoke_collections(series, host=host, packs=packs)
|
||||
else:
|
||||
nonempty_failures = _nonempty_from_lifecycle(report)
|
||||
|
||||
probe_failures = [
|
||||
{
|
||||
"service": r.service,
|
||||
"operation_id": r.operation_id,
|
||||
"method": r.method,
|
||||
"path": r.path,
|
||||
"status": r.status,
|
||||
"detail": r.detail,
|
||||
"ok": False,
|
||||
"nonempty": True,
|
||||
}
|
||||
for r in report.failures
|
||||
]
|
||||
|
||||
coverage_failures: list[dict[str, Any]] = []
|
||||
if coverage_incomplete:
|
||||
coverage_failures.append(
|
||||
{
|
||||
"service": "_coverage",
|
||||
"operation_id": "coverage_incomplete",
|
||||
"method": "*",
|
||||
"path": "*",
|
||||
"status": 0,
|
||||
"detail": f"coverage_incomplete: total={len(report.results)} expected_ops={expected_ops}",
|
||||
"ok": False,
|
||||
"nonempty": True,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"series": series,
|
||||
"host": host,
|
||||
"mode": report.mode,
|
||||
"total": len(report.results),
|
||||
"expected_ops": expected_ops,
|
||||
"coverage_incomplete": coverage_incomplete,
|
||||
"methods": methods,
|
||||
"ok_count": len(report.results) - len(report.failures),
|
||||
"fail_count": len(report.failures) + (1 if coverage_incomplete else 0),
|
||||
"nonempty_fail_count": len(nonempty_failures),
|
||||
"results": [
|
||||
{
|
||||
"service": r.service,
|
||||
"method": r.method,
|
||||
"path": r.path,
|
||||
"operation_id": r.operation_id,
|
||||
"status": r.status,
|
||||
"detail": r.detail,
|
||||
"mode": r.mode,
|
||||
"ok": r.ok,
|
||||
"succeeded": r.succeeded,
|
||||
}
|
||||
for r in report.results
|
||||
],
|
||||
"failures": coverage_failures + probe_failures + nonempty_failures,
|
||||
}
|
||||
|
||||
|
||||
def write_probe_json(payload: dict[str, Any], path: Path) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Render HTML report from Pulumi + HTTP coverage results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SERIES_ORDER = ("yoga", "antelope", "caracal", "dalmatian")
|
||||
|
||||
|
||||
def _methods_line(methods: dict[str, Any]) -> str:
|
||||
return " ".join(f"{m}={methods.get(m, 0)}" for m in ("GET", "POST", "PUT", "PATCH", "DELETE"))
|
||||
|
||||
|
||||
def render_html(summary: dict[str, Any], series_reports: list[dict[str, Any]]) -> str:
|
||||
generated = datetime.now(UTC).strftime("%Y-%m-%d %H:%M:%S UTC")
|
||||
cards = []
|
||||
for rep in series_reports:
|
||||
series = html.escape(str(rep.get("series", "?")))
|
||||
pu = rep.get("pulumi", {})
|
||||
http = rep.get("http", {})
|
||||
cards.append(
|
||||
f"""
|
||||
<div class="card">
|
||||
<h3>{series}</h3>
|
||||
<p class="muted">pulumi_openstack + HTTP pack probe</p>
|
||||
<div class="stats">
|
||||
<span class="ok">pulumi exports ok={len(pu.get("outputs", {})) - len(pu.get("empty_exports", []))}</span>
|
||||
<span class="fail">empty exports={len(pu.get("empty_exports", []))}</span>
|
||||
</div>
|
||||
<div class="stats">
|
||||
<span class="ok">http ok={http.get("ok_count", 0)}</span>
|
||||
<span class="fail">http fail={http.get("fail_count", 0)} nonempty_fail={http.get("nonempty_fail_count", 0)}</span>
|
||||
<span>http total={http.get("total", 0)}/{http.get("expected_ops", "?")}</span>
|
||||
</div>
|
||||
<div class="stats muted">
|
||||
methods: {html.escape(_methods_line(http.get("methods") or {}))}
|
||||
{" · <span class='fail'>coverage incomplete</span>" if http.get("coverage_incomplete") else ""}
|
||||
</div>
|
||||
</div>"""
|
||||
)
|
||||
|
||||
detail_rows = []
|
||||
for rep in series_reports:
|
||||
series = rep.get("series", "?")
|
||||
for item in rep.get("http", {}).get("failures", [])[:500]:
|
||||
detail_rows.append(
|
||||
'<tr class="fail">'
|
||||
f"<td>{html.escape(str(series))}</td>"
|
||||
f"<td>{html.escape(str(item.get('service', '')))}</td>"
|
||||
f"<td>{html.escape(str(item.get('operation_id', '')))}</td>"
|
||||
f"<td>{html.escape(str(item.get('method', '')))}</td>"
|
||||
f"<td><code>{html.escape(str(item.get('path', '')))}</code></td>"
|
||||
f"<td>{html.escape(str(item.get('status', '')))}</td>"
|
||||
f"<td>{html.escape(str(item.get('detail', ''))[:240])}</td>"
|
||||
"</tr>"
|
||||
)
|
||||
for empty in rep.get("pulumi", {}).get("empty_exports", []):
|
||||
detail_rows.append(
|
||||
'<tr class="fail">'
|
||||
f"<td>{html.escape(str(series))}</td>"
|
||||
f"<td>pulumi_openstack</td>"
|
||||
f"<td>export_nonempty</td>"
|
||||
f"<td>—</td><td><code>export</code></td><td>—</td>"
|
||||
f"<td>{html.escape(str(empty))}</td>"
|
||||
"</tr>"
|
||||
)
|
||||
|
||||
return f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8"/>
|
||||
<title>Pulumi OpenStack coverage</title>
|
||||
<style>
|
||||
:root {{ --bg:#0f1419; --panel:#1a222c; --text:#e7ecf3; --muted:#9aa7b8; --ok:#3dd68c; --fail:#ff6b6b; --border:#2a3544; }}
|
||||
body {{ margin:0; font-family:"IBM Plex Sans",sans-serif; background:radial-gradient(1000px 500px at 0% 0%,#1b2a3d,var(--bg)); color:var(--text); }}
|
||||
header, main {{ max-width:1200px; margin:0 auto; padding:1.5rem; }}
|
||||
.summary, .cards {{ display:grid; gap:.75rem; grid-template-columns:repeat(auto-fit,minmax(160px,1fr)); margin:1rem 0; }}
|
||||
.summary div, .card {{ background:var(--panel); border:1px solid var(--border); border-radius:10px; padding:1rem; }}
|
||||
.ok {{ color:var(--ok); font-weight:600; }} .fail {{ color:var(--fail); font-weight:600; }}
|
||||
.muted {{ color:var(--muted); }}
|
||||
table {{ width:100%; border-collapse:collapse; background:var(--panel); border:1px solid var(--border); border-radius:10px; overflow:hidden; font-size:.9rem; }}
|
||||
th, td {{ padding:.45rem .6rem; border-bottom:1px solid var(--border); text-align:left; vertical-align:top; }}
|
||||
th {{ color:var(--muted); background:#121820; }}
|
||||
tr.fail {{ background:rgba(255,107,107,.08); }}
|
||||
code {{ font-family:ui-monospace,monospace; font-size:.82rem; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>Pulumi OpenStack API coverage</h1>
|
||||
<p class="muted">Generated {html.escape(generated)} · pulumi_openstack primary + HTTP pack probe with non-empty checks</p>
|
||||
</header>
|
||||
<main>
|
||||
<div class="summary">
|
||||
<div><strong>{summary.get("series_count", 0)}</strong><span class="muted"> series</span></div>
|
||||
<div><strong class="ok">{summary.get("pulumi_ok", 0)}</strong><span class="muted"> pulumi stacks ok</span></div>
|
||||
<div><strong class="fail">{summary.get("pulumi_fail", 0)}</strong><span class="muted"> pulumi failures</span></div>
|
||||
<div><strong class="ok">{summary.get("http_ok", 0)}</strong><span class="muted"> http ops ok</span></div>
|
||||
<div><strong class="fail">{summary.get("http_fail", 0)}</strong><span class="muted"> http / nonempty fails</span></div>
|
||||
</div>
|
||||
<h2>Series</h2>
|
||||
<div class="cards">{"".join(cards)}</div>
|
||||
<h2>Failures</h2>
|
||||
<table>
|
||||
<thead><tr><th>Series</th><th>Service</th><th>Operation</th><th>Method</th><th>Path</th><th>HTTP</th><th>Detail</th></tr></thead>
|
||||
<tbody>{"".join(detail_rows) if detail_rows else '<tr><td colspan="7">No failures</td></tr>'}</tbody>
|
||||
</table>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
|
||||
def write_html(
|
||||
report_dir: Path, summary: dict[str, Any], series_reports: list[dict[str, Any]]
|
||||
) -> Path:
|
||||
path = report_dir / "pulumi-report.html"
|
||||
path.write_text(render_html(summary, series_reports), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def load_series_files(report_dir: Path) -> list[dict[str, Any]]:
|
||||
out = []
|
||||
for series in SERIES_ORDER:
|
||||
path = report_dir / f"series-{series}.json"
|
||||
if path.exists():
|
||||
out.append(json.loads(path.read_text(encoding="utf-8")))
|
||||
return out
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Helpers for series activation and non-empty validation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
|
||||
def activate_series(host: str, series: str) -> None:
|
||||
body = json.dumps({"series": series}).encode()
|
||||
req = urllib.request.Request(
|
||||
f"{host.rstrip('/')}/ui/api/openstack/contracts/activate",
|
||||
data=body,
|
||||
headers={"Content-Type": "application/json", "Accept": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as res:
|
||||
if res.status >= 400:
|
||||
raise RuntimeError(f"activate {series}: HTTP {res.status}")
|
||||
except urllib.error.HTTPError as exc:
|
||||
raw = exc.read().decode()
|
||||
raise RuntimeError(f"activate {series}: HTTP {exc.code} {raw[:300]}") from exc
|
||||
|
||||
|
||||
def is_nonempty(value: Any) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
if isinstance(value, str) and not value.strip():
|
||||
return False
|
||||
if isinstance(value, (list, tuple, set, dict)) and len(value) == 0:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def assert_outputs_nonempty(outputs: dict[str, Any], *, min_count: int = 15) -> list[str]:
|
||||
"""Return list of failing export names (empty if all good)."""
|
||||
failures: list[str] = []
|
||||
if len(outputs) < min_count:
|
||||
failures.append(f"__export_count__={len(outputs)}<{min_count}")
|
||||
for key, value in sorted(outputs.items()):
|
||||
if not is_nonempty(value):
|
||||
failures.append(f"{key}={value!r}")
|
||||
return failures
|
||||
|
||||
|
||||
def payload_nonempty(
|
||||
payload: Any, *, collection_key: str | None = None, method: str = "GET"
|
||||
) -> bool:
|
||||
"""True when a successful response body has meaningful content.
|
||||
|
||||
For GET list/show and POST create we require real data — empty ``[]`` /
|
||||
``{}`` / blank strings fail. DELETE/204-style empties are not checked here.
|
||||
"""
|
||||
if payload is None:
|
||||
return False
|
||||
if isinstance(payload, str):
|
||||
return bool(payload.strip())
|
||||
if isinstance(payload, list):
|
||||
return len(payload) > 0
|
||||
if not isinstance(payload, dict):
|
||||
return True
|
||||
if collection_key and collection_key in payload:
|
||||
value = payload[collection_key]
|
||||
if isinstance(value, list):
|
||||
return len(value) > 0
|
||||
return is_nonempty(value)
|
||||
# Common OpenStack envelopes
|
||||
for key, value in payload.items():
|
||||
if key in {"versions", "version", "id", "token", "links", "status", "name"}:
|
||||
if is_nonempty(value):
|
||||
return True
|
||||
if isinstance(value, list) and value:
|
||||
return True
|
||||
if isinstance(value, dict) and (value.get("id") or value.get("name") or value.get("uuid")):
|
||||
return True
|
||||
if isinstance(value, str) and value.strip():
|
||||
return True
|
||||
if isinstance(value, (int, float, bool)):
|
||||
return True
|
||||
# Non-empty dict with any nested content
|
||||
return any(is_nonempty(v) for v in payload.values())
|
||||
@@ -0,0 +1 @@
|
||||
encryptionsalt: v1:sHIQhiAs888=:v1:YU5x9MYzw3X3DaH0:SmpMQVn+3UVBkcz4HBf42IVasHoToQ==
|
||||
@@ -0,0 +1 @@
|
||||
encryptionsalt: v1:jXbSF2V0mog=:v1:ZhW0i106mP+RAph4:sS0KukNKHIsJ/z0Ozeoaa2Zf3Cj8nw==
|
||||
@@ -0,0 +1 @@
|
||||
encryptionsalt: v1:SXzsEq8DgEw=:v1:RQUIpYKUzggs7nqf:yUC20qV384YMu+fnGhjFKH7aSD5Dkg==
|
||||
@@ -0,0 +1 @@
|
||||
encryptionsalt: v1:B2EuV1+nh3k=:v1:9khdcorRMBI/WL1d:e08aEeItHe3flfvAWfDkSO9LUxEUrQ==
|
||||
@@ -0,0 +1,3 @@
|
||||
name: os-coverage
|
||||
runtime: python
|
||||
description: pulumi_openstack coverage stack for openstack-api-simulator
|
||||
@@ -0,0 +1,157 @@
|
||||
"""Maximize pulumi_openstack coverage against the OpenStack API simulator.
|
||||
|
||||
Creates / looks up resources via the official provider, then exports IDs so
|
||||
the suite can assert every stack output is non-empty.
|
||||
|
||||
Notes vs simulator limits:
|
||||
- flavor_id is seeded ``1`` (get_flavor fails on extra_specs shape)
|
||||
- skip Heat/Designate/Swift/Octavia creates (status codes or catalog URL
|
||||
shapes the lab does not yet match)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import uuid
|
||||
|
||||
import pulumi
|
||||
from pulumi_openstack import blockstorage, compute, identity, images, networking
|
||||
|
||||
series = os.environ.get("OPENSTACK_SERIES", "dalmatian")
|
||||
tag = f"pu-{series}-{uuid.uuid4().hex[:6]}"
|
||||
|
||||
# --- Data sources (reads) ---
|
||||
auth = identity.get_auth_scope(name="lab-token")
|
||||
image = images.get_image(name="cirros", most_recent=True)
|
||||
demo_net = networking.get_network(name="demo-net")
|
||||
flavor_id = "1" # seeded m1.tiny
|
||||
|
||||
# --- Identity (writes) ---
|
||||
project = identity.Project(
|
||||
f"{tag}-project",
|
||||
name=f"{tag}-project",
|
||||
description=f"pulumi coverage {series}",
|
||||
enabled=True,
|
||||
)
|
||||
user = identity.User(
|
||||
f"{tag}-user",
|
||||
name=f"{tag}-user",
|
||||
password="pulumi-lab-secret",
|
||||
description=f"pulumi coverage {series}",
|
||||
enabled=True,
|
||||
)
|
||||
|
||||
# --- Networking ---
|
||||
app_net = networking.Network(f"{tag}-net", name=f"{tag}-net", admin_state_up=True)
|
||||
app_subnet = networking.Subnet(
|
||||
f"{tag}-subnet",
|
||||
name=f"{tag}-subnet",
|
||||
network_id=app_net.id,
|
||||
cidr="10.88.0.0/24",
|
||||
ip_version=4,
|
||||
enable_dhcp=True,
|
||||
)
|
||||
router = networking.Router(f"{tag}-router", name=f"{tag}-router", admin_state_up=True)
|
||||
router_iface = networking.RouterInterface(
|
||||
f"{tag}-rtr-if",
|
||||
router_id=router.id,
|
||||
subnet_id=app_subnet.id,
|
||||
)
|
||||
sg = networking.SecGroup(
|
||||
f"{tag}-sg",
|
||||
name=f"{tag}-sg",
|
||||
description=f"pulumi {series}",
|
||||
delete_default_rules=True,
|
||||
)
|
||||
sg_rule = networking.SecGroupRule(
|
||||
f"{tag}-sg-ssh",
|
||||
direction="ingress",
|
||||
ethertype="IPv4",
|
||||
protocol="tcp",
|
||||
port_range_min=22,
|
||||
port_range_max=22,
|
||||
remote_ip_prefix="0.0.0.0/0",
|
||||
security_group_id=sg.id,
|
||||
)
|
||||
port = networking.Port(
|
||||
f"{tag}-port",
|
||||
name=f"{tag}-port",
|
||||
network_id=app_net.id,
|
||||
admin_state_up=True,
|
||||
security_group_ids=[sg.id],
|
||||
fixed_ips=[networking.PortFixedIpArgs(subnet_id=app_subnet.id)],
|
||||
)
|
||||
|
||||
# --- Compute ---
|
||||
keypair = compute.Keypair(f"{tag}-kp", name=f"{tag}-kp")
|
||||
server_group = compute.ServerGroup(
|
||||
f"{tag}-sgroup",
|
||||
name=f"{tag}-sgroup",
|
||||
policies="anti-affinity",
|
||||
)
|
||||
server = compute.Instance(
|
||||
f"{tag}-vm",
|
||||
name=f"{tag}-vm",
|
||||
flavor_id=flavor_id,
|
||||
image_id=image.id,
|
||||
key_pair=keypair.name,
|
||||
security_groups=[sg.name],
|
||||
networks=[compute.InstanceNetworkArgs(uuid=demo_net.id)],
|
||||
scheduler_hints=[compute.InstanceSchedulerHintArgs(group=server_group.id)],
|
||||
metadata={"managed_by": "pulumi", "series": series, "tag": tag},
|
||||
)
|
||||
iface = compute.InterfaceAttach(
|
||||
f"{tag}-iface",
|
||||
instance_id=server.id,
|
||||
port_id=port.id,
|
||||
)
|
||||
|
||||
# --- Block storage ---
|
||||
volume = blockstorage.Volume(
|
||||
f"{tag}-vol",
|
||||
name=f"{tag}-vol",
|
||||
size=1,
|
||||
description=f"pulumi coverage {series}",
|
||||
)
|
||||
vol_attach = compute.VolumeAttach(
|
||||
f"{tag}-attach",
|
||||
instance_id=server.id,
|
||||
volume_id=volume.id,
|
||||
)
|
||||
volume2 = blockstorage.Volume(
|
||||
f"{tag}-vol2",
|
||||
name=f"{tag}-vol2",
|
||||
size=1,
|
||||
description=f"pulumi second volume {series}",
|
||||
)
|
||||
|
||||
# Octavia / Designate / Swift / Heat omitted: catalog paths or status codes
|
||||
# in the lab gateway do not yet match what pulumi_openstack expects.
|
||||
|
||||
# --- Exports (all must be non-empty; suite requires >= 25) ---
|
||||
pulumi.export("series", series)
|
||||
pulumi.export("tag", tag)
|
||||
pulumi.export("auth_user_id", auth.user_id)
|
||||
pulumi.export("auth_project_id", auth.project_id)
|
||||
pulumi.export("project_name", auth.project_name)
|
||||
pulumi.export("flavor_id", flavor_id)
|
||||
pulumi.export("image_id", image.id)
|
||||
pulumi.export("image_name", image.name)
|
||||
pulumi.export("demo_net_id", demo_net.id)
|
||||
pulumi.export("created_project_id", project.id)
|
||||
pulumi.export("created_user_id", user.id)
|
||||
pulumi.export("network_id", app_net.id)
|
||||
pulumi.export("subnet_id", app_subnet.id)
|
||||
pulumi.export("router_id", router.id)
|
||||
pulumi.export("router_iface_id", router_iface.id)
|
||||
pulumi.export("secgroup_id", sg.id)
|
||||
pulumi.export("secgroup_rule_id", sg_rule.id)
|
||||
pulumi.export("port_id", port.id)
|
||||
pulumi.export("keypair_name", keypair.name)
|
||||
pulumi.export("server_group_id", server_group.id)
|
||||
pulumi.export("server_id", server.id)
|
||||
pulumi.export("server_name", server.name)
|
||||
pulumi.export("interface_id", iface.id)
|
||||
pulumi.export("volume_id", volume.id)
|
||||
pulumi.export("volume2_id", volume2.id)
|
||||
pulumi.export("volume_attach_id", vol_attach.id)
|
||||
@@ -0,0 +1,2 @@
|
||||
pulumi>=3.140,<4
|
||||
pulumi-openstack>=5.0,<6
|
||||
@@ -0,0 +1,294 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Pulumi OpenStack coverage runner.
|
||||
|
||||
For each series (yoga → dalmatian):
|
||||
1. Activate the OpenStack pack
|
||||
2. ``pulumi up`` a pulumi_openstack program (maximises provider coverage)
|
||||
3. Assert every stack export is non-empty
|
||||
4. HTTP-probe remaining pack operations; require non-empty GET/POST bodies
|
||||
5. ``pulumi destroy``
|
||||
6. Emit JUnit + HTML report
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SERIES_ORDER = ("yoga", "antelope", "caracal", "dalmatian")
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
REPO = Path(os.environ.get("WORKSPACE", "/workspace"))
|
||||
PROGRAM = ROOT / "pulumi" / "programs" / "os_coverage"
|
||||
REPORT_DIR = Path(os.environ.get("REPORT_DIR", "/reports"))
|
||||
|
||||
sys.path.insert(0, str(REPO))
|
||||
sys.path.insert(0, str(ROOT / "pulumi"))
|
||||
|
||||
|
||||
def _smoke() -> bool:
|
||||
return os.environ.get("TEST_SMOKE", "").strip().lower() in {"1", "true", "yes"}
|
||||
|
||||
|
||||
def _series_list() -> list[str]:
|
||||
raw = os.environ.get("OPENSTACK_SERIES_LIST", "").strip()
|
||||
if raw:
|
||||
return [s.strip() for s in raw.split(",") if s.strip()]
|
||||
return list(SERIES_ORDER)
|
||||
|
||||
|
||||
def _host() -> str:
|
||||
return os.environ.get("OS_GATEWAY_URL") or os.environ.get(
|
||||
"OS_AUTH_URL", "http://api-gateway:5000/v3"
|
||||
).removesuffix("/v3").rstrip("/")
|
||||
|
||||
|
||||
def _env_vars(series: str) -> dict[str, str]:
|
||||
backend = os.environ.get("PULUMI_BACKEND_URL", "file:///tmp/pulumi-state")
|
||||
if backend.startswith("file://"):
|
||||
Path(backend.removeprefix("file://").split("?", 1)[0]).mkdir(parents=True, exist_ok=True)
|
||||
return {
|
||||
"OS_AUTH_URL": os.environ.get("OS_AUTH_URL", f"{_host()}/v3"),
|
||||
"OS_USERNAME": os.environ.get("OS_USERNAME", "admin"),
|
||||
"OS_PASSWORD": os.environ.get("OS_PASSWORD", "secret"),
|
||||
"OS_PROJECT_NAME": os.environ.get("OS_PROJECT_NAME", "demo"),
|
||||
"OS_USER_DOMAIN_NAME": os.environ.get("OS_USER_DOMAIN_NAME", "Default"),
|
||||
"OS_PROJECT_DOMAIN_NAME": os.environ.get("OS_PROJECT_DOMAIN_NAME", "Default"),
|
||||
"OS_REGION_NAME": os.environ.get("OS_REGION_NAME", "RegionOne"),
|
||||
"OS_INTERFACE": "public",
|
||||
"OS_IDENTITY_API_VERSION": "3",
|
||||
"OPENSTACK_SERIES": series,
|
||||
"PULUMI_CONFIG_PASSPHRASE": os.environ.get("PULUMI_CONFIG_PASSPHRASE", "lab"),
|
||||
"PULUMI_BACKEND_URL": backend,
|
||||
"PYTHONPATH": f"{REPO}:{ROOT / 'pulumi'}:{os.environ.get('PYTHONPATH', '')}",
|
||||
"WORKSPACE": str(REPO),
|
||||
}
|
||||
|
||||
|
||||
def run_pulumi_stack(series: str) -> dict[str, Any]:
|
||||
from pulumi import automation as auto
|
||||
|
||||
from _lib.validate import activate_series, assert_outputs_nonempty
|
||||
|
||||
host = _host()
|
||||
activate_series(host, series)
|
||||
env_vars = _env_vars(series)
|
||||
stack_name = f"os-coverage-{series}"
|
||||
t0 = time.time()
|
||||
|
||||
stack = auto.create_or_select_stack(
|
||||
stack_name=stack_name,
|
||||
work_dir=str(PROGRAM),
|
||||
opts=auto.LocalWorkspaceOptions(
|
||||
env_vars=env_vars,
|
||||
# Install program requirements into workspace on first run.
|
||||
),
|
||||
)
|
||||
try:
|
||||
stack.workspace.install_plugin("openstack", "v5.3.1")
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
try:
|
||||
# Ensure python deps for the program
|
||||
stack.workspace.run_cmd(["python3", "-m", "pip", "install", "-q", "-r", "requirements.txt"])
|
||||
except Exception: # noqa: BLE001
|
||||
# LocalWorkspace may not expose run_cmd on all versions — pip in image instead.
|
||||
pass
|
||||
|
||||
empty: list[str] = []
|
||||
outputs: dict[str, Any] = {}
|
||||
error: str | None = None
|
||||
try:
|
||||
up = stack.up(on_output=lambda _: None)
|
||||
outputs = {
|
||||
k: (v.value if hasattr(v, "value") else v) for k, v in (up.outputs or {}).items()
|
||||
}
|
||||
empty = assert_outputs_nonempty(outputs, min_count=25)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
error = str(exc)[:4000]
|
||||
finally:
|
||||
try:
|
||||
stack.destroy(on_output=lambda _: None)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
return {
|
||||
"series": series,
|
||||
"elapsed_s": round(time.time() - t0, 2),
|
||||
"error": error,
|
||||
"outputs": {k: outputs[k] for k in sorted(outputs)},
|
||||
"empty_exports": empty,
|
||||
"ok": error is None and not empty,
|
||||
}
|
||||
|
||||
|
||||
def run_http_coverage(series: str, *, collections_only: bool) -> dict[str, Any]:
|
||||
from _lib.http_coverage import probe_pack_operations
|
||||
|
||||
return probe_pack_operations(
|
||||
series,
|
||||
host=_host(),
|
||||
collections_only=collections_only,
|
||||
require_nonempty=True,
|
||||
)
|
||||
|
||||
|
||||
def write_junit(series_reports: list[dict[str, Any]], path: Path) -> None:
|
||||
cases: list[tuple[str, bool, str]] = []
|
||||
for rep in series_reports:
|
||||
series = rep["series"]
|
||||
pu = rep["pulumi"]
|
||||
cases.append(
|
||||
(
|
||||
f"{series}.pulumi_openstack.stack",
|
||||
not pu.get("ok", False),
|
||||
pu.get("error") or (", ".join(pu.get("empty_exports") or []) or "ok"),
|
||||
)
|
||||
)
|
||||
for empty in pu.get("empty_exports") or []:
|
||||
cases.append((f"{series}.pulumi_openstack.nonempty.{empty.split('=')[0]}", True, empty))
|
||||
http = rep.get("http") or {}
|
||||
for item in http.get("results") or []:
|
||||
name = f"{series}.http.{item.get('service')}.{item.get('operation_id')}"
|
||||
failed = (
|
||||
http.get("mode") == "lifecycle"
|
||||
and item.get("mode") == "lifecycle"
|
||||
and not item.get("succeeded")
|
||||
) or (not item.get("ok"))
|
||||
# nonempty failures are separate entries in failures list
|
||||
cases.append(
|
||||
(
|
||||
name,
|
||||
bool(failed),
|
||||
f"{item.get('method')} {item.get('path')} → {item.get('status')}",
|
||||
)
|
||||
)
|
||||
for fail in http.get("failures") or []:
|
||||
detail = str(fail.get("detail") or "")
|
||||
if detail == "empty response body":
|
||||
cases.append(
|
||||
(
|
||||
f"{series}.http.nonempty.{fail.get('service')}.{fail.get('operation_id')}",
|
||||
True,
|
||||
"empty response body",
|
||||
)
|
||||
)
|
||||
elif fail.get("operation_id") == "coverage_incomplete" or detail.startswith(
|
||||
"coverage_incomplete"
|
||||
):
|
||||
cases.append(
|
||||
(
|
||||
f"{series}.http.coverage_incomplete",
|
||||
True,
|
||||
detail,
|
||||
)
|
||||
)
|
||||
|
||||
suite = ET.Element(
|
||||
"testsuite",
|
||||
name="pulumi-openstack-coverage",
|
||||
tests=str(len(cases)),
|
||||
failures=str(sum(1 for _, failed, _ in cases if failed)),
|
||||
)
|
||||
for name, failed, detail in cases:
|
||||
case = ET.SubElement(suite, "testcase", classname="pulumi", name=name)
|
||||
if failed:
|
||||
node = ET.SubElement(case, "failure", message=detail[:300])
|
||||
node.text = detail
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
ET.ElementTree(suite).write(path, encoding="utf-8", xml_declaration=True)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
REPORT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
collections_only = _smoke() or os.environ.get("COLLECTIONS_ONLY", "").lower() in {
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
}
|
||||
skip_http = os.environ.get("SKIP_HTTP_COVERAGE", "").lower() in {"1", "true", "yes"}
|
||||
series_list = _series_list()
|
||||
print(
|
||||
f"Pulumi coverage: series={','.join(series_list)} "
|
||||
f"collections_only={collections_only} host={_host()}"
|
||||
)
|
||||
|
||||
series_reports: list[dict[str, Any]] = []
|
||||
for series in series_list:
|
||||
print(f"==> series {series}: pulumi_openstack")
|
||||
pu = run_pulumi_stack(series)
|
||||
print(
|
||||
f"{series}: pulumi ok={pu.get('ok')} empty_exports={len(pu.get('empty_exports') or [])} "
|
||||
f"error={'yes' if pu.get('error') else 'no'}"
|
||||
)
|
||||
http: dict[str, Any]
|
||||
if skip_http:
|
||||
http = {
|
||||
"series": series,
|
||||
"total": 0,
|
||||
"expected_ops": 0,
|
||||
"coverage_incomplete": False,
|
||||
"methods": {"GET": 0, "POST": 0, "PUT": 0, "PATCH": 0, "DELETE": 0},
|
||||
"ok_count": 0,
|
||||
"fail_count": 0,
|
||||
"nonempty_fail_count": 0,
|
||||
"results": [],
|
||||
"failures": [],
|
||||
}
|
||||
else:
|
||||
print(f"==> series {series}: HTTP pack probe (nonempty)")
|
||||
http = run_http_coverage(series, collections_only=collections_only)
|
||||
methods = http.get("methods") or {}
|
||||
methods_s = " ".join(
|
||||
f"{m}={methods.get(m, 0)}" for m in ("GET", "POST", "PUT", "PATCH", "DELETE")
|
||||
)
|
||||
print(
|
||||
f"{series}: http ok={http.get('ok_count')} fail={http.get('fail_count')} "
|
||||
f"nonempty_fail={http.get('nonempty_fail_count')} "
|
||||
f"total={http.get('total')}/{http.get('expected_ops')} "
|
||||
f"coverage_incomplete={http.get('coverage_incomplete')} "
|
||||
f"methods[{methods_s}]"
|
||||
)
|
||||
rep = {"series": series, "pulumi": pu, "http": http}
|
||||
series_reports.append(rep)
|
||||
(REPORT_DIR / f"series-{series}.json").write_text(
|
||||
json.dumps(rep, indent=2) + "\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
write_junit(series_reports, REPORT_DIR / "pulumi-junit.xml")
|
||||
|
||||
from _lib.report_html import write_html
|
||||
|
||||
summary = {
|
||||
"series_count": len(series_reports),
|
||||
"pulumi_ok": sum(1 for r in series_reports if r["pulumi"].get("ok")),
|
||||
"pulumi_fail": sum(1 for r in series_reports if not r["pulumi"].get("ok")),
|
||||
"http_ok": sum(int(r["http"].get("ok_count", 0)) for r in series_reports),
|
||||
"http_fail": sum(
|
||||
int(r["http"].get("fail_count", 0)) + int(r["http"].get("nonempty_fail_count", 0))
|
||||
for r in series_reports
|
||||
),
|
||||
"coverage_incomplete": sum(
|
||||
1 for r in series_reports if r["http"].get("coverage_incomplete")
|
||||
),
|
||||
"collections_only": collections_only,
|
||||
}
|
||||
html_path = write_html(REPORT_DIR, summary, series_reports)
|
||||
(REPORT_DIR / "summary.json").write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
|
||||
print(f"JUnit: {REPORT_DIR / 'pulumi-junit.xml'}")
|
||||
print(f"HTML: {html_path}")
|
||||
print(json.dumps(summary, indent=2))
|
||||
|
||||
failed = (
|
||||
summary["pulumi_fail"] > 0 or summary["http_fail"] > 0 or summary["coverage_incomplete"] > 0
|
||||
)
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,111 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8"/>
|
||||
<title>Pulumi OpenStack coverage</title>
|
||||
<style>
|
||||
:root { --bg:#0f1419; --panel:#1a222c; --text:#e7ecf3; --muted:#9aa7b8; --ok:#3dd68c; --fail:#ff6b6b; --border:#2a3544; }
|
||||
body { margin:0; font-family:"IBM Plex Sans",sans-serif; background:radial-gradient(1000px 500px at 0% 0%,#1b2a3d,var(--bg)); color:var(--text); }
|
||||
header, main { max-width:1200px; margin:0 auto; padding:1.5rem; }
|
||||
.summary, .cards { display:grid; gap:.75rem; grid-template-columns:repeat(auto-fit,minmax(160px,1fr)); margin:1rem 0; }
|
||||
.summary div, .card { background:var(--panel); border:1px solid var(--border); border-radius:10px; padding:1rem; }
|
||||
.ok { color:var(--ok); font-weight:600; } .fail { color:var(--fail); font-weight:600; }
|
||||
.muted { color:var(--muted); }
|
||||
table { width:100%; border-collapse:collapse; background:var(--panel); border:1px solid var(--border); border-radius:10px; overflow:hidden; font-size:.9rem; }
|
||||
th, td { padding:.45rem .6rem; border-bottom:1px solid var(--border); text-align:left; vertical-align:top; }
|
||||
th { color:var(--muted); background:#121820; }
|
||||
tr.fail { background:rgba(255,107,107,.08); }
|
||||
code { font-family:ui-monospace,monospace; font-size:.82rem; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>Pulumi OpenStack API coverage</h1>
|
||||
<p class="muted">Generated 2026-07-17 13:12:06 UTC · pulumi_openstack primary + HTTP pack probe with non-empty checks</p>
|
||||
</header>
|
||||
<main>
|
||||
<div class="summary">
|
||||
<div><strong>4</strong><span class="muted"> series</span></div>
|
||||
<div><strong class="ok">4</strong><span class="muted"> pulumi stacks ok</span></div>
|
||||
<div><strong class="fail">0</strong><span class="muted"> pulumi failures</span></div>
|
||||
<div><strong class="ok">4721</strong><span class="muted"> http ops ok</span></div>
|
||||
<div><strong class="fail">0</strong><span class="muted"> http / nonempty fails</span></div>
|
||||
</div>
|
||||
<h2>Series</h2>
|
||||
<div class="cards">
|
||||
<div class="card">
|
||||
<h3>yoga</h3>
|
||||
<p class="muted">pulumi_openstack + HTTP pack probe</p>
|
||||
<div class="stats">
|
||||
<span class="ok">pulumi exports ok=26</span>
|
||||
<span class="fail">empty exports=0</span>
|
||||
</div>
|
||||
<div class="stats">
|
||||
<span class="ok">http ok=1060</span>
|
||||
<span class="fail">http fail=0 nonempty_fail=0</span>
|
||||
<span>http total=1060/1060</span>
|
||||
</div>
|
||||
<div class="stats muted">
|
||||
methods: GET=404 POST=168 PUT=171 PATCH=155 DELETE=162
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>antelope</h3>
|
||||
<p class="muted">pulumi_openstack + HTTP pack probe</p>
|
||||
<div class="stats">
|
||||
<span class="ok">pulumi exports ok=26</span>
|
||||
<span class="fail">empty exports=0</span>
|
||||
</div>
|
||||
<div class="stats">
|
||||
<span class="ok">http ok=1108</span>
|
||||
<span class="fail">http fail=0 nonempty_fail=0</span>
|
||||
<span>http total=1108/1108</span>
|
||||
</div>
|
||||
<div class="stats muted">
|
||||
methods: GET=422 POST=177 PUT=178 PATCH=162 DELETE=169
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>caracal</h3>
|
||||
<p class="muted">pulumi_openstack + HTTP pack probe</p>
|
||||
<div class="stats">
|
||||
<span class="ok">pulumi exports ok=26</span>
|
||||
<span class="fail">empty exports=0</span>
|
||||
</div>
|
||||
<div class="stats">
|
||||
<span class="ok">http ok=1196</span>
|
||||
<span class="fail">http fail=0 nonempty_fail=0</span>
|
||||
<span>http total=1196/1196</span>
|
||||
</div>
|
||||
<div class="stats muted">
|
||||
methods: GET=453 POST=192 PUT=192 PATCH=176 DELETE=183
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>dalmatian</h3>
|
||||
<p class="muted">pulumi_openstack + HTTP pack probe</p>
|
||||
<div class="stats">
|
||||
<span class="ok">pulumi exports ok=26</span>
|
||||
<span class="fail">empty exports=0</span>
|
||||
</div>
|
||||
<div class="stats">
|
||||
<span class="ok">http ok=1357</span>
|
||||
<span class="fail">http fail=0 nonempty_fail=0</span>
|
||||
<span>http total=1357/1357</span>
|
||||
</div>
|
||||
<div class="stats muted">
|
||||
methods: GET=514 POST=217 PUT=217 PATCH=201 DELETE=208
|
||||
|
||||
</div>
|
||||
</div></div>
|
||||
<h2>Failures</h2>
|
||||
<table>
|
||||
<thead><tr><th>Series</th><th>Service</th><th>Operation</th><th>Method</th><th>Path</th><th>HTTP</th><th>Detail</th></tr></thead>
|
||||
<tbody><tr><td colspan="7">No failures</td></tr></tbody>
|
||||
</table>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"series_count": 4,
|
||||
"pulumi_ok": 4,
|
||||
"pulumi_fail": 0,
|
||||
"http_ok": 4721,
|
||||
"http_fail": 0,
|
||||
"coverage_incomplete": 0,
|
||||
"collections_only": false
|
||||
}
|
||||
Reference in New Issue
Block a user