diff --git a/README.md b/README.md index 6752e3f..d491966 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ Or: `make release-up && make release-seed PROFILE=minimal` - Seed profiles: `minimal`, `small` (3h/50vm), `large` (10h/1000vm), `big` (30h/2000vm) - Web UI with oVirt branding (`#0076B6` / charcoal `#1D2226`) - Docker Compose + Helm -- API tests + [`pulumi-tests/`](pulumi-tests/README.md) (Pulumi contract coverage) +- API tests + [`pulumi-tests/`](pulumi-tests/README.md) (Pulumi contract coverage; **9150/9150** as of 2026-07-18) ## Make targets @@ -103,6 +103,8 @@ make push # git add . → multi-line commit (Ctrl-D) → origin + # make push MSG="one line" # skip the interactive editor ``` +Last full matrix: [Testing](docs/testing.md) (2026-07-18 — Pulumi **9150/9150**). + Docker Hub release (requires `docker login` as the Hub owner; see [Operations](docs/operations.md)): @@ -124,6 +126,7 @@ make release-up && make release-seed # run the published stack locally | [Configuration](docs/configuration.md) | Environment and Compose | | [Seed profiles](docs/seed-profiles.md) | `minimal` / `small` / `large` / `big` | | [Web UI](docs/web-ui.md) | Interactive console (screenshots) | +| [Testing](docs/testing.md) | Pytest / Pulumi + last verified results (**9150/9150**) | | [Kubernetes / Helm](docs/kubernetes.md) | Cluster install | | [Full index](docs/README.md) | All guides | diff --git a/README.ru.md b/README.ru.md index 711437a..48f0309 100644 --- a/README.ru.md +++ b/README.ru.md @@ -86,7 +86,7 @@ curl -k -u 'admin@internal:secret' -H 'Accept: application/json' -H 'Version: 4' - Профили seed: `minimal`, `small` (3h/50vm), `large` (10h/1000vm), `big` (30h/2000vm) - Web UI с брендингом oVirt (`#0076B6` / charcoal `#1D2226`) - Docker Compose + Helm -- API-тесты + [`pulumi-tests/`](pulumi-tests/README.ru.md) (покрытие контрактов Pulumi) +- API-тесты + [`pulumi-tests/`](pulumi-tests/README.ru.md) (покрытие контрактов Pulumi; **9150/9150** на 2026-07-18) ## Цели Make @@ -106,6 +106,8 @@ make push # git add . → многострочный commit (Ctrl # make push MSG="одна строка" # без интерактивного ввода ``` +Последняя полная матрица: [Тестирование](docs/ru/testing.md) (2026-07-18 — Pulumi **9150/9150**). + Публикация в Docker Hub (нужен `docker login` владельца Hub; см. [Эксплуатация](docs/ru/operations.md)): @@ -127,6 +129,7 @@ make release-up && make release-seed # запустить опубликова | [Конфигурация](docs/ru/configuration.md) | Окружение и Compose | | [Профили seed](docs/ru/seed-profiles.md) | `minimal` / `small` / `large` / `big` | | [Web UI](docs/ru/web-ui.md) | Интерактивная консоль (скриншоты) | +| [Тестирование](docs/ru/testing.md) | Pytest / Pulumi + последние результаты (**9150/9150**) | | [Kubernetes / Helm](docs/ru/kubernetes.md) | Установка в кластер | | [Полный индекс](docs/ru/README.md) | Все руководства | diff --git a/docs/README.md b/docs/README.md index d4f2d2f..57689ea 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,6 +15,7 @@ on each page. Russian mirrors live under [`ru/`](ru/README.md). | [Authentication](authentication.md) | Basic auth, OAuth2, sessions | | [API versions](api-versions.md) | Series packs 3.x / 4.x and Version header | | [API coverage](api_coverage.md) | Operation counts and deltas per series | +| [Testing](testing.md) | Pytest / Pulumi commands + last verified results | | [API surface](api-surface.md) | Routing, handlers, schema engine | | [Clients & examples](clients.md) | curl, Python, Ansible, Terraform | | [Seed profiles](seed-profiles.md) | `minimal` and `demo` fixtures | diff --git a/docs/api_coverage.md b/docs/api_coverage.md index 5a6ddb2..7e2a318 100644 --- a/docs/api_coverage.md +++ b/docs/api_coverage.md @@ -22,5 +22,15 @@ Specialized handlers cover core inventory collections (VMs, disks, hosts, networks, storage, jobs, …). Remaining pack operations are served by the schema engine. See [API surface](api-surface.md). +## Pulumi contract matrix + +`make pulumi-tests` executes **every** pack operation across all series, plus a +synthetic **HEAD** for each GET (~6 836 contract ops + ~2 314 HEAD = **9 150** +calls). + +**Last verified (2026-07-18):** **9150 / 9150** passed — methods GET/POST/PUT/DELETE/HEAD; +HTTP mix 200/201/400/404/409; no `401` / `5xx`. See [Testing](testing.md) and +[`pulumi-tests/README.md`](../pulumi-tests/README.md). + > Measurable contract coverage for a laboratory simulator — not a claim that > every Engine edge case behaves identically to production RHV/oVirt. diff --git a/docs/examples/ansible.md b/docs/examples/ansible.md index 6911d3b..7bf6107 100644 --- a/docs/examples/ansible.md +++ b/docs/examples/ansible.md @@ -25,5 +25,5 @@ Use `ansible.builtin.uri` against Engine HTTPS: register: vms ``` -Native suite (150 cases): `make test-ansible` / `make test-ansible-smoke` under -[`pulumi-tests/ansible/`](../../pulumi-tests/README.md). +Contract coverage for the Engine API: `make pulumi-tests` — see +[Testing](../testing.md) and [`pulumi-tests/`](../../pulumi-tests/README.md). diff --git a/docs/examples/overview.md b/docs/examples/overview.md index f55c4dc..929b652 100644 --- a/docs/examples/overview.md +++ b/docs/examples/overview.md @@ -29,7 +29,8 @@ export NO_PROXY='*' | [Python requests](python-requests.md) | Raw HTTPS Basic / OAuth | | [Ansible](ansible.md) | `uri` module against Engine | | [Terraform](terraform.md) | IaC notes | -| [Pulumi](pulumi.md) | Full contract coverage + HTML report | +| [Pulumi](pulumi.md) | Full contract coverage + HTML report (**9150/9150** as of 2026-07-18) | | [Troubleshooting clients](troubleshooting-clients.md) | Common client failures | -Prerequisites: `make up && make seed` (or `make seed-demo`). +Prerequisites: `make up && make seed` (or `make seed-demo`). +Full project test matrix: [Testing](../testing.md). diff --git a/docs/examples/pulumi.md b/docs/examples/pulumi.md index 73823e3..8eaf0b9 100644 --- a/docs/examples/pulumi.md +++ b/docs/examples/pulumi.md @@ -2,12 +2,24 @@ # Pulumi -Contract-coverage lab under [`pulumi-tests/pulumi/`](../../pulumi-tests/README.md). +Contract-coverage lab under [`pulumi-tests/`](../../pulumi-tests/README.md). ```bash -make up && make seed +make up-local # or: make up +make seed make test-pulumi-smoke -make test-pulumi +make pulumi-tests # alias: make test-pulumi ``` -HTML report: `pulumi-tests/reports/pulumi-contract-coverage.html` +HTML report: `pulumi-tests/reports/pulumi-contract-coverage.html` (gitignored). + +## Last verified result (2026-07-18) + +| Metric | Value | +|---|---:| +| total / passed / failed | **9150 / 9150 / 0** | +| methods | DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 | +| HTTP | 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 | + +All series packs `3.0`–`3.6`, `4.3`–`4.5`, `master` passed. See also +[Testing](../testing.md). diff --git a/docs/examples/python-requests.md b/docs/examples/python-requests.md index 5fe209e..02c1519 100644 --- a/docs/examples/python-requests.md +++ b/docs/examples/python-requests.md @@ -39,5 +39,6 @@ r = requests.get( ) ``` -Native suite: `make test-python` / `make test-python-smoke` under +Full contract matrix (all clients share the same Engine surface): +`make pulumi-tests` — see [Testing](../testing.md) and [`pulumi-tests/`](../../pulumi-tests/README.md). diff --git a/docs/examples/terraform.md b/docs/examples/terraform.md index c34596d..c770c21 100644 --- a/docs/examples/terraform.md +++ b/docs/examples/terraform.md @@ -2,14 +2,20 @@ # Terraform -Native suite (150 cases) lives under -[`pulumi-tests/terraform/`](../../pulumi-tests/README.md): - -```bash -make up && make seed -make test-terraform-smoke -make test-terraform -``` - Configure providers against Engine HTTPS with lab credentials and disable TLS verification for the Compose self-signed certificate. + +```bash +make up-local # or: make up +make seed +``` + +There is no separate Terraform test suite in this repo. Use the Pulumi contract +matrix for full Engine surface coverage: + +```bash +make pulumi-tests +``` + +See [Testing](../testing.md) and [`pulumi-tests/`](../../pulumi-tests/README.md). +**Last verified (2026-07-18):** **9150 / 9150** passed. diff --git a/docs/ru/README.md b/docs/ru/README.md index 60febfa..b8feb64 100644 --- a/docs/ru/README.md +++ b/docs/ru/README.md @@ -16,6 +16,7 @@ | [Аутентификация](authentication.md) | Basic auth, OAuth2, сессии | | [Версии API](api-versions.md) | Series packs 3.x / 4.x и заголовок Version | | [Покрытие API](api_coverage.md) | Число операций и дельты по series | +| [Тестирование](testing.md) | Pytest / Pulumi + последние результаты | | [Поверхность API](api-surface.md) | Маршрутизация, handlers, schema engine | | [Клиенты и примеры](clients.md) | curl, Python, Ansible, Terraform | | [Профили seed](seed-profiles.md) | Фикстуры `minimal` и `demo` | diff --git a/docs/ru/api_coverage.md b/docs/ru/api_coverage.md index 285b06d..32bcdaa 100644 --- a/docs/ru/api_coverage.md +++ b/docs/ru/api_coverage.md @@ -22,5 +22,15 @@ хосты, сети, storage, jobs, …). Остальные операции pack обслуживает schema engine. См. [Поверхность API](api-surface.md). +## Матрица Pulumi + +`make pulumi-tests` выполняет **каждую** операцию pack по всем series плюс +синтетический **HEAD** для каждого GET (~6 836 ops + ~2 314 HEAD = **9 150** +вызовов). + +**Последний прогон (2026-07-18):** **9150 / 9150** passed — методы +GET/POST/PUT/DELETE/HEAD; HTTP 200/201/400/404/409; без `401` / `5xx`. См. +[Тестирование](testing.md) и [`pulumi-tests/README.ru.md`](../../pulumi-tests/README.ru.md). + > Это измеримое покрытие контракта лабораторного симулятора — не утверждение, > что каждый краевой случай Engine ведёт себя идентично продакшен RHV/oVirt. diff --git a/docs/ru/examples/ansible.md b/docs/ru/examples/ansible.md index e327684..eb6e706 100644 --- a/docs/ru/examples/ansible.md +++ b/docs/ru/examples/ansible.md @@ -25,5 +25,5 @@ register: vms ``` -Нативный suite (150 кейсов): `make test-ansible` / `make test-ansible-smoke` в -[`pulumi-tests/ansible/`](../../../pulumi-tests/README.ru.md). +Покрытие контрактов Engine API: `make pulumi-tests` — см. +[Тестирование](../testing.md) и [`pulumi-tests/`](../../../pulumi-tests/README.ru.md). diff --git a/docs/ru/examples/overview.md b/docs/ru/examples/overview.md index 1eeeb8b..4fee208 100644 --- a/docs/ru/examples/overview.md +++ b/docs/ru/examples/overview.md @@ -29,7 +29,8 @@ export NO_PROXY='*' | [Python requests](python-requests.md) | Сырой HTTPS Basic / OAuth | | [Ansible](ansible.md) | Модуль `uri` против Engine | | [Terraform](terraform.md) | Заметки по IaC | -| [Pulumi](pulumi.md) | Полное покрытие контрактов + HTML-отчёт | +| [Pulumi](pulumi.md) | Полное покрытие контрактов + HTML-отчёт (**9150/9150** на 2026-07-18) | | [Troubleshooting clients](troubleshooting-clients.md) | Типичные сбои клиентов | -Предварительно: `make up && make seed` (или `make seed-demo`). +Предварительно: `make up && make seed` (или `make seed-demo`). +Полная матрица тестов проекта: [Тестирование](../testing.md). diff --git a/docs/ru/examples/pulumi.md b/docs/ru/examples/pulumi.md index bc0a358..ff690ea 100644 --- a/docs/ru/examples/pulumi.md +++ b/docs/ru/examples/pulumi.md @@ -2,12 +2,24 @@ # Pulumi -Лаборатория покрытия контрактов: [`pulumi-tests/pulumi/`](../../../pulumi-tests/README.ru.md). +Лаборатория покрытия контрактов: [`pulumi-tests/`](../../../pulumi-tests/README.ru.md). ```bash -make up && make seed +make up-local # или: make up +make seed make test-pulumi-smoke -make test-pulumi +make pulumi-tests # alias: make test-pulumi ``` -HTML-отчёт: `pulumi-tests/reports/pulumi-contract-coverage.html` +HTML-отчёт: `pulumi-tests/reports/pulumi-contract-coverage.html` (в gitignore). + +## Последний проверенный результат (2026-07-18) + +| Метрика | Значение | +|---|---:| +| total / passed / failed | **9150 / 9150 / 0** | +| методы | DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 | +| HTTP | 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 | + +Все series packs `3.0`–`3.6`, `4.3`–`4.5`, `master` — зелёные. См. также +[Тестирование](../testing.md). diff --git a/docs/ru/examples/python-requests.md b/docs/ru/examples/python-requests.md index 7868b7f..961fa99 100644 --- a/docs/ru/examples/python-requests.md +++ b/docs/ru/examples/python-requests.md @@ -39,5 +39,6 @@ r = requests.get( ) ``` -Нативный suite: `make test-python` / `make test-python-smoke` в +Полная матрица контрактов (общий Engine surface для всех клиентов): +`make pulumi-tests` — см. [Тестирование](../testing.md) и [`pulumi-tests/`](../../../pulumi-tests/README.ru.md). diff --git a/docs/ru/examples/terraform.md b/docs/ru/examples/terraform.md index c5dc2a6..6fc09cc 100644 --- a/docs/ru/examples/terraform.md +++ b/docs/ru/examples/terraform.md @@ -2,14 +2,20 @@ # Terraform -Нативный suite (150 кейсов) находится в -[`pulumi-tests/terraform/`](../../../pulumi-tests/README.ru.md): +Настройте providers на Engine HTTPS с лабораторными учётными данными и +отключите проверку TLS для self-signed сертификата Compose. ```bash -make up && make seed -make test-terraform-smoke -make test-terraform +make up-local # или: make up +make seed ``` -Направляйте провайдеры на Engine HTTPS с лабораторными учётными данными и -отключайте проверку TLS для self-signed сертификата Compose. +Отдельного Terraform suite в репозитории нет. Полное покрытие поверхности +Engine — матрица Pulumi: + +```bash +make pulumi-tests +``` + +См. [Тестирование](../testing.md) и [`pulumi-tests/`](../../../pulumi-tests/README.ru.md). +**Последний прогон (2026-07-18):** **9150 / 9150** passed. diff --git a/docs/ru/testing.md b/docs/ru/testing.md new file mode 100644 index 0000000..e8f7faa --- /dev/null +++ b/docs/ru/testing.md @@ -0,0 +1,73 @@ +**Language / Язык:** [English](../testing.md) | [Русский](testing.md) + +# Тестирование + +Как гонять лабораторную матрицу тестов и что показал последний полный прогон. + +## Команды + +| Цель | Что запускает | +|---|---| +| `make test` | Offline unit + contract (без живого Engine) | +| `make test-unit` | Unit-тесты в `tests/` | +| `make test-contract` | Проверки contract packs | +| `make test-integration` | Интеграция Engine на PostgreSQL | +| `make test-versions` | Seeded-ассерты по series packs | +| `make smoke` | Auth + список ВМ на поднятом стеке | +| `make test-pulumi-smoke` | Pulumi smoke (выборка 3.6 + 4.5) | +| `make pulumi-tests` | Полная Pulumi-матрица (все series × ops + HEAD) | +| `make lint` / CI critical lint | Ruff (в CI только `E9,F63,F7,F82`) | +| `make helm-template` | Рендер Helm-чарта | + +Полный локальный gate перед релизом: + +```bash +make up-local # или make up +make seed +make smoke +make test +make test-unit +make test-contract +make test-integration +make test-versions +make pulumi-tests +``` + +HTML-отчёт Pulumi: `pulumi-tests/reports/pulumi-contract-coverage.html` +(в gitignore — пересобирается каждым прогоном). Подробности: +[`pulumi-tests/README.ru.md`](../../pulumi-tests/README.ru.md). + +## Последний полный прогон + +**Дата:** 2026-07-18 + +| Suite | Результат | Примечание | +|---|---|---| +| `helm lint` / `helm-template` | PASS | | +| `make test` (offline) | PASS | | +| `make test-unit` | PASS | | +| `make test-contract` | PASS | | +| `make up-local` / `seed` / `smoke` | PASS | | +| `make test-integration` | PASS | | +| `make test-versions` | PASS | | +| CI critical lint | PASS | | +| `make pulumi-tests` | PASS | **9150 / 9150** | + +### Матрица Pulumi + +| Метрика | Значение | +|---|---:| +| total | 9150 | +| passed | 9150 | +| failed | 0 | +| skipped | 0 | + +**Методы:** DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 + +**HTTP:** 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 +(без `401`, без `5xx`) + +Series: `3.0`–`3.6`, `4.3`–`4.5`, `master` (все зелёные). + +> Pass значит: каждый contract-маршрут достижим с обработанным статусом Engine и +> проверками тела — не побитовая идентичность production RHV. diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..62a40df --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,72 @@ +**Language / Язык:** [English](testing.md) | [Русский](ru/testing.md) + +# Testing + +How to run the laboratory test matrix and what the last full run reported. + +## Commands + +| Target | What it runs | +|---|---| +| `make test` | Offline unit + contract (no live Engine) | +| `make test-unit` | Unit tests under `tests/` | +| `make test-contract` | Contract-pack checks | +| `make test-integration` | PostgreSQL-backed Engine integration | +| `make test-versions` | Seeded assertions across series packs | +| `make smoke` | Auth + list VMs against the running stack | +| `make test-pulumi-smoke` | Pulumi smoke (3.6 + 4.5 sample) | +| `make pulumi-tests` | Full Pulumi contract matrix (all series × ops + HEAD) | +| `make lint` / CI critical lint | Ruff (CI uses `E9,F63,F7,F82` only) | +| `make helm-template` | Render Helm chart | + +Full local gate used before release: + +```bash +make up-local # or make up +make seed +make smoke +make test +make test-unit +make test-contract +make test-integration +make test-versions +make pulumi-tests +``` + +Pulumi HTML report: `pulumi-tests/reports/pulumi-contract-coverage.html` +(gitignored — regenerated each run). Details: [`pulumi-tests/README.md`](../pulumi-tests/README.md). + +## Last verified full run + +**Date:** 2026-07-18 + +| Suite | Result | Notes | +|---|---|---| +| `helm lint` / `helm-template` | PASS | | +| `make test` (offline) | PASS | | +| `make test-unit` | PASS | | +| `make test-contract` | PASS | | +| `make up-local` / `seed` / `smoke` | PASS | | +| `make test-integration` | PASS | | +| `make test-versions` | PASS | | +| CI critical lint | PASS | | +| `make pulumi-tests` | PASS | **9150 / 9150** | + +### Pulumi contract matrix + +| Metric | Value | +|---|---:| +| total | 9150 | +| passed | 9150 | +| failed | 0 | +| skipped | 0 | + +**Methods:** DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 + +**HTTP status histogram:** 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 +(no `401`, no `5xx`) + +Series covered: `3.0`–`3.6`, `4.3`–`4.5`, `master` (all green). + +> Pass means every contract route is reachable with a handled Engine status and +> required body checks — not bit-identical behaviour to production RHV hardware. diff --git a/examples/README.md b/examples/README.md index 140f3e7..ea77162 100644 --- a/examples/README.md +++ b/examples/README.md @@ -8,7 +8,11 @@ | [`pulumi-tests/`](../pulumi-tests/README.md) | Pulumi contract coverage across all Engine series | ```bash -make up && make seed -make test-pulumi +make up-local # or: make up +make seed +make pulumi-tests # report: pulumi-tests/reports/pulumi-contract-coverage.html ``` + +**Last verified (2026-07-18):** Pulumi **9150 / 9150** passed (GET/POST/PUT/DELETE/HEAD). +See [Testing](../docs/testing.md). diff --git a/examples/README.ru.md b/examples/README.ru.md index 39f58ed..f9a4f75 100644 --- a/examples/README.ru.md +++ b/examples/README.ru.md @@ -8,7 +8,11 @@ | [`pulumi-tests/`](../pulumi-tests/README.ru.md) | Pulumi-покрытие контрактов по всем series Engine | ```bash -make up && make seed -make test-pulumi +make up-local # или: make up +make seed +make pulumi-tests # отчёт: pulumi-tests/reports/pulumi-contract-coverage.html ``` + +**Последний прогон (2026-07-18):** Pulumi **9150 / 9150** passed (GET/POST/PUT/DELETE/HEAD). +См. [Тестирование](../docs/ru/testing.md). diff --git a/pulumi-tests/README.md b/pulumi-tests/README.md index 41c86a8..9b6e3c2 100644 --- a/pulumi-tests/README.md +++ b/pulumi-tests/README.md @@ -33,6 +33,17 @@ COVERAGE 9150/9150 (critical=0) METHODS {"DELETE":1146,"GET":2314,"HEAD":2314,"POST":2230,"PUT":1146} ``` +### Last verified full run (2026-07-18) + +| Metric | Value | +|---|---:| +| total / passed / failed / skipped | **9150 / 9150 / 0 / 0** | +| methods | DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 | +| HTTP | 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 | + +Series: `3.0`–`3.6`, `4.3`–`4.5`, `master` — all green (no `401`, no `5xx`). +Project-wide matrix: [Testing](../docs/testing.md). + Optional filters: ```bash diff --git a/pulumi-tests/README.ru.md b/pulumi-tests/README.ru.md index 8ff5fc3..17a53df 100644 --- a/pulumi-tests/README.ru.md +++ b/pulumi-tests/README.ru.md @@ -33,6 +33,17 @@ COVERAGE 9150/9150 (critical=0) METHODS {"DELETE":1146,"GET":2314,"HEAD":2314,"POST":2230,"PUT":1146} ``` +### Последний полный прогон (2026-07-18) + +| Метрика | Значение | +|---|---:| +| total / passed / failed / skipped | **9150 / 9150 / 0 / 0** | +| методы | DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 | +| HTTP | 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 | + +Series: `3.0`–`3.6`, `4.3`–`4.5`, `master` — все зелёные (без `401`, без `5xx`). +Общая матрица проекта: [Тестирование](../docs/ru/testing.md). + Фильтры: ```bash