DevOpsTools/ovirt-api-simulator
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Document the test matrix and record the 2026-07-18 Pulumi 9150/9150 run.

Browse Source 
Add EN/RU testing guides, link them from README and docs indexes, and drop
stale native Python/Ansible/Terraform suite references.
This commit is contained in:
Sergey Antropoff
2026-07-18 09:39:03 +03:00
parent 0942c57482
commit 9c94df119b
22 changed files with 283 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -1
View File
@@ -83,7 +83,7 @@ Or: `make release-up && make release-seed PROFILE=minimal`
- Seed profiles: `minimal`, `small` (3h/50vm), `large` (10h/1000vm), `big` (30h/2000vm)
- Web UI with oVirt branding (`#0076B6` / charcoal `#1D2226`)
- Docker Compose + Helm
- API tests + [`pulumi-tests/`](pulumi-tests/README.md) (Pulumi contract coverage)
- API tests + [`pulumi-tests/`](pulumi-tests/README.md) (Pulumi contract coverage; **9150/9150** as of 2026-07-18)
## Make targets
@@ -103,6 +103,8 @@ make push # git add . → multi-line commit (Ctrl-D) → origin +
# make push MSG="one line" # skip the interactive editor
```
Last full matrix: [Testing](docs/testing.md) (2026-07-18 — Pulumi **9150/9150**).
Docker Hub release (requires `docker login` as the Hub owner; see
[Operations](docs/operations.md)):
@@ -124,6 +126,7 @@ make release-up && make release-seed # run the published stack locally
| [Configuration](docs/configuration.md) | Environment and Compose |
| [Seed profiles](docs/seed-profiles.md) | `minimal` / `small` / `large` / `big` |
| [Web UI](docs/web-ui.md) | Interactive console (screenshots) |
| [Testing](docs/testing.md) | Pytest / Pulumi + last verified results (**9150/9150**) |
| [Kubernetes / Helm](docs/kubernetes.md) | Cluster install |
| [Full index](docs/README.md) | All guides |
README.ru.md
+4 -1
View File
@@ -86,7 +86,7 @@ curl -k -u 'admin@internal:secret' -H 'Accept: application/json' -H 'Version: 4'
- Профили seed: `minimal`, `small` (3h/50vm), `large` (10h/1000vm), `big` (30h/2000vm)
- Web UI с брендингом oVirt (`#0076B6` / charcoal `#1D2226`)
- Docker Compose + Helm
- API-тесты + [`pulumi-tests/`](pulumi-tests/README.ru.md) (покрытие контрактов Pulumi)
- API-тесты + [`pulumi-tests/`](pulumi-tests/README.ru.md) (покрытие контрактов Pulumi; **9150/9150** на 2026-07-18)
## Цели Make
@@ -106,6 +106,8 @@ make push # git add . → многострочный commit (Ctrl
# make push MSG="одна строка" # без интерактивного ввода
```
Последняя полная матрица: [Тестирование](docs/ru/testing.md) (2026-07-18 — Pulumi **9150/9150**).
Публикация в Docker Hub (нужен `docker login` владельца Hub; см.
[Эксплуатация](docs/ru/operations.md)):
@@ -127,6 +129,7 @@ make release-up && make release-seed # запустить опубликова
| [Конфигурация](docs/ru/configuration.md) | Окружение и Compose |
| [Профили seed](docs/ru/seed-profiles.md) | `minimal` / `small` / `large` / `big` |
| [Web UI](docs/ru/web-ui.md) | Интерактивная консоль (скриншоты) |
| [Тестирование](docs/ru/testing.md) | Pytest / Pulumi + последние результаты (**9150/9150**) |
| [Kubernetes / Helm](docs/ru/kubernetes.md) | Установка в кластер |
| [Полный индекс](docs/ru/README.md) | Все руководства |
docs/README.md
+1
View File
@@ -15,6 +15,7 @@ on each page. Russian mirrors live under [`ru/`](ru/README.md).
| [Authentication](authentication.md) | Basic auth, OAuth2, sessions |
| [API versions](api-versions.md) | Series packs 3.x / 4.x and Version header |
| [API coverage](api_coverage.md) | Operation counts and deltas per series |
| [Testing](testing.md) | Pytest / Pulumi commands + last verified results |
| [API surface](api-surface.md) | Routing, handlers, schema engine |
| [Clients & examples](clients.md) | curl, Python, Ansible, Terraform |
| [Seed profiles](seed-profiles.md) | `minimal` and `demo` fixtures |
docs/api_coverage.md
+10
View File
@@ -22,5 +22,15 @@ Specialized handlers cover core inventory collections (VMs, disks, hosts,
networks, storage, jobs, …). Remaining pack operations are served by the schema
engine. See [API surface](api-surface.md).
## Pulumi contract matrix
`make pulumi-tests` executes **every** pack operation across all series, plus a
synthetic **HEAD** for each GET (~6 836 contract ops + ~2 314 HEAD = **9 150**
calls).
**Last verified (2026-07-18):** **9150 / 9150** passed — methods GET/POST/PUT/DELETE/HEAD;
HTTP mix 200/201/400/404/409; no `401` / `5xx`. See [Testing](testing.md) and
[`pulumi-tests/README.md`](../pulumi-tests/README.md).
> Measurable contract coverage for a laboratory simulator — not a claim that
> every Engine edge case behaves identically to production RHV/oVirt.
docs/examples/ansible.md
+2 -2
View File
@@ -25,5 +25,5 @@ Use `ansible.builtin.uri` against Engine HTTPS:
register: vms
```
Native suite (150 cases): `make test-ansible` / `make test-ansible-smoke` under
[`pulumi-tests/ansible/`](../../pulumi-tests/README.md).
Contract coverage for the Engine API: `make pulumi-tests` — see
[Testing](../testing.md) and [`pulumi-tests/`](../../pulumi-tests/README.md).
docs/examples/overview.md
+2 -1
View File
@@ -29,7 +29,8 @@ export NO_PROXY='*'
| [Python requests](python-requests.md) | Raw HTTPS Basic / OAuth |
| [Ansible](ansible.md) | `uri` module against Engine |
| [Terraform](terraform.md) | IaC notes |
| [Pulumi](pulumi.md) | Full contract coverage + HTML report |
| [Pulumi](pulumi.md) | Full contract coverage + HTML report (**9150/9150** as of 2026-07-18) |
| [Troubleshooting clients](troubleshooting-clients.md) | Common client failures |
Prerequisites: `make up && make seed` (or `make seed-demo`).
Full project test matrix: [Testing](../testing.md).
docs/examples/pulumi.md
+16 -4
View File
@@ -2,12 +2,24 @@
# Pulumi
Contract-coverage lab under [`pulumi-tests/pulumi/`](../../pulumi-tests/README.md).
Contract-coverage lab under [`pulumi-tests/`](../../pulumi-tests/README.md).
```bash
make up && make seed
make up-local # or: make up
make seed
make test-pulumi-smoke
make test-pulumi
make pulumi-tests # alias: make test-pulumi
```
HTML report: `pulumi-tests/reports/pulumi-contract-coverage.html`
HTML report: `pulumi-tests/reports/pulumi-contract-coverage.html` (gitignored).
## Last verified result (2026-07-18)
| Metric | Value |
|---|---:|
| total / passed / failed | **9150 / 9150 / 0** |
| methods | DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 |
| HTTP | 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 |
All series packs `3.0`–`3.6`, `4.3`–`4.5`, `master` passed. See also
[Testing](../testing.md).
docs/examples/python-requests.md
+2 -1
View File
@@ -39,5 +39,6 @@ r = requests.get(
)
```
Native suite: `make test-python` / `make test-python-smoke` under
Full contract matrix (all clients share the same Engine surface):
`make pulumi-tests` — see [Testing](../testing.md) and
[`pulumi-tests/`](../../pulumi-tests/README.md).
docs/examples/terraform.md
+15 -9
View File
@@ -2,14 +2,20 @@
# Terraform
Native suite (150 cases) lives under
[`pulumi-tests/terraform/`](../../pulumi-tests/README.md):
```bash
make up && make seed
make test-terraform-smoke
make test-terraform
```
Configure providers against Engine HTTPS with lab credentials and disable TLS
verification for the Compose self-signed certificate.
```bash
make up-local # or: make up
make seed
```
There is no separate Terraform test suite in this repo. Use the Pulumi contract
matrix for full Engine surface coverage:
```bash
make pulumi-tests
```
See [Testing](../testing.md) and [`pulumi-tests/`](../../pulumi-tests/README.md).
**Last verified (2026-07-18):** **9150 / 9150** passed.
docs/ru/README.md
+1
View File
@@ -16,6 +16,7 @@
| [Аутентификация](authentication.md) | Basic auth, OAuth2, сессии |
| [Версии API](api-versions.md) | Series packs 3.x / 4.x и заголовок Version |
| [Покрытие API](api_coverage.md) | Число операций и дельты по series |
| [Тестирование](testing.md) | Pytest / Pulumi + последние результаты |
| [Поверхность API](api-surface.md) | Маршрутизация, handlers, schema engine |
| [Клиенты и примеры](clients.md) | curl, Python, Ansible, Terraform |
| [Профили seed](seed-profiles.md) | Фикстуры `minimal` и `demo` |
docs/ru/api_coverage.md
+10
View File
@@ -22,5 +22,15 @@
хосты, сети, storage, jobs, …). Остальные операции pack обслуживает schema
engine. См. [Поверхность API](api-surface.md).
## Матрица Pulumi
`make pulumi-tests` выполняет **каждую** операцию pack по всем series плюс
синтетический **HEAD** для каждого GET (~6 836 ops + ~2 314 HEAD = **9 150**
вызовов).
**Последний прогон (2026-07-18):** **9150 / 9150** passed — методы
GET/POST/PUT/DELETE/HEAD; HTTP 200/201/400/404/409; без `401` / `5xx`. См.
[Тестирование](testing.md) и [`pulumi-tests/README.ru.md`](../../pulumi-tests/README.ru.md).
> Это измеримое покрытие контракта лабораторного симулятора — не утверждение,
> что каждый краевой случай Engine ведёт себя идентично продакшен RHV/oVirt.
docs/ru/examples/ansible.md
+2 -2
View File
@@ -25,5 +25,5 @@
register: vms
```
Нативный suite (150 кейсов): `make test-ansible` / `make test-ansible-smoke` в
[`pulumi-tests/ansible/`](../../../pulumi-tests/README.ru.md).
Покрытие контрактов Engine API: `make pulumi-tests` — см.
[Тестирование](../testing.md) и [`pulumi-tests/`](../../../pulumi-tests/README.ru.md).
docs/ru/examples/overview.md
+2 -1
View File
@@ -29,7 +29,8 @@ export NO_PROXY='*'
| [Python requests](python-requests.md) | Сырой HTTPS Basic / OAuth |
| [Ansible](ansible.md) | Модуль `uri` против Engine |
| [Terraform](terraform.md) | Заметки по IaC |
| [Pulumi](pulumi.md) | Полное покрытие контрактов + HTML-отчёт |
| [Pulumi](pulumi.md) | Полное покрытие контрактов + HTML-отчёт (**9150/9150** на 2026-07-18) |
| [Troubleshooting clients](troubleshooting-clients.md) | Типичные сбои клиентов |
Предварительно: `make up && make seed` (или `make seed-demo`).
Полная матрица тестов проекта: [Тестирование](../testing.md).
docs/ru/examples/pulumi.md
+16 -4
View File
@@ -2,12 +2,24 @@
# Pulumi
Лаборатория покрытия контрактов: [`pulumi-tests/pulumi/`](../../../pulumi-tests/README.ru.md).
Лаборатория покрытия контрактов: [`pulumi-tests/`](../../../pulumi-tests/README.ru.md).
```bash
make up && make seed
make up-local # или: make up
make seed
make test-pulumi-smoke
make test-pulumi
make pulumi-tests # alias: make test-pulumi
```
HTML-отчёт: `pulumi-tests/reports/pulumi-contract-coverage.html`
HTML-отчёт: `pulumi-tests/reports/pulumi-contract-coverage.html` (в gitignore).
## Последний проверенный результат (2026-07-18)
| Метрика | Значение |
|---|---:|
| total / passed / failed | **9150 / 9150 / 0** |
| методы | DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 |
| HTTP | 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 |
Все series packs `3.0`–`3.6`, `4.3`–`4.5`, `master` — зелёные. См. также
[Тестирование](../testing.md).
docs/ru/examples/python-requests.md
+2 -1
View File
@@ -39,5 +39,6 @@ r = requests.get(
)
```
Нативный suite: `make test-python` / `make test-python-smoke` в
Полная матрица контрактов (общий Engine surface для всех клиентов):
`make pulumi-tests` — см. [Тестирование](../testing.md) и
[`pulumi-tests/`](../../../pulumi-tests/README.ru.md).
docs/ru/examples/terraform.md
+13 -7
View File
@@ -2,14 +2,20 @@
# Terraform
Нативный suite (150 кейсов) находится в
[`pulumi-tests/terraform/`](../../../pulumi-tests/README.ru.md):
Настройте providers на Engine HTTPS с лабораторными учётными данными и
отключите проверку TLS для self-signed сертификата Compose.
```bash
make up && make seed
make test-terraform-smoke
make test-terraform
make up-local # или: make up
make seed
```
Направляйте провайдеры на Engine HTTPS с лабораторными учётными данными и
отключайте проверку TLS для self-signed сертификата Compose.
Отдельного Terraform suite в репозитории нет. Полное покрытие поверхности
Engine — матрица Pulumi:
```bash
make pulumi-tests
```
См. [Тестирование](../testing.md) и [`pulumi-tests/`](../../../pulumi-tests/README.ru.md).
**Последний прогон (2026-07-18):** **9150 / 9150** passed.
docs/ru/testing.md
+73
View File
@@ -0,0 +1,73 @@
**Language / Язык:** [English](../testing.md) | [Русский](testing.md)
# Тестирование
Как гонять лабораторную матрицу тестов и что показал последний полный прогон.
## Команды
| Цель | Что запускает |
|---|---|
| `make test` | Offline unit + contract (без живого Engine) |
| `make test-unit` | Unit-тесты в `tests/` |
| `make test-contract` | Проверки contract packs |
| `make test-integration` | Интеграция Engine на PostgreSQL |
| `make test-versions` | Seeded-ассерты по series packs |
| `make smoke` | Auth + список ВМ на поднятом стеке |
| `make test-pulumi-smoke` | Pulumi smoke (выборка 3.6 + 4.5) |
| `make pulumi-tests` | Полная Pulumi-матрица (все series × ops + HEAD) |
| `make lint` / CI critical lint | Ruff (в CI только `E9,F63,F7,F82`) |
| `make helm-template` | Рендер Helm-чарта |
Полный локальный gate перед релизом:
```bash
make up-local # или make up
make seed
make smoke
make test
make test-unit
make test-contract
make test-integration
make test-versions
make pulumi-tests
```
HTML-отчёт Pulumi: `pulumi-tests/reports/pulumi-contract-coverage.html`
(в gitignore — пересобирается каждым прогоном). Подробности:
[`pulumi-tests/README.ru.md`](../../pulumi-tests/README.ru.md).
## Последний полный прогон
**Дата:** 2026-07-18
| Suite | Результат | Примечание |
|---|---|---|
| `helm lint` / `helm-template` | PASS | |
| `make test` (offline) | PASS | |
| `make test-unit` | PASS | |
| `make test-contract` | PASS | |
| `make up-local` / `seed` / `smoke` | PASS | |
| `make test-integration` | PASS | |
| `make test-versions` | PASS | |
| CI critical lint | PASS | |
| `make pulumi-tests` | PASS | **9150 / 9150** |
### Матрица Pulumi
| Метрика | Значение |
|---|---:|
| total | 9150 |
| passed | 9150 |
| failed | 0 |
| skipped | 0 |
**Методы:** DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146
**HTTP:** 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11
(без `401`, без `5xx`)
Series: `3.0`–`3.6`, `4.3`–`4.5`, `master` (все зелёные).
> Pass значит: каждый contract-маршрут достижим с обработанным статусом Engine и
> проверками тела — не побитовая идентичность production RHV.
docs/testing.md
+72
View File
@@ -0,0 +1,72 @@
**Language / Язык:** [English](testing.md) | [Русский](ru/testing.md)
# Testing
How to run the laboratory test matrix and what the last full run reported.
## Commands
| Target | What it runs |
|---|---|
| `make test` | Offline unit + contract (no live Engine) |
| `make test-unit` | Unit tests under `tests/` |
| `make test-contract` | Contract-pack checks |
| `make test-integration` | PostgreSQL-backed Engine integration |
| `make test-versions` | Seeded assertions across series packs |
| `make smoke` | Auth + list VMs against the running stack |
| `make test-pulumi-smoke` | Pulumi smoke (3.6 + 4.5 sample) |
| `make pulumi-tests` | Full Pulumi contract matrix (all series × ops + HEAD) |
| `make lint` / CI critical lint | Ruff (CI uses `E9,F63,F7,F82` only) |
| `make helm-template` | Render Helm chart |
Full local gate used before release:
```bash
make up-local # or make up
make seed
make smoke
make test
make test-unit
make test-contract
make test-integration
make test-versions
make pulumi-tests
```
Pulumi HTML report: `pulumi-tests/reports/pulumi-contract-coverage.html`
(gitignored — regenerated each run). Details: [`pulumi-tests/README.md`](../pulumi-tests/README.md).
## Last verified full run
**Date:** 2026-07-18
| Suite | Result | Notes |
|---|---|---|
| `helm lint` / `helm-template` | PASS | |
| `make test` (offline) | PASS | |
| `make test-unit` | PASS | |
| `make test-contract` | PASS | |
| `make up-local` / `seed` / `smoke` | PASS | |
| `make test-integration` | PASS | |
| `make test-versions` | PASS | |
| CI critical lint | PASS | |
| `make pulumi-tests` | PASS | **9150 / 9150** |
### Pulumi contract matrix
| Metric | Value |
|---|---:|
| total | 9150 |
| passed | 9150 |
| failed | 0 |
| skipped | 0 |
**Methods:** DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146
**HTTP status histogram:** 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11
(no `401`, no `5xx`)
Series covered: `3.0`–`3.6`, `4.3`–`4.5`, `master` (all green).
> Pass means every contract route is reachable with a handled Engine status and
> required body checks — not bit-identical behaviour to production RHV hardware.
examples/README.md
+6 -2
View File
@@ -8,7 +8,11 @@
| [`pulumi-tests/`](../pulumi-tests/README.md) | Pulumi contract coverage across all Engine series |
```bash
make up && make seed
make test-pulumi
make up-local # or: make up
make seed
make pulumi-tests
# report: pulumi-tests/reports/pulumi-contract-coverage.html
```
**Last verified (2026-07-18):** Pulumi **9150 / 9150** passed (GET/POST/PUT/DELETE/HEAD).
See [Testing](../docs/testing.md).
examples/README.ru.md
+6 -2
View File
@@ -8,7 +8,11 @@
| [`pulumi-tests/`](../pulumi-tests/README.ru.md) | Pulumi-покрытие контрактов по всем series Engine |
```bash
make up && make seed
make test-pulumi
make up-local # или: make up
make seed
make pulumi-tests
# отчёт: pulumi-tests/reports/pulumi-contract-coverage.html
```
**Последний прогон (2026-07-18):** Pulumi **9150 / 9150** passed (GET/POST/PUT/DELETE/HEAD).
См. [Тестирование](../docs/ru/testing.md).
pulumi-tests/README.md
+11
View File
@@ -33,6 +33,17 @@ COVERAGE 9150/9150 (critical=0)
METHODS {"DELETE":1146,"GET":2314,"HEAD":2314,"POST":2230,"PUT":1146}
```
### Last verified full run (2026-07-18)
| Metric | Value |
|---|---:|
| total / passed / failed / skipped | **9150 / 9150 / 0 / 0** |
| methods | DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 |
| HTTP | 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 |
Series: `3.0`–`3.6`, `4.3`–`4.5`, `master` — all green (no `401`, no `5xx`).
Project-wide matrix: [Testing](../docs/testing.md).
Optional filters:
```bash
pulumi-tests/README.ru.md
+11
View File
@@ -33,6 +33,17 @@ COVERAGE 9150/9150 (critical=0)
METHODS {"DELETE":1146,"GET":2314,"HEAD":2314,"POST":2230,"PUT":1146}
```
### Последний полный прогон (2026-07-18)
| Метрика | Значение |
|---|---:|
| total / passed / failed / skipped | **9150 / 9150 / 0 / 0** |
| методы | DELETE 1146 · GET 2314 · HEAD 2314 · POST 2230 · PUT 1146 |
| HTTP | 200: 6739 · 404: 1514 · 201: 864 · 400: 22 · 409: 11 |
Series: `3.0`–`3.6`, `4.3`–`4.5`, `master` — все зелёные (без `401`, без `5xx`).
Общая матрица проекта: [Тестирование](../docs/ru/testing.md).
Фильтры:
```bash