Document 2026-07-18 lab test results across EN/RU guides.
Record offline 161 passed, green test-vsphere probes, and full pulumi-tests 7/7 with REST matrix 1987/1987 (critical=0); note the Terraform cookbook binary mismatch separately.
This commit is contained in:
@@ -158,6 +158,10 @@ Runnable cookbooks live under [`examples/`](examples/README.md). The
|
|||||||
(nonempty output checks, HTML report) lives under
|
(nonempty output checks, HTML report) lives under
|
||||||
[`pulumi-tests/`](pulumi-tests/README.md) — run with `make pulumi-tests`.
|
[`pulumi-tests/`](pulumi-tests/README.md) — run with `make pulumi-tests`.
|
||||||
|
|
||||||
|
Latest lab run (**2026-07-18**): offline **161 passed**; `test-vsphere` probes
|
||||||
|
green; full `pulumi-tests` **7/7** with REST matrix **1987/1987** (`critical=0`).
|
||||||
|
See [pulumi-tests/README.md](pulumi-tests/README.md#latest-lab-run-2026-07-18).
|
||||||
|
|
||||||
## Python (requests) against the HTTPS gateway
|
## Python (requests) against the HTTPS gateway
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
|||||||
@@ -159,6 +159,11 @@ majors 6–9, редактирование request/response и runtime contract
|
|||||||
(проверки непустых export'ов, HTML-отчёт) — в
|
(проверки непустых export'ов, HTML-отчёт) — в
|
||||||
[`pulumi-tests/`](pulumi-tests/README.ru.md); запуск: `make pulumi-tests`.
|
[`pulumi-tests/`](pulumi-tests/README.ru.md); запуск: `make pulumi-tests`.
|
||||||
|
|
||||||
|
Последний lab-прогон (**2026-07-18**): offline **161 passed**; probes
|
||||||
|
`test-vsphere` зелёные; полный `pulumi-tests` **7/7**, REST-матрица
|
||||||
|
**1987/1987** (`critical=0`). См.
|
||||||
|
[pulumi-tests/README.ru.md](pulumi-tests/README.ru.md#последний-lab-прогон-2026-07-18).
|
||||||
|
|
||||||
## Python (requests) через HTTPS gateway
|
## Python (requests) через HTTPS gateway
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
|||||||
@@ -73,7 +73,7 @@ registered paths — see `.cursor/rules/durable-simulator.mdc`.
|
|||||||
| External systems | NSX/LDAP/SAML/OIDC/ACME do not contact real remotes; state is simulated locally |
|
| External systems | NSX/LDAP/SAML/OIDC/ACME do not contact real remotes; state is simulated locally |
|
||||||
| TLS | Local nginx gateway with a checked-in self-signed development key only |
|
| TLS | Local nginx gateway with a checked-in self-signed development key only |
|
||||||
| Client certification | pyvmomi/govmomi-style SOAP smoke + Ansible/Terraform/Pulumi cookbooks; not a formal certification suite for every provider version |
|
| Client certification | pyvmomi/govmomi-style SOAP smoke + Ansible/Terraform/Pulumi cookbooks; not a formal certification suite for every provider version |
|
||||||
| Provider smoke | The `pulumi-vsphere` suite under `pulumi-tests/` (`make pulumi-tests`) exercises SOAP-backed inventory/VM/tag resources with nonempty export checks; semantic depth still varies (deep handlers vs DB-backed stubs) |
|
| Provider smoke | The `pulumi-vsphere` suite under `pulumi-tests/` (`make pulumi-tests`) exercises SOAP-backed inventory/VM/tag resources with nonempty export checks; semantic depth still varies (deep handlers vs DB-backed stubs). Latest full run (2026-07-18): **7/7** cases, REST **1987/1987**, `critical=0` — see [`pulumi-tests/README.md`](../pulumi-tests/README.md#latest-lab-run-2026-07-18) |
|
||||||
|
|
||||||
Full registry coverage at major 9 means HTTP 501 "handler pending" should not
|
Full registry coverage at major 9 means HTTP 501 "handler pending" should not
|
||||||
appear for any route in the simulator's registry. Compatibility *quality*
|
appear for any route in the simulator's registry. Compatibility *quality*
|
||||||
|
|||||||
@@ -62,6 +62,16 @@ Additional cookbooks under [`examples/`](../examples/README.md) and the
|
|||||||
`pulumi-vsphere` lab suite under [`pulumi-tests/`](../pulumi-tests/README.md)
|
`pulumi-vsphere` lab suite under [`pulumi-tests/`](../pulumi-tests/README.md)
|
||||||
(`make pulumi-tests`) are manual or CI-optional depending on the stack.
|
(`make pulumi-tests`) are manual or CI-optional depending on the stack.
|
||||||
|
|
||||||
|
### Latest lab run (2026-07-18)
|
||||||
|
|
||||||
|
| Suite | Result |
|
||||||
|
|---|---|
|
||||||
|
| `make test` (offline) | **161 passed** |
|
||||||
|
| `make test-vsphere` (pytest + surface/matrix/dump/nonempty) | **73** pytest passed; probes **0 failures** (Terraform cookbook binary mismatch only) |
|
||||||
|
| `make pulumi-tests` (full hybrid) | **7 / 7 passed**; Layer A REST **1987 / 1987**, `critical=0` |
|
||||||
|
|
||||||
|
Details and report paths: [`pulumi-tests/README.md`](../pulumi-tests/README.md#latest-lab-run-2026-07-18).
|
||||||
|
|
||||||
## Known behavioural limits
|
## Known behavioural limits
|
||||||
|
|
||||||
| Area | Behaviour |
|
| Area | Behaviour |
|
||||||
|
|||||||
+3
-2
@@ -134,8 +134,9 @@ chart — see [Kubernetes / Helm](kubernetes.md).
|
|||||||
2. Run migrations (`make db-migrate`).
|
2. Run migrations (`make db-migrate`).
|
||||||
3. Confirm `/health/ready`.
|
3. Confirm `/health/ready`.
|
||||||
4. Re-check `/ui/api/compatibility?major=9` and `/api/appliance/system/version`.
|
4. Re-check `/ui/api/compatibility?major=9` and `/api/appliance/system/version`.
|
||||||
5. Re-run `make test-vsphere` / `make vsphere-matrix` if you validate the
|
5. Re-run `make test-vsphere` / `make vsphere-matrix` / `make pulumi-tests`
|
||||||
surface after upgrading.
|
if you validate the surface after upgrading. Latest recorded lab numbers:
|
||||||
|
[Compatibility — Latest lab run](compatibility.md#latest-lab-run-2026-07-18).
|
||||||
|
|
||||||
## Resetting a lab
|
## Resetting a lab
|
||||||
|
|
||||||
|
|||||||
@@ -73,7 +73,7 @@ evidence-отчётов, перегенерируется через `make evide
|
|||||||
| Внешние системы | NSX/LDAP/SAML/OIDC/ACME не обращаются к реальным remotes; состояние симулируется локально |
|
| Внешние системы | NSX/LDAP/SAML/OIDC/ACME не обращаются к реальным remotes; состояние симулируется локально |
|
||||||
| TLS | Локальный nginx gateway только с закоммиченным self-signed development key |
|
| TLS | Локальный nginx gateway только с закоммиченным self-signed development key |
|
||||||
| Сертификация клиентов | SOAP smoke в стиле pyvmomi/govmomi + cookbook'и Ansible/Terraform/Pulumi; не формальный certification suite для каждой версии провайдера |
|
| Сертификация клиентов | SOAP smoke в стиле pyvmomi/govmomi + cookbook'и Ansible/Terraform/Pulumi; не формальный certification suite для каждой версии провайдера |
|
||||||
| Smoke провайдера | Набор `pulumi-vsphere` в `pulumi-tests/` (`make pulumi-tests`) гоняет SOAP inventory/VM/tag с проверкой непустых export'ов; семантическая глубина по-прежнему разная (deep handlers vs DB-backed stubs) |
|
| Smoke провайдера | Набор `pulumi-vsphere` в `pulumi-tests/` (`make pulumi-tests`) гоняет SOAP inventory/VM/tag с проверкой непустых export'ов; семантическая глубина по-прежнему разная (deep handlers vs DB-backed stubs). Последний полный прогон (2026-07-18): **7/7** кейсов, REST **1987/1987**, `critical=0` — см. [`pulumi-tests/README.ru.md`](../../pulumi-tests/README.ru.md#последний-lab-прогон-2026-07-18) |
|
||||||
|
|
||||||
Полное покрытие реестра на major 9 означает, что HTTP 501 «handler pending»
|
Полное покрытие реестра на major 9 означает, что HTTP 501 «handler pending»
|
||||||
не должен появляться ни для одного маршрута в реестре симулятора. *Качество*
|
не должен появляться ни для одного маршрута в реестре симулятора. *Качество*
|
||||||
|
|||||||
@@ -67,6 +67,17 @@ make vsphere-bundles # стаб-бандлы OpenAPI + журналы evidence
|
|||||||
[`pulumi-tests/`](../../pulumi-tests/README.ru.md) (`make pulumi-tests`)
|
[`pulumi-tests/`](../../pulumi-tests/README.ru.md) (`make pulumi-tests`)
|
||||||
выполняются вручную или опционально в CI.
|
выполняются вручную или опционально в CI.
|
||||||
|
|
||||||
|
### Последний lab-прогон (2026-07-18)
|
||||||
|
|
||||||
|
| Набор | Результат |
|
||||||
|
|---|---|
|
||||||
|
| `make test` (offline) | **161 passed** |
|
||||||
|
| `make test-vsphere` (pytest + surface/matrix/dump/nonempty) | **73** pytest passed; probes **0 failures** (только mismatch бинаря Terraform cookbook) |
|
||||||
|
| `make pulumi-tests` (полный гибрид) | **7 / 7 passed**; Layer A REST **1987 / 1987**, `critical=0` |
|
||||||
|
|
||||||
|
Подробности и пути к отчёту:
|
||||||
|
[`pulumi-tests/README.ru.md`](../../pulumi-tests/README.ru.md#последний-lab-прогон-2026-07-18).
|
||||||
|
|
||||||
## Известные поведенческие ограничения
|
## Известные поведенческие ограничения
|
||||||
|
|
||||||
| Область | Поведение |
|
| Область | Поведение |
|
||||||
|
|||||||
@@ -138,7 +138,9 @@ Helm-чарт — см. [Kubernetes / Helm](kubernetes.md).
|
|||||||
4. Перепроверьте `/ui/api/compatibility?major=9` и
|
4. Перепроверьте `/ui/api/compatibility?major=9` и
|
||||||
`/api/appliance/system/version`.
|
`/api/appliance/system/version`.
|
||||||
5. При необходимости заново запустите `make test-vsphere` /
|
5. При необходимости заново запустите `make test-vsphere` /
|
||||||
`make vsphere-matrix`, если проверяете поверхность после обновления.
|
`make vsphere-matrix` / `make pulumi-tests`, если проверяете поверхность
|
||||||
|
после обновления. Последние зафиксированные lab-цифры:
|
||||||
|
[Совместимость — Последний lab-прогон](compatibility.md#последний-lab-прогон-2026-07-18).
|
||||||
|
|
||||||
## Сброс лаборатории
|
## Сброс лаборатории
|
||||||
|
|
||||||
|
|||||||
+35
-2
@@ -1,5 +1,7 @@
|
|||||||
# pulumi hybrid lab suite for vmware-api-simulator
|
# pulumi hybrid lab suite for vmware-api-simulator
|
||||||
|
|
||||||
|
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
|
||||||
|
|
||||||
Hybrid suite under `pulumi-tests/`:
|
Hybrid suite under `pulumi-tests/`:
|
||||||
|
|
||||||
1. **Layer A (required for “100%”)** — HTTP contract matrix: all `IMPLEMENTED`
|
1. **Layer A (required for “100%”)** — HTTP contract matrix: all `IMPLEMENTED`
|
||||||
@@ -12,6 +14,28 @@ Hybrid suite under `pulumi-tests/`:
|
|||||||
`pulumi-vsphere` alone cannot hit ~1092 REST routes — **100% means the HTTP
|
`pulumi-vsphere` alone cannot hit ~1092 REST routes — **100% means the HTTP
|
||||||
matrix** (`coverage probed/declared`, `critical=0`), not provider resource count.
|
matrix** (`coverage probed/declared`, `critical=0`), not provider resource count.
|
||||||
|
|
||||||
|
## Latest lab run (2026-07-18)
|
||||||
|
|
||||||
|
Full hybrid (`make pulumi-tests`, seed profile `small`):
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|-------|--------|
|
||||||
|
| Cases | **7 / 7 passed** (`PU-INV`, `PU-FOLDER`, `PU-VM`, `PU-TAG`, `PU-REST`, `PU-CRUD`, `PU-SOAP`) |
|
||||||
|
| Layer A REST matrix | **1987 / 1987** (`critical=0`) across majors 6–9 |
|
||||||
|
| Verb histogram | GET 682 · HEAD 682 · POST 301 · PUT 96 · PATCH 91 · DELETE 135 |
|
||||||
|
|
||||||
|
Artifacts (local copy from the `lab-reports` volume; directory is gitignored):
|
||||||
|
`pulumi-tests/reports/pulumi-report.html`,
|
||||||
|
`pulumi-tests/reports/pulumi-summary.json`.
|
||||||
|
|
||||||
|
Related offline / gateway probes from the same day:
|
||||||
|
|
||||||
|
| Suite | Result |
|
||||||
|
|-------|--------|
|
||||||
|
| `make test` (offline unit + contract) | **161 passed** |
|
||||||
|
| `make test-vsphere` pytest + surface/matrix/dump/nonempty | **73** pytest passed; matrix/surface/nonempty **0 failures** |
|
||||||
|
| Client cookbooks (part of `test-vsphere`) | Python / Ansible URI / Pulumi-style OK; **Terraform cookbook** failed (`Exec format error` on `.tools/terraform` binary — host/arch mismatch, not an API regression) |
|
||||||
|
|
||||||
## Pass rules (Layer A)
|
## Pass rules (Layer A)
|
||||||
|
|
||||||
| Check | Rule |
|
| Check | Rule |
|
||||||
@@ -44,8 +68,17 @@ matrix** (`coverage probed/declared`, `critical=0`), not provider resource count
|
|||||||
| `PU-CRUD` | Deep REST | Session, folder, tagging, content library, VM |
|
| `PU-CRUD` | Deep REST | Session, folder, tagging, content library, VM |
|
||||||
| `PU-SOAP` | SOAP | Existing WSDL ops only |
|
| `PU-SOAP` | SOAP | Existing WSDL ops only |
|
||||||
|
|
||||||
Artifacts: HTML + JSON + JUnit under the `lab-reports` volume. Look for
|
Artifacts: HTML + JSON + JUnit under the `lab-reports` volume (and optionally
|
||||||
`rest.coverage_line`, `rest.critical`, `rest.by_verb` (incl. HEAD).
|
`pulumi-tests/reports/` on the host). Look for `rest.coverage_line`,
|
||||||
|
`rest.critical`, `rest.by_verb` (incl. HEAD). Open the HTML report after a run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p pulumi-tests/reports
|
||||||
|
cd pulumi-tests && docker compose --profile test run --rm --no-deps \
|
||||||
|
-v "$(pwd)/reports:/out" pulumi-runner \
|
||||||
|
cp /reports/pulumi-report.html /reports/pulumi-summary.json /out/
|
||||||
|
open reports/pulumi-report.html
|
||||||
|
```
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
# Гибридный lab-набор pulumi для vmware-api-simulator
|
# Гибридный lab-набор pulumi для vmware-api-simulator
|
||||||
|
|
||||||
|
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
|
||||||
|
|
||||||
Гибрид под `pulumi-tests/`:
|
Гибрид под `pulumi-tests/`:
|
||||||
|
|
||||||
1. **Layer A (обязателен для «100%»)** — HTTP contract matrix: все `IMPLEMENTED`
|
1. **Layer A (обязателен для «100%»)** — HTTP contract matrix: все `IMPLEMENTED`
|
||||||
@@ -12,6 +14,28 @@
|
|||||||
Одним `pulumi-vsphere` закрыть ~1092 REST `verb×path` нельзя — **100% = HTTP-матрица**
|
Одним `pulumi-vsphere` закрыть ~1092 REST `verb×path` нельзя — **100% = HTTP-матрица**
|
||||||
(`coverage probed/declared`, `critical=0`), а не число provider-ресурсов.
|
(`coverage probed/declared`, `critical=0`), а не число provider-ресурсов.
|
||||||
|
|
||||||
|
## Последний lab-прогон (2026-07-18)
|
||||||
|
|
||||||
|
Полный гибрид (`make pulumi-tests`, seed-профиль `small`):
|
||||||
|
|
||||||
|
| Проверка | Результат |
|
||||||
|
|----------|-----------|
|
||||||
|
| Кейсы | **7 / 7 passed** (`PU-INV`, `PU-FOLDER`, `PU-VM`, `PU-TAG`, `PU-REST`, `PU-CRUD`, `PU-SOAP`) |
|
||||||
|
| Layer A REST matrix | **1987 / 1987** (`critical=0`) по majors 6–9 |
|
||||||
|
| Verb histogram | GET 682 · HEAD 682 · POST 301 · PUT 96 · PATCH 91 · DELETE 135 |
|
||||||
|
|
||||||
|
Артефакты (локальная копия с тома `lab-reports`; каталог в `.gitignore`):
|
||||||
|
`pulumi-tests/reports/pulumi-report.html`,
|
||||||
|
`pulumi-tests/reports/pulumi-summary.json`.
|
||||||
|
|
||||||
|
Связанные прогоны того же дня:
|
||||||
|
|
||||||
|
| Набор | Результат |
|
||||||
|
|-------|-----------|
|
||||||
|
| `make test` (offline unit + contract) | **161 passed** |
|
||||||
|
| `make test-vsphere` pytest + surface/matrix/dump/nonempty | **73** pytest passed; matrix/surface/nonempty **0 failures** |
|
||||||
|
| Client cookbooks (часть `test-vsphere`) | Python / Ansible URI / Pulumi-style OK; **Terraform cookbook** упал (`Exec format error` на `.tools/terraform` — несовпадение host/arch, не регрессия API) |
|
||||||
|
|
||||||
## Критерии pass (Layer A)
|
## Критерии pass (Layer A)
|
||||||
|
|
||||||
| Проверка | Правило |
|
| Проверка | Правило |
|
||||||
@@ -37,6 +61,16 @@ make pulumi-tests # полный гибрид
|
|||||||
make pulumi-tests-smoke # PU-INV + Layer A major-9
|
make pulumi-tests-smoke # PU-INV + Layer A major-9
|
||||||
```
|
```
|
||||||
|
|
||||||
|
После прогона HTML-отчёт:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p pulumi-tests/reports
|
||||||
|
cd pulumi-tests && docker compose --profile test run --rm --no-deps \
|
||||||
|
-v "$(pwd)/reports:/out" pulumi-runner \
|
||||||
|
cp /reports/pulumi-report.html /reports/pulumi-summary.json /out/
|
||||||
|
open reports/pulumi-report.html
|
||||||
|
```
|
||||||
|
|
||||||
Шлюз (в compose): `https://api-gateway`. Хост-алиас: `127.0.0.1:28443`
|
Шлюз (в compose): `https://api-gateway`. Хост-алиас: `127.0.0.1:28443`
|
||||||
(не занимает `https://localhost` / `:18443` основного lab — после тестов
|
(не занимает `https://localhost` / `:18443` основного lab — после тестов
|
||||||
`make down`).
|
`make down`).
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user