**Language / Язык:** [English](troubleshooting.md) | [Русский](ru/troubleshooting.md)

# Troubleshooting

## Ready stays unavailable

1. Confirm Postgres: `make logs` / Compose health.
2. Run `make db-migrate`.
3. Hit `/health/ready` again.

Task workers may retry until migrations catch up after a late migrate.

## Unexpected HTTP 501

Every registered route should have a real handler or DB-backed stub — 501
should not appear for a known path. If you see it:

- Confirm you are calling the exact registered path/verb (check
  `app/vsphere/rest/coverage.py` or `/docs`).
- 501 from the optional legacy stub (`ENABLE_PVE_STUB=true`) is expected for
  undeclared PVE-style methods when `CONTRACT_FALLBACK=error`; it is
  unrelated to the vSphere surface.
- Report a regression — full registry coverage is expected on the native
  vSphere plane.

## 401 / 403

- Session expired (2-hour sliding TTL) or `vmware-api-session-id` header/cookie
  not sent.
- Basic auth malformed on `/api/session` (missing header, wrong
  `user:password` base64).
- Privilege denial — try `administrator@vsphere.local` vs
  `readonly@vsphere.local` to compare (see [Authorization](domains/authz.md)).
- In the Web UI, a 401 clears the stored session and switches the header badge
  to **Guest** with a “Session expired — sign in again” toast.

## Ingress returns branded HTML 404 / nginx 405 instead of JSON

The simulator answers API errors as JSON (`detail` / `error_type` /
`messages`). If you see a site HTML “page not found” or plain nginx **405**
page, the **Ingress / reverse proxy** replaced the upstream body (often via
`custom-http-errors` on the ingress-nginx controller).

Fix on the Ingress for this host (see
`helm/vmware-api-simulator/values-ingress-example.yaml`):

```yaml
annotations:
  nginx.ingress.kubernetes.io/proxy-intercept-errors: "false"
  nginx.ingress.kubernetes.io/custom-http-errors: "502,503"
```

Then re-check with `Accept: application/json`. A missing host should look like:

```json
{
  "detail": {
    "error_type": "not_found",
    "messages": [
      {
        "default_message": "No such host ('host-999')",
        "id": "com.vmware.vapi.std.errors.not_found",
        "args": ["host-999"]
      }
    ],
    "data": { "host": "No such host ('host-999')" }
  }
}
```

Seeded host ids for `small` / `large` / `big` start at `host-11`. Cookbook VMs
include `vm-101` (`web-01`).

### Correct authenticated mutation (vSphere Automation)

Session cookie/header + JSON body (not form-urlencoded):

```bash
SID=$(curl -sk -u 'administrator@vsphere.local:VMware1!' -X POST \
  "https://HOST/api/session" | tr -d '"')
curl -sk -X POST "https://HOST/api/vcenter/vm" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "vmware-api-session-id: $SID" \
  -d '{"name":"lab-vm","guest_os":"OTHER_GUEST_64","placement":{"folder":"group-v23","host":"host-11","datastore":"datastore-31","resource_pool":"resgroup-22"}}'
```

## Task never finishes

- Inspect `/api/cis/tasks/{task}`.
- Check worker/simulator logs (`make logs`).
- Verify `TASK_WORKER_CONCURRENCY` > 0 and database leases can be claimed.
- Extremely high `SIMULATION_TIME_SCALE` slowdowns are unusual (higher =
  faster simulation); mis-set worker leases are more common culprits.

## TLS / gateway failures

- Use host port **443** (gateway) for TLS clients — pyvmomi, govmomi,
  Terraform's `hashicorp/vsphere` provider, Pulumi.
- Set `verify_ssl=False` / `allow_unverified_ssl=true` **only** for the local
  self-signed development certificate.
- Inside Compose, target `simulator:8080` directly (plain HTTP, no gateway).
- Seeded VM names for `small` are `web-01`, `web-02`, `db-01`, `app-01`,
  `jumpbox` — not Proxmox-style `pve01`/VMIDs.

## Terraform / Pulumi / Ansible drift after reseed

Reseed replaces PostgreSQL inventory (MOIDs and VM names can change);
external tool state does not update automatically. Refresh, import, or
rebuild stacks after `make seed`.

## Hot-swap "did nothing"

- Catalog browse ≠ apply. Use **Apply as runtime** or
  `POST /ui/api/contract/apply?major=N`.
- Applying a major changes the **Web UI catalog / evidence view**, not the
  live route table — the runtime always serves the full registered surface.
  See [API versions](api-versions.md).
- Apply is process-local; a Compose restart returns to the default (major 9).

## Demo unload surprised you

`POST /ui/api/demo/unload` clears API-created state and loads `small`. Re-run
`make seed` (or load `demo-cluster` again) to restore a richer fixture.