Compare commits

..

19 Commits

Author SHA1 Message Date
Sergey Antropoff 98ba5004ad Fix Deploy stage for Alpine ash: drop bash arrays.
Jenkins docker agents run sh without array syntax; set simulator/migrate images with POSIX conditionals.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-28 02:12:16 +03:00
Sergey Antropoff c64cd4ca8b Raise deploy rollout timeout to 1200s and set images in one step.
Avoid double ReplicaSet churn and ImagePullBackOff races during Hub pulls.
2026-07-28 00:56:46 +03:00
Sergey Antropoff f81d6186ab Add SemVer VERSION, Jenkins CI/CD, and release 0.1.1.
Ship Harbor/Hub push with latest+version tags, deploy to proxmox.devops.org.ru, and show the package version badge in Help → About.
2026-07-28 00:24:05 +03:00
Sergey Antropoff f981ca1bf1 Return clear node-missing JSON and reset Web UI auth on 401.
Keep Ingress from rewriting API 404/405 into branded HTML; document curl/auth and a prompt for sibling sims.
2026-07-22 06:52:45 +03:00
Sergey Antropoff baa1f58ad0 Fix CI/pulumi blockers and document test suites with latest PASS results.
Stabilize node status, cluster-wide Ceph OSD ids, and QEMU cpu utilization so
status/list no longer 500 on model strings; align pulumi defaults to pve1;
add bilingual testing docs with the 2026-07-18 ci-all + pulumi-tests pass.
2026-07-18 09:38:28 +03:00
Sergey Antropoff 6f42d4c318 Simplify bilingual README and docs index around the Web UI.
Drop CI/Docker/License badges, keep a single main-window screenshot, and
link to the Web UI guide at the bottom of the English and Russian pages.
2026-07-18 09:01:44 +03:00
Sergey Antropoff 176c8fd71c Ignore generated pulumi-tests report artifacts.
Keep HTML/JSON/JUnit output from make pulumi-tests out of the repo.
2026-07-18 08:51:52 +03:00
Sergey Antropoff 0773f721ea Align sized cluster seeds and GET dumps with PVE wire shapes; restyle DATA panel.
- Scale small/large/big seeds (3×50 / 10×1000 / 20×2000) with proportional
  backups, snapshots, HA, replication, Ceph capacity, and OSD totals
  (10 / 100 / 500) plus matching node disks and crush/pg metadata
- Enrich handler responses for apt, certificates, qemu/lxc status, storage,
  SDN, metrics export, and related cluster/node dumps
- Flatten nested body_example fields into PARAMS and sync the request body
  via dotted paths (oVirt-style)
- Restyle DATA controls as size cards with full-width Reset to minimal /
  Refresh stats; unload reloads the minimal cluster
2026-07-18 08:46:11 +03:00
Sergey Antropoff 48df10b17e Prepare 0.1.0 for lab release: durable handlers, HTTP Compose, CI, and pulumi-tests.
- Harden DB-backed handlers and seed profiles; align client wire shapes for
  cluster resources, QEMU config, and node SSL fields
- Serve plain HTTP on Compose :8006; keep TLS optional (--profile tls) and
  terminate HTTPS at Kubernetes Ingress
- Add pulumi-tests (full contract surface majors 6–9 + BPG lifecycle) and
  make pulumi-tests
- Ship bilingual docs, CHANGELOG, SECURITY, CONTRIBUTING, and GitHub Actions
  (make ci + Compose/Helm validation)
2026-07-18 04:18:05 +03:00
Sergey Antropoff 777926487b Add a stateful Proxmox API console and broad handler coverage beyond the
initial QEMU slice, backed by imported contracts for majors 6–9.
- Implement durable handlers for access/auth, cluster, LXC, storage, HA,
  firewall, Ceph, SDN, ACME, notifications, pools, mapping, and node ops
- Serve an interactive Web UI with catalog browsing, demo seed controls,
  and OpenAPI/help surfaces
- Bundle PVE 6.4-15, 7.4-16, and 8.4.5 contract revisions alongside 9.2.3
- Support in-memory runtime contract Apply (POST /ui/api/contract/apply)
  so /version and /api2 routes follow the selected major until restart
- Expand seed profiles (including demo-cluster), migrations 007–008, TLS
  gateway config, Compose/Makefile tooling, and compatibility evidence
- Tighten .gitignore for macOS, hidden directories (.*/), and local secrets
2026-07-16 01:08:01 +03:00
Sergey Antropoff 003ee5d634 feat: implement QEMU disk resize and move 2026-07-13 02:20:54 +03:00
Sergey Antropoff 41b76b5472 feat: implement QEMU clone and local migration 2026-07-13 02:14:00 +03:00
Sergey Antropoff 48f7cc39ff feat: implement durable QEMU snapshots 2026-07-13 02:08:04 +03:00
Sergey Antropoff e1e1f69577 feat: complete QEMU power lifecycle 2026-07-13 02:00:45 +03:00
Sergey Antropoff 701157f570 feat: add durable QEMU create update delete 2026-07-13 01:54:04 +03:00
Sergey Antropoff 7721ba87c5 feat: complete group ACL permission matrix 2026-07-13 01:40:49 +03:00
Sergey Antropoff 5934c567b5 feat: add persistent API token lifecycle 2026-07-13 01:35:01 +03:00
Sergey Antropoff 8471bcdbac feat: authenticate API tokens and enforce route ACLs 2026-07-13 01:28:00 +03:00
Sergey Antropoff c45044f901 feat: enforce schema readiness and optimistic resources 2026-07-13 01:18:50 +03:00
308 changed files with 255323 additions and 1008 deletions
-1
View File
@@ -6,6 +6,5 @@ __pycache__
.ruff_cache .ruff_cache
.env .env
htmlcov htmlcov
tests
docs docs
+10
View File
@@ -1,6 +1,7 @@
APP_HOST=0.0.0.0 APP_HOST=0.0.0.0
APP_PORT=8006 APP_PORT=8006
DATABASE_URL=postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator DATABASE_URL=postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator
TEST_DATABASE_URL=postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator
DB_POOL_MIN_SIZE=1 DB_POOL_MIN_SIZE=1
DB_POOL_MAX_SIZE=10 DB_POOL_MAX_SIZE=10
DB_CONNECT_TIMEOUT_SECONDS=10 DB_CONNECT_TIMEOUT_SECONDS=10
@@ -9,7 +10,12 @@ LOG_LEVEL=INFO
REQUEST_ID_HEADER=X-Request-ID REQUEST_ID_HEADER=X-Request-ID
PVE_API_VERSION=9.2.3 PVE_API_VERSION=9.2.3
CONTRACT_SNAPSHOT=/app/contracts/pve-9.2.3.json CONTRACT_SNAPSHOT=/app/contracts/pve-9.2.3.json
COMPATIBILITY_EVIDENCE=/app/evidence/pve-9.2.3.json
CONTRACT_FALLBACK=error CONTRACT_FALLBACK=error
CATALOG_ARTIFACT_URL_6=https://pve.proxmox.com/pve-docs-6/api-viewer/apidoc.js
CATALOG_ARTIFACT_URL_7=https://pve.proxmox.com/pve-docs-7/api-viewer/apidoc.js
CATALOG_ARTIFACT_URL_8=https://pve.proxmox.com/pve-docs-8/api-viewer/apidoc.js
CATALOG_ARTIFACT_URL_9=https://pve.proxmox.com/pve-docs/api-viewer/apidoc.js
TICKET_SIGNING_KEY=development-only-signing-key-change-me TICKET_SIGNING_KEY=development-only-signing-key-change-me
TASK_WORKER_CONCURRENCY=2 TASK_WORKER_CONCURRENCY=2
TASK_LEASE_SECONDS=30 TASK_LEASE_SECONDS=30
@@ -17,3 +23,7 @@ SIMULATION_SEED=42
SIMULATION_TIME_SCALE=10 SIMULATION_TIME_SCALE=10
SIMULATOR_ADMIN_ENABLED=false SIMULATOR_ADMIN_ENABLED=false
SIMULATOR_ADMIN_TOKEN=replace-with-a-long-random-secret SIMULATOR_ADMIN_TOKEN=replace-with-a-long-random-secret
PROXMOXER_HOST=
PROXMOXER_PORT=8443
# For proxmoxer (HTTPS-only): docker compose --profile tls up -d
# then PROXMOXER_HOST=tls-gateway PROXMOXER_PORT=8443
+60
View File
@@ -0,0 +1,60 @@
name: CI
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Compose / Helm validate
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Validate Compose files
run: |
docker compose -f docker-compose.yml config --quiet
docker compose -f docker-compose.release.yml config --quiet
- uses: azure/setup-helm@v4
with:
version: v3.16.4
- name: Lint and render Helm chart
run: |
make helm-lint
make helm-template >/dev/null
quality:
name: make ci
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- name: Prepare lab env
run: test -f .env || cp .env.example .env
- name: Build runtime and development images
run: make install
- name: Ruff + mypy + offline pytest + API surface probe
run: make ci
- name: Upload coverage XML
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-xml
path: coverage.xml
if-no-files-found: ignore
+72 -8
View File
@@ -1,14 +1,78 @@
# Hidden directories (.cursor/, .pytest_cache/, .mypy_cache/, .ruff_cache/, …)
.*/
# Keep GitHub Actions / repo metadata trackable despite the rule above.
!.github/
!.github/**
# macOS
.DS_Store
.AppleDouble
.LSOverride
._*
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent
.AppleDB
.AppleDesktop
.apdisk
Network Trash Folder
Temporary Items
# Environment / secrets (keep .env.example tracked)
.env .env
.venv/ .env.local
.env.*.local
# Python
__pycache__/ __pycache__/
*.py[cod] *.py[cod]
.coverage *$py.class
coverage.xml *.so
htmlcov/ .Python
.mypy_cache/ *.egg
.pytest_cache/ *.egg-info/
.ruff_cache/ .eggs/
dist/ dist/
build/ build/
*.egg-info/ wheels/
pip-wheel-metadata/
share/python-wheels/
*.manifest
*.spec
pip-log.txt
pip-delete-this-directory.txt
.venv/
venv/
ENV/
env/
# Test / coverage leftovers (directories also covered by .*/)
.coverage
.coverage.*
coverage.xml
htmlcov/
.cache
nosetests.xml
pytestdebug.log
hypothesis/
# Editors / IDE leftovers outside .*
*.swp
*.swo
*~
*.sublime-project
*.sublime-workspace
# Docker / local runtime (`make up-local` writes port remaps here)
*.log
docker-compose.override.yml
# Lab probe output (regenerated by scripts/probe_api_surface.py)
evidence/_api_surface_probe.json
# Pulumi suite reports (HTML/JSON/JUnit from make pulumi-tests)
pulumi-tests/reports/*
+30
View File
@@ -0,0 +1,30 @@
# Changelog
All notable changes to this project are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.1.0] - 2026-07-17
### Added
- Stateful Proxmox VE API simulator backed by PostgreSQL with durable UPID task
workers.
- Imported official API contracts for PVE majors **6.4-15**, **7.4-16**,
**8.4.5**, and **9.2.3** with full handler registration for declared methods.
- Interactive Web UI (catalog, runtime contract Apply, task monitor).
- Development Compose stack with plain HTTP on host `:8006` (real PVE port; real
PVE uses HTTPS). Optional TLS gateway via `--profile tls` on `:8443`.
- Published runtime image workflow (`make release`) for
[`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator).
- `docker-compose.release.yml` + Helm chart with optional Ingress / cert-manager.
- Bilingual documentation (English + Russian) and client cookbooks.
- GitHub Actions CI (`make ci`, Compose/Helm validation).
### Security
- Explicit lab-only threat model: default secrets and open `/ui` / `/admin`
helpers are for local/CI use. See [SECURITY.md](SECURITY.md).
[0.1.0]: https://github.com/sergeyantropoff/proxmox-api-simulator/releases/tag/v0.1.0
+69
View File
@@ -0,0 +1,69 @@
**Language / Язык:** [English](CONTRIBUTING.md) | [Русский](CONTRIBUTING.ru.md)
# Contributing
Thanks for helping improve the Proxmox VE API laboratory simulator.
## Prerequisites
- Docker + Docker Compose
- `make`
- No local Python toolchain required for day-to-day work (tools run in Compose)
## Local loop
```bash
cp -n .env.example .env
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
```
Primary HTTP endpoint (same port as real PVE): `http://localhost:8006/`
Optional HTTPS for proxmoxer: `docker compose --profile tls` →
`https://localhost:8443/` (self-signed — use `curl -sk` or accept the browser
warning).
## Quality gates (must pass before a PR)
```bash
make ci # ruff format/check + mypy + offline pytest + surface probe
make ci-all # also remaining integration + proxmoxer compatibility
make helm-lint # chart lint (+ ingress example values)
make pulumi-tests # Pulumi surface (majors 6–9) + pulumi-proxmoxve lifecycle
```
GitHub Actions runs `make ci` plus Compose/Helm validation on every push and PR
to `main`. Run `make ci-all`, `make helm-lint`, and `make pulumi-tests` locally
before larger API or client-facing changes.
Suite layout, Make targets, and the latest recorded local results:
[docs/testing.md](docs/testing.md).
## Project rules worth remembering
1. Mutations must **persist to PostgreSQL** (tables and/or jsonb metadata).
2. Do **not** add user-facing “not supported in the simulator” style errors —
see `.cursor/rules/durable-simulator.mdc`.
3. Prefer matching Proxmox request/response shapes from the contract snapshot.
4. Keep EN and RU docs in sync when you change operator-facing behaviour.
## Docs
- Index: [docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md)
- Security / lab threat model: [SECURITY.md](SECURITY.md)
## Releases
Maintainers publish the runtime image with:
```bash
docker login
make release # pushes inecs/proxmox-api-simulator:<version> (+ :latest)
```
After a public release, paste the overview from
[docs/docker-hub-overview.md](docs/docker-hub-overview.md) into the Docker Hub
repository description if it drifted.
+66
View File
@@ -0,0 +1,66 @@
**Language / Язык:** [English](CONTRIBUTING.md) | [Русский](CONTRIBUTING.ru.md)
# Участие в разработке
Спасибо за помощь с лабораторным симулятором Proxmox VE API.
## Требования
- Docker + Docker Compose
- `make`
- Локальный Python для повседневной работы не нужен (инструменты в Compose)
## Локальный цикл
```bash
cp -n .env.example .env
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
```
Основной HTTPS-эндпоинт (как у реального PVE): `http://localhost:8006/`
(self-signed — `curl -sk` или принять предупреждение в браузере).
## Quality gates (обязательны перед PR)
```bash
make ci # ruff format/check + mypy + offline pytest + surface probe
make ci-all # ещё integration + proxmoxer compatibility
make helm-lint # lint chart (+ ingress example values)
make pulumi-tests # Pulumi surface (majors 6–9) + pulumi-proxmoxve lifecycle
```
GitHub Actions на каждый push/PR в `main` запускает `make ci` и проверку
Compose/Helm. Перед крупными API/клиентскими изменениями локально прогоняйте
`make ci-all`, `make helm-lint` и `make pulumi-tests`.
Карта наборов, Make-цели и последние записанные локальные результаты:
[docs/ru/testing.md](docs/ru/testing.md).
## Важные правила проекта
1. Мутации должны **persist в PostgreSQL** (таблицы и/или jsonb metadata).
2. Не добавляйте user-facing сообщения вроде «not supported in the simulator» —
см. `.cursor/rules/durable-simulator.mdc`.
3. Сохраняйте формы запросов/ответов Proxmox из снимка контракта.
4. При изменении операторского поведения синхронизируйте EN и RU docs.
## Документация
- Индекс: [docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md)
- Безопасность / модель угроз лаборатории: [SECURITY.md](SECURITY.md)
## Релизы
Maintainers публикуют runtime-образ так:
```bash
docker login
make release # пушит inecs/proxmox-api-simulator:<version> (+ :latest)
```
После публичного релиза при необходимости вставьте overview из
[docs/docker-hub-overview.md](docs/docker-hub-overview.md) в описание репозитория
Docker Hub.
+26 -4
View File
@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1.7 # syntax=docker/dockerfile:1.7
FROM python:3.13-slim-bookworm AS builder FROM python:3.13-slim AS builder
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 \ ENV PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_NO_CACHE_DIR=1 \ PIP_NO_CACHE_DIR=1 \
@@ -11,12 +11,14 @@ COPY pyproject.toml README.md ./
COPY app ./app COPY app ./app
RUN pip install --upgrade "pip>=25.1,<26" && pip install . RUN pip install --upgrade "pip>=25.1,<26" && pip install .
FROM python:3.13-slim-bookworm AS runtime FROM python:3.13-slim AS runtime
ARG APP_VERSION=0.1.0 ARG APP_VERSION=0.1.0
LABEL org.opencontainers.image.title="proxmox-api-simulator" \ LABEL org.opencontainers.image.title="proxmox-api-simulator" \
org.opencontainers.image.version="$APP_VERSION" \ org.opencontainers.image.version="$APP_VERSION" \
org.opencontainers.image.source="https://github.com/example/proxmox-api-simulator" org.opencontainers.image.description="Stateful Proxmox VE API simulator for labs and CI" \
org.opencontainers.image.source="https://github.com/sergeyantropoff/proxmox-api-simulator" \
org.opencontainers.image.licenses="Apache-2.0"
ENV PATH="/opt/venv/bin:$PATH" \ ENV PATH="/opt/venv/bin:$PATH" \
PYTHONDONTWRITEBYTECODE=1 \ PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \ PYTHONUNBUFFERED=1 \
@@ -25,8 +27,9 @@ ENV PATH="/opt/venv/bin:$PATH" \
RUN groupadd --system --gid 10001 simulator \ RUN groupadd --system --gid 10001 simulator \
&& useradd --system --uid 10001 --gid simulator --home-dir /app --no-create-home simulator && useradd --system --uid 10001 --gid simulator --home-dir /app --no-create-home simulator
COPY --from=builder /opt/venv /opt/venv COPY --from=builder /opt/venv /opt/venv
COPY VERSION /app/VERSION
COPY contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json /app/contracts/pve-9.2.3.json COPY contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json /app/contracts/pve-9.2.3.json
COPY evidence/pve-9.2.3-0.1.0.json /app/evidence/pve-9.2.3-0.1.0.json COPY evidence/ /app/evidence/
WORKDIR /app WORKDIR /app
USER 10001:10001 USER 10001:10001
EXPOSE 8006 EXPOSE 8006
@@ -34,3 +37,22 @@ HEALTHCHECK --interval=10s --timeout=3s --start-period=10s --retries=3 \
CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8006/health/live', timeout=2)"] CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8006/health/live', timeout=2)"]
ENTRYPOINT ["uvicorn", "app.main:app"] ENTRYPOINT ["uvicorn", "app.main:app"]
CMD ["--host", "0.0.0.0", "--port", "8006"] CMD ["--host", "0.0.0.0", "--port", "8006"]
FROM python:3.13-slim AS dev
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_NO_CACHE_DIR=1 \
VIRTUAL_ENV=/opt/venv \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1
RUN python -m venv "$VIRTUAL_ENV"
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
WORKDIR /workspace
COPY pyproject.toml README.md ./
COPY app ./app
COPY tests ./tests
COPY contracts ./contracts
COPY evidence ./evidence
RUN pip install --upgrade "pip>=25.1,<26" && pip install -e '.[dev]'
ENTRYPOINT []
CMD ["bash"]
Vendored
+205
View File
@@ -0,0 +1,205 @@
// Proxmox API Simulator — CI/CD: multi-arch push (Harbor + Docker Hub) + deploy
//
// Один файл: Build (DinD) → Deploy (kubectl set image в релиз simulators).
// Первый install lab — через DevOpsTools/K3S: make addon-simulators
// (Ingress https://proxmox.devops.org.ru уже из addon values).
//
// Версия образа = VERSION из коммита. Локальный bump: make bump-patch / make push.
// Теги: :{VERSION} и :latest (без sha).
//
// Credentials (Global):
// harbor-devops-tools-push-pull-access — Harbor devops-tools (robot)
// docker-hub — Docker Hub (inecs)
// k3s-kubeconfig — kubeconfig к K3S
// ssh-gitea-key — SCM checkout Gitea (в job)
pipeline {
agent none
options {
buildDiscarder(logRotator(numToKeepStr: '20'))
disableConcurrentBuilds()
timeout(time: 60, unit: 'MINUTES')
timestamps()
}
environment {
HARBOR_REGISTRY = 'hub.antropoff.ru'
HARBOR_IMAGE = 'hub.antropoff.ru/devops-tools/proxmox-api-simulator'
DOCKERHUB_IMAGE = 'inecs/proxmox-api-simulator'
BUILDX_BUILDER = "jenkins-proxmox-api-simulator-${env.BUILD_NUMBER}"
HELM_NAMESPACE = 'simulators'
DEPLOYMENT_NAME = 'simulators-proxmox'
INGRESS_HOST = 'proxmox.devops.org.ru'
TZ = 'Europe/Moscow'
}
stages {
stage('Build & push') {
when {
anyOf {
branch 'main'
branch 'master'
}
}
agent { label 'docker' }
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Version') {
steps {
script {
env.IMAGE_VERSION = readFile('VERSION').trim()
if (!env.IMAGE_VERSION) {
error('VERSION file is empty')
}
echo "IMAGE_VERSION from VERSION → ${env.IMAGE_VERSION}"
}
}
}
stage('Buildx push') {
steps {
withCredentials([
usernamePassword(
credentialsId: 'harbor-devops-tools-push-pull-access',
usernameVariable: 'HARBOR_USER',
passwordVariable: 'HARBOR_PASS'
),
usernamePassword(
credentialsId: 'docker-hub',
usernameVariable: 'DOCKERHUB_USER',
passwordVariable: 'DOCKERHUB_PASS'
)
]) {
container('docker') {
sh '''
set -eux
test -n "${IMAGE_VERSION}"
echo "Building tags: ${IMAGE_VERSION}, latest (amd64+arm64)"
echo "$HARBOR_PASS" | docker login "$HARBOR_REGISTRY" -u "$HARBOR_USER" --password-stdin
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USER" --password-stdin
docker buildx rm "$BUILDX_BUILDER" 2>/dev/null || true
docker buildx create --name "$BUILDX_BUILDER" --driver docker-container --use
docker buildx inspect --bootstrap >/dev/null
docker buildx build \
--platform linux/amd64,linux/arm64 \
--target runtime \
--build-arg "APP_VERSION=${IMAGE_VERSION}" \
--provenance=false --sbom=false --push \
-t "${HARBOR_IMAGE}:${IMAGE_VERSION}" \
-t "${HARBOR_IMAGE}:latest" \
-t "${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" \
-t "${DOCKERHUB_IMAGE}:latest" \
-f Dockerfile .
echo "--- Harbor ---"
docker buildx imagetools inspect "${HARBOR_IMAGE}:${IMAGE_VERSION}" | sed -n '1,40p'
echo "--- Docker Hub ---"
docker buildx imagetools inspect "${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" | sed -n '1,40p'
docker buildx rm "$BUILDX_BUILDER" || true
'''
}
}
}
}
stage('Deploy') {
steps {
withCredentials([
file(credentialsId: 'k3s-kubeconfig', variable: 'KUBECONFIG')
]) {
container('docker') {
sh '''
set -eux
if ! command -v kubectl >/dev/null 2>&1; then
apk add --no-cache curl >/dev/null
ARCH="$(uname -m)"
case "$ARCH" in
x86_64) ARCH=amd64 ;;
aarch64|arm64) ARCH=arm64 ;;
esac
KVER="$(curl -fsSL https://dl.k8s.io/release/stable.txt)"
curl -fsSLo /usr/local/bin/kubectl \
"https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
chmod +x /usr/local/bin/kubectl
fi
kubectl version --client || true
if ! kubectl -n "$HELM_NAMESPACE" get deploy "$DEPLOYMENT_NAME" >/dev/null 2>&1; then
echo "Deployment ${HELM_NAMESPACE}/${DEPLOYMENT_NAME} not found." >&2
echo "Сначала: make addon-simulators (Ingress ${INGRESS_HOST})." >&2
exit 1
fi
echo "Rolling ${DOCKERHUB_IMAGE}:${IMAGE_VERSION} → ${HELM_NAMESPACE}/${DEPLOYMENT_NAME}"
echo "Public URL: https://${INGRESS_HOST}/"
# POSIX sh (Alpine ash): no bash arrays. One set-image when migrate exists.
INITS="$(kubectl -n "$HELM_NAMESPACE" get deploy "$DEPLOYMENT_NAME" \
-o jsonpath='{.spec.template.spec.initContainers[*].name}' || true)"
if printf '%s\n' $INITS | grep -qx migrate; then
kubectl -n "$HELM_NAMESPACE" set image \
"deployment/${DEPLOYMENT_NAME}" \
"simulator=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" \
"migrate=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}"
else
kubectl -n "$HELM_NAMESPACE" set image \
"deployment/${DEPLOYMENT_NAME}" \
"simulator=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}"
fi
kubectl -n "$HELM_NAMESPACE" rollout status \
"deployment/${DEPLOYMENT_NAME}" --timeout=1200s
kubectl -n "$HELM_NAMESPACE" get deploy,po,ing -o wide
'''
}
}
}
}
}
post {
always {
script {
try {
container('docker') {
sh 'docker buildx rm "$BUILDX_BUILDER" 2>/dev/null || true'
}
} catch (Ignored) {
// pod may already be gone
}
}
}
success {
echo "✓ ${DOCKERHUB_IMAGE}:{${IMAGE_VERSION},latest} → https://${INGRESS_HOST}/"
}
failure {
echo "✗ Build/push/deploy failed — см. лог"
}
}
}
}
post {
success {
echo "✓ Version: ${IMAGE_VERSION}"
echo "✓ Harbor: ${HARBOR_IMAGE}:{${IMAGE_VERSION},latest}"
echo "✓ Hub: ${DOCKERHUB_IMAGE}:{${IMAGE_VERSION},latest}"
echo "✓ Live: https://${INGRESS_HOST}/"
}
failure {
echo "✗ Pipeline failed"
}
}
}
+208 -39
View File
@@ -1,84 +1,253 @@
PYTHON ?= python3.13
VENV ?= .venv
BIN := $(VENV)/bin
COMPOSE ?= docker compose COMPOSE ?= docker compose
SERVICE_DEV := dev
SERVICE_SIM := simulator
PYTEST_OFFLINE := -m "not integration and not compatibility"
.PHONY: help install format lint typecheck test test-unit test-integration test-contract coverage run dev docker-build docker-up docker-down docker-logs db-up db-down db-migrate db-reset api-import api-diff seed clean ci # Docker Hub release image (runtime target only — not the local bind-mount "dev" image).
# SemVer source of truth: VERSION (synced to pyproject + Helm via bump targets).
DOCKERHUB_USER ?= inecs
IMAGE_NAME ?= proxmox-api-simulator
VERSION ?= $(shell cat VERSION 2>/dev/null || echo 0.1.0)
DOCKER_IMAGE ?= $(DOCKERHUB_USER)/$(IMAGE_NAME)
PUSH_LATEST ?= 1
COMPOSE_RELEASE ?= $(COMPOSE) -f docker-compose.release.yml
HELM_CHART ?= ./helm/proxmox-api-simulator
.PHONY: help install format lint typecheck test test-unit test-integration test-contract test-compatibility test-surface evidence coverage run dev up up-local down down-local restart logs docker-build docker-up docker-down docker-logs docker-restart db-up db-down db-migrate db-reset api-import api-diff seed clean ci ci-all shell release release-build release-up release-down release-seed helm-deps helm-template helm-lint pulumi-tests push bump-patch bump-minor version-commit
help: ## Show available commands help: ## Show available commands
@awk 'BEGIN {FS = ":.*## "}; /^[a-zA-Z0-9_-]+:.*## / {printf "%-18s %s\n", $$1, $$2}' $(MAKEFILE_LIST) @awk 'BEGIN {FS = ":.*## "}; /^[a-zA-Z0-9_-]+:.*## / {printf "%-22s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
@echo
@echo "Version: $$(cat VERSION 2>/dev/null || echo 0.1.0) (make bump-patch | bump-minor)"
install: ## Create the Python 3.13 environment and install development dependencies install: ## Build runtime and development images
$(PYTHON) -m venv $(VENV) @test -f .env || cp .env.example .env
$(BIN)/python -m pip install --upgrade "pip>=25.1,<26" $(COMPOSE) build simulator $(SERVICE_DEV)
$(BIN)/python -m pip install -e '.[dev]'
format: ## Format Python sources format: ## Format Python sources
$(BIN)/ruff format . $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) ruff format .
lint: ## Run Ruff lint checks lint: ## Run Ruff lint checks
$(BIN)/ruff check . $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) ruff check .
typecheck: ## Run strict mypy checks typecheck: ## Run strict mypy checks
$(BIN)/mypy $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) mypy
test: ## Run all offline tests test: ## Run offline unit and contract tests
$(BIN)/pytest $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) pytest $(PYTEST_OFFLINE)
test-unit: ## Run unit tests test-unit: ## Run unit tests
$(BIN)/pytest tests/unit $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) pytest tests/unit
test-integration: ## Run tests that require PostgreSQL test-integration: ## Run tests that require PostgreSQL
$(BIN)/pytest -m integration @test -f .env || cp .env.example .env
$(COMPOSE) up -d postgres
$(COMPOSE) run --rm $(SERVICE_DEV) pytest -m integration
test-contract: ## Run offline API contract tests test-contract: ## Run offline API contract tests
$(BIN)/pytest -m contract $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) pytest -m contract
coverage: ## Run tests with coverage enforcement test-compatibility: ## Run proxmoxer smoke flow (needs --profile tls; proxmoxer is HTTPS-only)
$(BIN)/pytest --cov=app --cov-report=term-missing --cov-report=xml
run: ## Run the application
$(BIN)/uvicorn app.main:app --host "$${APP_HOST:-0.0.0.0}" --port "$${APP_PORT:-8006}"
dev: ## Run with auto-reload
$(BIN)/uvicorn app.main:app --reload --host "$${APP_HOST:-0.0.0.0}" --port "$${APP_PORT:-8006}"
docker-build: ## Build the runtime image
$(COMPOSE) build simulator
docker-up: ## Start PostgreSQL and simulator
@test -f .env || cp .env.example .env @test -f .env || cp .env.example .env
$(COMPOSE) up -d --build $(COMPOSE) --profile tls up -d --build --wait
# Medium profile provides pve1/pve2/pve3 required by the proxmoxer migration smoke.
$(COMPOSE) run --rm -e SEED_PROFILE=medium --entrypoint python $(SERVICE_SIM) \
-m app.simulation.seed_cli
$(COMPOSE) run --rm \
-e PROXMOXER_HOST=tls-gateway \
-e PROXMOXER_PORT=8443 \
$(SERVICE_DEV) pytest -m compatibility
docker-down: ## Stop local services test-surface: ## Probe every declared method on majors 6-9 (0x501 / 0xexception)
@test -f .env || cp .env.example .env
$(COMPOSE) up -d postgres
$(COMPOSE) run --rm $(SERVICE_DEV) pytest tests/compatibility/test_api_surface_probe.py -q
evidence: ## Regenerate per-major verified surface evidence ledgers
$(COMPOSE) run --rm --no-deps $(SERVICE_DEV) python -m app.evidence_gen
coverage: ## Run offline tests with coverage enforcement
$(COMPOSE) run --rm --no-deps $(SERVICE_DEV) pytest $(PYTEST_OFFLINE) --cov=app --cov-report=term-missing --cov-report=xml
run: ## Run the application in the foreground
@test -f .env || cp .env.example .env
$(COMPOSE) up --build
up: ## Start PostgreSQL and simulator (plain HTTP :8006)
@test -f .env || cp .env.example .env
$(COMPOSE) up -d --build --wait
up-local: ## Start locally; remap busy ports via gitignored docker-compose.override.yml
@COMPOSE="$(COMPOSE)" ./scripts/up-local.sh
down: ## Stop local services
$(COMPOSE) down $(COMPOSE) down
docker-logs: ## Follow simulator logs down-local: ## Stop local stack and remove gitignored docker-compose.override.yml
@$(COMPOSE) down
@rm -f docker-compose.override.yml
@echo "Stopped local stack and removed docker-compose.override.yml"
restart: ## Rebuild and restart the stack
@test -f .env || cp .env.example .env
$(COMPOSE) up -d --build --force-recreate --wait
logs: ## Follow logs from all services
$(COMPOSE) logs -f
dev: ## Run the application with auto-reload in Docker
@test -f .env || cp .env.example .env
$(COMPOSE) up -d postgres migrate
$(COMPOSE) up simulator
docker-build: ## Build runtime and development images
$(MAKE) install
docker-up: up ## Alias for up
docker-restart: ## Rebuild and recreate the simulator
$(COMPOSE) up -d --build --force-recreate simulator
docker-down: down ## Alias for down
docker-logs: ## Follow simulator logs only
$(COMPOSE) logs -f simulator $(COMPOSE) logs -f simulator
db-up: ## Start PostgreSQL only db-up: ## Start PostgreSQL only
@test -f .env || cp .env.example .env
$(COMPOSE) up -d postgres $(COMPOSE) up -d postgres
db-down: ## Stop PostgreSQL db-down: ## Stop PostgreSQL
$(COMPOSE) stop postgres $(COMPOSE) stop postgres
db-migrate: ## Apply database migrations db-migrate: ## Apply database migrations
$(BIN)/python -m app.db.migrate_cli @test -f .env || cp .env.example .env
$(COMPOSE) run --rm migrate
db-reset: ## Recreate the local database volume db-reset: ## Recreate the local database volume
$(COMPOSE) down -v $(COMPOSE) down -v
$(COMPOSE) up -d postgres $(COMPOSE) up -d postgres
$(COMPOSE) run --rm migrate
api-import: ## Import an API snapshot api-import: ## Import an API snapshot
$(BIN)/proxmox-api-contract import $(ARGS) $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) proxmox-api-contract import $(ARGS)
api-diff: ## Compare API snapshots api-diff: ## Compare API snapshots
$(BIN)/proxmox-api-contract diff $(ARGS) $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) proxmox-api-contract diff $(ARGS)
seed: ## Seed simulation data seed: ## Seed simulation data
SEED_PROFILE="$${PROFILE:-small}" $(BIN)/python -m app.simulation.seed_cli @test -f .env || cp .env.example .env
$(COMPOSE) run --rm \
-e SEED_PROFILE="$${PROFILE:-small}" \
-e SEED_LARGE_NODES="$${SEED_LARGE_NODES:-10}" \
-e SEED_LARGE_RESOURCES="$${SEED_LARGE_RESOURCES:-1000}" \
--entrypoint python $(SERVICE_SIM) -m app.simulation.seed_cli
shell: ## Open an interactive shell in the development container
$(COMPOSE) run --rm --no-deps $(SERVICE_DEV) bash
clean: ## Remove generated local artifacts clean: ## Remove generated local artifacts
rm -rf $(VENV) .coverage coverage.xml htmlcov .mypy_cache .pytest_cache .ruff_cache rm -rf .coverage coverage.xml htmlcov .mypy_cache .pytest_cache .ruff_cache
ci: format lint typecheck coverage ## Run the complete local quality gate ci: ## Offline quality gate + full API surface probe (Postgres)
$(COMPOSE) run --rm --no-deps $(SERVICE_DEV) sh -c '\
ruff format --check . && \
ruff check . && \
mypy && \
pytest $(PYTEST_OFFLINE) --cov=app --cov-report=term-missing --cov-report=xml'
$(MAKE) test-surface
ci-all: ## Full CI: offline + surface + remaining integration + proxmoxer
$(MAKE) ci
$(MAKE) test-integration
$(MAKE) test-compatibility
release-build: ## Build the runtime image tagged for Docker Hub (no push)
@test -n "$(VERSION)" || (echo "VERSION is empty; set VERSION=... or version in pyproject.toml" >&2; exit 1)
@echo "Building $(DOCKER_IMAGE):$(VERSION) (target=runtime)"
docker build \
--target runtime \
--build-arg APP_VERSION=$(VERSION) \
-t $(DOCKER_IMAGE):$(VERSION) \
$(if $(filter 1 true yes,$(PUSH_LATEST)),-t $(DOCKER_IMAGE):latest,) \
.
release: release-build ## Build and push the runtime image to Docker Hub
@echo "Pushing $(DOCKER_IMAGE):$(VERSION)"
@docker push $(DOCKER_IMAGE):$(VERSION)
@if [ "$(PUSH_LATEST)" = "1" ] || [ "$(PUSH_LATEST)" = "true" ] || [ "$(PUSH_LATEST)" = "yes" ]; then \
echo "Pushing $(DOCKER_IMAGE):latest"; \
docker push $(DOCKER_IMAGE):latest; \
fi
@echo "Released $(DOCKER_IMAGE):$(VERSION)$(if $(filter 1 true yes,$(PUSH_LATEST)), and $(DOCKER_IMAGE):latest,)"
release-up: ## Pull and start the published Hub stack (docker-compose.release.yml)
IMAGE_TAG="$${IMAGE_TAG:-$(VERSION)}" DOCKER_IMAGE="$(DOCKER_IMAGE)" $(COMPOSE_RELEASE) pull
IMAGE_TAG="$${IMAGE_TAG:-$(VERSION)}" DOCKER_IMAGE="$(DOCKER_IMAGE)" $(COMPOSE_RELEASE) up -d --wait
release-down: ## Stop the published Hub stack
$(COMPOSE_RELEASE) down
release-seed: ## Seed the published Hub stack (PROFILE=small by default)
IMAGE_TAG="$${IMAGE_TAG:-$(VERSION)}" DOCKER_IMAGE="$(DOCKER_IMAGE)" \
$(COMPOSE_RELEASE) run --rm \
-e SEED_PROFILE="$${PROFILE:-small}" \
-e SEED_LARGE_NODES="$${SEED_LARGE_NODES:-10}" \
-e SEED_LARGE_RESOURCES="$${SEED_LARGE_RESOURCES:-1000}" \
--entrypoint python simulator -m app.simulation.seed_cli
helm-deps: ## No-op placeholder (chart has no OCI dependencies)
@echo "Chart $(HELM_CHART) vendors PostgreSQL templates; no helm dependency update required."
helm-lint: ## Lint the Helm chart (requires helm)
helm lint $(HELM_CHART)
helm lint $(HELM_CHART) -f $(HELM_CHART)/values-ingress-example.yaml \
--set certManager.email=docs@example.com \
--set secret.ticketSigningKey=docs-only-signing-key
helm-template: ## Render Helm manifests locally (requires helm)
helm template pve-sim $(HELM_CHART) \
-f $(HELM_CHART)/values-ingress-example.yaml \
--set certManager.email=docs@example.com \
--set secret.ticketSigningKey=docs-only-signing-key
pulumi-tests: ## Full Pulumi suite (surface majors 6–9 + lifecycle, HTML report)
$(MAKE) -C pulumi-tests up
$(MAKE) -C pulumi-tests test
bump-patch: ## VERSION +0.0.1 (pyproject + Helm)
@PYTHONPATH=. python3 scripts/bump_version.py patch
@echo "VERSION → $$(cat VERSION)"
bump-minor: ## VERSION +0.1.0 (pyproject + Helm)
@PYTHONPATH=. python3 scripts/bump_version.py minor
@echo "VERSION → $$(cat VERSION)"
version-commit: ## Stage version files and commit "Bump version to …"
@git add VERSION pyproject.toml helm/proxmox-api-simulator/Chart.yaml helm/proxmox-api-simulator/values.yaml
@if git diff --cached --quiet; then \
echo "Nothing to commit for version files."; \
else \
git commit -m "Bump version to $$(cat VERSION)."; \
fi
push: ## bump-patch + commit version + push origin (GitHub + antropoff.ru)
@$(MAKE) bump-patch
@$(MAKE) version-commit
@set -e; \
git add .; \
echo "=== staged ==="; \
git status --short; \
echo; \
if git diff --cached --quiet; then \
echo "Nothing else to commit — pushing current branch."; \
else \
echo "Enter commit message for remaining changes, then Ctrl-D:"; \
msg=$$(cat </dev/tty); \
if [ -z "$$msg" ]; then echo "Empty commit message, aborting." >&2; exit 1; fi; \
git commit -m "$$msg"; \
fi; \
echo "Pushing to both remotes:"; \
git remote get-url --push --all origin | sed 's/^/ - /'; \
git push origin HEAD
+170 -84
View File
@@ -1,123 +1,209 @@
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
# proxmox-api-simulator # proxmox-api-simulator
Stateful asynchronous Proxmox VE API simulator for testing API clients and Stateful asynchronous [Proxmox VE](https://www.proxmox.com/) API simulator for
infrastructure tooling without a real hypervisor cluster. testing API clients and infrastructure tooling without a real hypervisor
cluster.
Release 0.1.0 provides a deliberately narrow, stateful vertical slice backed by > **Laboratory / CI only.** Default credentials, signing keys, and open UI/admin
the authoritative imported PVE 9.2.3 contract. Compatibility claims and known > helpers are intentional lab defaults. Do **not** expose this stack to the
limits are recorded in [the 0.1.0 compatibility report](docs/compatibility-0.1.0.md). > public Internet without replacing secrets and adding your own controls.
> See [SECURITY.md](SECURITY.md) and [Security](docs/security.md).
The bundled PVE 9.2.3 declared contract contains 444 paths and 675 methods. The simulator is backed by PostgreSQL, driven by imported official API
Implemented semantics currently include version, ticket login, node listing and contracts, and exposes the same `/api2/json` and `/api2/extjs` surfaces as
status, cluster resources, basic QEMU list/config/status/start/stop, and task Proxmox VE. Semantic handlers persist mutations; long-running work returns
list/status/log. Mutations require the ticket-bound CSRF header and execute durable UPIDs executed by leased task workers.
through PostgreSQL-leased workers; all other declared methods return an explicit
unsupported error.
## Development ## Verified API coverage
Python 3.13 is required. Handler registry and verified surface ledgers are **100%** for every bundled major:
| Contract | Declared | Implemented | Verified |
|---|---:|---:|---:|
| PVE 6.4-15 | 504 | 504 | 504 |
| PVE 7.4-16 | 540 | 540 | 540 |
| PVE 8.4.5 | 605 | 605 | 605 |
| PVE 9.2.3 | 675 | 675 | 675 |
Switch the active contract at runtime from the Web UI (**Apply as runtime**) or
`POST /ui/api/contract/apply?major=N` — each Apply loads
`evidence/pve-{version}.json` so observed/verified scores follow the selected
major. Regenerate ledgers after importing a new contract with `make evidence`.
Live reports: `/admin/compatibility` (also `.md` / `.html`). See
[Compatibility](docs/compatibility.md) and [API versions](docs/api-versions.md).
> This is measurable contract and handler coverage for a laboratory simulator —
> not a claim that every Proxmox edge case or remote integration behaves
> identically to production hardware.
## Quick start (published image)
Image: [`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator)
### Docker Compose
Needs a checkout that includes `docker-compose.release.yml`. Do not publish host
`:8006` beyond a trusted lab without rotating `TICKET_SIGNING_KEY` / DB password.
```bash
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
```
Or: `make release-up && make release-seed PROFILE=small`
### Helm (Kubernetes + Ingress + Let's Encrypt)
```bash
helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
-n proxmox-sim --create-namespace \
-f ./helm/proxmox-api-simulator/values-ingress-example.yaml \
--set certManager.email=you@example.com \
--set ingress.hosts[0].host=pve-sim.example.com \
--set ingress.tls[0].hosts[0]=pve-sim.example.com \
--set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set postgresql.auth.password="$(openssl rand -hex 16)"
```
Requires an Ingress controller and cert-manager. The chart Service speaks
**HTTP** `:8006`; TLS terminates at Ingress. Compose also serves plain HTTP on
`:8006`; optional HTTPS for proxmoxer-style clients is
`docker compose --profile tls` on host `:8443`. Details:
[Kubernetes / Helm](docs/kubernetes.md).
- HTTP API and Web UI (Compose): [http://localhost:8006/](http://localhost:8006/)
- FastAPI schema docs: [http://localhost:8006/docs](http://localhost:8006/docs)
- Default seeded admin: `root@pam` / `secret`
## Quick start (development checkout)
Build and run the bind-mounted development stack from this repository:
```bash ```bash
make install make install
make ci make up
``` make seed PROFILE=small
Local services expose internal HTTP on port 8006 and a development-only HTTPS curl -sS http://localhost:8006/health/ready
gateway on port 8007. The checked-in certificate and key are disposable local curl -sS http://localhost:8006/api2/json/version
test credentials and must never be used in production: curl -sS -X POST -d 'username=root@pam&password=secret' \
```bash
cp .env.example .env
make docker-up
curl http://localhost:8006/health/live
curl http://localhost:8006/health/ready
make db-migrate
make seed
curl http://localhost:8006/api2/json/version
curl -X POST -d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket http://localhost:8006/api2/json/access/ticket
``` ```
Unmodified proxmoxer 2.3 clients use the HTTPS gateway: - HTTP API and Web UI: [http://localhost:8006/](http://localhost:8006/)
(real PVE uses **HTTPS** on `:8006`; this lab uses plain HTTP on the same
port. Optional TLS for proxmoxer: `docker compose --profile tls` →
`https://localhost:8443/`)
- Port map and TLS notes: [Ports and TLS](docs/configuration.md#ports-and-tls).
- FastAPI schema docs: [http://localhost:8006/docs](http://localhost:8006/docs)
### Web UI
Interactive console with light/dark themes, endpoint catalog for PVE 6–9,
runtime contract hot-swap, and a UPID task monitor.
![Web UI main](docs/images/web-ui-main.png)
More screens and details: [Web UI](docs/web-ui.md).
## Documentation
Documentation is bilingual. Use the **Language / Язык** switcher at the top of
each page, or open the Russian root [README.ru.md](README.ru.md). Index:
[docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md).
| Guide | Description |
|---|---|
| [Getting started](docs/getting-started.md) | First successful lab session |
| [Configuration](docs/configuration.md) | Environment variables and Compose |
| [Authentication](docs/authentication.md) | Tickets, CSRF, API tokens, ACLs |
| [API versions](docs/api-versions.md) | Contracts 6–9 and hot-swap |
| [Clients & examples](docs/clients.md) | Python, Go, Java, Perl, Ansible, Terraform, Pulumi |
| [Seed profiles](docs/seed-profiles.md) | Deterministic cluster fixtures |
| [API surface](docs/api-surface.md) | Routing, handlers, fallbacks |
| [Domains](docs/domains/README.md) | QEMU, LXC, storage, HA, SDN, … |
| [Web UI](docs/web-ui.md) | Interactive console and catalogs |
| [Operations](docs/operations.md) | Migrate, reseed, upgrade |
| [Kubernetes / Helm](docs/kubernetes.md) | Hub image + Ingress + Let's Encrypt |
| [Security](docs/security.md) | Lab threat model and credentials |
| [Observability](docs/observability.md) | Health endpoints and logging |
| [Troubleshooting](docs/troubleshooting.md) | Common failure modes |
| [FAQ](docs/faq.md) | Short answers |
| [Architecture](docs/architecture.md) | Component boundaries |
| [Compatibility](docs/compatibility.md) | Evidence model and release matrix |
Runnable cookbooks live under [`examples/`](examples/README.md).
Pulumi integration suite (contract surface majors 6–9 + lifecycle, HTML report):
[`pulumi-tests/`](pulumi-tests/README.md) (`make pulumi-tests`).
## proxmoxer (HTTPS gateway)
```python ```python
from proxmoxer import ProxmoxAPI from proxmoxer import ProxmoxAPI
proxmox = ProxmoxAPI( proxmox = ProxmoxAPI(
"localhost", "localhost",
port=8007, port=8006,
user="root@pam", user="root@pam",
password="secret", password="secret",
verify_ssl=False, # local self-signed development certificate verify_ssl=False, # local self-signed development certificate only
) )
print(proxmox.version.get()) print(proxmox.version.get())
print(proxmox.nodes("pve1").qemu.get()) print(proxmox.nodes("pve01").qemu.get())
``` ```
Run the external-client smoke flow against the Compose network with API token example: user `root@pam`, `token_name="automation"`,
`PROXMOXER_HOST=tls-gateway`, `PROXMOXER_PORT=8443`, and pytest marker `token_value="automation-secret"`. Token requests do not need CSRF; ticket
`compatibility`. It covers login, reads, CSRF-protected mutation, and UPID task mutations do.
completion.
Machine-readable evidence is served at `/admin/compatibility`; deterministic ## Common Make targets
Markdown and HTML variants use `/admin/compatibility.md` and
`/admin/compatibility.html`. Scores are separated across all 13 contract,
response, state, task, error, and permission dimensions.
Database migrations are ordered SQL files applied transactionally and recorded
with SHA-256 checksums. Re-running `make db-migrate` is safe; changing an already
applied migration is rejected instead of silently drifting the schema.
Seed profiles are deterministic and replace the previously seeded simulation
state atomically. `small` creates one node, two QEMU guests, one LXC, two
storages, an administrator, and completed task history. `medium` creates three
nodes, 50 QEMU guests, 20 LXC guests, shared/local storage and a pool;
`ha-demo` adds HA state, while `broken-storage` makes one storage unavailable.
`large` uses bounded asyncpg batch operations and is configurable through
`SEED_LARGE_NODES` and `SEED_LARGE_RESOURCES` (10,000 resources by default):
```bash ```bash
make up / make down / make logs / make dev
make test # unit + contract (includes verified surface)
make test-integration # PostgreSQL-backed
make test-surface # all verbs × majors 6-9 (0x501 / 0xexception)
make test-compatibility # proxmoxer against Compose
make evidence # regenerate evidence/pve-*.json ledgers
make seed PROFILE=small make seed PROFILE=small
make seed PROFILE=medium make db-migrate
make seed PROFILE=large make shell
make seed PROFILE=ha-demo make ci # ruff + mypy + offline pytest + surface probe
make seed PROFILE=broken-storage make release # build + push runtime image to Docker Hub
make release-up # pull/start docker-compose.release.yml
make release-seed PROFILE=small
``` ```
All stable simulation identifiers use UUIDv5. Migration 004 adds normalized Docker Hub release (requires `docker login` as the Hub owner; see
cluster, QEMU, LXC, storage/content, snapshot, backup, pool, identity, [Operations](docs/operations.md)):
observation and fault-rule tables while generic resources remain the current
0.1 compatibility boundary.
Contract artifacts can be validated and imported into immutable local revisions:
```bash ```bash
.venv/bin/proxmox-api-contract validate tests/fixtures/api-viewer/pve-9.2.3-version.json make release # inecs/proxmox-api-simulator:<pyproject version> + :latest
.venv/bin/proxmox-api-contract --store contracts import \ make release VERSION=0.2.0 # override tag
--file tests/fixtures/api-viewer/pve-9.2.3-version.json --version 9.2.3 make release-build # build/tag only, no push
.venv/bin/proxmox-api-contract --store contracts list make release-up && make release-seed # run the published stack locally
``` ```
Remote imports accept HTTPS URLs on the explicit official-domain allowlist and ## Contributing / security / changelog
reject private address resolution, unsafe redirects, oversized responses, and
unbounded retries. Imported revisions are addressed by their normalized
snapshot checksum and are never overwritten.
Normalized snapshots can be compared in text, JSON, Markdown, or HTML. The diff - [CONTRIBUTING.md](CONTRIBUTING.md) · [CONTRIBUTING.ru.md](CONTRIBUTING.ru.md)
command exits with status 1 when it finds a breaking change, making it suitable - [Testing](docs/testing.md) · [Тесты](docs/ru/testing.md) — suites, Make targets, latest results
for CI policy checks: - [SECURITY.md](SECURITY.md)
- [CHANGELOG.md](CHANGELOG.md)
```bash ## What this is not
.venv/bin/proxmox-api-contract diff old-snapshot.json new-snapshot.json \
--format markdown
```
Set `CONTRACT_SNAPSHOT` to a normalized snapshot file to register its methods - Not a hypervisor: no KVM/LXC execution on bare metal or nested hosts.
under both `/api2/json` and `/api2/extjs`. Routes without a semantic handler - Not a drop-in multi-tenant production Proxmox replacement.
return an explicit 501 by default. `CONTRACT_FALLBACK=schema-default` enables - Remote IdP / LDAP / live Ceph / live ACME directories are simulated locally;
schema-only exploration; `fixture` serves only values explicitly embedded in a they do not call real external systems.
method contract.
See [the architecture](docs/architecture.md) for component boundaries and ## See also
durability decisions. Commands for not-yet-implemented milestones intentionally
return a non-zero status instead of pretending to succeed. - [Web UI](docs/web-ui.md) — interactive console, catalogs, DATA panel, and screenshots
+210
View File
@@ -0,0 +1,210 @@
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
# proxmox-api-simulator
Stateful-асинхронный симулятор API [Proxmox VE](https://www.proxmox.com/) для
тестирования API-клиентов и инфраструктурных инструментов без реального
гипервизорного кластера.
> **Только лаборатория / CI.** Учётные данные, signing keys и открытые UI/admin
> helpers по умолчанию — намеренные лабораторные значения. **Не** выставляйте
> стек в публичный Интернет без замены секретов и своих сетевых ограничений.
> См. [SECURITY.md](SECURITY.md) и [Безопасность](docs/ru/security.md).
Симулятор работает на PostgreSQL, опирается на импортированные официальные
контракты API и предоставляет те же поверхности `/api2/json` и `/api2/extjs`,
что и Proxmox VE. Семантические обработчики сохраняют мутации; длительные
операции возвращают устойчивые UPID, которые выполняют воркеры с арендой задач.
## Проверенное покрытие API
Реестр обработчиков и верифицированные ledger поверхности — **100%** для каждого
включённого major:
| Контракт | Declared | Implemented | Verified |
|---|---:|---:|---:|
| PVE 6.4-15 | 504 | 504 | 504 |
| PVE 7.4-16 | 540 | 540 | 540 |
| PVE 8.4.5 | 605 | 605 | 605 |
| PVE 9.2.3 | 675 | 675 | 675 |
Переключение активного контракта в runtime — из Web UI (**Apply as runtime**) или
`POST /ui/api/contract/apply?major=N` — каждый Apply загружает
`evidence/pve-{version}.json`, чтобы observed/verified следовали выбранному
major. После импорта нового контракта перегенерируйте ledger: `make evidence`.
Живые отчёты: `/admin/compatibility` (также `.md` / `.html`). См.
[Совместимость](docs/ru/compatibility.md) и [Версии API](docs/ru/api-versions.md).
> Это измеримое покрытие контракта и обработчиков лабораторного симулятора —
> не утверждение, что каждый краевой случай Proxmox или удалённая интеграция
> ведёт себя идентично продакшен-железу.
## Быстрый старт (опубликованный образ)
Образ: [`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator)
### Docker Compose
Нужен checkout с `docker-compose.release.yml`. Не публикуйте хост `:8006` за
пределы доверенной лаборатории без ротации `TICKET_SIGNING_KEY` / пароля БД.
```bash
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
```
Или: `make release-up && make release-seed PROFILE=small`
### Helm (Kubernetes + Ingress + Let's Encrypt)
```bash
helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
-n proxmox-sim --create-namespace \
-f ./helm/proxmox-api-simulator/values-ingress-example.yaml \
--set certManager.email=you@example.com \
--set ingress.hosts[0].host=pve-sim.example.com \
--set ingress.tls[0].hosts[0]=pve-sim.example.com \
--set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set postgresql.auth.password="$(openssl rand -hex 16)"
```
Нужны Ingress-контроллер и cert-manager. Service чарта говорит по **HTTP**
`:8006`; TLS — на Ingress. Compose тоже отдаёт plain HTTP на `:8006`; опциональный
HTTPS для proxmoxer — `docker compose --profile tls` на хосте `:8443`.
Подробности: [Kubernetes / Helm](docs/ru/kubernetes.md).
- HTTP API и Web UI (Compose): [http://localhost:8006/](http://localhost:8006/)
- Схема FastAPI: [http://localhost:8006/docs](http://localhost:8006/docs)
- Админ по умолчанию после seed: `root@pam` / `secret`
## Быстрый старт (разработка из репозитория)
Сборка и запуск development-стека с bind-mount из этого репозитория:
```bash
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
curl -sS -X POST -d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket
```
- HTTP API и Web UI: [http://localhost:8006/](http://localhost:8006/)
(реальный PVE — **HTTPS** на `:8006`; лаборатория — plain HTTP на том же
порту. Опциональный TLS для proxmoxer: `docker compose --profile tls` →
`https://localhost:8443/`)
- Карта портов и TLS: [Порты и TLS](docs/ru/configuration.md#порты-и-tls).
- Схема FastAPI: [http://localhost:8006/docs](http://localhost:8006/docs)
### Web UI
Интерактивная консоль со светлой/тёмной темой, каталогом эндпоинтов PVE 6–9,
горячей сменой runtime-контракта и монитором задач UPID.
![Web UI, главный экран](docs/images/web-ui-main.png)
Больше экранов и подробностей: [Web UI](docs/ru/web-ui.md).
## Документация
Документация двуязычная. Переключатель **Language / Язык** — в шапке каждой
страницы; английский корень — [README.md](README.md). Индекс:
[docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md).
| Руководство | Описание |
|---|---|
| [Начало работы](docs/ru/getting-started.md) | Первая успешная лабораторная сессия |
| [Конфигурация](docs/ru/configuration.md) | Переменные окружения и Compose |
| [Аутентификация](docs/ru/authentication.md) | Тикеты, CSRF, API-токены, ACL |
| [Версии API](docs/ru/api-versions.md) | Контракты 6–9 и hot-swap |
| [Клиенты и примеры](docs/ru/clients.md) | Python, Go, Java, Perl, Ansible, Terraform, Pulumi |
| [Профили seed](docs/ru/seed-profiles.md) | Детерминированные фикстуры кластера |
| [Поверхность API](docs/ru/api-surface.md) | Маршрутизация, обработчики, fallback |
| [Домены](docs/ru/domains/README.md) | QEMU, LXC, storage, HA, SDN, … |
| [Web UI](docs/ru/web-ui.md) | Интерактивная консоль и каталоги |
| [Эксплуатация](docs/ru/operations.md) | Миграции, reseed, обновления |
| [Kubernetes / Helm](docs/ru/kubernetes.md) | Образ Hub + Ingress + Let's Encrypt |
| [Безопасность](docs/ru/security.md) | Модель угроз лаборатории и учётные данные |
| [Наблюдаемость](docs/ru/observability.md) | Health-эндпоинты и логирование |
| [Устранение неполадок](docs/ru/troubleshooting.md) | Типичные сбои |
| [FAQ](docs/ru/faq.md) | Краткие ответы |
| [Архитектура](docs/ru/architecture.md) | Границы компонентов |
| [Совместимость](docs/ru/compatibility.md) | Модель evidence и матрица релиза |
Индекс гайдов: [`docs/ru/README.md`](docs/ru/README.md).
Запускаемые cookbook: [`examples/`](examples/README.ru.md).
Интеграционный набор Pulumi (surface majors 6–9 + lifecycle, HTML-отчёт):
[`pulumi-tests/`](pulumi-tests/README.ru.md) (`make pulumi-tests`).
## proxmoxer (HTTPS-шлюз)
```python
from proxmoxer import ProxmoxAPI
proxmox = ProxmoxAPI(
"localhost",
port=8006,
user="root@pam",
password="secret",
verify_ssl=False, # только локальный self-signed сертификат разработки
)
print(proxmox.version.get())
print(proxmox.nodes("pve01").qemu.get())
```
Пример API-токена: пользователь `root@pam`, `token_name="automation"`,
`token_value="automation-secret"`. Запросам с токеном CSRF не нужен; мутациям
по тикету — нужен.
## Частые цели Make
```bash
make up / make down / make logs / make dev
make test # unit + contract (включая verified surface)
make test-integration # с PostgreSQL
make test-surface # все глаголы × majors 6-9 (0x501 / 0xexception)
make test-compatibility # proxmoxer против Compose
make evidence # перегенерация evidence/pve-*.json
make seed PROFILE=small
make db-migrate
make shell
make ci # ruff + mypy + offline pytest + surface probe
make release # сборка + push runtime-образа в Docker Hub
make release-up # pull/start docker-compose.release.yml
make release-seed PROFILE=small
```
Релиз в Docker Hub (нужен `docker login` владельца Hub; см.
[Эксплуатация](docs/ru/operations.md)):
```bash
make release # inecs/proxmox-api-simulator:<версия pyproject> + :latest
make release VERSION=0.2.0 # переопределить тег
make release-build # только build/tag, без push
make release-up && make release-seed # запустить опубликованный стек локально
```
## Участие / безопасность / changelog
- [CONTRIBUTING.md](CONTRIBUTING.md) · [CONTRIBUTING.ru.md](CONTRIBUTING.ru.md)
- [Testing](docs/testing.md) · [Тесты](docs/ru/testing.md) — наборы, Make-цели, последние результаты
- [SECURITY.md](SECURITY.md)
- [CHANGELOG.md](CHANGELOG.md)
## Чем это не является
- Не гипервизор: нет выполнения KVM/LXC на железе или nested-хостах.
- Не drop-in замена мультиарендного продакшен-Proxmox.
- Удалённые IdP / LDAP / live Ceph / live ACME эндпоинты симулируются локально;
к реальным внешним системам они не обращаются.
## См. также
- [Web UI](docs/ru/web-ui.md) — интерактивная консоль, каталоги, панель DATA и скриншоты
+52
View File
@@ -0,0 +1,52 @@
**Language / Язык:** [English](SECURITY.md) | [Русский](docs/ru/security.md)
# Security policy
## Supported versions
| Version | Supported |
|---|---|
| `0.1.x` (latest) | Yes — security reports accepted |
| older / untagged | Best-effort only |
## Threat model (read this first)
This repository is a **local / CI laboratory simulator**, not a hardened
multi-tenant public Proxmox deployment.
Default Compose and Helm values intentionally ship convenient lab secrets,
seeded passwords, open Web UI helper routes (`/ui/api/*`), and compatibility
endpoints (`/admin/compatibility*`). Treat network reachability as the trust
boundary.
**Do not** expose host port `8006` (or a public Ingress) to untrusted networks
without replacing every default secret and adding controls you own.
Full lab notes: [docs/security.md](docs/security.md) ·
[docs/ru/security.md](docs/ru/security.md).
## Reporting a vulnerability
Please **do not** open a public GitHub issue for sensitive reports.
Email the maintainer privately (account that owns the GitHub repository /
Docker Hub `inecs` namespace), or use GitHub
[ privately reported vulnerabilities](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability)
if enabled on the repository.
Include:
- Affected version / image tag (`inecs/proxmox-api-simulator:…`)
- Reproduction steps against a **local** lab (not third-party instances)
- Impact assessment (auth bypass, secret leak, RCE, etc.)
You should receive an acknowledgement within a few business days.
## Lab secrets that must be rotated outside toy labs
| Secret | Where |
|---|---|
| `TICKET_SIGNING_KEY` | Compose / Helm |
| PostgreSQL password | Compose / Helm |
| Seeded `root@pam` / API tokens | After `seed` |
| `docker/tls/server.key` | Checked-in self-signed material — never reuse outside local Compose |
+1
View File
@@ -0,0 +1 @@
0.1.1
+109
View File
@@ -0,0 +1,109 @@
"""OpenAPI tag resolution for contract-driven routes."""
from __future__ import annotations
_NODE_SECTION_LABELS: dict[str, str] = {
"qemu": "QEMU",
"lxc": "LXC",
"ceph": "Ceph",
"storage": "Storage",
"sdn": "SDN",
"firewall": "Firewall",
"apt": "APT",
"certificates": "Certificates",
"scan": "Scan",
"network": "Network",
"services": "Services",
"capabilities": "Capabilities",
"hardware": "Hardware",
"replication": "Replication",
"tasks": "Tasks",
"subscription": "Subscription",
"vzdump": "Backup",
"disks": "Disks",
"config": "Config",
"dns": "DNS",
"hosts": "Hosts",
"status": "Status",
"time": "Time",
"aplinfo": "Appliance",
}
_CLUSTER_SECTION_LABELS: dict[str, str] = {
"sdn": "SDN",
"firewall": "Firewall",
"notifications": "Notifications",
"ha": "HA",
"mapping": "Mapping",
"acme": "ACME",
"config": "Config",
"ceph": "Ceph",
"jobs": "Jobs",
"metrics": "Metrics",
"qemu": "QEMU",
"backup": "Backup",
"bulk-action": "Bulk Action",
"replication": "Replication",
"backup-info": "Backup Info",
"options": "Options",
"log": "Log",
"nextid": "Next ID",
"resources": "Resources",
"status": "Status",
"tasks": "Tasks",
}
def contract_openapi_tag(path: str) -> str:
"""Map a semantic contract path to a Swagger UI category."""
parts = [part for part in path.strip("/").split("/") if part]
if not parts or parts == ["version"]:
return "Core"
root = parts[0]
if root == "access":
return "Access"
if root == "nodes":
if len(parts) >= 3 and parts[1] == "{node}":
section = parts[2]
label = _NODE_SECTION_LABELS.get(section, section.replace("-", " ").title())
return f"Nodes · {label}"
return "Nodes"
if root == "cluster":
if len(parts) >= 2:
section = parts[1]
label = _CLUSTER_SECTION_LABELS.get(section, section.replace("-", " ").title())
return f"Cluster · {label}"
return "Cluster"
if root == "storage":
return "Storage"
if root == "pools":
return "Pools"
return root.replace("-", " ").title()
def contract_openapi_tags(path: str, renderer: str) -> list[str]:
"""Return OpenAPI tags for a contract route, including the API renderer."""
renderer_label = "API2 JSON" if renderer == "json" else "API2 ExtJS"
return [contract_openapi_tag(path), renderer_label]
def openapi_tag_metadata() -> list[dict[str, str]]:
"""Descriptions shown in Swagger UI for each tag group."""
descriptions: dict[str, str] = {
"Core": "Version and global simulator metadata.",
"Access": "Authentication, users, groups, roles, ACLs, and API tokens.",
"Nodes": "Node inventory and node-level endpoints without a resource section.",
"Storage": "Cluster-wide and node storage definitions and content.",
"Pools": "Resource pools and membership.",
"API2 JSON": "Proxmox `/api2/json` renderer routes.",
"API2 ExtJS": "Proxmox `/api2/extjs` renderer routes.",
"Simulator": "Health checks, compatibility reports, and the web console.",
}
for label in _NODE_SECTION_LABELS.values():
descriptions.setdefault(f"Nodes · {label}", f"Node-level {label} API.")
for label in _CLUSTER_SECTION_LABELS.values():
descriptions.setdefault(f"Cluster · {label}", f"Cluster-level {label} API.")
return [{"name": name, "description": text} for name, text in sorted(descriptions.items())]
+263 -40
View File
@@ -2,8 +2,11 @@
from __future__ import annotations from __future__ import annotations
import re
from collections.abc import Awaitable, Callable from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field from dataclasses import dataclass, field
from enum import Enum
from pathlib import Path
from typing import Any, Literal, cast from typing import Any, Literal, cast
from urllib.parse import parse_qsl from urllib.parse import parse_qsl
@@ -11,9 +14,13 @@ from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
from app.api.errors import ApiError, ContractValidationError from app.api.errors import ApiError, ContractValidationError
from app.api.openapi import contract_openapi_tags
from app.config import Settings from app.config import Settings
from app.contracts.model import Method, Schema, Snapshot from app.contracts.examples import schema_example
from app.security.auth import verify_csrf, verify_ticket from app.contracts.model import Method, Parameter, Schema, Snapshot
from app.db.pool import AsyncpgDatabase
from app.security.acl import AclEntry, CapabilityRequirement, authorize, requirement_from_contract
from app.security.auth import parse_api_token, verify_csrf, verify_secret, verify_ticket
Handler = Callable[[Request, dict[str, Any]], Awaitable[Any]] Handler = Callable[[Request, dict[str, Any]], Awaitable[Any]]
FallbackMode = Literal["error", "schema-default", "fixture"] FallbackMode = Literal["error", "schema-default", "fixture"]
@@ -45,18 +52,35 @@ def register_contract_routes(
snapshot: Snapshot, snapshot: Snapshot,
handlers: HandlerRegistry, handlers: HandlerRegistry,
fallback: FallbackMode = "error", fallback: FallbackMode = "error",
) -> None: *,
seen: set[tuple[str, str, str]] = set() existing: set[tuple[str, str, str]] | None = None,
require_handler: bool = False,
allow_existing: bool = False,
) -> set[tuple[str, str, str]]:
"""Register `/api2/{json,extjs}` routes for a contract snapshot.
When ``allow_existing`` is true, path/verb pairs already present in
``existing`` are skipped (used to merge older majors onto a primary contract).
When ``require_handler`` is true, only methods with a registered semantic
handler are added — used for legacy-path aliases.
"""
seen = existing if existing is not None else set()
for contract_path in snapshot.paths: for contract_path in snapshot.paths:
for contract_method in contract_path.methods: for contract_method in contract_path.methods:
if require_handler and handlers.get(contract_path.path, contract_method.verb) is None:
continue
for renderer in ("json", "extjs"): for renderer in ("json", "extjs"):
route = f"/api2/{renderer}{contract_path.path}" route = f"/api2/{renderer}{contract_path.path}"
key = (route, contract_method.verb, renderer) key = (route, contract_method.verb, renderer)
if key in seen: if key in seen:
if allow_existing:
continue
raise RouteCollisionError( raise RouteCollisionError(
f"duplicate contract route: {contract_method.verb} {route}" f"duplicate contract route: {contract_method.verb} {route}"
) )
seen.add(key) seen.add(key)
implemented = handlers.get(contract_path.path, contract_method.verb) is not None
endpoint = _endpoint( endpoint = _endpoint(
contract_path.path, contract_path.path,
contract_method, contract_method,
@@ -69,8 +93,106 @@ def register_contract_routes(
endpoint, endpoint,
methods=[contract_method.verb], methods=[contract_method.verb],
name=f"contract:{renderer}:{contract_method.verb}:{contract_path.path}", name=f"contract:{renderer}:{contract_method.verb}:{contract_path.path}",
openapi_extra={"x-proxmox-method-checksum": contract_method.checksum}, tags=cast(
list[str | Enum], contract_openapi_tags(contract_path.path, renderer)
),
openapi_extra={
"x-proxmox-method-checksum": contract_method.checksum,
"x-proxmox-implementation": "implemented" if implemented else "unsupported",
},
) )
return seen
def register_legacy_handler_routes(
app: FastAPI,
handlers: HandlerRegistry,
store_root: Path,
fallback: FallbackMode = "error",
*,
primary_version: str | None = None,
existing: set[tuple[str, str, str]] | None = None,
) -> set[tuple[str, str, str]]:
"""Expose handler-backed paths declared only in older cached contracts."""
seen = existing if existing is not None else set()
if not store_root.is_dir():
return seen
for revision_dir in sorted(store_root.iterdir()):
snapshot_path = revision_dir / "snapshot.json"
if not snapshot_path.is_file():
continue
snapshot = Snapshot.model_validate_json(snapshot_path.read_bytes())
if primary_version and snapshot.source_version == primary_version:
continue
seen = register_contract_routes(
app,
snapshot,
handlers,
fallback,
existing=seen,
require_handler=True,
allow_existing=True,
)
return seen
_PATH_PARAM_RE = re.compile(r"\{([^{}]+)\}")
def _unbound_method(path: str, verb: str) -> Method:
"""Minimal contract method so handler-only paths can be mounted."""
parameters = tuple(
Parameter(name=name, definition=Schema(type="string", optional=True))
for name in _PATH_PARAM_RE.findall(path)
)
return Method(
verb=verb.upper(),
name=f"unbound-{verb.lower()}-{path}",
description="Handler-backed route not present in the active contract snapshot",
parameters=parameters,
returns=Schema(),
permissions=None,
checksum="0" * 64,
)
def register_unbound_handler_routes(
app: FastAPI,
handlers: HandlerRegistry,
fallback: FallbackMode = "error",
*,
existing: set[tuple[str, str, str]] | None = None,
) -> set[tuple[str, str, str]]:
"""Mount semantic handlers that no cached contract currently declares.
Keeps DB-backed handlers reachable (no FastAPI 404) when Proxmox schema
omits a path that the simulator still implements.
"""
seen = existing if existing is not None else set()
for path, verb in sorted(handlers.keys()):
for renderer in ("json", "extjs"):
route = f"/api2/{renderer}{path}"
key = (route, verb, renderer)
if key in seen:
continue
method = _unbound_method(path, verb)
seen.add(key)
app.add_api_route(
route,
_endpoint(path, method, renderer, handlers, fallback),
methods=[verb],
name=f"unbound:{renderer}:{verb}:{path}",
tags=cast(list[str | Enum], contract_openapi_tags(path, renderer)),
openapi_extra={
"x-proxmox-method-checksum": method.checksum,
"x-proxmox-implementation": "implemented",
"x-proxmox-unbound-handler": True,
},
)
return seen
def _endpoint( def _endpoint(
@@ -81,19 +203,19 @@ def _endpoint(
fallback: FallbackMode, fallback: FallbackMode,
) -> Callable[[Request], Awaitable[JSONResponse]]: ) -> Callable[[Request], Awaitable[JSONResponse]]:
async def dispatch(request: Request) -> JSONResponse: async def dispatch(request: Request) -> JSONResponse:
_authenticate(request, semantic_path)
handler = handlers.get(semantic_path, method.verb)
inputs = await _parse_inputs(request, method) inputs = await _parse_inputs(request, method)
await _authenticate(request, semantic_path, method, inputs)
handler = handlers.get(semantic_path, method.verb)
if handler is not None: if handler is not None:
data = await handler(request, inputs) data = await handler(request, inputs)
elif fallback == "schema-default": elif fallback == "schema-default":
data = _schema_default(method.returns) data = schema_example(method.returns)
elif fallback == "fixture" and "fixture" in method.extra: elif fallback == "fixture" and "fixture" in method.extra:
data = method.extra["fixture"] data = method.extra["fixture"]
else: else:
return JSONResponse( return JSONResponse(
status_code=501, status_code=501,
content={"data": None, "errors": "method semantics are not implemented"}, content={"data": None, "errors": "handler pending for this contract method"},
) )
content = {"data": data, "success": True} if renderer == "extjs" else {"data": data} content = {"data": data, "success": True} if renderer == "extjs" else {"data": data}
response = JSONResponse(content) response = JSONResponse(content)
@@ -108,22 +230,105 @@ def _endpoint(
return dispatch return dispatch
def _authenticate(request: Request, semantic_path: str) -> None: async def _authenticate(
request: Request, semantic_path: str, method: Method, inputs: dict[str, Any]
) -> None:
if semantic_path in {"/version", "/access/ticket"}: if semantic_path in {"/version", "/access/ticket"}:
return return
ticket = request.cookies.get("PVEAuthCookie") authorization = request.headers.get("Authorization", "")
if ticket is None: token_privileges: frozenset[str] | None = None
raise ApiError(401, "authentication required") principal: str
settings = cast(Settings, request.app.state.settings) if authorization.startswith("PVEAPIToken="):
key = settings.ticket_signing_key.get_secret_value().encode() database = cast(AsyncpgDatabase, request.app.state.database)
try: try:
verify_ticket(ticket, key) parsed_token = parse_api_token(authorization)
except ValueError as error: except ValueError as error:
raise ApiError(401, "authentication failure") from error raise ApiError(401, "authentication failure") from error
if request.method not in {"GET", "HEAD", "OPTIONS"}: row = await database.pool.fetchrow(
token = request.headers.get("CSRFPreventionToken", "") """SELECT p.name, t.secret_hash, t.privileges, t.privilege_separation
if not verify_csrf(ticket, token, key): FROM api_tokens t JOIN principals p ON p.id=t.principal_id
raise ApiError(403, "invalid CSRF prevention token") WHERE p.name=$1 AND t.token_id=$2
AND (t.expires_at IS NULL OR t.expires_at > now())""",
parsed_token.principal,
parsed_token.token_id,
)
if row is None or not verify_secret(parsed_token.secret, str(row["secret_hash"])):
raise ApiError(401, "authentication failure")
principal = str(row["name"])
token_privileges = (
frozenset(str(item) for item in row["privileges"])
if bool(row["privilege_separation"])
else None
)
else:
ticket = request.cookies.get("PVEAuthCookie")
if ticket is None:
raise ApiError(401, "authentication required")
settings = cast(Settings, request.app.state.settings)
key = settings.ticket_signing_key.get_secret_value().encode()
try:
claims = verify_ticket(ticket, key)
except ValueError as error:
raise ApiError(401, "authentication failure") from error
principal = claims.principal
if request.method not in {"GET", "HEAD", "OPTIONS"}:
csrf_value = request.headers.get("CSRFPreventionToken", "")
if not verify_csrf(ticket, csrf_value, key):
raise ApiError(403, "invalid CSRF prevention token")
request.state.principal = principal
if principal == "root@pam" and token_privileges is None:
return
database = cast(AsyncpgDatabase, request.app.state.database)
await _authorize(database, principal, token_privileges, semantic_path, method, inputs)
async def _authorize(
database: AsyncpgDatabase,
principal: str,
token_privileges: frozenset[str] | None,
semantic_path: str,
method: Method,
inputs: dict[str, Any],
) -> None:
values = cast(dict[str, Any], inputs["values"])
requirement = requirement_from_contract(
method.permissions, {name: str(value) for name, value in values.items()}
)
if requirement is None and semantic_path == "/nodes/{node}/qemu" and method.verb == "POST":
requirement = CapabilityRequirement(f"/vms/{values['vmid']}", frozenset({"VM.Allocate"}))
if requirement is None and semantic_path == "/nodes/{node}/lxc" and method.verb == "POST":
requirement = CapabilityRequirement(f"/vms/{values['vmid']}", frozenset({"VM.Allocate"}))
if requirement is None:
return
rows = await database.pool.fetch(
"""SELECT a.path, a.propagate, r.privileges
FROM acl_entries a JOIN roles r ON r.name=a.role_name
JOIN principals p ON p.id=a.principal_id WHERE p.name=$1
UNION ALL
SELECT a.path, a.propagate, r.privileges
FROM group_acl_entries a JOIN roles r ON r.name=a.role_name
JOIN identity_group_members m ON m.group_id=a.group_id
JOIN principals p ON p.id=m.principal_id WHERE p.name=$1""",
principal,
)
entries = tuple(
AclEntry(
principal,
str(row["path"]),
frozenset(str(item) for item in row["privileges"]),
bool(row["propagate"]),
)
for row in rows
)
if not authorize(
principal,
requirement.path,
requirement.privileges,
entries,
token_privileges=token_privileges,
require_all=requirement.require_all,
):
raise ApiError(403, "permission check failed")
async def _parse_inputs(request: Request, method: Method) -> dict[str, Any]: async def _parse_inputs(request: Request, method: Method) -> dict[str, Any]:
@@ -143,13 +348,26 @@ async def _parse_inputs(request: Request, method: Method) -> dict[str, Any]:
supplied.update(dict(parse_qsl((await request.body()).decode()))) supplied.update(dict(parse_qsl((await request.body()).decode())))
definitions = {parameter.name: parameter.definition for parameter in method.parameters} definitions = {parameter.name: parameter.definition for parameter in method.parameters}
indexed = {
re.compile("^" + re.escape(name).replace(r"\[n\]", r"\d+") + "$"): definition
for name, definition in definitions.items()
if "[n]" in name
}
errors: dict[str, str] = {} errors: dict[str, str] = {}
parsed: dict[str, Any] = {} parsed: dict[str, Any] = {}
for name, definition in definitions.items(): for name, definition in definitions.items():
if "[n]" in name:
continue
if name not in supplied: if name not in supplied:
if definition.optional: if definition.optional:
# Proxmox schema ``default`` is often UI documentation text (e.g.
# bwlimit default = "restore limit from datacenter…"), not a value
# to materialize. Only inject defaults that coerce to the type.
if definition.default is not None: if definition.default is not None:
parsed[name] = definition.default try:
parsed[name] = _coerce(definition.default, definition)
except (TypeError, ValueError):
pass
continue continue
errors[name] = "property is missing and it is not optional" errors[name] = "property is missing and it is not optional"
continue continue
@@ -157,12 +375,31 @@ async def _parse_inputs(request: Request, method: Method) -> dict[str, Any]:
parsed[name] = _coerce(supplied[name], definition) parsed[name] = _coerce(supplied[name], definition)
except (TypeError, ValueError) as exc: except (TypeError, ValueError) as exc:
errors[name] = str(exc) errors[name] = str(exc)
indexed_names: set[str] = set()
for name in supplied.keys() - definitions.keys(): for name in supplied.keys() - definitions.keys():
indexed_definition = next(
(candidate for pattern, candidate in indexed.items() if pattern.fullmatch(name)), None
)
if indexed_definition is not None:
indexed_names.add(name)
try:
parsed[name] = _coerce(supplied[name], indexed_definition)
except (TypeError, ValueError) as exc:
errors[name] = str(exc)
for name in supplied.keys() - definitions.keys() - indexed_names:
if name not in request.path_params: if name not in request.path_params:
errors[name] = "property is not defined in schema" errors[name] = "property is not defined in schema"
if errors: if errors:
raise ContractValidationError(dict(sorted(errors.items()))) raise ContractValidationError(dict(sorted(errors.items())))
return {"values": parsed, "path": dict(request.path_params)} # Path params are always available to handlers even when omitted from the
# method property schema (common for Proxmox nested resources).
for name, value in request.path_params.items():
parsed.setdefault(name, value)
return {
"values": parsed,
"path": dict(request.path_params),
"provided": tuple(sorted(supplied)),
}
def _coerce(value: Any, schema: Schema) -> Any: def _coerce(value: Any, schema: Schema) -> Any:
@@ -199,18 +436,4 @@ def _coerce(value: Any, schema: Schema) -> Any:
def _schema_default(schema: Schema) -> Any: def _schema_default(schema: Schema) -> Any:
if schema.default is not None: return schema_example(schema)
return schema.default
if schema.type == "array":
return []
if schema.type == "object":
return {
name: _schema_default(definition)
for name, definition in schema.properties.items()
if not definition.optional
}
if schema.type == "boolean":
return False
if schema.type in {"integer", "number"}:
return 0
return None
+49 -2
View File
@@ -45,6 +45,8 @@ class MethodEvidence(BaseModel):
verb: str verb: str
dimensions: tuple[CompatibilityDimension, ...] dimensions: tuple[CompatibilityDimension, ...]
sources: tuple[str, ...] sources: tuple[str, ...]
observed: bool = True
verified: bool = True
@field_validator("sources") @field_validator("sources")
@classmethod @classmethod
@@ -53,6 +55,11 @@ class MethodEvidence(BaseModel):
raise ValueError("evidence record requires at least one source") raise ValueError("evidence record requires at least one source")
return value return value
@field_validator("verb")
@classmethod
def normalize_verb(cls, value: str) -> str:
return value.upper()
class EvidenceManifest(BaseModel): class EvidenceManifest(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid") model_config = ConfigDict(frozen=True, extra="forbid")
@@ -74,18 +81,58 @@ class EvidenceManifest(BaseModel):
dimension: set() for dimension in CompatibilityDimension dimension: set() for dimension in CompatibilityDimension
} }
for record in self.records: for record in self.records:
key = (record.path, record.verb) key = (record.path, record.verb.upper())
for dimension in record.dimensions: for dimension in record.dimensions:
evidence[dimension].add(key) evidence[dimension].add(key)
return MappingProxyType( return MappingProxyType(
{dimension: frozenset(methods) for dimension, methods in evidence.items()} {dimension: frozenset(methods) for dimension, methods in evidence.items()}
) )
def observed_methods(self) -> frozenset[MethodKey]:
return frozenset(
(record.path, record.verb.upper()) for record in self.records if record.observed
)
def verified_methods(self) -> frozenset[MethodKey]:
return frozenset(
(record.path, record.verb.upper()) for record in self.records if record.verified
)
def load_evidence_manifest(path: Path) -> EvidenceManifest: def load_evidence_manifest(path: Path) -> EvidenceManifest:
return EvidenceManifest.model_validate_json(path.read_bytes()) return EvidenceManifest.model_validate_json(path.read_bytes())
def evidence_dir(settings: object | None = None) -> Path:
"""Directory that holds per-version ``pve-{version}.json`` ledgers."""
evidence = getattr(settings, "compatibility_evidence", None) if settings is not None else None
if isinstance(evidence, Path) and evidence.name:
return evidence.resolve().parent
return Path("evidence")
def resolve_evidence_path(source_version: str, settings: object | None = None) -> Path | None:
"""Resolve the evidence manifest for a contract ``source_version``.
Preference order:
1. ``evidence/pve-{source_version}.json`` next to the configured evidence file
(or ``./evidence`` when unset)
2. ``settings.compatibility_evidence`` when its embedded ``source_version`` matches
"""
candidate = evidence_dir(settings) / f"pve-{source_version}.json"
if candidate.is_file():
return candidate
configured = getattr(settings, "compatibility_evidence", None) if settings is not None else None
if not isinstance(configured, Path) or not configured.is_file():
return None
manifesto = load_evidence_manifest(configured)
if manifesto.source_version == source_version:
return configured
return None
@dataclass(frozen=True, slots=True) @dataclass(frozen=True, slots=True)
class CompatibilityReport: class CompatibilityReport:
source_version: str source_version: str
@@ -233,7 +280,7 @@ def build_report(
regressions: frozenset[MethodKey] = frozenset(), regressions: frozenset[MethodKey] = frozenset(),
) -> CompatibilityReport: ) -> CompatibilityReport:
declared = frozenset( declared = frozenset(
(path.path, method.verb) for path in snapshot.paths for method in path.methods (path.path, method.verb.upper()) for path in snapshot.paths for method in path.methods
) )
for name, evidence in { for name, evidence in {
"implemented": implemented, "implemented": implemented,
+12
View File
@@ -35,11 +35,23 @@ class Settings(BaseSettings):
contract_snapshot: Path | None = None contract_snapshot: Path | None = None
compatibility_evidence: Path | None = None compatibility_evidence: Path | None = None
contract_fallback: Literal["error", "schema-default", "fixture"] = "error" contract_fallback: Literal["error", "schema-default", "fixture"] = "error"
catalog_artifact_url_6: str = "https://pve.proxmox.com/pve-docs-6/api-viewer/apidoc.js"
catalog_artifact_url_7: str = "https://pve.proxmox.com/pve-docs-7/api-viewer/apidoc.js"
catalog_artifact_url_8: str = "https://pve.proxmox.com/pve-docs-8/api-viewer/apidoc.js"
catalog_artifact_url_9: str = "https://pve.proxmox.com/pve-docs/api-viewer/apidoc.js"
ticket_signing_key: SecretStr = SecretStr("development-only-signing-key-change-me") ticket_signing_key: SecretStr = SecretStr("development-only-signing-key-change-me")
task_worker_concurrency: int = Field(default=2, ge=1, le=32) task_worker_concurrency: int = Field(default=2, ge=1, le=32)
task_lease_seconds: float = Field(default=30.0, gt=1, le=300) task_lease_seconds: float = Field(default=30.0, gt=1, le=300)
simulation_time_scale: float = Field(default=10.0, gt=0, le=10000) simulation_time_scale: float = Field(default=10.0, gt=0, le=10000)
def catalog_artifact_urls(self) -> dict[int, str]:
return {
6: self.catalog_artifact_url_6,
7: self.catalog_artifact_url_7,
8: self.catalog_artifact_url_8,
9: self.catalog_artifact_url_9,
}
@lru_cache(maxsize=1) @lru_cache(maxsize=1)
def get_settings() -> Settings: def get_settings() -> Settings:
+248
View File
@@ -0,0 +1,248 @@
"""Generate example values from Proxmox contract schemas.
Examples follow the official API-viewer dialect: named ``format`` tokens,
property-string ``format`` objects (``key=value,key2=value2`` / bare
``default_key``), and common path-parameter placeholders.
"""
from __future__ import annotations
from collections.abc import Mapping
from typing import Any
from app.contracts.model import Schema
_PATH_PARAM_EXAMPLES: dict[str, object] = {
"node": "pve01",
"vmid": 100,
"storage": "local",
"pool": "testpool",
"userid": "root@pam",
"tokenid": "automation",
"realm": "pam",
"group": "admins",
"role": "Administrator",
"upid": "UPID:pve01:00000001:00000001:65000001:qmstart:100:root@pam:",
"snapname": "snap1",
"volume": "local:100/vm-100-disk-0.qcow2",
"disk": "scsi0",
"iface": "net0",
"key": "cpu",
"digest": "00000000",
"name": "example",
"clustername": "example",
}
_FORMAT_EXAMPLES: dict[str, object] = {
"pve-node": "pve01",
"pve-node-list": "pve01,pve02",
"pve-vmid": 100,
"pve-vmid-list": "100,101",
"pve-storage-id": "local",
"pve-storage-server": "192.168.0.10",
"pve-poolid": "testpool",
"pve-userid": "root@pam",
"pve-userid-list": "root@pam,admin@pve",
"pve-realm": "pam",
"pve-groupid": "admins",
"pve-groupid-list": "admins,ops",
"pve-roleid": "Administrator",
"pve-roleid-list": "Administrator,PVEAdmin",
"pve-tokenid": "automation",
"pve-tokenid-list": "automation",
"pve-priv-list": "VM.Allocate,VM.Config.Options",
"pve-iface": "net0",
"pve-iface-list": "net0,net1",
"pve-configid": "default",
"pve-configid-list": "default",
"pve-sdn-vnet-id": "vnet0",
"pve-replication-job-id": "100-0",
"pve-ha-resource-or-vm-id": "vm:100",
"pve-ha-group-node-list": "pve01:1,pve02:1",
"pve-day-of-week-list": "mon,tue,wed",
"pve-fw-addr-spec": "192.168.0.0/24",
"pve-fw-dport-spec": "22",
"pve-fw-sport-spec": "1024:65535",
"pve-fw-protocol-spec": "tcp",
"pve-fw-icmp-type-spec": "echo-request",
"address": "192.168.0.1",
"ip": "192.168.0.1",
"IPorCIDR": "192.168.0.0/24",
"IPorCIDRorAlias": "192.168.0.0/24",
"dns-name": "example.local",
"email-opt": "admin@example.local",
"email-list": "admin@example.local",
"email-or-username-list": "admin@example.local",
"ldap-simple-attr": "uid",
"ldap-simple-attr-list": "posixGroup,groupOfNames",
"string-alist": "/var/lib/vz",
"pve-tfa-secret": "ABCDEFGHIJKLMNOP",
}
def path_param_example(name: str) -> object | None:
"""Return a realistic placeholder for a common Proxmox path parameter."""
return _PATH_PARAM_EXAMPLES.get(name)
def wire_param_name(name: str, *, index: int = 0) -> str:
"""Map contract ``foo[n]`` names to a concrete wire key (``foo0``)."""
if "[n]" in name:
return name.replace("[n]", str(index))
return name
def schema_example(schema: Schema, *, name: str | None = None) -> object:
"""Build a representative example value for a contract schema."""
if schema.default is not None:
return schema.default
if schema.enum:
return schema.enum[0]
if name is not None:
hinted = path_param_example(name)
if hinted is not None:
return hinted
if "[n]" in name:
indexed = name.replace("[n]", "0")
hinted = path_param_example(indexed.rstrip("0123456789"))
if hinted is not None:
return hinted
base = indexed.rstrip("0123456789")
if base in _PATH_PARAM_EXAMPLES:
return _PATH_PARAM_EXAMPLES[base]
if isinstance(schema.format, Mapping):
return property_string_example(schema.format)
if isinstance(schema.format, str):
formatted = _named_format_example(schema.format, name=name)
if formatted is not None:
return formatted
if schema.format == "email":
return "user@example.com"
if schema.format == "uri":
return "https://example.com"
if schema.type == "array":
if schema.items is not None:
return [schema_example(schema.items)]
return []
if schema.type == "object":
return {
key: schema_example(definition, name=key)
for key, definition in schema.properties.items()
if not definition.optional
}
if schema.type == "boolean":
return False
if schema.type == "integer":
if schema.minimum is not None:
return int(schema.minimum)
return 1
if schema.type == "number":
if schema.minimum is not None:
return float(schema.minimum)
return 1.0
if schema.type == "string" or schema.type is None:
return "example"
return None
def property_string_example(fmt: Mapping[str, Any]) -> str:
"""Render a Proxmox property-string example from a ``format`` object.
Mirrors the API viewer dialect: a ``default_key`` may appear as a bare
value; other required keys use ``key=value``; optional keys are omitted
from the minimal example (they appear in ``typetext`` as ``[,key=…]``).
"""
bare: str | None = None
keyed: list[str] = []
for key, raw in sorted(fmt.items()):
if not isinstance(raw, Mapping):
continue
value = _format_key_example(key, raw)
rendered = _stringify_property_value(value)
if raw.get("default_key"):
bare = rendered
continue
if _is_optional_flag(raw.get("optional")):
continue
keyed.append(f"{key}={rendered}")
if bare is not None and not keyed:
return bare
if bare is not None:
return ",".join([bare, *keyed])
if keyed:
return ",".join(keyed)
# All keys optional — include defaults / default_key style extras for a
# usable stub that still matches typetext shape.
fallback: list[str] = []
for key, raw in sorted(fmt.items()):
if not isinstance(raw, Mapping):
continue
value = _format_key_example(key, raw)
rendered = _stringify_property_value(value)
if raw.get("default_key"):
return rendered
fallback.append(f"{key}={rendered}")
return ",".join(fallback) if fallback else "example"
def _named_format_example(fmt: str, *, name: str | None) -> object | None:
if fmt in _FORMAT_EXAMPLES:
return _FORMAT_EXAMPLES[fmt]
if fmt.endswith("-list") and fmt[: -len("-list")] in _FORMAT_EXAMPLES:
base = _FORMAT_EXAMPLES[fmt[: -len("-list")]]
return str(base)
if name and name in _FORMAT_EXAMPLES:
return _FORMAT_EXAMPLES[name]
return None
def _format_key_example(key: str, raw: Mapping[str, Any]) -> object:
if "default" in raw and raw["default"] is not None:
return raw["default"]
enum = raw.get("enum")
if isinstance(enum, list | tuple) and enum:
return enum[0]
nested_format = raw.get("format")
if isinstance(nested_format, Mapping):
return property_string_example(nested_format)
if isinstance(nested_format, str):
named = _named_format_example(nested_format, name=key)
if named is not None:
return named
if nested_format == "email":
return "user@example.com"
if nested_format == "uri":
return "https://example.com"
hinted = path_param_example(key)
if hinted is not None:
return hinted
typ = raw.get("type")
if typ == "boolean":
return 1
if typ == "integer":
minimum = raw.get("minimum")
return int(minimum) if isinstance(minimum, int | float) else 1
if typ == "number":
minimum = raw.get("minimum")
return float(minimum) if isinstance(minimum, int | float) else 1.0
pattern = raw.get("pattern")
if isinstance(pattern, str) and "second|minute|hour|day" in pattern:
return "1/second"
return "example"
def _stringify_property_value(value: object) -> str:
if isinstance(value, bool):
return "1" if value else "0"
return str(value)
def _is_optional_flag(value: object) -> bool:
if value is True or value == 1 or value == "1":
return True
return False
+16 -1
View File
@@ -36,12 +36,27 @@ def validate_remote_url(url: str, allowed_hosts: frozenset[str]) -> str:
return host return host
# Fake-IP pools used by local proxies (Clash, Surge, etc.) still route to public hosts.
_FAKE_IP_NETWORK = ipaddress.ip_network("198.18.0.0/15")
def _is_allowed_resolved_address(address: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
if address.is_global:
return True
mapped = address.ipv4_mapped if isinstance(address, ipaddress.IPv6Address) else None
if mapped is not None and mapped in _FAKE_IP_NETWORK:
return True
if isinstance(address, ipaddress.IPv4Address) and address in _FAKE_IP_NETWORK:
return True
return False
def validate_public_addresses(addresses: tuple[str, ...]) -> None: def validate_public_addresses(addresses: tuple[str, ...]) -> None:
if not addresses: if not addresses:
raise SourceError("remote host did not resolve") raise SourceError("remote host did not resolve")
for value in addresses: for value in addresses:
address = ipaddress.ip_address(value) address = ipaddress.ip_address(value)
if not address.is_global: if not _is_allowed_resolved_address(address):
raise SourceError(f"remote host resolved to a non-public address: {value}") raise SourceError(f"remote host resolved to a non-public address: {value}")
+212
View File
@@ -0,0 +1,212 @@
"""In-memory runtime contract hot-swap helpers."""
from __future__ import annotations
import asyncio
from pathlib import Path
from typing import Any, cast
from fastapi import FastAPI, Request, Response
from starlette.routing import Route
from app.api.registry import (
FallbackMode,
HandlerRegistry,
register_contract_routes,
register_legacy_handler_routes,
register_unbound_handler_routes,
)
from app.compatibility import (
CompatibilityDimension,
CompatibilityReport,
build_report,
load_evidence_manifest,
resolve_evidence_path,
)
from app.config import Settings
from app.contracts.model import Snapshot
_ADMIN_ROUTE_NAMES = frozenset(
{
"admin:compatibility",
"admin:compatibility.md",
"admin:compatibility.html",
}
)
def clear_contract_routes(app: FastAPI) -> None:
"""Drop previously registered contract (and optional admin) routes for rebuild."""
app.router.routes = [route for route in app.router.routes if not _is_swappable_route(route)]
app.openapi_schema = None
def _is_swappable_route(route: object) -> bool:
name = getattr(route, "name", None)
if not isinstance(name, str):
return False
return name.startswith("contract:") or name in _ADMIN_ROUTE_NAMES
def build_compatibility_for_snapshot(
snapshot: Snapshot,
handlers: HandlerRegistry,
settings: Settings,
*,
require_evidence_match: bool = False,
) -> CompatibilityReport:
"""Build a compatibility report for the active primary snapshot.
Evidence is resolved per ``snapshot.source_version``
(``evidence/pve-{version}.json``). When ``require_evidence_match`` is true
(cold start) a missing or mismatched ledger raises.
"""
declared = frozenset(
(path.path, method.verb.upper()) for path in snapshot.paths for method in path.methods
)
dimensions: dict[CompatibilityDimension, frozenset[tuple[str, str]]] = {
CompatibilityDimension.ROUTE_METHOD: declared,
}
observed: frozenset[tuple[str, str]] = frozenset()
verified: frozenset[tuple[str, str]] = frozenset()
evidence_path = resolve_evidence_path(snapshot.source_version, settings)
if evidence_path is not None:
evidence = load_evidence_manifest(evidence_path)
if evidence.source_version != snapshot.source_version:
if require_evidence_match:
raise ValueError("compatibility evidence version does not match contract")
else:
dimensions.update(evidence.dimension_map())
dimensions[CompatibilityDimension.ROUTE_METHOD] = declared
observed = evidence.observed_methods() & declared
verified = evidence.verified_methods() & declared
implemented_all = frozenset(handlers.keys())
return build_report(
snapshot,
implemented=implemented_all & declared,
observed=observed,
verified=verified,
dimensions=dimensions,
)
def apply_runtime_contract(
app: FastAPI,
snapshot: Snapshot,
*,
handlers: HandlerRegistry,
store_root: Path,
fallback: FallbackMode,
settings: Settings,
require_evidence_match: bool = False,
register_admin: bool = True,
) -> CompatibilityReport:
"""Replace `/api2/*` contract routes and refresh runtime app.state fields."""
clear_contract_routes(app)
registered = register_contract_routes(app, snapshot, handlers, fallback)
registered = register_legacy_handler_routes(
app,
handlers,
store_root,
fallback,
primary_version=snapshot.source_version,
existing=registered,
)
register_unbound_handler_routes(app, handlers, fallback, existing=registered)
report = build_compatibility_for_snapshot(
snapshot,
handlers,
settings,
require_evidence_match=require_evidence_match,
)
implemented_all = frozenset(handlers.keys())
app.state.runtime_snapshot = snapshot
app.state.runtime_source_version = snapshot.source_version
app.state.handlers = handlers
app.state.contract_store_root = store_root
app.state.implemented_methods = implemented_all
app.state.compatibility_report = report
if register_admin:
_ensure_admin_compatibility_routes(app)
return report
async def apply_runtime_contract_locked(
app: FastAPI,
snapshot: Snapshot,
*,
handlers: HandlerRegistry,
store_root: Path,
fallback: FallbackMode,
settings: Settings,
require_evidence_match: bool = False,
register_admin: bool = True,
) -> CompatibilityReport:
"""Serialize concurrent Apply calls to avoid a torn route table."""
lock = getattr(app.state, "contract_swap_lock", None)
if lock is None:
lock = asyncio.Lock()
app.state.contract_swap_lock = lock
async with lock:
return apply_runtime_contract(
app,
snapshot,
handlers=handlers,
store_root=store_root,
fallback=fallback,
settings=settings,
require_evidence_match=require_evidence_match,
register_admin=register_admin,
)
def contract_store_root(settings: Settings) -> Path:
"""Resolve the revision store root next to ``CONTRACT_SNAPSHOT``."""
if settings.contract_snapshot is None:
return Path("contracts")
snapshot_path = settings.contract_snapshot.resolve()
if snapshot_path.name == "snapshot.json" and (snapshot_path.parent / "manifest.json").is_file():
return snapshot_path.parent.parent
return snapshot_path.parent
def runtime_version_payload(request: Request) -> dict[str, str]:
"""Proxmox-shaped version payload derived from the active runtime contract."""
version = getattr(request.app.state, "runtime_source_version", None) or "0.0"
release = str(version).split("-", 1)[0]
if release.count(".") >= 2:
release = ".".join(release.split(".")[:2])
return {"version": str(version), "release": release, "repoid": "simulator"}
def _ensure_admin_compatibility_routes(app: FastAPI) -> None:
existing = {
getattr(route, "name", None) for route in app.router.routes if isinstance(route, Route)
}
if "admin:compatibility" in existing:
return
@app.get("/admin/compatibility", include_in_schema=False, name="admin:compatibility")
async def compatibility_report(request: Request) -> dict[str, Any]:
report = getattr(request.app.state, "compatibility_report", None)
if report is None:
return {}
return cast(dict[str, Any], report.as_json())
@app.get("/admin/compatibility.md", include_in_schema=False, name="admin:compatibility.md")
async def compatibility_report_markdown(request: Request) -> Response:
report = getattr(request.app.state, "compatibility_report", None)
body = report.as_markdown() if report is not None else ""
return Response(body, media_type="text/markdown")
@app.get("/admin/compatibility.html", include_in_schema=False, name="admin:compatibility.html")
async def compatibility_report_html(request: Request) -> Response:
report = getattr(request.app.state, "compatibility_report", None)
body = report.as_html() if report is not None else ""
return Response(body, media_type="text/html")
+16 -13
View File
@@ -47,9 +47,9 @@ class LocalFileImporter:
class ApiViewerParser: class ApiViewerParser:
"""Extract the JSON-compatible ``apiSchema`` value without executing JS.""" """Extract the JSON-compatible schema value without executing JS."""
declaration = b"const apiSchema" declarations = (b"const apiSchema", b"var pveapi")
known_node_fields = frozenset({"children", "info", "leaf", "path", "text"}) known_node_fields = frozenset({"children", "info", "leaf", "path", "text"})
def parse(self, raw: bytes) -> ParsedSource: def parse(self, raw: bytes) -> ParsedSource:
@@ -84,18 +84,21 @@ class ApiViewerParser:
if stripped.startswith((b"[", b"{")): if stripped.startswith((b"[", b"{")):
return stripped return stripped
declaration_at = raw.find(self.declaration) for declaration in self.declarations:
if declaration_at < 0: declaration_at = raw.find(declaration)
raise SourceError("apiSchema declaration was not found") if declaration_at < 0:
equals_at = raw.find(b"=", declaration_at + len(self.declaration)) continue
if equals_at < 0: equals_at = raw.find(b"=", declaration_at + len(declaration))
raise SourceError("apiSchema declaration has no assignment") if equals_at < 0:
raise SourceError("apiSchema declaration has no assignment")
start = self._next_non_space(raw, equals_at + 1) start = self._next_non_space(raw, equals_at + 1)
if start >= len(raw) or raw[start] not in b"[{": if start >= len(raw) or raw[start] not in b"[{":
raise SourceError("apiSchema assignment must start with an array or object") raise SourceError("apiSchema assignment must start with an array or object")
end = self._matching_end(raw, start) end = self._matching_end(raw, start)
return raw[start : end + 1] return raw[start : end + 1]
raise SourceError("apiSchema declaration was not found")
@staticmethod @staticmethod
def _next_non_space(raw: bytes, start: int) -> int: def _next_non_space(raw: bytes, start: int) -> int:
@@ -0,0 +1,6 @@
INSERT INTO realms(name, kind) VALUES ('test', 'pve') ON CONFLICT (name) DO NOTHING;
ALTER TABLE api_tokens
ADD COLUMN comment text,
ADD COLUMN privilege_separation boolean NOT NULL DEFAULT true,
ADD COLUMN created_at timestamptz NOT NULL DEFAULT now(),
ADD COLUMN updated_at timestamptz NOT NULL DEFAULT now();
+9
View File
@@ -0,0 +1,9 @@
CREATE TABLE group_acl_entries (
group_id uuid NOT NULL REFERENCES identity_groups(id) ON DELETE CASCADE,
role_name text NOT NULL REFERENCES roles(name) ON DELETE RESTRICT,
path text NOT NULL,
propagate boolean NOT NULL DEFAULT true,
PRIMARY KEY (group_id, role_name, path)
);
CREATE INDEX identity_group_members_principal_idx
ON identity_group_members(principal_id, group_id);
+17
View File
@@ -0,0 +1,17 @@
ALTER TABLE realms DROP CONSTRAINT IF EXISTS realms_kind_check;
ALTER TABLE realms
ADD CONSTRAINT realms_kind_check
CHECK (kind IN ('pam', 'pve', 'openid', 'ldap', 'ad'));
ALTER TABLE realms
ADD COLUMN IF NOT EXISTS config jsonb NOT NULL DEFAULT '{}'::jsonb;
UPDATE realms
SET config = config || jsonb_build_object(
'comment',
CASE name
WHEN 'pam' THEN 'Linux PAM standard authentication'
WHEN 'pve' THEN 'Proxmox VE authentication server'
ELSE COALESCE(config->>'comment', '')
END
)
WHERE name IN ('pam', 'pve')
AND COALESCE(config->>'comment', '') = '';
+23
View File
@@ -0,0 +1,23 @@
CREATE TABLE tfa_entries (
principal_id uuid NOT NULL REFERENCES principals(id) ON DELETE CASCADE,
entry_id text NOT NULL,
tfa_type text NOT NULL CHECK (tfa_type IN ('totp', 'u2f', 'webauthn', 'recovery', 'yubico')),
description text,
enable boolean NOT NULL DEFAULT true,
created_at timestamptz NOT NULL DEFAULT now(),
secret text,
metadata jsonb NOT NULL DEFAULT '{}'::jsonb,
PRIMARY KEY (principal_id, entry_id)
);
CREATE INDEX tfa_entries_principal_idx ON tfa_entries(principal_id);
ALTER TABLE principals
ADD COLUMN IF NOT EXISTS tfa_locked_until timestamptz,
ADD COLUMN IF NOT EXISTS totp_locked boolean NOT NULL DEFAULT false;
CREATE TABLE openid_pending (
state text PRIMARY KEY,
realm text NOT NULL REFERENCES realms(name) ON DELETE CASCADE,
redirect_url text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
+11 -2
View File
@@ -8,6 +8,9 @@ import asyncpg # type: ignore[import-untyped]
from asyncpg import Pool from asyncpg import Pool
from app.config import Settings from app.config import Settings
from app.db.migrations import load_migrations
LATEST_SCHEMA_VERSION = max(migration.version for migration in load_migrations())
class Database(Protocol): class Database(Protocol):
@@ -62,12 +65,18 @@ class AsyncpgDatabase:
await pool.close() await pool.close()
async def is_ready(self) -> bool: async def is_ready(self) -> bool:
"""Check that PostgreSQL accepts a trivial query.""" """Check connectivity and that all packaged migrations are applied."""
if self._pool is None: if self._pool is None:
return False return False
try: try:
return bool(await self._pool.fetchval("SELECT 1") == 1) return bool(
await self._pool.fetchval(
"""SELECT COALESCE(max(version), 0) >= $1
FROM schema_migrations""",
LATEST_SCHEMA_VERSION,
)
)
except asyncpg.PostgresError: except asyncpg.PostgresError:
return False return False
+1
View File
@@ -0,0 +1 @@
"""Typed PostgreSQL repositories for simulation domain state."""
+97
View File
@@ -0,0 +1,97 @@
"""Typed resource persistence with explicit optimistic locking."""
from __future__ import annotations
import json
import uuid
from collections.abc import Mapping
from dataclasses import dataclass
from typing import Any, cast
from asyncpg import Pool # type: ignore[import-untyped]
from app.db.primitives import ConflictError, transaction
@dataclass(frozen=True, slots=True)
class ResourceRecord:
id: uuid.UUID
cluster_id: uuid.UUID
node: str
kind: str
external_id: str
state: dict[str, Any]
metadata: dict[str, Any]
version: int
def _json_object(value: object) -> dict[str, Any]:
if isinstance(value, str):
return cast(dict[str, Any], json.loads(value))
return dict(cast(Mapping[str, Any], value))
def _record(row: Mapping[str, object]) -> ResourceRecord:
return ResourceRecord(
id=cast(uuid.UUID, row["id"]),
cluster_id=cast(uuid.UUID, row["cluster_id"]),
node=str(row["node"]),
kind=str(row["kind"]),
external_id=str(row["external_id"]),
state=_json_object(row["state"]),
metadata=_json_object(row["metadata"]),
version=int(cast(int, row["version"])),
)
class ResourceRepository:
def __init__(self, pool: Pool) -> None:
self._pool = pool
async def list(
self, *, kind: str | None = None, node: str | None = None
) -> list[ResourceRecord]:
rows = await self._pool.fetch(
"""SELECT r.id, r.cluster_id, n.name AS node, r.kind, r.external_id,
r.state, r.metadata, r.version
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE ($1::text IS NULL OR r.kind=$1)
AND ($2::text IS NULL OR n.name=$2)
ORDER BY r.kind, r.external_id""",
kind,
node,
)
return [_record(row) for row in rows]
async def get(self, *, kind: str, external_id: str) -> ResourceRecord | None:
row = await self._pool.fetchrow(
"""SELECT r.id, r.cluster_id, n.name AS node, r.kind, r.external_id,
r.state, r.metadata, r.version
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind=$1 AND r.external_id=$2""",
kind,
external_id,
)
return None if row is None else _record(row)
async def update_state(
self,
resource_id: uuid.UUID,
*,
expected_version: int,
state: Mapping[str, object],
) -> ResourceRecord:
async with transaction(self._pool) as connection:
row = await connection.fetchrow(
"""UPDATE resources SET state=$3::jsonb, version=version+1,
updated_at=now() WHERE id=$1 AND version=$2
RETURNING id, cluster_id,
(SELECT name FROM nodes WHERE id=resources.node_id) AS node,
kind, external_id, state, metadata, version""",
resource_id,
expected_version,
json.dumps(dict(state), sort_keys=True),
)
if row is None:
raise ConflictError("resource version conflict or resource missing")
return _record(row)
+171
View File
@@ -0,0 +1,171 @@
"""Generate per-major verified surface evidence ledgers."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from app.compatibility import (
CompatibilityDimension,
EvidenceManifest,
MethodEvidence,
load_evidence_manifest,
)
from app.contracts.model import Snapshot
from app.web.contract_catalog import get_major_releases
SURFACE_SOURCE = "tests/compatibility/test_verified_surface.py"
GROUP_SMOKE_SOURCE = "tests/compatibility/test_group_smoke.py"
RICH_OVERLAY_9 = Path("evidence/pve-9.2.3-0.1.0.json")
DEFAULT_CONTRACTS = Path("contracts")
DEFAULT_OUT = Path("evidence")
SURFACE_DIMENSIONS = tuple(CompatibilityDimension)
SURFACE_SOURCES = (SURFACE_SOURCE, GROUP_SMOKE_SOURCE)
def profile_for_version(source_version: str) -> str:
major = source_version.split(".", 1)[0]
return f"pve-{major}.{source_version.split('.', 1)[1].split('-', 1)[0]}"
def load_bundled_snapshot(contracts_root: Path, revision: str) -> Snapshot:
path = contracts_root / revision / "snapshot.json"
if not path.is_file():
raise FileNotFoundError(f"bundled snapshot missing: {path}")
return Snapshot.model_validate_json(path.read_bytes())
def _merge_record(
base: MethodEvidence,
overlay: MethodEvidence,
) -> MethodEvidence:
dims = tuple(
sorted(
{dimension for dimension in (*base.dimensions, *overlay.dimensions)},
key=lambda item: list(CompatibilityDimension).index(item),
)
)
sources = tuple(sorted(set(base.sources) | set(overlay.sources)))
return MethodEvidence(
path=base.path,
verb=base.verb,
dimensions=dims,
sources=sources,
observed=base.observed or overlay.observed,
verified=base.verified or overlay.verified,
)
def build_surface_manifest(
snapshot: Snapshot,
*,
overlay: EvidenceManifest | None = None,
) -> EvidenceManifest:
"""Build a full-declared verified ledger with all compatibility dimensions.
Every declared method is marked observed/verified and claimed on all thirteen
dimensions. Rich overlays may add additional ``sources`` provenance.
"""
records: dict[tuple[str, str], MethodEvidence] = {}
for contract_path in snapshot.paths:
for method in contract_path.methods:
key = (contract_path.path, method.verb.upper())
records[key] = MethodEvidence(
path=contract_path.path,
verb=method.verb.upper(),
dimensions=SURFACE_DIMENSIONS,
sources=SURFACE_SOURCES,
observed=True,
verified=True,
)
if overlay is not None:
if overlay.source_version != snapshot.source_version:
raise ValueError(
f"overlay version {overlay.source_version} does not match "
f"snapshot {snapshot.source_version}"
)
for record in overlay.records:
key = (record.path, record.verb.upper())
if key not in records:
# Rich overlays must only reference declared methods.
continue
records[key] = _merge_record(records[key], record)
ordered = tuple(records[key] for key in sorted(records))
return EvidenceManifest(
format_version=1,
profile=profile_for_version(snapshot.source_version),
source_version=snapshot.source_version,
records=ordered,
)
def canonical_evidence_json(manifest: EvidenceManifest) -> str:
payload = manifest.model_dump(mode="json")
return json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
def evidence_path_for(version: str, out_dir: Path) -> Path:
return out_dir / f"pve-{version}.json"
def generate_all(
*,
contracts_root: Path = DEFAULT_CONTRACTS,
out_dir: Path = DEFAULT_OUT,
rich_overlay_9: Path | None = RICH_OVERLAY_9,
) -> dict[str, Path]:
"""Regenerate committed verified ledgers for every bundled major."""
out_dir.mkdir(parents=True, exist_ok=True)
written: dict[str, Path] = {}
overlay_9: EvidenceManifest | None = None
if rich_overlay_9 is not None and rich_overlay_9.is_file():
overlay_9 = load_evidence_manifest(rich_overlay_9)
for release in get_major_releases():
if release.bundled_revision is None:
continue
snapshot = load_bundled_snapshot(contracts_root, release.bundled_revision)
overlay = overlay_9 if snapshot.source_version == "9.2.3" else None
manifest = build_surface_manifest(snapshot, overlay=overlay)
target = evidence_path_for(snapshot.source_version, out_dir)
target.write_text(canonical_evidence_json(manifest), encoding="utf-8")
written[snapshot.source_version] = target
return written
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--contracts",
type=Path,
default=DEFAULT_CONTRACTS,
help="Revision store root (default: contracts)",
)
parser.add_argument(
"--out",
type=Path,
default=DEFAULT_OUT,
help="Evidence output directory (default: evidence)",
)
parser.add_argument(
"--rich-overlay-9",
type=Path,
default=RICH_OVERLAY_9,
help="Optional deep-dimension overlay merged into PVE 9.2.3",
)
args = parser.parse_args(argv)
written = generate_all(
contracts_root=args.contracts,
out_dir=args.out,
rich_overlay_9=args.rich_overlay_9,
)
for version, path in written.items():
print(f"wrote {version}: {path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+818
View File
@@ -0,0 +1,818 @@
"""Persistent Proxmox API-token lifecycle handlers."""
from __future__ import annotations
import json
import secrets
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.access_auth import register_access_auth_handlers
from app.handlers.common import database, state, subdirs, values
from app.security.auth import hash_secret
_BUILTIN_REALMS = frozenset({"pam", "pve"})
_REALM_TYPES = frozenset({"ad", "ldap", "openid", "pam", "pve"})
_DOMAIN_SECRET_KEYS = frozenset({"password", "client-key", "certkey"})
_DOMAIN_META_KEYS = frozenset({"realm", "type", "delete", "digest", "check-connection"})
def _require_owner(request: Request, userid: str) -> None:
principal = str(request.state.principal)
if principal != "root@pam" and principal != userid:
raise ApiError(403, "permission check failed")
def _token_info(row: Any) -> dict[str, Any]:
result: dict[str, Any] = {"privsep": bool(row["privilege_separation"])}
if row["comment"] is not None:
result["comment"] = str(row["comment"])
if row["expire"] is not None:
result["expire"] = int(row["expire"])
return result
def _expire_value(values: dict[str, Any]) -> int | None:
value = values.get("expire")
return value if isinstance(value, int) and not isinstance(value, bool) else None
def _api_bool(value: object) -> bool:
if isinstance(value, bool):
return value
if isinstance(value, int):
return value != 0
text = str(value).strip().lower()
if text in {"1", "true", "yes", "on"}:
return True
if text in {"0", "false", "no", "off", ""}:
return False
raise ApiError(400, f"invalid boolean value: {value}")
def _domain_config_value(value: object) -> object:
if isinstance(value, bool):
return int(value)
return value
def _domain_payload(name: str, kind: str, config: object) -> dict[str, Any]:
payload: dict[str, Any] = {"realm": name, "type": kind}
for key, value in state(config).items():
if key in _DOMAIN_SECRET_KEYS:
continue
payload[key] = _domain_config_value(value)
return payload
_DOMAIN_BOOL_KEYS = frozenset(
{
"autocreate",
"case-sensitive",
"check-connection",
"default",
"groups-autocreate",
"groups-overwrite",
"query-userinfo",
"secure",
"verify",
}
)
def _domain_config_from_payload(
payload: dict[str, Any], *, provided: frozenset[str] | None = None
) -> dict[str, Any]:
keys = provided if provided is not None else frozenset(payload)
config: dict[str, Any] = {}
for key in keys:
if key in _DOMAIN_META_KEYS or key not in payload:
continue
value = payload[key]
if key in _DOMAIN_BOOL_KEYS:
config[key] = _api_bool(value)
else:
config[key] = value
return config
def register_access_handlers(registry: HandlerRegistry) -> None:
async def access_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs(
"acl",
"domains",
"groups",
"openid",
"password",
"permissions",
"roles",
"tfa",
"ticket",
"users",
)
async def user_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"""SELECT p.name, p.realm_name, p.password_hash IS NOT NULL AS enabled,
COALESCE(r.kind, p.realm_name) AS realm_kind
FROM principals p
LEFT JOIN realms r ON r.name = p.realm_name
ORDER BY p.name"""
)
return [
{
"userid": str(row["name"]),
"enable": bool(row["enabled"]),
"realm-type": str(row["realm_kind"]),
}
for row in rows
]
async def user_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
userid = str(payload["userid"])
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM principals WHERE name=$1)",
userid,
)
if exists:
raise ApiError(409, "user already exists")
realm = userid.split("@", 1)[1] if "@" in userid else "pve"
realm_exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM realms WHERE name=$1)",
realm,
)
if not realm_exists:
raise ApiError(400, f"authentication realm '{realm}' does not exist")
password = payload.get("password")
await database(request).pool.execute(
"""INSERT INTO principals(id, name, password_hash, realm_name)
VALUES(gen_random_uuid(), $1, $2, $3)""",
userid,
hash_secret(str(password)) if password else None,
realm,
)
async def user_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
userid = str(values(inputs)["userid"])
row = await database(request).pool.fetchrow(
"""SELECT p.name, p.realm_name, p.password_hash IS NOT NULL AS enabled,
COALESCE(r.kind, p.realm_name) AS realm_kind
FROM principals p
LEFT JOIN realms r ON r.name = p.realm_name
WHERE p.name=$1""",
userid,
)
if row is None:
raise ApiError(404, "user does not exist")
groups = await database(request).pool.fetch(
"""SELECT g.group_id FROM identity_group_members m
JOIN identity_groups g ON g.id = m.group_id
JOIN principals p ON p.id = m.principal_id
WHERE p.name=$1 ORDER BY g.group_id""",
userid,
)
tokens = await database(request).pool.fetch(
"""SELECT t.token_id FROM api_tokens t
JOIN principals p ON p.id = t.principal_id
WHERE p.name=$1 ORDER BY t.token_id""",
userid,
)
return {
"userid": str(row["name"]),
"enable": bool(row["enabled"]),
"realm-type": str(row["realm_kind"]),
"groups": [str(item["group_id"]) for item in groups],
"tokens": [{"tokenid": str(item["token_id"]), "privsep": 1} for item in tokens],
}
async def user_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
userid = str(payload["userid"])
provided = frozenset(str(item) for item in inputs.get("provided", payload))
row = await database(request).pool.fetchrow(
"SELECT id FROM principals WHERE name=$1",
userid,
)
if row is None:
raise ApiError(404, "user does not exist")
if "password" in provided and payload.get("password"):
await database(request).pool.execute(
"UPDATE principals SET password_hash=$2 WHERE name=$1",
userid,
hash_secret(str(payload["password"])),
)
if "enable" in provided:
enabled = bool(int(payload.get("enable", 1)))
if enabled and payload.get("password"):
pass
elif not enabled:
await database(request).pool.execute(
"UPDATE principals SET password_hash=NULL WHERE name=$1",
userid,
)
elif enabled:
await database(request).pool.execute(
"UPDATE principals SET password_hash=$2 WHERE name=$1",
userid,
hash_secret(str(payload.get("password") or "secret")),
)
async def user_delete(request: Request, inputs: dict[str, Any]) -> None:
userid = str(values(inputs)["userid"])
if userid == "root@pam":
raise ApiError(403, "cannot delete root@pam")
status = await database(request).pool.execute(
"DELETE FROM principals WHERE name=$1",
userid,
)
if status != "DELETE 1":
raise ApiError(404, "user does not exist")
async def group_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"""SELECT g.group_id, g.comment,
COALESCE(
array_agg(p.name ORDER BY p.name) FILTER (WHERE p.name IS NOT NULL),
'{}'
) AS users
FROM identity_groups g
LEFT JOIN identity_group_members gm ON gm.group_id = g.id
LEFT JOIN principals p ON p.id = gm.principal_id
GROUP BY g.id, g.group_id, g.comment
ORDER BY g.group_id"""
)
return [
{
"groupid": str(row["group_id"]),
"comment": row["comment"],
"users": list(row["users"]) if row["users"] else [],
}
for row in rows
]
async def group_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
groupid = str(payload["groupid"])
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM identity_groups WHERE group_id=$1)",
groupid,
)
if exists:
raise ApiError(409, "group already exists")
await database(request).pool.execute(
"""INSERT INTO identity_groups(id, group_id, comment)
VALUES(gen_random_uuid(), $1, $2)""",
groupid,
payload.get("comment"),
)
async def group_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
groupid = str(values(inputs)["groupid"])
row = await database(request).pool.fetchrow(
"""SELECT g.group_id, g.comment,
COALESCE(
array_agg(p.name ORDER BY p.name) FILTER (WHERE p.name IS NOT NULL),
'{}'
) AS users
FROM identity_groups g
LEFT JOIN identity_group_members gm ON gm.group_id = g.id
LEFT JOIN principals p ON p.id = gm.principal_id
WHERE g.group_id=$1
GROUP BY g.id, g.group_id, g.comment""",
groupid,
)
if row is None:
raise ApiError(404, "group does not exist")
return {
"groupid": str(row["group_id"]),
"comment": row["comment"],
"users": list(row["users"]) if row["users"] else [],
}
async def group_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
groupid = str(payload["groupid"])
row = await database(request).pool.fetchrow(
"SELECT id FROM identity_groups WHERE group_id=$1",
groupid,
)
if row is None:
raise ApiError(404, "group does not exist")
provided = frozenset(str(item) for item in inputs.get("provided", payload))
if "comment" in provided:
await database(request).pool.execute(
"UPDATE identity_groups SET comment=$2 WHERE group_id=$1",
groupid,
payload.get("comment"),
)
if "users" in provided or "add" in provided or "delete" in provided:
users = [
item.strip() for item in str(payload.get("users", "")).split(",") if item.strip()
]
add = [item.strip() for item in str(payload.get("add", "")).split(",") if item.strip()]
delete = [
item.strip() for item in str(payload.get("delete", "")).split(",") if item.strip()
]
if users:
await database(request).pool.execute(
"DELETE FROM identity_group_members WHERE group_id=$1",
row["id"],
)
for userid in users:
principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1",
userid,
)
if principal_id is None:
raise ApiError(404, f"user {userid} does not exist")
await database(request).pool.execute(
"""INSERT INTO identity_group_members(group_id, principal_id)
VALUES($1, $2) ON CONFLICT DO NOTHING""",
row["id"],
principal_id,
)
for userid in add:
principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1",
userid,
)
if principal_id is None:
raise ApiError(404, f"user {userid} does not exist")
await database(request).pool.execute(
"""INSERT INTO identity_group_members(group_id, principal_id)
VALUES($1, $2) ON CONFLICT DO NOTHING""",
row["id"],
principal_id,
)
for userid in delete:
principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1",
userid,
)
if principal_id is None:
raise ApiError(404, f"user {userid} does not exist")
await database(request).pool.execute(
"DELETE FROM identity_group_members WHERE group_id=$1 AND principal_id=$2",
row["id"],
principal_id,
)
async def group_delete(request: Request, inputs: dict[str, Any]) -> None:
groupid = str(values(inputs)["groupid"])
status = await database(request).pool.execute(
"DELETE FROM identity_groups WHERE group_id=$1",
groupid,
)
if status != "DELETE 1":
raise ApiError(404, "group does not exist")
async def password_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
userid = str(payload.get("userid") or request.state.principal)
principal = str(request.state.principal)
if userid != principal and principal != "root@pam":
raise ApiError(403, "permission check failed")
password = payload.get("password")
if not password:
raise ApiError(400, "parameter password is required")
status = await database(request).pool.execute(
"UPDATE principals SET password_hash=$2 WHERE name=$1",
userid,
hash_secret(str(password)),
)
if status != "UPDATE 1":
raise ApiError(404, "user does not exist")
async def acl_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"""SELECT p.name AS ugid, 'user' AS type, a.role_name AS roleid, a.path, a.propagate
FROM acl_entries a JOIN principals p ON p.id=a.principal_id
UNION ALL
SELECT g.group_id AS ugid, 'group' AS type, a.role_name AS roleid, a.path, a.propagate
FROM group_acl_entries a JOIN identity_groups g ON g.id=a.group_id
ORDER BY path, ugid"""
)
return [
{
"ugid": str(row["ugid"]),
"type": str(row["type"]),
"roleid": str(row["roleid"]),
"path": str(row["path"]),
"propagate": 1 if row["propagate"] else 0,
}
for row in rows
]
async def acl_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
path = str(payload["path"])
roleid = str(payload["roles"])
propagate = bool(int(payload.get("propagate", 1)))
delete = bool(int(payload.get("delete", 0)))
users = [item.strip() for item in str(payload.get("users", "")).split(",") if item.strip()]
groups = [
item.strip() for item in str(payload.get("groups", "")).split(",") if item.strip()
]
tokens = [
item.strip() for item in str(payload.get("tokens", "")).split(",") if item.strip()
]
await database(request).pool.execute(
"""INSERT INTO roles(name) VALUES($1) ON CONFLICT DO NOTHING""",
roleid,
)
for userid in users:
principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1",
userid,
)
if principal_id is None:
raise ApiError(404, f"user {userid} does not exist")
if delete:
await database(request).pool.execute(
"""DELETE FROM acl_entries
WHERE principal_id=$1 AND role_name=$2 AND path=$3""",
principal_id,
roleid,
path,
)
continue
await database(request).pool.execute(
"""INSERT INTO acl_entries(principal_id, role_name, path, propagate)
VALUES($1, $2, $3, $4)
ON CONFLICT (principal_id, role_name, path) DO UPDATE
SET propagate=EXCLUDED.propagate""",
principal_id,
roleid,
path,
propagate,
)
for groupid in groups:
group_id = await database(request).pool.fetchval(
"SELECT id FROM identity_groups WHERE group_id=$1",
groupid,
)
if group_id is None:
raise ApiError(404, f"group {groupid} does not exist")
if delete:
await database(request).pool.execute(
"""DELETE FROM group_acl_entries
WHERE group_id=$1 AND role_name=$2 AND path=$3""",
group_id,
roleid,
path,
)
continue
await database(request).pool.execute(
"""INSERT INTO group_acl_entries(group_id, role_name, path, propagate)
VALUES($1, $2, $3, $4)
ON CONFLICT (group_id, role_name, path) DO UPDATE
SET propagate=EXCLUDED.propagate""",
group_id,
roleid,
path,
propagate,
)
for token_ref in tokens:
# Wire form: userid!tokenid
userid, _, tokenid = token_ref.partition("!")
if not userid or not tokenid:
raise ApiError(400, f"invalid token ACL subject '{token_ref}'")
token_row = await database(request).pool.fetchrow(
"""SELECT t.id FROM api_tokens t
JOIN principals p ON p.id = t.principal_id
WHERE p.name=$1 AND t.token_id=$2""",
userid,
tokenid,
)
if token_row is None:
raise ApiError(404, f"token {token_ref} does not exist")
# Persist as principal ACL under the owning user with token scope in path metadata.
# Keep durable by tagging path; GET ACL still lists principal entries.
scoped_path = f"{path}#token:{tokenid}" if path != "/" else f"/#token:{tokenid}"
principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1",
userid,
)
if delete:
await database(request).pool.execute(
"""DELETE FROM acl_entries
WHERE principal_id=$1 AND role_name=$2 AND path=$3""",
principal_id,
roleid,
scoped_path,
)
else:
await database(request).pool.execute(
"""INSERT INTO acl_entries(principal_id, role_name, path, propagate)
VALUES($1, $2, $3, $4)
ON CONFLICT (principal_id, role_name, path) DO UPDATE
SET propagate=EXCLUDED.propagate""",
principal_id,
roleid,
scoped_path,
propagate,
)
async def token_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
userid = str(values(inputs)["userid"])
_require_owner(request, userid)
rows = await database(request).pool.fetch(
"""SELECT t.token_id, t.comment, t.privilege_separation,
extract(epoch from t.expires_at)::bigint AS expire
FROM api_tokens t JOIN principals p ON p.id=t.principal_id
WHERE p.name=$1 ORDER BY t.token_id""",
userid,
)
return [{"tokenid": str(row["token_id"]), **_token_info(row)} for row in rows]
async def token_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
userid, tokenid = str(payload["userid"]), str(payload["tokenid"])
_require_owner(request, userid)
row = await database(request).pool.fetchrow(
"""SELECT t.comment, t.privilege_separation,
extract(epoch from t.expires_at)::bigint AS expire
FROM api_tokens t JOIN principals p ON p.id=t.principal_id
WHERE p.name=$1 AND t.token_id=$2""",
userid,
tokenid,
)
if row is None:
raise ApiError(404, "API token does not exist")
return _token_info(row)
async def token_create(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
userid, tokenid = str(payload["userid"]), str(payload["tokenid"])
_require_owner(request, userid)
secret = secrets.token_urlsafe(32)
row = await database(request).pool.fetchrow(
"""INSERT INTO api_tokens(
principal_id, token_id, secret_hash, comment, expires_at,
privilege_separation
) SELECT id, $2, $3, $4,
CASE WHEN $5::bigint IS NULL OR $5=0 THEN NULL ELSE to_timestamp($5) END,
$6 FROM principals WHERE name=$1
ON CONFLICT (principal_id, token_id) DO NOTHING
RETURNING comment, privilege_separation,
extract(epoch from expires_at)::bigint AS expire""",
userid,
tokenid,
hash_secret(secret),
payload.get("comment"),
_expire_value(payload),
bool(payload.get("privsep", True)),
)
if row is None:
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM principals WHERE name=$1)", userid
)
raise ApiError(409 if exists else 404, "user or API token conflict")
return {"full-tokenid": f"{userid}!{tokenid}", "info": _token_info(row), "value": secret}
async def token_update(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
provided = frozenset(str(item) for item in inputs.get("provided", payload))
userid, tokenid = str(payload["userid"]), str(payload["tokenid"])
_require_owner(request, userid)
regenerate = bool(payload.get("regenerate", False))
secret = secrets.token_urlsafe(32) if regenerate else None
row = await database(request).pool.fetchrow(
"""UPDATE api_tokens t SET
comment=COALESCE($3::text, comment),
expires_at=CASE WHEN $4::bigint IS NULL THEN expires_at
WHEN $4=0 THEN NULL ELSE to_timestamp($4) END,
privilege_separation=COALESCE($5::boolean, privilege_separation),
secret_hash=COALESCE($6::text, secret_hash), updated_at=now()
FROM principals p WHERE p.id=t.principal_id AND p.name=$1 AND t.token_id=$2
RETURNING t.comment, t.privilege_separation,
extract(epoch from t.expires_at)::bigint AS expire""",
userid,
tokenid,
payload.get("comment") if "comment" in provided else None,
_expire_value(payload) if "expire" in provided else None,
payload.get("privsep") if "privsep" in provided else None,
hash_secret(secret) if secret is not None else None,
)
if row is None:
raise ApiError(404, "API token does not exist")
result = _token_info(row)
if secret is not None:
result.update({"full-tokenid": f"{userid}!{tokenid}", "value": secret})
return result
async def token_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
userid, tokenid = str(payload["userid"]), str(payload["tokenid"])
_require_owner(request, userid)
status = await database(request).pool.execute(
"""DELETE FROM api_tokens t USING principals p
WHERE p.id=t.principal_id AND p.name=$1 AND t.token_id=$2""",
userid,
tokenid,
)
if status != "DELETE 1":
raise ApiError(404, "API token does not exist")
async def role_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"SELECT name, privileges FROM roles ORDER BY name"
)
return [
{"roleid": str(row["name"]), "privs": ",".join(str(item) for item in row["privileges"])}
for row in rows
]
async def role_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
roleid = str(values(inputs)["roleid"])
row = await database(request).pool.fetchrow(
"SELECT name, privileges FROM roles WHERE name=$1",
roleid,
)
if row is None:
raise ApiError(404, "role does not exist")
# Contract returns privilege→boolean map (additionalProperties: 0 on named privs).
return {str(priv): 1 for priv in row["privileges"]}
async def role_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
roleid = str(payload["roleid"])
privs = [item.strip() for item in str(payload.get("privs", "")).split(",") if item.strip()]
await database(request).pool.execute(
"""INSERT INTO roles(name, privileges) VALUES($1, $2)
ON CONFLICT (name) DO UPDATE SET privileges=EXCLUDED.privileges""",
roleid,
privs,
)
async def role_update(request: Request, inputs: dict[str, Any]) -> None:
await role_create(request, inputs)
async def role_delete(request: Request, inputs: dict[str, Any]) -> None:
roleid = str(values(inputs)["roleid"])
status = await database(request).pool.execute(
"DELETE FROM roles WHERE name=$1",
roleid,
)
if status != "DELETE 1":
raise ApiError(404, "role does not exist")
async def domain_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"SELECT name, kind, config FROM realms ORDER BY name"
)
return [_domain_payload(str(row["name"]), str(row["kind"]), row["config"]) for row in rows]
async def domain_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
realm = str(values(inputs)["realm"])
row = await database(request).pool.fetchrow(
"SELECT name, kind, config FROM realms WHERE name=$1",
realm,
)
if row is None:
raise ApiError(404, "realm does not exist")
return _domain_payload(str(row["name"]), str(row["kind"]), row["config"])
async def domain_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
realm = str(payload["realm"])
realm_type = str(payload.get("type") or "")
if realm_type not in _REALM_TYPES:
missing = realm_type or "<missing>"
raise ApiError(400, f"parameter verification failed - type: {missing}")
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM realms WHERE name=$1)",
realm,
)
if exists:
raise ApiError(400, f"realm '{realm}' already exists")
config = _domain_config_from_payload(payload)
if config.get("default"):
await database(request).pool.execute(
"""UPDATE realms
SET config = config - 'default'
WHERE COALESCE((config->>'default')::boolean, false)"""
)
await database(request).pool.execute(
"INSERT INTO realms(name, kind, config) VALUES($1, $2, $3::jsonb)",
realm,
realm_type,
json.dumps(config, sort_keys=True),
)
async def domain_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
realm = str(payload["realm"])
provided = frozenset(str(item) for item in inputs.get("provided", payload))
row = await database(request).pool.fetchrow(
"SELECT name, kind, config FROM realms WHERE name=$1",
realm,
)
if row is None:
raise ApiError(404, "realm does not exist")
if "type" in provided and payload.get("type") is not None:
raise ApiError(400, "realm type cannot be changed")
current = state(row["config"])
delete_raw = str(payload.get("delete") or "")
for key in [item.strip() for item in delete_raw.split(",") if item.strip()]:
current.pop(key, None)
updates = _domain_config_from_payload(payload, provided=provided)
updated = {**current, **updates}
if updates.get("default"):
await database(request).pool.execute(
"""UPDATE realms
SET config = config - 'default'
WHERE name <> $1 AND COALESCE((config->>'default')::boolean, false)""",
realm,
)
await database(request).pool.execute(
"UPDATE realms SET config=$2::jsonb WHERE name=$1",
realm,
json.dumps(updated, sort_keys=True),
)
async def domain_delete(request: Request, inputs: dict[str, Any]) -> None:
realm = str(values(inputs)["realm"])
if realm in _BUILTIN_REALMS:
raise ApiError(400, "builtin authentication server can't be removed")
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM realms WHERE name=$1)",
realm,
)
if not exists:
raise ApiError(404, "realm does not exist")
in_use = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM principals WHERE realm_name=$1)",
realm,
)
if in_use:
raise ApiError(400, f"realm '{realm}' is still in use by users")
await database(request).pool.execute("DELETE FROM realms WHERE name=$1", realm)
async def domain_sync(request: Request, inputs: dict[str, Any]) -> None:
realm = str(values(inputs)["realm"])
payload = values(inputs)
row = await database(request).pool.fetchrow(
"SELECT kind, config FROM realms WHERE name=$1",
realm,
)
if row is None:
raise ApiError(404, "realm does not exist")
if str(row["kind"]) not in {"ldap", "ad"}:
raise ApiError(400, "sync is only supported for ldap/ad realms")
config = state(row["config"])
now = int(await database(request).pool.fetchval("SELECT extract(epoch from now())::bigint"))
config["last_sync"] = now
config["last_sync_options"] = {
key: payload[key]
for key in (
"dry-run",
"enable-new",
"full",
"purge",
"remove-vanished",
"scope",
)
if key in payload
}
await database(request).pool.execute(
"UPDATE realms SET config=$2::jsonb WHERE name=$1",
realm,
json.dumps(config, sort_keys=True),
)
registry.register("/access", "GET", access_index)
registry.register("/access/users", "GET", user_list)
registry.register("/access/users", "POST", user_create)
registry.register("/access/users/{userid}", "GET", user_get)
registry.register("/access/users/{userid}", "PUT", user_update)
registry.register("/access/users/{userid}", "DELETE", user_delete)
registry.register("/access/groups", "GET", group_list)
registry.register("/access/groups", "POST", group_create)
registry.register("/access/groups/{groupid}", "GET", group_get)
registry.register("/access/groups/{groupid}", "PUT", group_update)
registry.register("/access/groups/{groupid}", "DELETE", group_delete)
registry.register("/access/password", "PUT", password_update)
registry.register("/access/acl", "GET", acl_list)
registry.register("/access/acl", "PUT", acl_update)
registry.register("/access/roles", "GET", role_list)
registry.register("/access/roles", "POST", role_create)
registry.register("/access/roles/{roleid}", "GET", role_get)
registry.register("/access/roles/{roleid}", "PUT", role_update)
registry.register("/access/roles/{roleid}", "DELETE", role_delete)
registry.register("/access/domains", "GET", domain_list)
registry.register("/access/domains", "POST", domain_create)
registry.register("/access/domains/{realm}", "GET", domain_get)
registry.register("/access/domains/{realm}", "PUT", domain_update)
registry.register("/access/domains/{realm}", "DELETE", domain_delete)
registry.register("/access/domains/{realm}/sync", "POST", domain_sync)
registry.register("/access/users/{userid}/token", "GET", token_list)
registry.register("/access/users/{userid}/token/{tokenid}", "GET", token_get)
registry.register("/access/users/{userid}/token/{tokenid}", "POST", token_create)
registry.register("/access/users/{userid}/token/{tokenid}", "PUT", token_update)
registry.register("/access/users/{userid}/token/{tokenid}", "DELETE", token_delete)
register_access_auth_handlers(registry)
+381
View File
@@ -0,0 +1,381 @@
"""Access TFA, OpenID, permissions, and ticket helpers with durable state."""
from __future__ import annotations
import json
import secrets
from typing import Any, cast
from urllib.parse import urlencode
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.config import Settings
from app.handlers.common import database, values
from app.security.auth import AuthenticationError, csrf_token, issue_ticket, verify_ticket
_TFA_TYPES = frozenset({"totp", "u2f", "webauthn", "recovery", "yubico"})
def _settings(request: Request) -> Settings:
return cast(Settings, request.app.state.settings)
def _tfa_public(row: Any) -> dict[str, Any]:
created = row["created_at"]
created_ts = int(created.timestamp()) if hasattr(created, "timestamp") else int(created or 0)
return {
"id": str(row["entry_id"]),
"type": str(row["tfa_type"]),
"description": row["description"] or "",
"enable": int(bool(row["enable"])),
"created": created_ts,
}
async def _principal_row(request: Request, userid: str) -> Any:
row = await database(request).pool.fetchrow(
"""SELECT id, name, tfa_locked_until, totp_locked
FROM principals WHERE name=$1""",
userid,
)
if row is None:
raise ApiError(404, "user does not exist")
return row
def register_access_auth_handlers(registry: HandlerRegistry) -> None:
async def ticket_get(_request: Request, _inputs: dict[str, Any]) -> None:
return None
async def permissions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
userid = str(payload.get("userid") or request.state.principal)
path_filter = payload.get("path")
if userid == "root@pam":
caps = {
"/": {
"Permissions.Modify": 1,
"Sys.Audit": 1,
"Sys.Modify": 1,
"VM.Allocate": 1,
"VM.Audit": 1,
"VM.PowerMgmt": 1,
"Datastore.Allocate": 1,
"Datastore.Audit": 1,
}
}
if path_filter:
return {str(path_filter): caps["/"]}
return caps
rows = await database(request).pool.fetch(
"""SELECT a.path, r.privileges
FROM acl_entries a
JOIN principals p ON p.id=a.principal_id
JOIN roles r ON r.name=a.role_name
WHERE p.name=$1
UNION ALL
SELECT a.path, r.privileges
FROM group_acl_entries a
JOIN identity_groups g ON g.id=a.group_id
JOIN identity_group_members m ON m.group_id=g.id
JOIN principals p ON p.id=m.principal_id
JOIN roles r ON r.name=a.role_name
WHERE p.name=$1""",
userid,
)
result: dict[str, dict[str, int]] = {}
for row in rows:
path = str(row["path"])
bucket = result.setdefault(path, {})
for privilege in row["privileges"] or []:
bucket[str(privilege)] = 1
if path_filter:
target = str(path_filter)
merged: dict[str, int] = {}
for path, privs in result.items():
if target == path or target.startswith(path.rstrip("/") + "/") or path == "/":
merged.update(privs)
return {target: merged} if merged else {}
return result
async def vncticket(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
ticket = str(payload["vncticket"])
key = _settings(request).ticket_signing_key.get_secret_value().encode()
try:
claims = verify_ticket(ticket, key)
except AuthenticationError as error:
raise ApiError(401, "authentication failure") from error
authid = str(payload["authid"])
if claims.principal != authid:
raise ApiError(401, "authentication failure")
return None
async def openid_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return [{"subdir": "auth-url"}, {"subdir": "login"}]
async def openid_auth_url(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
realm = str(payload["realm"])
redirect_url = str(payload["redirect-url"])
row = await database(request).pool.fetchrow(
"SELECT name, kind, config FROM realms WHERE name=$1",
realm,
)
if row is None:
raise ApiError(404, "realm does not exist")
if str(row["kind"]) != "openid":
raise ApiError(400, "realm is not an OpenID realm")
state = secrets.token_urlsafe(16)
await database(request).pool.execute(
"""INSERT INTO openid_pending(state, realm, redirect_url)
VALUES($1, $2, $3)
ON CONFLICT (state) DO UPDATE
SET realm=EXCLUDED.realm, redirect_url=EXCLUDED.redirect_url,
created_at=now()""",
state,
realm,
redirect_url,
)
config = row["config"]
if isinstance(config, str):
config = json.loads(config)
config = config or {}
issuer = str(config.get("issuer-url") or "https://openid.example.local")
client_id = str(config.get("client-id") or "pve-simulator")
query = urlencode(
{
"client_id": client_id,
"redirect_uri": redirect_url,
"response_type": "code",
"scope": str(config.get("scopes") or "openid email profile"),
"state": state,
}
)
return f"{issuer.rstrip('/')}/authorize?{query}"
async def openid_login(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
state = str(payload["state"])
pending = await database(request).pool.fetchrow(
"SELECT realm, redirect_url FROM openid_pending WHERE state=$1",
state,
)
if pending is None:
raise ApiError(400, "invalid OpenID state")
redirect = payload.get("redirect-url")
if redirect is not None and str(redirect) != str(pending["redirect_url"]):
raise ApiError(400, "redirect-url mismatch")
realm = str(pending["realm"])
code = str(payload["code"])
username = f"openid-{code[:12]}@{realm}"
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM principals WHERE name=$1)",
username,
)
if not exists:
await database(request).pool.execute(
"""INSERT INTO principals(id, name, password_hash, realm_name)
VALUES(gen_random_uuid(), $1, NULL, $2)""",
username,
realm,
)
await database(request).pool.execute(
"DELETE FROM openid_pending WHERE state=$1",
state,
)
key = _settings(request).ticket_signing_key.get_secret_value().encode()
ticket = issue_ticket(username, key)
return {
"username": username,
"ticket": ticket,
"CSRFPreventionToken": csrf_token(ticket, key),
"clustername": "pve-simulator",
"cap": {},
}
async def tfa_list_all(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(request).pool.fetch(
"""SELECT p.name AS userid, p.tfa_locked_until, p.totp_locked,
t.entry_id, t.tfa_type, t.description, t.enable, t.created_at
FROM principals p
LEFT JOIN tfa_entries t ON t.principal_id=p.id
ORDER BY p.name, t.entry_id"""
)
by_user: dict[str, dict[str, Any]] = {}
for row in rows:
userid = str(row["userid"])
item = by_user.setdefault(
userid,
{
"userid": userid,
"entries": [],
"totp-locked": int(bool(row["totp_locked"])),
},
)
if row["tfa_locked_until"] is not None:
locked = row["tfa_locked_until"]
item["tfa-locked-until"] = (
int(locked.timestamp()) if hasattr(locked, "timestamp") else int(locked)
)
if row["entry_id"] is not None:
item["entries"].append(_tfa_public(row))
return list(by_user.values())
async def tfa_list_user(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
userid = str(values(inputs)["userid"])
principal = await _principal_row(request, userid)
rows = await database(request).pool.fetch(
"""SELECT entry_id, tfa_type, description, enable, created_at
FROM tfa_entries WHERE principal_id=$1 ORDER BY entry_id""",
principal["id"],
)
return [_tfa_public(row) for row in rows]
async def tfa_add(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
userid = payload.get("userid")
if userid in {None, ""}:
raise ApiError(400, "parameter 'userid' is required")
userid = str(userid)
tfa_type = payload.get("type")
if tfa_type in {None, ""}:
raise ApiError(400, "parameter 'type' is required")
tfa_type = str(tfa_type)
if tfa_type not in _TFA_TYPES:
raise ApiError(400, f"invalid TFA type: {tfa_type}")
principal = await _principal_row(request, userid)
entry_id = secrets.token_hex(8)
secret = str(payload.get("value") or payload.get("totp") or secrets.token_hex(20))
description = str(payload.get("description") or tfa_type)
recovery: list[str] = []
metadata: dict[str, Any] = {}
if tfa_type == "recovery":
recovery = [secrets.token_hex(5) for _ in range(8)]
metadata["recovery"] = recovery
await database(request).pool.execute(
"""INSERT INTO tfa_entries(
principal_id, entry_id, tfa_type, description, enable, secret, metadata
) VALUES($1, $2, $3, $4, true, $5, $6::jsonb)""",
principal["id"],
entry_id,
tfa_type,
description,
secret,
json.dumps(metadata, sort_keys=True),
)
result: dict[str, Any] = {"id": entry_id}
if recovery:
result["recovery"] = recovery
if payload.get("challenge") is not None:
result["challenge"] = payload.get("challenge")
return result
async def tfa_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
userid = str(values(inputs)["userid"])
entry_id = str(values(inputs)["id"])
principal = await _principal_row(request, userid)
row = await database(request).pool.fetchrow(
"""SELECT entry_id, tfa_type, description, enable, created_at
FROM tfa_entries WHERE principal_id=$1 AND entry_id=$2""",
principal["id"],
entry_id,
)
if row is None:
raise ApiError(404, "TFA entry does not exist")
return _tfa_public(row)
async def tfa_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
userid = payload.get("userid")
if userid in {None, ""}:
raise ApiError(400, "parameter 'userid' is required")
userid = str(userid)
entry_id = payload.get("id")
if entry_id in {None, ""}:
raise ApiError(400, "parameter 'id' is required")
entry_id = str(entry_id)
provided = frozenset(str(item) for item in inputs.get("provided", payload))
principal = await _principal_row(request, userid)
row = await database(request).pool.fetchrow(
"SELECT entry_id FROM tfa_entries WHERE principal_id=$1 AND entry_id=$2",
principal["id"],
entry_id,
)
if row is None:
raise ApiError(404, "TFA entry does not exist")
if "description" in provided:
await database(request).pool.execute(
"""UPDATE tfa_entries SET description=$3
WHERE principal_id=$1 AND entry_id=$2""",
principal["id"],
entry_id,
payload.get("description"),
)
if "enable" in provided:
enabled = payload.get("enable")
if isinstance(enabled, bool):
value = enabled
else:
value = str(enabled).lower() in {"1", "true", "yes", "on"}
await database(request).pool.execute(
"""UPDATE tfa_entries SET enable=$3
WHERE principal_id=$1 AND entry_id=$2""",
principal["id"],
entry_id,
value,
)
async def tfa_delete(request: Request, inputs: dict[str, Any]) -> None:
userid = str(values(inputs)["userid"])
entry_id = str(values(inputs)["id"])
principal = await _principal_row(request, userid)
status = await database(request).pool.execute(
"DELETE FROM tfa_entries WHERE principal_id=$1 AND entry_id=$2",
principal["id"],
entry_id,
)
if status != "DELETE 1":
raise ApiError(404, "TFA entry does not exist")
async def user_tfa_types(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
userid = str(values(inputs)["userid"])
principal = await _principal_row(request, userid)
rows = await database(request).pool.fetch(
"""SELECT DISTINCT tfa_type FROM tfa_entries
WHERE principal_id=$1 AND enable=true ORDER BY tfa_type""",
principal["id"],
)
types = [str(row["tfa_type"]) for row in rows]
realm = userid.split("@", 1)[1] if "@" in userid else "pam"
return {"user": types, "types": types, "realm": realm}
async def unlock_tfa(request: Request, inputs: dict[str, Any]) -> bool:
userid = str(values(inputs)["userid"])
status = await database(request).pool.execute(
"""UPDATE principals
SET tfa_locked_until=NULL, totp_locked=false
WHERE name=$1""",
userid,
)
if status != "UPDATE 1":
raise ApiError(404, "user does not exist")
return True
registry.register("/access/ticket", "GET", ticket_get)
registry.register("/access/permissions", "GET", permissions)
registry.register("/access/vncticket", "POST", vncticket)
registry.register("/access/openid", "GET", openid_index)
registry.register("/access/openid/auth-url", "POST", openid_auth_url)
registry.register("/access/openid/login", "POST", openid_login)
registry.register("/access/tfa", "GET", tfa_list_all)
registry.register("/access/tfa/{userid}", "GET", tfa_list_user)
registry.register("/access/tfa/{userid}", "POST", tfa_add)
registry.register("/access/tfa/{userid}/{id}", "GET", tfa_get)
registry.register("/access/tfa/{userid}/{id}", "PUT", tfa_update)
registry.register("/access/tfa/{userid}/{id}", "DELETE", tfa_delete)
registry.register("/access/users/{userid}/tfa", "GET", user_tfa_types)
registry.register("/access/users/{userid}/unlock-tfa", "PUT", unlock_tfa)
+232
View File
@@ -0,0 +1,232 @@
"""Cluster ACME accounts and DNS plugins."""
from __future__ import annotations
import hashlib
import json
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs, values
def _acme(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.get("acme")
if not isinstance(current, dict):
current = {}
metadata["acme"] = current
if not isinstance(current.get("accounts"), dict):
current["accounts"] = {}
if not isinstance(current.get("plugins"), dict):
current["plugins"] = {}
if not isinstance(current.get("meta"), dict):
current["meta"] = {}
if not isinstance(current.get("directories"), list):
current["directories"] = []
if not isinstance(current.get("challenge_schema"), list):
current["challenge_schema"] = []
return current
def _default_directory(acme: dict[str, Any]) -> str:
directories = acme.get("directories")
if isinstance(directories, list):
for item in directories:
if isinstance(item, dict) and item.get("url"):
return str(item["url"])
return "https://acme-v02.api.letsencrypt.org/directory"
def register_acme_handlers(registry: HandlerRegistry) -> None:
async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs(
"account",
"challenge-schema",
"directories",
"meta",
"plugins",
"tos",
)
async def account_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
accounts = _acme(metadata)["accounts"]
return [
{
"name": name,
"contact": item.get("contact", []),
"directory": item.get("directory"),
}
for name, item in sorted(accounts.items())
]
async def account_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
name = str(payload.get("name") or "default")
metadata = await cluster_metadata(request)
acme = _acme(metadata)
accounts = acme["accounts"]
if name in accounts:
raise ApiError(400, f"ACME account '{name}' already exists")
directory = str(payload.get("directory") or _default_directory(acme))
accounts[name] = {
"name": name,
"contact": payload.get("contact"),
"directory": directory,
"tos_url": payload.get("tos_url"),
"eab-kid": payload.get("eab-kid"),
"eab-hmac-key": payload.get("eab-hmac-key"),
"location": f"https://acme.example.local/acct/{name}",
}
await save_cluster_metadata(request, metadata)
return name
async def account_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
account = _acme(metadata)["accounts"].get(name)
if not isinstance(account, dict):
raise ApiError(404, "ACME account does not exist")
return {
"account": {
"name": name,
"contact": account.get("contact"),
},
"directory": account.get("directory"),
"tos": account.get("tos_url"),
"location": account.get("location"),
}
async def account_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
accounts = _acme(metadata)["accounts"]
if name not in accounts:
raise ApiError(404, "ACME account does not exist")
if "contact" in payload:
accounts[name]["contact"] = payload["contact"]
await save_cluster_metadata(request, metadata)
async def account_delete(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
accounts = _acme(metadata)["accounts"]
if name not in accounts:
raise ApiError(404, "ACME account does not exist")
del accounts[name]
await save_cluster_metadata(request, metadata)
async def plugins_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
plugins = _acme(metadata)["plugins"]
plugin_type = values(inputs).get("type")
result = []
for plugin_id, item in sorted(plugins.items()):
if plugin_type and item.get("type") != plugin_type:
continue
entry = {"plugin": plugin_id, **{k: v for k, v in item.items() if k != "data"}}
if "digest" not in entry:
material = {k: v for k, v in entry.items() if k != "digest"}
entry["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(material, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
result.append(entry)
return result
async def plugins_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
plugin_id = str(payload["id"])
metadata = await cluster_metadata(request)
plugins = _acme(metadata)["plugins"]
if plugin_id in plugins:
raise ApiError(400, f"ACME plugin '{plugin_id}' already exists")
plugins[plugin_id] = {
key: value for key, value in payload.items() if key not in {"delete", "digest"}
}
await save_cluster_metadata(request, metadata)
async def plugins_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
plugin_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
plugin = _acme(metadata)["plugins"].get(plugin_id)
if not isinstance(plugin, dict):
raise ApiError(404, "ACME plugin does not exist")
return {"id": plugin_id, **plugin}
async def plugins_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
plugin_id = str(payload["id"])
metadata = await cluster_metadata(request)
plugins = _acme(metadata)["plugins"]
if plugin_id not in plugins:
raise ApiError(404, "ACME plugin does not exist")
current = dict(plugins[plugin_id])
for key in [
item.strip() for item in str(payload.get("delete") or "").split(",") if item.strip()
]:
current.pop(key, None)
for key, value in payload.items():
if key in {"id", "delete", "digest"}:
continue
current[key] = value
current["id"] = plugin_id
plugins[plugin_id] = current
await save_cluster_metadata(request, metadata)
async def plugins_delete(request: Request, inputs: dict[str, Any]) -> None:
plugin_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
plugins = _acme(metadata)["plugins"]
if plugin_id not in plugins:
raise ApiError(404, "ACME plugin does not exist")
del plugins[plugin_id]
await save_cluster_metadata(request, metadata)
async def directories(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
directories = _acme(metadata).get("directories")
if isinstance(directories, list):
return [dict(item) for item in directories if isinstance(item, dict)]
return []
async def challenge_schema(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
schema = _acme(metadata).get("challenge_schema")
if isinstance(schema, list):
return [dict(item) for item in schema if isinstance(item, dict)]
return []
async def meta(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
acme = _acme(metadata)
directory = str(values(inputs).get("directory") or _default_directory(acme))
meta_raw = acme.get("meta")
meta_store: dict[str, Any] = dict(meta_raw) if isinstance(meta_raw, dict) else {}
payload = meta_store.get(directory)
if not isinstance(payload, dict):
return {}
return dict(payload)
async def tos(request: Request, inputs: dict[str, Any]) -> str:
result = await meta(request, inputs)
return str(result.get("termsOfService") or "")
registry.register("/cluster/acme", "GET", index)
registry.register("/cluster/acme/account", "GET", account_list)
registry.register("/cluster/acme/account", "POST", account_create)
registry.register("/cluster/acme/account/{name}", "GET", account_get)
registry.register("/cluster/acme/account/{name}", "PUT", account_update)
registry.register("/cluster/acme/account/{name}", "DELETE", account_delete)
registry.register("/cluster/acme/plugins", "GET", plugins_list)
registry.register("/cluster/acme/plugins", "POST", plugins_create)
registry.register("/cluster/acme/plugins/{id}", "GET", plugins_get)
registry.register("/cluster/acme/plugins/{id}", "PUT", plugins_update)
registry.register("/cluster/acme/plugins/{id}", "DELETE", plugins_delete)
registry.register("/cluster/acme/directories", "GET", directories)
registry.register("/cluster/acme/challenge-schema", "GET", challenge_schema)
registry.register("/cluster/acme/meta", "GET", meta)
registry.register("/cluster/acme/tos", "GET", tos)
+291
View File
@@ -0,0 +1,291 @@
"""Cluster backup jobs and node vzdump handlers."""
from __future__ import annotations
import secrets
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.db.primitives import ConflictError
from app.handlers.common import (
cluster_metadata,
database,
require_node,
save_cluster_metadata,
state,
subdirs,
values,
)
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
def _backup_jobs(metadata: dict[str, Any]) -> dict[str, dict[str, Any]]:
jobs = metadata.get("backup_jobs")
if not isinstance(jobs, dict):
return {}
return {
str(job_id): dict(payload) for job_id, payload in jobs.items() if isinstance(payload, dict)
}
def _job_view(job_id: str, payload: dict[str, Any]) -> dict[str, Any]:
view = dict(payload)
view["id"] = job_id
enabled = view.get("enabled", 1)
view["enabled"] = bool(int(enabled)) if not isinstance(enabled, bool) else enabled
return view
def register_backup_handlers(registry: HandlerRegistry) -> None:
async def backup_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
return [_job_view(job_id, payload) for job_id, payload in sorted(jobs.items())]
async def backup_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
payload = jobs.get(job_id)
if payload is None:
raise ApiError(404, "backup job does not exist")
return _job_view(job_id, payload)
async def backup_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
job_id = str(payload.get("id") or f"backup-{secrets.token_hex(3)}")
if job_id in jobs:
raise ApiError(400, f"backup job '{job_id}' already exists")
entry = {
key: value
for key, value in payload.items()
if key not in {"delete", "digest", "node", "target"}
}
entry["id"] = job_id
entry.setdefault("enabled", 1)
entry.setdefault("storage", payload.get("storage") or "local")
jobs[job_id] = entry
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_update(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
payload = values(inputs)
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
current = jobs.get(job_id)
if current is None:
raise ApiError(404, "backup job does not exist")
updated = dict(current)
for key, value in payload.items():
if key in {"id", "delete", "digest"}:
continue
updated[key] = value
updated["id"] = job_id
jobs[job_id] = updated
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_delete(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
if job_id not in jobs:
raise ApiError(404, "backup job does not exist")
del jobs[job_id]
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_info(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("not-backed-up")
async def backupinfo_stub(_request: Request, _inputs: dict[str, Any]) -> str:
# PVE 6.x `/cluster/backupinfo` returns a stub string; 7+ uses `/backup-info` index.
return "Please use the 'not-backed-up' API"
async def backup_not_backed_up(
request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
covered: set[str] = set()
for payload in jobs.values():
if int(payload.get("all") or 0):
covered.add("*")
for item in str(payload.get("vmid") or "").split(","):
item = item.strip()
if item:
covered.add(item)
rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.kind, r.state FROM resources r
WHERE r.kind IN ('qemu', 'lxc')
ORDER BY r.external_id::integer"""
)
result: list[dict[str, Any]] = []
if "*" in covered:
return result
for row in rows:
vmid = str(row["external_id"])
if vmid in covered:
continue
guest_state = state(row["state"])
result.append(
{
"vmid": int(vmid),
"type": "qemu" if row["kind"] == "qemu" else "lxc",
"name": str(guest_state.get("name") or f"{row['kind']}-{vmid}"),
}
)
return result
async def backup_included_volumes(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
job = jobs.get(job_id)
if job is None:
raise ApiError(404, "backup job does not exist")
vmids = [item.strip() for item in str(job.get("vmid") or "").split(",") if item.strip()]
guest_select = """SELECT r.external_id, r.kind, r.state,
v.config AS qemu_config, c.config AS lxc_config
FROM resources r
LEFT JOIN virtual_machines v ON v.resource_id = r.id
LEFT JOIN containers c ON c.resource_id = r.id"""
if int(job.get("all") or 0):
rows = await database(request).pool.fetch(
f"""{guest_select}
WHERE r.kind IN ('qemu', 'lxc')
ORDER BY r.external_id::integer"""
)
else:
rows = await database(request).pool.fetch(
f"""{guest_select}
WHERE r.kind IN ('qemu', 'lxc') AND r.external_id = ANY($1::text[])
ORDER BY r.external_id::integer""",
vmids,
)
children: list[dict[str, Any]] = []
for row in rows:
guest_state = state(row["state"])
config_raw = row["qemu_config"] if row["kind"] == "qemu" else row["lxc_config"]
config = state(config_raw) if config_raw is not None else {}
volumes: list[dict[str, Any]] = []
for key, value in sorted(config.items()):
if not (
key.startswith("scsi")
or key.startswith("virtio")
or key.startswith("sata")
or key.startswith("ide")
or key in {"rootfs", "mp0", "mp1", "mp2", "mp3"}
):
continue
volumes.append(
{
"id": key,
"name": str(value).split(",", 1)[0],
"included": True,
"reason": "included by default",
}
)
children.append(
{
"id": int(row["external_id"]),
"name": str(guest_state.get("name") or f"{row['kind']}-{row['external_id']}"),
"type": "qemu" if row["kind"] == "qemu" else "lxc",
"children": volumes,
}
)
return {"children": children}
async def vzdump_defaults(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
from app.handlers.nodes import load_node_ops
node = str(values(inputs)["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
vzdump = ops.get("vzdump")
defaults = vzdump.get("defaults") if isinstance(vzdump, dict) else None
return dict(defaults) if isinstance(defaults, dict) else {}
async def vzdump_extractconfig(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import load_node_ops
payload = values(inputs)
node = str(payload["node"])
volid = str(payload.get("volume") or payload.get("volid") or "")
if not volid:
raise ApiError(400, "volume parameter required")
await require_node(request, node)
ops = await load_node_ops(request, node)
vzdump = ops.get("vzdump")
configs = vzdump.get("extractconfig") if isinstance(vzdump, dict) else None
if not isinstance(configs, dict):
return ""
return str(configs.get(volid) or "")
async def vzdump_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
vmids = payload.get("vmid")
selected = None
if vmids is not None:
selected = [str(item) for item in str(vmids).split(",") if item.strip()]
return await _schedule_vzdump(request, node=node, vmids=selected, payload=payload)
registry.register("/cluster/backup", "GET", backup_list)
registry.register("/cluster/backup", "POST", backup_create)
registry.register("/cluster/backup-info", "GET", backup_info)
registry.register("/cluster/backupinfo", "GET", backupinfo_stub)
registry.register("/cluster/backup-info/not-backed-up", "GET", backup_not_backed_up)
registry.register("/cluster/backup/{id}", "GET", backup_get)
registry.register("/cluster/backup/{id}", "PUT", backup_update)
registry.register("/cluster/backup/{id}", "DELETE", backup_delete)
registry.register("/cluster/backup/{id}/included_volumes", "GET", backup_included_volumes)
registry.register("/nodes/{node}/vzdump", "POST", vzdump_create)
registry.register("/nodes/{node}/vzdump/defaults", "GET", vzdump_defaults)
registry.register("/nodes/{node}/vzdump/extractconfig", "GET", vzdump_extractconfig)
async def _schedule_vzdump(
request: Request,
*,
node: str,
vmids: list[str] | None,
payload: dict[str, Any],
) -> str:
pool = database(request).pool
if vmids is None:
rows = await pool.fetch(
"""SELECT external_id FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='qemu' ORDER BY external_id::integer LIMIT 100""",
node,
)
vmids = [str(row["external_id"]) for row in rows]
if not vmids:
raise ApiError(400, "no virtual machines selected for backup")
vmid = vmids[0]
upid = str(Upid.allocate(node, "vzdump", vmid, str(request.state.principal)))
try:
task = await TaskRepository(pool).create(
upid=upid,
task_type="vzdump",
payload={
"node": node,
"vmids": vmids,
"storage": str(payload.get("storage") or "local"),
"mode": str(payload.get("mode") or "snapshot"),
"compress": str(payload.get("compress") or "zstd"),
},
resource_key=f"backup:{node}:{secrets.token_hex(4)}",
idempotency_key=request.headers.get("Idempotency-Key"),
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
+830
View File
@@ -0,0 +1,830 @@
"""Ceph semantic handlers with durable cluster/node state."""
from __future__ import annotations
import json
import secrets
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import (
database,
node_metadata,
require_node,
save_node_metadata,
state,
subdirs,
values,
)
from app.simulation.seed import CLUSTER_ID
async def _load_cluster_ceph(request: Request) -> dict[str, Any]:
row = await database(request).pool.fetchrow(
"SELECT metadata FROM clusters WHERE id=$1",
CLUSTER_ID,
)
metadata = state(row["metadata"]) if row is not None else {}
ceph = metadata.get("ceph")
if not isinstance(ceph, dict):
return {}
return dict(ceph)
async def _save_cluster_ceph(request: Request, ceph: dict[str, Any]) -> None:
await database(request).pool.execute(
"""UPDATE clusters SET metadata = jsonb_set(
COALESCE(metadata, '{}'::jsonb), '{ceph}', $2::jsonb, true
), updated_at=now() WHERE id=$1""",
CLUSTER_ID,
json.dumps(ceph, sort_keys=True),
)
async def _load_node_ceph(request: Request, node: str) -> dict[str, Any]:
metadata = await node_metadata(request, node)
ops = metadata.get("ops")
if not isinstance(ops, dict):
return {"mds": {}, "mgr": {}, "mon": {}, "log": []}
ceph = ops.get("ceph")
if not isinstance(ceph, dict):
return {"mds": {}, "mgr": {}, "mon": {}, "log": []}
return {
"mds": ceph.get("mds") if isinstance(ceph.get("mds"), dict) else {},
"mgr": ceph.get("mgr") if isinstance(ceph.get("mgr"), dict) else {},
"mon": ceph.get("mon") if isinstance(ceph.get("mon"), dict) else {},
"log": list(ceph.get("log") or []) if isinstance(ceph.get("log"), list) else [],
**{key: value for key, value in ceph.items() if key not in {"mds", "mgr", "mon", "log"}},
}
async def _save_node_ceph(request: Request, node: str, ceph: dict[str, Any]) -> None:
metadata = await node_metadata(request, node)
ops = metadata.get("ops")
if not isinstance(ops, dict):
ops = {}
metadata["ops"] = ops
ops["ceph"] = ceph
await save_node_metadata(request, node, metadata)
def _upid(node: str, kind: str) -> str:
return f"UPID:{node}:{secrets.token_hex(4)}:{kind}:root@pam:"
async def _osd_row(request: Request, node: str, osdid: str) -> Any:
row = await database(request).pool.fetchrow(
"""SELECT r.id, r.external_id, r.state
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='ceph-osd'
AND (r.external_id=$2 OR r.external_id=$3)""",
node,
osdid,
f"osd.{osdid}",
)
if row is None:
raise ApiError(404, "OSD does not exist")
return row
def register_ceph_handlers(registry: HandlerRegistry) -> None:
async def ceph_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
node = str(values(inputs)["node"])
await require_node(request, node)
return subdirs(
"cfg",
"cmd-safety",
"crush",
"fs",
"init",
"log",
"mds",
"mgr",
"mon",
"osd",
"pool",
"restart",
"rules",
"start",
"status",
"stop",
)
async def cfg_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"]))
return subdirs("db", "raw", "value")
async def cfg_db(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
ceph = await _load_cluster_ceph(request)
return list(ceph.get("cfg_db") or [])
async def cfg_raw(request: Request, inputs: dict[str, Any]) -> str:
await require_node(request, str(values(inputs)["node"]))
ceph = await _load_cluster_ceph(request)
return str(ceph.get("cfg_raw") or "")
async def cfg_value(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
ceph = await _load_cluster_ceph(request)
keys = [item.strip() for item in str(payload.get("config-keys") or "").split(",") if item]
stored = ceph.get("cfg_values")
if not isinstance(stored, dict):
stored = {}
if keys:
return {key: stored.get(key, "") for key in keys}
return dict(stored)
async def crush(request: Request, inputs: dict[str, Any]) -> str:
await require_node(request, str(values(inputs)["node"]))
ceph = await _load_cluster_ceph(request)
return str(ceph.get("crush") or "")
async def rules(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
ceph = await _load_cluster_ceph(request)
return list(ceph.get("rules") or [])
async def log(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
ceph = await _load_node_ceph(request, node)
entries = list(ceph.get("log") or [])
start = int(values(inputs).get("start") or 0)
limit = int(values(inputs).get("limit") or 50)
return entries[start : start + limit]
async def cmd_safety(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
ceph = await _load_cluster_ceph(request)
safety = ceph.get("cmd_safety")
result = dict(safety) if isinstance(safety, dict) else {}
for key in ("action", "service", "id"):
if key in payload:
result[key] = payload[key]
return result
async def init(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
ceph = await _load_cluster_ceph(request)
config = ceph.setdefault("config", {})
for key in (
"network",
"cluster-network",
"size",
"min_size",
"pg_bits",
"disable_cephx",
):
if key in payload:
config[key] = payload[key]
ceph["initialized"] = True
await _save_cluster_ceph(request, ceph)
node_ceph = await _load_node_ceph(request, node)
node_ceph.setdefault("mon", {})[node] = {
"name": node,
"addr": f"{node}.local:6789",
"rank": 0,
}
await _save_node_ceph(request, node, node_ceph)
async def status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
await require_node(request, node)
return await cluster_ceph_status(request, inputs)
async def start(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
ceph = await _load_cluster_ceph(request)
ceph["running"] = True
ceph["last_service_action"] = {
"action": "start",
"service": payload.get("service"),
"node": node,
}
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephstart")
async def stop(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
ceph = await _load_cluster_ceph(request)
ceph["running"] = False
ceph["last_service_action"] = {
"action": "stop",
"service": payload.get("service"),
"node": node,
}
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephstop")
async def restart(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
ceph = await _load_cluster_ceph(request)
ceph["running"] = True
ceph["last_service_action"] = {
"action": "restart",
"service": payload.get("service"),
"node": node,
}
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephrestart")
def _wire_ceph_pool(name: str, item: dict[str, Any], *, index: int) -> dict[str, Any]:
pool_id = item.get("pool_id", item.get("pool"))
try:
pool_num = int(pool_id) if pool_id is not None else index
except (TypeError, ValueError):
pool_num = index
crush_raw = item.get("crush_rule", 0)
try:
crush_id = int(crush_raw)
crush_name = str(item.get("crush_rule_name") or "replicated_rule")
except (TypeError, ValueError):
crush_name = str(crush_raw or "replicated_rule")
crush_id = 0
application = str(item.get("application") or "rbd")
return {
"pool": pool_num,
"pool_name": str(item.get("pool_name") or name),
"size": int(item.get("size") or 3),
"min_size": int(item.get("min_size") or 2),
"pg_num": int(item.get("pg_num") or 128),
"crush_rule": crush_id,
"crush_rule_name": crush_name,
"application_metadata": item.get("application_metadata")
if isinstance(item.get("application_metadata"), dict)
else {application: {}},
"bytes_used": int(item.get("bytes_used") or 0),
"percent_used": float(item.get("percent_used") or 0.0),
"healthy": bool(item.get("healthy", True)),
}
async def pool_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
ceph = await _load_cluster_ceph(request)
pools = ceph.get("pools") or {}
return [
_wire_ceph_pool(name, item, index=index)
for index, (name, item) in enumerate(sorted(pools.items()))
if isinstance(item, dict)
]
async def pool_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
name = str(payload["name"])
ceph = await _load_cluster_ceph(request)
pools = ceph.setdefault("pools", {})
if name in pools:
raise ApiError(400, f"pool '{name}' already exists")
pools[name] = {
"pool": name,
"size": int(payload.get("size") or 3),
"min_size": int(payload.get("min_size") or 2),
"pg_num": int(payload.get("pg_num") or 128),
"application": str(payload.get("application") or "rbd"),
"crush_rule": str(payload.get("crush_rule") or "replicated_rule"),
"pg_autoscale_mode": payload.get("pg_autoscale_mode", "warn"),
"target_size": payload.get("target_size"),
}
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephcreatepool")
async def pool_get(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
name = str(payload["name"])
ceph = await _load_cluster_ceph(request)
pool = (ceph.get("pools") or {}).get(name)
if not isinstance(pool, dict):
raise ApiError(404, "pool does not exist")
# Reuse list wire shape for detail GETs.
names = sorted((ceph.get("pools") or {}).keys())
index = names.index(name) if name in names else 0
return [_wire_ceph_pool(name, pool, index=index)]
async def pool_update(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
name = str(payload["name"])
ceph = await _load_cluster_ceph(request)
pools = ceph.setdefault("pools", {})
if name not in pools or not isinstance(pools[name], dict):
raise ApiError(404, "pool does not exist")
current = dict(pools[name])
for key in (
"application",
"crush_rule",
"min_size",
"pg_autoscale_mode",
"pg_num",
"pg_num_min",
"size",
"target_size",
"target_size_ratio",
):
if key in payload:
current[key] = payload[key]
pools[name] = current
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephsetpool")
async def pool_delete(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
name = str(payload["name"])
ceph = await _load_cluster_ceph(request)
pools = ceph.setdefault("pools", {})
if name not in pools:
raise ApiError(404, "pool does not exist")
del pools[name]
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephdestroypool")
async def pool_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
name = str(payload["name"])
ceph = await _load_cluster_ceph(request)
pool = (ceph.get("pools") or {}).get(name)
if not isinstance(pool, dict):
raise ApiError(404, "pool does not exist")
return dict(pool)
async def fs_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
ceph = await _load_cluster_ceph(request)
return [dict(item) for _, item in sorted((ceph.get("fs") or {}).items())]
async def fs_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
name = str(payload["name"])
ceph = await _load_cluster_ceph(request)
filesystems = ceph.setdefault("fs", {})
if name in filesystems:
raise ApiError(400, f"fs '{name}' already exists")
filesystems[name] = {
"name": name,
"metadata": f"{name}_meta",
"data": f"{name}_data",
"pg_num": int(payload.get("pg_num") or 32),
}
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephcreatefs")
async def fs_delete(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
name = str(payload["name"])
ceph = await _load_cluster_ceph(request)
filesystems = ceph.setdefault("fs", {})
if name not in filesystems:
raise ApiError(404, "fs does not exist")
del filesystems[name]
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephdestroyfs")
async def mds_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
ceph = await _load_node_ceph(request, node)
return [
{"name": name, **data}
for name, data in sorted((ceph.get("mds") or {}).items())
if isinstance(data, dict)
]
async def mds_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
name = str(payload["name"])
ceph = await _load_node_ceph(request, node)
mds = ceph.setdefault("mds", {})
if name in mds:
raise ApiError(400, f"mds '{name}' already exists")
mds[name] = {
"name": name,
"state": "up:active",
"hotstandby": int(bool(payload.get("hotstandby"))),
}
await _save_node_ceph(request, node, ceph)
return _upid(node, "cephcreatemds")
async def mds_delete(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
name = str(payload["name"])
ceph = await _load_node_ceph(request, node)
mds = ceph.setdefault("mds", {})
if name not in mds:
raise ApiError(404, "mds does not exist")
del mds[name]
await _save_node_ceph(request, node, ceph)
return _upid(node, "cephdestroymds")
async def mgr_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
ceph = await _load_node_ceph(request, node)
return [
{"name": name, **data}
for name, data in sorted((ceph.get("mgr") or {}).items())
if isinstance(data, dict)
]
async def mgr_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
mgr_id = str(payload["id"])
ceph = await _load_node_ceph(request, node)
mgr = ceph.setdefault("mgr", {})
if mgr_id in mgr:
raise ApiError(400, f"mgr '{mgr_id}' already exists")
mgr[mgr_id] = {"name": mgr_id, "state": "active"}
await _save_node_ceph(request, node, ceph)
return _upid(node, "cephcreatemgr")
async def mgr_delete(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
mgr_id = str(payload["id"])
ceph = await _load_node_ceph(request, node)
mgr = ceph.setdefault("mgr", {})
if mgr_id not in mgr:
raise ApiError(404, "mgr does not exist")
del mgr[mgr_id]
await _save_node_ceph(request, node, ceph)
return _upid(node, "cephdestroymgr")
async def mon_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
ceph = await _load_node_ceph(request, node)
return [
{"name": name, **data}
for name, data in sorted((ceph.get("mon") or {}).items())
if isinstance(data, dict)
]
async def mon_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
monid = str(payload["monid"])
ceph = await _load_node_ceph(request, node)
mons = ceph.setdefault("mon", {})
if monid in mons:
raise ApiError(400, f"mon '{monid}' already exists")
mons[monid] = {
"name": monid,
"addr": str(payload.get("mon-address") or f"{node}.local:6789"),
"rank": len(mons),
}
await _save_node_ceph(request, node, ceph)
return _upid(node, "cephcreatemon")
async def mon_delete(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
monid = str(payload["monid"])
ceph = await _load_node_ceph(request, node)
mons = ceph.setdefault("mon", {})
if monid not in mons:
raise ApiError(404, "mon does not exist")
del mons[monid]
await _save_node_ceph(request, node, ceph)
return _upid(node, "cephdestroymon")
async def osd_list(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
await require_node(request, node)
rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind='ceph-osd'
ORDER BY n.name, r.external_id"""
)
hosts: dict[str, dict[str, Any]] = {}
host_id = -2
for row in rows:
host_name = str(row["node"])
host = hosts.get(host_name)
if host is None:
host = {
"id": host_id,
"name": host_name,
"type": "host",
"type_id": 1,
"children": [],
}
hosts[host_name] = host
host_id -= 1
payload = state(row["state"])
osd_raw = payload.get("osd_id", row["external_id"])
try:
osd_id = int(osd_raw)
except (TypeError, ValueError):
digits = "".join(ch for ch in str(osd_raw) if ch.isdigit())
osd_id = int(digits) if digits else 0
host["children"].append(
{
"id": osd_id,
"name": f"osd.{osd_id}",
"type": "osd",
"type_id": 0,
"status": payload.get("status", "up"),
"in": 1 if payload.get("in", True) else 0,
"weight": float(payload.get("weight", 1.0)),
"device_class": payload.get("device_class", "hdd"),
"host": host_name,
}
)
root_children = list(hosts.values())
# Prefer the requested node first in the tree for local dumps.
root_children.sort(key=lambda item: (0 if item["name"] == node else 1, item["name"]))
return {
"flags": "",
"root": {
"id": -1,
"name": "default",
"type": "root",
"type_id": 10,
"children": root_children,
},
}
async def osd_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
next_id = await database(request).pool.fetchval(
"""SELECT COALESCE(
MAX(NULLIF(regexp_replace(external_id, '\\D', '', 'g'), '')::int),
-1
) + 1
FROM resources
WHERE kind='ceph-osd'""",
)
osd_id = int(next_id or 0)
external_id = f"osd.{osd_id}"
node_id = await database(request).pool.fetchval(
"SELECT id FROM nodes WHERE name=$1",
node,
)
osd_state = {
"osd_id": osd_id,
"status": "up",
"in": True,
"weight": 1.0,
"device_class": payload.get("crush-device-class") or "hdd",
"dev": payload.get("dev"),
"size_bytes": 0,
"used_bytes": 0,
}
await database(request).pool.execute(
"""INSERT INTO resources(id, node_id, cluster_id, kind, external_id, state)
VALUES(
gen_random_uuid(),
$1,
(SELECT cluster_id FROM nodes WHERE id=$1),
'ceph-osd',
$2,
$3::jsonb
)""",
node_id,
external_id,
json.dumps(osd_state, sort_keys=True),
)
return _upid(node, "cephcreateosd")
async def osd_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
osdid = str(values(inputs)["osdid"])
await require_node(request, node)
row = await _osd_row(request, node, osdid)
payload = state(row["state"])
return {
"osd": int(osdid) if osdid.isdigit() else osdid,
"status": payload.get("status", "up"),
"in": 1 if payload.get("in", True) else 0,
"weight": payload.get("weight", 1.0),
"size": payload.get("size_bytes", 0),
"used": payload.get("used_bytes", 0),
"device_class": payload.get("device_class", "hdd"),
}
async def osd_delete(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
osdid = str(payload["osdid"])
await require_node(request, node)
row = await _osd_row(request, node, osdid)
await database(request).pool.execute("DELETE FROM resources WHERE id=$1", row["id"])
return _upid(node, "cephdestroyosd")
async def osd_in(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
node = str(payload["node"])
osdid = str(payload["osdid"])
await require_node(request, node)
row = await _osd_row(request, node, osdid)
current = state(row["state"])
current["in"] = True
await database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb, version=version+1 WHERE id=$1",
row["id"],
json.dumps(current, sort_keys=True),
)
async def osd_out(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
node = str(payload["node"])
osdid = str(payload["osdid"])
await require_node(request, node)
row = await _osd_row(request, node, osdid)
current = state(row["state"])
current["in"] = False
await database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb, version=version+1 WHERE id=$1",
row["id"],
json.dumps(current, sort_keys=True),
)
async def osd_scrub(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
node = str(payload["node"])
osdid = str(payload["osdid"])
await require_node(request, node)
row = await _osd_row(request, node, osdid)
current = state(row["state"])
current["last_scrub"] = {
"deep": int(bool(payload.get("deep"))),
"token": secrets.token_hex(4),
}
await database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb, version=version+1 WHERE id=$1",
row["id"],
json.dumps(current, sort_keys=True),
)
async def osd_lv_info(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
node = str(payload["node"])
osdid = str(payload["osdid"])
await require_node(request, node)
row = await _osd_row(request, node, osdid)
current = state(row["state"])
return {
"lv_name": f"osd-block-{osdid}",
"lv_path": f"/dev/ceph/{osdid}",
"lv_size": current.get("size_bytes", 0),
"type": payload.get("type") or "block",
}
async def osd_metadata(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
node = str(payload["node"])
osdid = str(payload["osdid"])
await require_node(request, node)
row = await _osd_row(request, node, osdid)
current = state(row["state"])
metadata = current.get("metadata")
if isinstance(metadata, dict):
return dict(metadata)
return {
"osd": {
"id": int(osdid) if osdid.isdigit() else osdid,
"uuid": current.get("uuid", ""),
"device_class": current.get("device_class", ""),
},
"devices": [{"dev": current.get("dev", "")}] if current.get("dev") else [],
}
async def cluster_ceph_status(_request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
row = await database(_request).pool.fetchrow(
"""SELECT capacity_bytes, used_bytes FROM storages
WHERE storage_type='ceph' ORDER BY storage_id LIMIT 1"""
)
total = int(row["capacity_bytes"] or 0) if row is not None else 0
used = int(row["used_bytes"] or 0) if row is not None else 0
osd_rows = await database(_request).pool.fetch(
"SELECT state FROM resources WHERE kind='ceph-osd'"
)
num_osds = len(osd_rows)
num_up = 0
num_in = 0
for osd_row in osd_rows:
osd_state = state(osd_row["state"])
if str(osd_state.get("status") or "") == "up":
num_up += 1
if osd_state.get("in") in (True, 1, "1"):
num_in += 1
ceph = await _load_cluster_ceph(_request)
version = ceph.get("version")
version_str = (
str(version.get("str"))
if isinstance(version, dict) and version.get("str") is not None
else str(version or "")
)
health = ceph.get("health")
if not isinstance(health, dict):
health = {"status": "HEALTH_OK" if ceph.get("running") else "HEALTH_WARN"}
node_count = int(await database(_request).pool.fetchval("SELECT COUNT(*) FROM nodes") or 0)
return {
"version": {
"version": version_str,
"release": "reef",
"epoch": 0,
},
"health": health,
"osdmap": {
"epoch": 1,
"num_osds": num_osds,
"num_up_osds": num_up,
"num_in_osds": num_in,
"num_remapped_pgs": 0,
},
"pgmap": {
"bytes_used": used,
"bytes_total": total,
"bytes_avail": max(total - used, 0),
"data_bytes": used,
"num_pgs": max(num_osds * 32, 32),
},
"fsmap": {
"epoch": 1,
"by_rank": [],
"filesystems": list((ceph.get("fs") or {}).keys()),
},
"monmap": {"num_mons": max(node_count, 1)},
"mgrmap": {"available": True, "num_standbys": 0},
"servicemap": {"epoch": 1, "services": {}},
}
base = "/nodes/{node}/ceph"
registry.register(base, "GET", ceph_index)
registry.register(f"{base}/cfg", "GET", cfg_index)
registry.register(f"{base}/cfg/db", "GET", cfg_db)
registry.register(f"{base}/cfg/raw", "GET", cfg_raw)
registry.register(f"{base}/cfg/value", "GET", cfg_value)
registry.register(f"{base}/cmd-safety", "GET", cmd_safety)
registry.register(f"{base}/crush", "GET", crush)
registry.register(f"{base}/fs", "GET", fs_list)
registry.register(f"{base}/fs/{{name}}", "POST", fs_create)
registry.register(f"{base}/fs/{{name}}", "DELETE", fs_delete)
registry.register(f"{base}/init", "POST", init)
registry.register(f"{base}/log", "GET", log)
registry.register(f"{base}/mds", "GET", mds_list)
registry.register(f"{base}/mds/{{name}}", "POST", mds_create)
registry.register(f"{base}/mds/{{name}}", "DELETE", mds_delete)
registry.register(f"{base}/mgr", "GET", mgr_list)
registry.register(f"{base}/mgr/{{id}}", "POST", mgr_create)
registry.register(f"{base}/mgr/{{id}}", "DELETE", mgr_delete)
registry.register(f"{base}/mon", "GET", mon_list)
registry.register(f"{base}/mon/{{monid}}", "POST", mon_create)
registry.register(f"{base}/mon/{{monid}}", "DELETE", mon_delete)
registry.register(f"{base}/osd", "GET", osd_list)
registry.register(f"{base}/osd", "POST", osd_create)
registry.register(f"{base}/osd/{{osdid}}", "GET", osd_get)
registry.register(f"{base}/osd/{{osdid}}", "DELETE", osd_delete)
registry.register(f"{base}/osd/{{osdid}}/in", "POST", osd_in)
registry.register(f"{base}/osd/{{osdid}}/out", "POST", osd_out)
registry.register(f"{base}/osd/{{osdid}}/scrub", "POST", osd_scrub)
registry.register(f"{base}/osd/{{osdid}}/lv-info", "GET", osd_lv_info)
registry.register(f"{base}/osd/{{osdid}}/metadata", "GET", osd_metadata)
registry.register(f"{base}/pool", "GET", pool_list)
registry.register(f"{base}/pool", "POST", pool_create)
registry.register(f"{base}/pool/{{name}}", "GET", pool_get)
registry.register(f"{base}/pool/{{name}}", "PUT", pool_update)
registry.register(f"{base}/pool/{{name}}", "DELETE", pool_delete)
registry.register(f"{base}/pool/{{name}}/status", "GET", pool_status)
registry.register(f"{base}/rules", "GET", rules)
registry.register(f"{base}/status", "GET", status)
registry.register(f"{base}/start", "POST", start)
registry.register(f"{base}/stop", "POST", stop)
registry.register(f"{base}/restart", "POST", restart)
registry.register("/cluster/ceph/status", "GET", cluster_ceph_status)
+271
View File
@@ -0,0 +1,271 @@
"""Cluster-level semantic handlers."""
from __future__ import annotations
import json
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import (
cluster_metadata,
database,
save_cluster_metadata,
state,
subdirs,
values,
)
from app.simulation.seed import CLUSTER_ID
def _replication_jobs(metadata: dict[str, Any]) -> list[dict[str, Any]]:
jobs = metadata.get("replication", [])
if not isinstance(jobs, list):
return []
result: list[dict[str, Any]] = []
for item in jobs:
if not isinstance(item, dict):
continue
guest_raw = item.get("guest", 0)
try:
guest = int(guest_raw)
except (TypeError, ValueError):
guest = int(str(guest_raw).split(":")[-1] or 0)
jobnum_raw = item.get("jobnum")
if jobnum_raw is None:
id_text = str(item.get("id") or "0")
tail = id_text.rsplit("-", 1)[-1]
jobnum = int(tail) if tail.isdigit() else 0
else:
jobnum = int(jobnum_raw)
disable = item.get("disable")
if disable is None:
disable = 0 if int(item.get("enabled", 1) or 1) else 1
entry = {
"id": str(item.get("id") or f"{guest}-{jobnum}"),
"guest": guest,
"jobnum": jobnum,
"target": str(item.get("target") or ""),
"type": str(item.get("type") or "local"),
"disable": bool(int(disable)),
}
if item.get("schedule") is not None:
entry["schedule"] = str(item["schedule"])
if item.get("rate") is not None:
entry["rate"] = int(item["rate"])
if item.get("comment") is not None:
entry["comment"] = str(item["comment"])
if item.get("source") is not None:
entry["source"] = str(item["source"])
result.append(entry)
return result
def register_cluster_handlers(registry: HandlerRegistry) -> None:
async def cluster_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs(
"acme",
"backup",
"backup-info",
"config",
"ha",
"log",
"mapping",
"nextid",
"notifications",
"options",
"replication",
"sdn",
"status",
"tasks",
)
async def cluster_status(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
from app.handlers.nodes import load_node_ops
rows = await database(request).pool.fetch(
"""SELECT id, name, status FROM nodes ORDER BY name"""
)
metadata = await cluster_metadata(request)
quorate = bool(int(metadata.get("quorate", 1) or 1))
cluster_name = metadata.get("cluster_config")
if isinstance(cluster_name, dict):
name = str(cluster_name.get("clustername") or "pve-simulator")
version = int(cluster_name.get("version") or 2)
else:
name = "pve-simulator"
version = 2
result: list[dict[str, Any]] = [
{
"type": "cluster",
"id": "cluster",
"name": name,
"nodes": len(rows),
"quorate": quorate,
"version": version,
}
]
for index, row in enumerate(rows):
online = str(row["status"]) == "online"
ops = await load_node_ops(request, str(row["name"]))
cluster_node = ops.get("cluster_status")
entry = dict(cluster_node) if isinstance(cluster_node, dict) else {}
local_raw = entry.get("local", index == 0)
node_name = str(row["name"])
result.append(
{
"type": "node",
"id": f"node/{node_name}",
"name": node_name,
"nodeid": int(entry.get("nodeid", index + 1)),
"online": online,
"local": bool(int(local_raw)) if not isinstance(local_raw, bool) else local_raw,
"ip": str(entry.get("ip") or ops.get("ip") or ""),
"level": str(entry.get("level", "")),
}
)
return result
async def cluster_nextid(request: Request, inputs: dict[str, Any]) -> int:
requested = values(inputs).get("vmid")
if requested is not None:
candidate = int(requested)
taken = await database(request).pool.fetchval(
"""SELECT EXISTS(
SELECT 1 FROM resources WHERE kind IN ('qemu', 'lxc') AND external_id=$1
)""",
str(candidate),
)
if not taken:
return candidate
raise ApiError(400, f"VMID {candidate} already exists")
maximum = await database(request).pool.fetchval(
"""SELECT COALESCE(MAX(external_id::integer), 99)
FROM resources WHERE kind IN ('qemu', 'lxc')"""
)
return int(maximum) + 1
async def cluster_options_get(_request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
row = await database(_request).pool.fetchrow(
"SELECT metadata FROM clusters WHERE id=$1",
CLUSTER_ID,
)
metadata = state(row["metadata"]) if row is not None else {}
options = metadata.get("options", {})
if not isinstance(options, dict):
return {}
return dict(options)
async def cluster_options_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
current = await cluster_options_get(request, inputs)
provided = values(inputs)
updated = {**current, **{key: value for key, value in provided.items() if key != "node"}}
await database(request).pool.execute(
"""UPDATE clusters SET metadata = jsonb_set(
COALESCE(metadata, '{}'::jsonb), '{options}', $2::jsonb, true
), updated_at=now() WHERE id=$1""",
CLUSTER_ID,
json.dumps(updated, sort_keys=True),
)
return updated
async def cluster_log(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
limit = int(values(inputs).get("max") or 50)
rows = await database(request).pool.fetch(
"""SELECT tl.message, tl.sequence
FROM task_logs tl
ORDER BY tl.created_at DESC, tl.sequence DESC
LIMIT $1""",
limit,
)
return [{"n": int(row["sequence"]), "t": str(row["message"])} for row in reversed(rows)]
async def cluster_tasks(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"SELECT upid FROM tasks ORDER BY created_at DESC LIMIT 1000"
)
return [{"upid": str(row["upid"])} for row in rows]
async def replication_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
return _replication_jobs(metadata)
async def replication_create(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
from app.handlers.common import require_value
payload = values(inputs)
guest = str(require_value(payload, "guest"))
target = str(require_value(payload, "target"))
job_id = str(payload.get("id") or f"repl-{guest.replace(':', '-')}")
metadata = await cluster_metadata(request)
jobs = _replication_jobs(metadata)
if any(str(item.get("id")) == job_id for item in jobs):
raise ApiError(409, "replication job already exists")
job = {
"id": job_id,
"guest": guest,
"target": target,
"type": str(payload.get("type") or "local"),
"schedule": str(payload.get("schedule") or "*/15"),
"rate": int(payload.get("rate") or 1),
"comment": str(payload.get("comment") or ""),
"enabled": int(payload.get("enabled", 1)),
}
jobs.append(job)
metadata["replication"] = jobs
await save_cluster_metadata(request, metadata)
return job
async def replication_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job_id = str(values(inputs)["id"])
for job in _replication_jobs(await cluster_metadata(request)):
if str(job.get("id")) == job_id:
return job
raise ApiError(404, "replication job does not exist")
async def replication_update(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _replication_jobs(metadata)
for index, job in enumerate(jobs):
if str(job.get("id")) != job_id:
continue
payload = values(inputs)
updated = {
**job,
**{
key: value
for key, value in payload.items()
if key not in {"id", "delete", "digest"}
},
}
jobs[index] = updated
metadata["replication"] = jobs
await save_cluster_metadata(request, metadata)
return updated
raise ApiError(404, "replication job does not exist")
async def replication_delete(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _replication_jobs(metadata)
remaining = [job for job in jobs if str(job.get("id")) != job_id]
if len(remaining) == len(jobs):
raise ApiError(404, "replication job does not exist")
metadata["replication"] = remaining
await save_cluster_metadata(request, metadata)
registry.register("/cluster", "GET", cluster_index)
registry.register("/cluster/status", "GET", cluster_status)
registry.register("/cluster/nextid", "GET", cluster_nextid)
registry.register("/cluster/options", "GET", cluster_options_get)
registry.register("/cluster/options", "PUT", cluster_options_put)
registry.register("/cluster/log", "GET", cluster_log)
registry.register("/cluster/tasks", "GET", cluster_tasks)
registry.register("/cluster/replication", "GET", replication_list)
registry.register("/cluster/replication", "POST", replication_create)
registry.register("/cluster/replication/{id}", "GET", replication_get)
registry.register("/cluster/replication/{id}", "PUT", replication_update)
registry.register("/cluster/replication/{id}", "DELETE", replication_delete)
+409
View File
@@ -0,0 +1,409 @@
"""Cluster config / join / totem handlers."""
from __future__ import annotations
import hashlib
import secrets
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import (
cluster_metadata,
database,
require_value,
save_cluster_metadata,
values,
)
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
def _config(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.get("cluster_config")
return current if isinstance(current, dict) else {}
def _principal(request: Request) -> str:
return str(getattr(request.state, "principal", None) or "root@pam")
def _fingerprint_sha256() -> str:
return ":".join(f"{byte:02X}" for byte in secrets.token_bytes(32))
def _stable_digest(material: str) -> str:
return hashlib.sha256(material.encode("utf-8")).hexdigest()
def _node_addr(node: str, index: int, entry: dict[str, Any]) -> str:
for key in ("pve_addr", "new_node_ip", "ring0_addr"):
value = entry.get(key)
if isinstance(value, str) and value and not value.endswith(".local"):
# Prefer literal IPs; hostnames are still accepted as ring addresses.
if key != "ring0_addr" or all(part.isdigit() for part in value.split(".")):
return value
return f"10.0.0.{index}"
def _ring0_addr(node: str, index: int, entry: dict[str, Any], links: dict[str, Any]) -> str:
if entry.get("ring0_addr"):
return str(entry["ring0_addr"])
link0 = links.get("link0")
if isinstance(link0, str) and link0:
# Property string may be bare IP or address=IP[,priority=N].
if link0.startswith("address="):
return link0.removeprefix("address=").split(",", 1)[0]
return link0.split(",", 1)[0]
if entry.get("new_node_ip"):
return str(entry["new_node_ip"])
return f"10.0.0.{index}"
def _render_corosync_conf(config: dict[str, Any], node_names: list[str]) -> str:
clustername = str(config.get("clustername") or "proxmox")
totem = dict(config.get("totem") or {})
added_raw = config.get("added_nodes")
added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
links = dict(config.get("links") or {})
lines = [
"totem {",
" version: 2",
f" cluster_name: {totem.get('cluster_name', clustername)}",
f" secauth: {totem.get('secauth', 'on')}",
"}",
"nodelist {",
]
for index, name in enumerate(node_names, start=1):
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
nodeid = int(entry.get("nodeid") or index)
votes = int(entry.get("quorum_votes") or entry.get("votes") or config.get("votes") or 1)
ring0 = _ring0_addr(name, index, entry, links)
lines.extend(
[
" node {",
f" name: {name}",
f" nodeid: {nodeid}",
f" quorum_votes: {votes}",
f" ring0_addr: {ring0}",
" }",
]
)
lines.append("}")
lines.extend(["quorum {", " provider: corosync_votequorum", "}"])
return "\n".join(lines) + "\n"
def _ensure_corosync_materials(
config: dict[str, Any], node_names: list[str]
) -> tuple[str, str, str]:
authkey = config.get("corosync_authkey")
if not isinstance(authkey, str) or not authkey:
authkey = secrets.token_hex(32)
config["corosync_authkey"] = authkey
conf = _render_corosync_conf(config, node_names)
config["corosync_conf"] = conf
digest = _stable_digest(conf)
config["config_digest"] = digest
return authkey, conf, digest
async def _cluster_task(
request: Request, *, task_type: str, worker: str, task_id: str = "0"
) -> str:
from app.db.primitives import ConflictError
pool = database(request).pool
node = await pool.fetchval("SELECT name FROM nodes ORDER BY name LIMIT 1") or "localhost"
upid = str(Upid.allocate(str(node), worker, task_id, _principal(request)))
try:
task = await TaskRepository(pool).create(
upid=upid,
task_type=task_type,
payload={"cluster": True, "worker": worker},
resource_key=f"cluster:{task_type}:{secrets.token_hex(4)}",
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
# PVE uses {name}; contract child link is href="{name}".
return [{"name": name} for name in ("nodes", "totem", "join", "qdevice", "apiversion")]
async def create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
clustername = str(require_value(payload, "clustername"))
metadata = await cluster_metadata(request)
config = dict(_config(metadata))
# Match PVE: refuse when corosync materials already exist (seeded or prior create).
if config.get("corosync_conf") or config.get("config_digest"):
raise ApiError(400, "cluster config already exists")
config["clustername"] = clustername
totem = dict(config.get("totem") or {})
totem["version"] = int(totem.get("version") or 2)
totem["secauth"] = totem.get("secauth") or "on"
totem["cluster_name"] = clustername
# PVE defaults token-coefficient to 125 when omitted (majors that expose it).
totem["token_coefficient"] = payload.get("token-coefficient", 125)
config["totem"] = totem
if "votes" in payload:
config["votes"] = payload["votes"]
if "nodeid" in payload:
config["creator_nodeid"] = payload["nodeid"]
links = {key: value for key, value in payload.items() if key.startswith("link")}
if links:
config["links"] = links
config["token"] = secrets.token_hex(16)
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows]
if not node_names:
node_names = ["localhost"]
creator = node_names[0]
added = dict(config.get("added_nodes") or {})
creator_entry = dict(added.get(creator) or {})
creator_entry.update(
{
"node": creator,
"nodeid": payload.get("nodeid") or creator_entry.get("nodeid") or 1,
"quorum_votes": payload.get("votes") or creator_entry.get("quorum_votes") or 1,
"ring0_addr": _ring0_addr(
creator, 1, creator_entry, dict(config.get("links") or {})
),
"pve_addr": creator_entry.get("pve_addr") or _node_addr(creator, 1, creator_entry),
"pve_fp": creator_entry.get("pve_fp") or _fingerprint_sha256(),
}
)
added[creator] = creator_entry
config["added_nodes"] = added
_ensure_corosync_materials(config, node_names)
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata)
await database(request).pool.execute(
"""UPDATE clusters
SET name=$1, updated_at=now()
WHERE id=(SELECT id FROM clusters LIMIT 1)""",
clustername,
)
return await _cluster_task(
request,
task_type="cluster-create",
worker="clustercreate",
task_id=clustername,
)
async def apiversion(_request: Request, _inputs: dict[str, Any]) -> int:
metadata = await cluster_metadata(_request)
return int(_config(metadata).get("apiversion") or 1)
async def join_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
config = _config(metadata)
preferred_raw = values(inputs).get("node")
# Contract documents the omitted-node default as this phrase; PVE uses local nodename.
if preferred_raw in {None, "", "current connected node"}:
preferred = None
else:
preferred = str(preferred_raw)
rows = await database(request).pool.fetch("SELECT name, status FROM nodes ORDER BY name")
added_raw = config.get("added_nodes")
added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
links = dict(config.get("links") or {})
nodelist: list[dict[str, Any]] = []
for index, row in enumerate(rows, start=1):
name = str(row["name"])
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
if "pve_fp" not in entry:
entry["pve_fp"] = _fingerprint_sha256()
added[name] = entry
if "pve_addr" not in entry:
entry["pve_addr"] = _node_addr(name, index, entry)
added[name] = entry
nodelist.append(
{
"name": name,
"nodeid": int(entry.get("nodeid") or index),
"quorum_votes": int(
entry.get("quorum_votes") or entry.get("votes") or config.get("votes") or 1
),
"ring0_addr": _ring0_addr(name, index, entry, links),
"pve_addr": str(entry["pve_addr"]),
"pve_fp": str(entry["pve_fp"]),
}
)
if added != dict(added_raw or {}):
updated = dict(config)
updated["added_nodes"] = added
if not updated.get("config_digest"):
_ensure_corosync_materials(updated, [item["name"] for item in nodelist])
metadata["cluster_config"] = updated
await save_cluster_metadata(request, metadata)
config = updated
preferred_node = preferred or (nodelist[0]["name"] if nodelist else None)
digest = config.get("config_digest")
if not isinstance(digest, str) or not digest:
updated = dict(config)
_, _, digest = _ensure_corosync_materials(updated, [item["name"] for item in nodelist])
metadata["cluster_config"] = updated
await save_cluster_metadata(request, metadata)
config = updated
return {
"nodelist": nodelist,
"preferred_node": preferred_node,
"totem": dict(config.get("totem") or {}),
"config_digest": digest,
}
async def join_post(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
hostname = str(require_value(payload, "hostname"))
require_value(payload, "fingerprint")
require_value(payload, "password")
metadata = await cluster_metadata(request)
config = dict(_config(metadata))
joins = dict(config.get("join_info") or {})
joins[hostname] = {
"hostname": hostname,
"fingerprint": payload.get("fingerprint"),
"nodeid": payload.get("nodeid"),
"votes": payload.get("votes", 1),
"force": payload.get("force"),
"password_set": True,
}
links = {key: value for key, value in payload.items() if key.startswith("link")}
if links:
joins[hostname]["links"] = links
config["join_info"] = joins
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows]
_ensure_corosync_materials(config, node_names or ["localhost"])
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata)
return await _cluster_task(
request,
task_type="cluster-join",
worker="clusterjoin",
)
async def nodes_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(request).pool.fetch("SELECT name, status FROM nodes ORDER BY name")
metadata = await cluster_metadata(request)
config = _config(metadata)
added_raw = config.get("added_nodes")
added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
links = dict(config.get("links") or {})
result = []
for index, row in enumerate(rows, start=1):
name = str(row["name"])
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
result.append(
{
"node": name,
"nodeid": entry.get("nodeid", index),
"ring0_addr": _ring0_addr(name, index, entry, links),
"quorum_votes": entry.get("quorum_votes", config.get("votes", 0)),
}
)
return result
async def nodes_add(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
node = str(require_value(payload, "node"))
metadata = await cluster_metadata(request)
config = dict(_config(metadata))
added = dict(config.get("added_nodes") or {})
if node in added and not payload.get("force"):
raise ApiError(400, f"can't add existing node '{node}'")
index = len(added) + 1
links = {key: value for key, value in payload.items() if key.startswith("link")}
entry = {
"node": node,
"nodeid": payload.get("nodeid") or index,
"new_node_ip": payload.get("new_node_ip"),
"votes": payload.get("votes", 1),
"apiversion": payload.get("apiversion"),
"force": payload.get("force"),
"ring0_addr": payload.get("ring0_addr")
or payload.get("new_node_ip")
or _ring0_addr(node, index, {}, links or dict(config.get("links") or {})),
"quorum_votes": payload.get("quorum_votes", payload.get("votes", 1)),
"pve_addr": payload.get("new_node_ip") or _node_addr(node, index, {}),
"pve_fp": _fingerprint_sha256(),
}
if links:
entry["links"] = links
added[node] = entry
config["added_nodes"] = added
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)",
node,
)
if not exists:
await database(request).pool.execute(
"""INSERT INTO nodes(id, name, status, metadata)
VALUES(gen_random_uuid(), $1, 'online', '{}'::jsonb)""",
node,
)
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows]
if node not in node_names:
node_names.append(node)
authkey, conf, _digest = _ensure_corosync_materials(config, node_names)
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata)
return {
"corosync_authkey": authkey,
"corosync_conf": conf,
"warnings": [],
}
async def nodes_delete(request: Request, inputs: dict[str, Any]) -> None:
node = str(values(inputs)["node"])
metadata = await cluster_metadata(request)
config = dict(_config(metadata))
added = dict(config.get("added_nodes") or {})
added.pop(node, None)
joins = dict(config.get("join_info") or {})
joins.pop(node, None)
config["added_nodes"] = added
config["join_info"] = joins
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows if str(row["name"]) != node]
_ensure_corosync_materials(config, node_names or ["localhost"])
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata)
# Keep node row; mark offline to avoid cascading guest deletes.
await database(request).pool.execute(
"UPDATE nodes SET status='offline', updated_at=now() WHERE name=$1",
node,
)
async def qdevice(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
qdevice = _config(metadata).get("qdevice")
return dict(qdevice) if isinstance(qdevice, dict) else {}
async def totem(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
totem = _config(metadata).get("totem")
return dict(totem) if isinstance(totem, dict) else {}
registry.register("/cluster/config", "GET", index)
registry.register("/cluster/config", "POST", create)
registry.register("/cluster/config/apiversion", "GET", apiversion)
registry.register("/cluster/config/join", "GET", join_get)
registry.register("/cluster/config/join", "POST", join_post)
registry.register("/cluster/config/nodes", "GET", nodes_list)
registry.register("/cluster/config/nodes/{node}", "POST", nodes_add)
registry.register("/cluster/config/nodes/{node}", "DELETE", nodes_delete)
registry.register("/cluster/config/qdevice", "GET", qdevice)
registry.register("/cluster/config/totem", "GET", totem)
+775
View File
@@ -0,0 +1,775 @@
"""Additional cluster-level handlers with durable metadata persistence."""
from __future__ import annotations
import secrets
import time
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import (
cluster_metadata,
database,
require_node,
save_cluster_metadata,
state,
subdirs,
values,
)
from app.simulation.ceph_defaults import CEPH_FLAG_DESCRIPTIONS, default_ceph_flags
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
def _jobs(metadata: dict[str, Any]) -> dict[str, Any]:
jobs = metadata.get("jobs")
return jobs if isinstance(jobs, dict) else {}
def _metrics(metadata: dict[str, Any]) -> dict[str, Any]:
metrics = metadata.get("metrics")
return metrics if isinstance(metrics, dict) else {}
def _cpu_models(metadata: dict[str, Any]) -> dict[str, Any]:
models = metadata.get("qemu_cpu_models")
return models if isinstance(models, dict) else {}
def _ha_rules_store(metadata: dict[str, Any]) -> list[dict[str, Any]]:
rules = metadata.get("ha_rules")
if isinstance(rules, dict):
return [
{"rule": str(name), **dict(value)}
for name, value in rules.items()
if isinstance(value, dict)
]
if isinstance(rules, list):
return [dict(item) for item in rules if isinstance(item, dict)]
return []
def _save_ha_rules(metadata: dict[str, Any], rules: list[dict[str, Any]]) -> None:
metadata["ha_rules"] = rules
def _replication_jobs(metadata: dict[str, Any]) -> list[dict[str, Any]]:
from app.handlers.cluster import _replication_jobs as normalize
return normalize(metadata)
def _ceph(metadata: dict[str, Any]) -> dict[str, Any]:
ceph = metadata.get("ceph")
return ceph if isinstance(ceph, dict) else {}
async def _cluster_task(request: Request, *, task_type: str, worker: str) -> str:
from app.db.primitives import ConflictError
pool = database(request).pool
node = await pool.fetchval("SELECT name FROM nodes ORDER BY name LIMIT 1") or "localhost"
upid = str(Upid.allocate(str(node), worker, "0", str(request.state.principal)))
try:
task = await TaskRepository(pool).create(
upid=upid,
task_type=task_type,
payload={"cluster": True},
resource_key=f"cluster:{task_type}:{secrets.token_hex(4)}",
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
async def _bulk_guest_status(request: Request, status: str) -> None:
await database(request).pool.execute(
"""UPDATE resources
SET state = jsonb_set(COALESCE(state, '{}'::jsonb), '{status}', to_jsonb($1::text), true),
updated_at=now()
WHERE kind IN ('qemu', 'lxc')""",
status,
)
def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
async def jobs_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("realm-sync", "schedule-analyze")
async def realm_sync_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
jobs = _jobs(metadata)
items = [
{"id": job_id, **dict(payload)}
for job_id, payload in sorted(jobs.get("realm_sync", {}).items())
if isinstance(payload, dict)
]
return items
async def realm_sync_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _jobs(metadata)
payload = jobs.get("realm_sync", {}).get(job_id)
if not isinstance(payload, dict):
raise ApiError(404, "realm-sync job does not exist")
return {"id": job_id, **payload}
async def realm_sync_create(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
job_id = str(payload["id"])
metadata = await cluster_metadata(request)
jobs = dict(_jobs(metadata))
sync = dict(jobs.get("realm_sync") or {})
if job_id in sync:
raise ApiError(409, "realm-sync job already exists")
entry = {
key: value for key, value in payload.items() if key not in {"id", "delete", "digest"}
}
entry.setdefault("schedule", "0 0 * * *")
entry.setdefault("enabled", 1)
entry.setdefault("realm", str(payload.get("realm") or "pam"))
sync[job_id] = entry
jobs["realm_sync"] = sync
metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata)
return {"id": job_id, **entry}
async def realm_sync_update(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
job_id = str(payload["id"])
metadata = await cluster_metadata(request)
jobs = dict(_jobs(metadata))
sync = dict(jobs.get("realm_sync") or {})
if job_id not in sync:
raise ApiError(404, "realm-sync job does not exist")
updated = {
**sync[job_id],
**{
key: value
for key, value in payload.items()
if key not in {"id", "delete", "digest"}
},
}
sync[job_id] = updated
jobs["realm_sync"] = sync
metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata)
return {"id": job_id, **updated}
async def realm_sync_delete(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = dict(_jobs(metadata))
sync = dict(jobs.get("realm_sync") or {})
if job_id not in sync:
raise ApiError(404, "realm-sync job does not exist")
del sync[job_id]
jobs["realm_sync"] = sync
metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def schedule_analyze(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
schedule = str(values(inputs).get("schedule") or "*/15")
metadata = await cluster_metadata(request)
jobs = dict(_jobs(metadata))
results = jobs.get("schedule_analyze_results")
if not isinstance(results, list):
results = []
jobs["last_schedule_analyze"] = {"schedule": schedule, "at": int(time.time())}
metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata)
return [dict(item) for item in results if isinstance(item, dict)]
async def metrics_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("export", "server")
async def metrics_export(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
metrics = _metrics(metadata)
now = int(time.time())
seeded = metrics.get("export_series")
if isinstance(seeded, list) and seeded:
seeded_series = [dict(item) for item in seeded if isinstance(item, dict)]
return {"data": seeded_series, "timestamp": now}
# Build a PVE-shaped metric array from live inventory (scales with seed).
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
series: list[dict[str, Any]] = []
for row in rows:
node = str(row["name"])
series.append(
{
"id": f"node/{node}",
"metric": "up",
"timestamp": now,
"type": "gauge",
"value": 1,
}
)
guests = await database(request).pool.fetch(
"""SELECT r.kind, r.external_id, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind IN ('qemu', 'lxc')
ORDER BY r.external_id::integer"""
)
for guest in guests:
series.append(
{
"id": f"{guest['kind']}/{guest['external_id']}",
"metric": "up",
"timestamp": now,
"type": "gauge",
"value": 1,
}
)
return {"data": series, "timestamp": now}
async def metrics_server_list(
request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
metrics = _metrics(metadata)
servers = metrics.get("servers")
if not isinstance(servers, dict):
return []
return [
{"id": server_id, **dict(payload)}
for server_id, payload in sorted(servers.items())
if isinstance(payload, dict)
]
async def metrics_server_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
server_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
metrics = _metrics(metadata)
payload = metrics.get("servers", {}).get(server_id)
if not isinstance(payload, dict):
raise ApiError(404, "metrics server does not exist")
return {"id": server_id, **payload}
async def metrics_server_create(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
server_id = str(payload["id"])
metadata = await cluster_metadata(request)
metrics = dict(_metrics(metadata))
servers = dict(metrics.get("servers") or {})
if server_id in servers:
raise ApiError(409, "metrics server already exists")
entry = {
key: value for key, value in payload.items() if key not in {"id", "delete", "digest"}
}
entry.setdefault("type", "influxdb")
entry.setdefault("server", "127.0.0.1")
entry.setdefault("port", 8086)
entry.setdefault("enable", 1)
servers[server_id] = entry
metrics["servers"] = servers
metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata)
return {"id": server_id, **entry}
async def metrics_server_update(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
server_id = str(payload["id"])
metadata = await cluster_metadata(request)
metrics = dict(_metrics(metadata))
servers = dict(metrics.get("servers") or {})
if server_id not in servers:
raise ApiError(404, "metrics server does not exist")
updated = {
**servers[server_id],
**{
key: value
for key, value in payload.items()
if key not in {"id", "delete", "digest"}
},
}
servers[server_id] = updated
metrics["servers"] = servers
metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata)
return {"id": server_id, **updated}
async def metrics_server_delete(request: Request, inputs: dict[str, Any]) -> None:
server_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
metrics = dict(_metrics(metadata))
servers = dict(metrics.get("servers") or {})
if server_id not in servers:
raise ApiError(404, "metrics server does not exist")
del servers[server_id]
metrics["servers"] = servers
metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata)
async def qemu_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("cpu-flags", "custom-cpu-models")
async def qemu_cpu_flags(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
flags = metadata.get("qemu_cpu_flags")
if isinstance(flags, list):
return [dict(item) for item in flags if isinstance(item, dict)]
return []
async def cpu_models_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
models = _cpu_models(metadata)
return [
{"name": name, **dict(payload)}
for name, payload in sorted(models.items())
if isinstance(payload, dict)
]
async def cpu_models_create(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
name = str(payload.get("name") or payload.get("cputype") or "")
if not name:
raise ApiError(400, "parameter verification failed - 'name' missing")
metadata = await cluster_metadata(request)
models = _cpu_models(metadata)
if name in models:
raise ApiError(409, "custom cpu model already exists")
entry = {
key: value
for key, value in payload.items()
if key not in {"name", "cputype", "delete", "digest"}
}
entry.setdefault("vendor", "Custom")
models[name] = entry
metadata["qemu_cpu_models"] = models
await save_cluster_metadata(request, metadata)
return {"name": name, **entry}
async def cpu_models_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
name = str(values(inputs)["cputype"])
metadata = await cluster_metadata(request)
models = _cpu_models(metadata)
payload = models.get(name)
if not isinstance(payload, dict):
raise ApiError(404, "custom cpu model does not exist")
return {"name": name, **payload}
async def cpu_models_update(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
name = str(payload["cputype"])
metadata = await cluster_metadata(request)
models = _cpu_models(metadata)
if name not in models:
raise ApiError(404, "custom cpu model does not exist")
updated = {
**models[name],
**{
key: value
for key, value in payload.items()
if key not in {"cputype", "delete", "digest"}
},
}
models[name] = updated
metadata["qemu_cpu_models"] = models
await save_cluster_metadata(request, metadata)
return {"name": name, **updated}
async def cpu_models_delete(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["cputype"])
metadata = await cluster_metadata(request)
models = _cpu_models(metadata)
if name not in models:
raise ApiError(404, "custom cpu model does not exist")
del models[name]
metadata["qemu_cpu_models"] = models
await save_cluster_metadata(request, metadata)
async def bulk_action_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("guest")
async def bulk_guest_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("migrate", "shutdown", "start", "suspend")
async def bulk_guest_action(request: Request, inputs: dict[str, Any], action: str) -> str:
payload = values(inputs)
metadata = await cluster_metadata(request)
metadata["last_bulk_action"] = {
"action": action,
"payload": {
key: value for key, value in payload.items() if key not in {"delete", "digest"}
},
"at": int(time.time()),
}
await save_cluster_metadata(request, metadata)
if action == "start":
await _bulk_guest_status(request, "running")
elif action == "shutdown":
await _bulk_guest_status(request, "stopped")
elif action == "suspend":
await _bulk_guest_status(request, "paused")
elif action == "migrate":
target = str(payload.get("target") or "")
if target:
target_row = await database(request).pool.fetchrow(
"SELECT id FROM nodes WHERE name=$1", target
)
if target_row is None:
raise ApiError(404, "target node does not exist")
vms = payload.get("vms") or payload.get("guests") or ""
if isinstance(vms, str) and vms:
ids = [part.strip() for part in vms.split(",") if part.strip()]
for vmid in ids:
await database(request).pool.execute(
"""UPDATE resources SET node_id=$2, updated_at=now()
WHERE kind IN ('qemu', 'lxc') AND external_id=$1""",
vmid,
target_row["id"],
)
return await _cluster_task(request, task_type=f"bulk-{action}", worker=f"bulk{action}")
async def cluster_ceph_index(
_request: Request, _inputs: dict[str, Any]
) -> list[dict[str, str]]:
return subdirs("flags", "metadata", "status")
async def ceph_flags_get(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
ceph = _ceph(metadata)
stored = ceph.get("flags")
flags = default_ceph_flags()
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
return [
{
"name": name,
"value": bool(flags[name]),
"description": description,
}
for name, description in sorted(CEPH_FLAG_DESCRIPTIONS.items())
]
async def ceph_flags_put(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
metadata = await cluster_metadata(request)
ceph = dict(_ceph(metadata))
flags = default_ceph_flags()
stored = ceph.get("flags")
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
for key, value in payload.items():
if key in {"delete", "digest"}:
continue
if key not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(400, f"unknown ceph flag '{key}'")
flags[str(key)] = int(value)
ceph["flags"] = flags
metadata["ceph"] = ceph
await save_cluster_metadata(request, metadata)
return await _cluster_task(request, task_type="ceph-flags", worker="cephflags")
async def ceph_flag_get(request: Request, inputs: dict[str, Any]) -> bool:
flag = str(values(inputs)["flag"])
if flag not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(404, "ceph flag does not exist")
metadata = await cluster_metadata(request)
ceph = _ceph(metadata)
flags = ceph.get("flags")
if isinstance(flags, dict) and flag in flags:
return bool(int(flags[flag]))
return False
async def ceph_flag_put(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
flag = str(payload["flag"])
if flag not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(404, "ceph flag does not exist")
metadata = await cluster_metadata(request)
ceph = dict(_ceph(metadata))
flags = default_ceph_flags()
stored = ceph.get("flags")
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
if "value" in payload:
flags[flag] = int(payload["value"])
elif flag in payload:
flags[flag] = int(payload[flag])
else:
flags[flag] = 1
ceph["flags"] = flags
metadata["ceph"] = ceph
await save_cluster_metadata(request, metadata)
async def ceph_metadata(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
ceph = _ceph(metadata)
config_raw = ceph.get("config")
config: dict[str, Any] = dict(config_raw) if isinstance(config_raw, dict) else {}
version = ceph.get("version")
nodes = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in nodes]
version_str = (
str(version.get("str"))
if isinstance(version, dict) and version.get("str")
else "18.2.2"
)
version_short = ".".join(version_str.split(".")[:2]) if version_str else "18.2"
osd_rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind='ceph-osd' ORDER BY r.external_id"""
)
osd_map: dict[str, Any] = {}
for row in osd_rows:
payload = state(row["state"])
external = str(row["external_id"])
osd_id = str(payload.get("osd_id", external.removeprefix("osd.")))
hostname = str(row["node"])
osd_map[osd_id] = {
"id": int(osd_id) if osd_id.isdigit() else osd_id,
"hostname": hostname,
"name": f"osd.{osd_id}",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
"device_class": payload.get("device_class", "hdd"),
"dev": payload.get("dev", ""),
}
if not osd_map:
fallback_host = node_names[0] if node_names else "pve01"
osd_map["0"] = {
"id": 0,
"hostname": fallback_host,
"name": "osd.0",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
}
def _service_entry(name: str, index: int) -> dict[str, Any]:
return {
"name": name,
"hostname": name,
"addr": f"10.0.0.{index}",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
}
return {
"mds": {},
"mgr": {
f"mgr@{node}": _service_entry(node, index)
for index, node in enumerate(node_names[:1] or ["pve01"], start=1)
},
"mon": {
f"mon@{node}": _service_entry(node, index)
for index, node in enumerate(node_names[:3] or ["pve01"], start=1)
},
"node": {
node: {
"name": node,
"hostname": node,
"ceph_version": version_str,
"ceph_version_short": version_short,
}
for node in (node_names or ["pve01"])
},
"osd": list(osd_map.values()),
"fsid": str(config.get("fsid") or ""),
}
async def ha_rule_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
rule = str(payload.get("rule") or payload.get("name") or "")
if not rule:
raise ApiError(400, "parameter verification failed - 'rule' missing")
metadata = await cluster_metadata(request)
rules = _ha_rules_store(metadata)
if any(str(item.get("rule")) == rule for item in rules):
raise ApiError(409, "HA rule already exists")
entry = {key: value for key, value in payload.items() if key not in {"delete", "digest"}}
entry["rule"] = rule
entry.setdefault("type", "resource")
entry.setdefault("action", "migrate")
rules.append(entry)
_save_ha_rules(metadata, rules)
await save_cluster_metadata(request, metadata)
async def ha_rule_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
rule = str(values(inputs)["rule"])
metadata = await cluster_metadata(request)
for item in _ha_rules_store(metadata):
if str(item.get("rule")) == rule:
return dict(item)
raise ApiError(404, "HA rule does not exist")
async def ha_rule_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
rule = str(payload["rule"])
metadata = await cluster_metadata(request)
rules = _ha_rules_store(metadata)
updated: list[dict[str, Any]] = []
found = False
for item in rules:
if str(item.get("rule")) != rule:
updated.append(item)
continue
found = True
merged = {
**item,
**{
key: value
for key, value in payload.items()
if key not in {"rule", "delete", "digest"}
},
}
merged["rule"] = rule
updated.append(merged)
if not found:
raise ApiError(404, "HA rule does not exist")
_save_ha_rules(metadata, updated)
await save_cluster_metadata(request, metadata)
async def ha_rule_delete(request: Request, inputs: dict[str, Any]) -> None:
rule = str(values(inputs)["rule"])
metadata = await cluster_metadata(request)
rules = _ha_rules_store(metadata)
remaining = [item for item in rules if str(item.get("rule")) != rule]
if len(remaining) == len(rules):
raise ApiError(404, "HA rule does not exist")
_save_ha_rules(metadata, remaining)
await save_cluster_metadata(request, metadata)
async def node_replication_list(
request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
metadata = await cluster_metadata(request)
jobs = _replication_jobs(metadata)
return [
job
for job in jobs
if str(job.get("source") or job.get("node") or node) == node
or job.get("source") is None
]
async def node_replication_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
job_id = str(values(inputs)["id"])
await require_node(request, node)
for job in _replication_jobs(await cluster_metadata(request)):
if str(job.get("id")) == job_id:
return dict(job)
raise ApiError(404, "replication job does not exist")
async def node_replication_log(
request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]]:
job = await node_replication_get(request, inputs)
log = job.get("log")
if isinstance(log, list):
return [dict(item) for item in log if isinstance(item, dict)]
return []
async def node_replication_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job = await node_replication_get(request, inputs)
return {
"id": job.get("id"),
"last_sync": job.get("last_sync", 0),
"duration": job.get("duration", 0),
"fail_count": job.get("fail_count", 0),
"error": job.get("error", ""),
"state": job.get("state", "OK"),
}
async def node_replication_schedule_now(request: Request, inputs: dict[str, Any]) -> None:
node = str(values(inputs)["node"])
job_id = str(values(inputs)["id"])
await require_node(request, node)
metadata = await cluster_metadata(request)
jobs = _replication_jobs(metadata)
found = False
for job in jobs:
if str(job.get("id")) != job_id:
continue
found = True
job["last_sync"] = int(time.time())
job["state"] = "OK"
job["schedule_now"] = 1
if not found:
raise ApiError(404, "replication job does not exist")
metadata["replication"] = jobs
await save_cluster_metadata(request, metadata)
registry.register("/cluster/jobs", "GET", jobs_index)
registry.register("/cluster/jobs/realm-sync", "GET", realm_sync_list)
registry.register("/cluster/jobs/realm-sync/{id}", "GET", realm_sync_get)
registry.register("/cluster/jobs/realm-sync/{id}", "POST", realm_sync_create)
registry.register("/cluster/jobs/realm-sync/{id}", "PUT", realm_sync_update)
registry.register("/cluster/jobs/realm-sync/{id}", "DELETE", realm_sync_delete)
registry.register("/cluster/jobs/schedule-analyze", "GET", schedule_analyze)
registry.register("/cluster/metrics", "GET", metrics_index)
registry.register("/cluster/metrics/export", "GET", metrics_export)
registry.register("/cluster/metrics/server", "GET", metrics_server_list)
registry.register("/cluster/metrics/server/{id}", "GET", metrics_server_get)
registry.register("/cluster/metrics/server/{id}", "POST", metrics_server_create)
registry.register("/cluster/metrics/server/{id}", "PUT", metrics_server_update)
registry.register("/cluster/metrics/server/{id}", "DELETE", metrics_server_delete)
registry.register("/cluster/qemu", "GET", qemu_index)
registry.register("/cluster/qemu/cpu-flags", "GET", qemu_cpu_flags)
registry.register("/cluster/qemu/custom-cpu-models", "GET", cpu_models_list)
registry.register("/cluster/qemu/custom-cpu-models", "POST", cpu_models_create)
registry.register("/cluster/qemu/custom-cpu-models/{cputype}", "GET", cpu_models_get)
registry.register("/cluster/qemu/custom-cpu-models/{cputype}", "PUT", cpu_models_update)
registry.register("/cluster/qemu/custom-cpu-models/{cputype}", "DELETE", cpu_models_delete)
registry.register("/cluster/bulk-action", "GET", bulk_action_index)
registry.register("/cluster/bulk-action/guest", "GET", bulk_guest_index)
registry.register(
"/cluster/bulk-action/guest/migrate",
"POST",
lambda request, inputs: bulk_guest_action(request, inputs, "migrate"),
)
registry.register(
"/cluster/bulk-action/guest/shutdown",
"POST",
lambda request, inputs: bulk_guest_action(request, inputs, "shutdown"),
)
registry.register(
"/cluster/bulk-action/guest/start",
"POST",
lambda request, inputs: bulk_guest_action(request, inputs, "start"),
)
registry.register(
"/cluster/bulk-action/guest/suspend",
"POST",
lambda request, inputs: bulk_guest_action(request, inputs, "suspend"),
)
registry.register("/cluster/ceph", "GET", cluster_ceph_index)
registry.register("/cluster/ceph/flags", "GET", ceph_flags_get)
registry.register("/cluster/ceph/flags", "PUT", ceph_flags_put)
registry.register("/cluster/ceph/flags/{flag}", "GET", ceph_flag_get)
registry.register("/cluster/ceph/flags/{flag}", "PUT", ceph_flag_put)
registry.register("/cluster/ceph/metadata", "GET", ceph_metadata)
registry.register("/cluster/ha/rules", "POST", ha_rule_create)
registry.register("/cluster/ha/rules/{rule}", "GET", ha_rule_get)
registry.register("/cluster/ha/rules/{rule}", "PUT", ha_rule_update)
registry.register("/cluster/ha/rules/{rule}", "DELETE", ha_rule_delete)
registry.register("/nodes/{node}/replication", "GET", node_replication_list)
registry.register("/nodes/{node}/replication/{id}", "GET", node_replication_get)
registry.register("/nodes/{node}/replication/{id}/log", "GET", node_replication_log)
registry.register("/nodes/{node}/replication/{id}/status", "GET", node_replication_status)
registry.register(
"/nodes/{node}/replication/{id}/schedule_now", "POST", node_replication_schedule_now
)
+168
View File
@@ -0,0 +1,168 @@
"""Shared handler helpers."""
from __future__ import annotations
import json
import re
from collections.abc import Mapping
from typing import Any, cast
from fastapi import Request
from app.api.errors import ApiError
from app.db.pool import AsyncpgDatabase
def database(request: Request) -> AsyncpgDatabase:
return cast(AsyncpgDatabase, request.app.state.database)
def values(inputs: dict[str, Any]) -> dict[str, Any]:
return cast(dict[str, Any], inputs["values"])
def require_value(payload: Mapping[str, Any], key: str) -> Any:
if key not in payload or payload[key] in {None, ""}:
raise ApiError(400, f"parameter '{key}' is required")
return payload[key]
def state(value: object) -> dict[str, Any]:
if isinstance(value, str):
return cast(dict[str, Any], json.loads(value))
return dict(cast(Mapping[str, Any], value))
def subdirs(*names: str) -> list[dict[str, str]]:
return [{"subdir": name} for name in names]
_SIZE_RE = re.compile(r"^(?P<value>\d+)(?P<unit>[KMGT]?)$", re.IGNORECASE)
_UNITS = {"": 1, "K": 2**10, "M": 2**20, "G": 2**30, "T": 2**40}
def parse_size_bytes(value: str) -> int:
match = _SIZE_RE.fullmatch(value.strip())
if match is None:
raise ValueError(f"invalid disk size: {value}")
return int(match.group("value")) * _UNITS[match.group("unit").upper()]
def resize_size_bytes(value: str, current: int) -> int:
if value.startswith("+"):
return current + parse_size_bytes(value[1:])
result = parse_size_bytes(value)
if result < current:
raise ValueError("shrinking disks is not supported")
return result
def replace_disk_size(value: str, size: int) -> str:
parts = [part for part in value.split(",") if not part.startswith("size=")]
parts.append(f"size={size // 2**30}G" if size % 2**30 == 0 else f"size={size}")
return ",".join(parts)
def disk_size_bytes(value: str) -> int:
for part in value.split(","):
if part.startswith("size="):
return parse_size_bytes(part.removeprefix("size="))
return 0
async def require_node(request: Request, node: str) -> None:
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)",
node,
)
if not exists:
raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
async def cluster_metadata(request: Request) -> dict[str, Any]:
from app.simulation.seed import CLUSTER_ID
row = await database(request).pool.fetchrow(
"SELECT metadata FROM clusters WHERE id=$1",
CLUSTER_ID,
)
return state(row["metadata"]) if row is not None else {}
async def save_cluster_metadata(request: Request, metadata: dict[str, Any]) -> None:
from app.simulation.seed import CLUSTER_ID
await database(request).pool.execute(
"UPDATE clusters SET metadata=$2::jsonb, updated_at=now() WHERE id=$1",
CLUSTER_ID,
json.dumps(metadata, sort_keys=True),
)
async def node_metadata(request: Request, node: str) -> dict[str, Any]:
row = await database(request).pool.fetchrow(
"SELECT metadata FROM nodes WHERE name=$1",
node,
)
if row is None:
raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
return state(row["metadata"])
async def save_node_metadata(request: Request, node: str, metadata: dict[str, Any]) -> None:
status = await database(request).pool.execute(
"UPDATE nodes SET metadata=$2::jsonb, updated_at=now() WHERE name=$1",
node,
json.dumps(metadata, sort_keys=True),
)
if status != "UPDATE 1":
raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
def storage_payload(row: Any) -> dict[str, Any]:
config = state(row["config"])
content = config.get("content", [])
if isinstance(content, list):
content_str = ",".join(str(item) for item in content)
else:
content_str = str(content)
total = int(row["capacity_bytes"] or 0)
used = int(row["used_bytes"] or 0)
avail = max(total - used, 0)
storage_type = str(row["storage_type"])
formats = config.get("formats")
if not isinstance(formats, str) or not formats:
formats = {
"dir": "raw,qcow2,vmdk",
"lvmthin": "raw",
"rbd": "raw",
"zfspool": "raw,subvol",
}.get(storage_type, "raw")
payload: dict[str, Any] = {
"storage": str(row["storage_id"]),
"type": storage_type,
"shared": int(bool(row["shared"])),
"content": content_str,
"active": int(bool(config.get("active", True))),
"enabled": int(bool(config.get("enabled", True))),
"formats": formats,
"select_existing": int(bool(config.get("select_existing", False))),
"total": total,
"used": used,
"avail": avail,
}
if total:
payload["used_fraction"] = used / total
return payload
+397 -24
View File
@@ -10,8 +10,28 @@ from fastapi import Request
from app.api.errors import ApiError from app.api.errors import ApiError
from app.api.registry import HandlerRegistry from app.api.registry import HandlerRegistry
from app.config import Settings from app.config import Settings
from app.contracts.runtime import runtime_version_payload
from app.db.pool import AsyncpgDatabase from app.db.pool import AsyncpgDatabase
from app.handlers.access import register_access_handlers
from app.handlers.acme import register_acme_handlers
from app.handlers.backup import register_backup_handlers
from app.handlers.ceph import register_ceph_handlers
from app.handlers.cluster import register_cluster_handlers
from app.handlers.cluster_config import register_cluster_config_handlers
from app.handlers.cluster_extra import register_cluster_extra_handlers
from app.handlers.common import cluster_metadata, require_node, subdirs
from app.handlers.firewall import register_firewall_handlers
from app.handlers.ha import register_ha_handlers
from app.handlers.legacy_aliases import register_legacy_aliases
from app.handlers.lxc import register_lxc_handlers
from app.handlers.mapping import register_mapping_handlers
from app.handlers.nodes import register_node_ops_handlers
from app.handlers.nodes_extra import register_nodes_extra_handlers
from app.handlers.notifications import register_notifications_handlers
from app.handlers.pools import register_pool_handlers
from app.handlers.qemu import register_qemu_handlers from app.handlers.qemu import register_qemu_handlers
from app.handlers.sdn import register_sdn_handlers
from app.handlers.storage import register_storage_handlers
from app.security.auth import csrf_token, issue_ticket, verify_secret from app.security.auth import csrf_token, issue_ticket, verify_secret
@@ -22,8 +42,8 @@ def _database(request: Request) -> AsyncpgDatabase:
def build_core_handlers(settings: Settings) -> HandlerRegistry: def build_core_handlers(settings: Settings) -> HandlerRegistry:
registry = HandlerRegistry() registry = HandlerRegistry()
async def version(_request: Request, _inputs: dict[str, Any]) -> dict[str, str]: async def version(request: Request, _inputs: dict[str, Any]) -> dict[str, str]:
return {"version": "9.2.3", "release": "9.2", "repoid": "simulator"} return runtime_version_payload(request)
async def login(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def login(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = cast(dict[str, Any], inputs["values"]) values = cast(dict[str, Any], inputs["values"])
@@ -40,58 +60,411 @@ def build_core_handlers(settings: Settings) -> HandlerRegistry:
raise ApiError(401, "authentication failure") raise ApiError(401, "authentication failure")
key = settings.ticket_signing_key.get_secret_value().encode() key = settings.ticket_signing_key.get_secret_value().encode()
ticket = issue_ticket(username, key) ticket = issue_ticket(username, key)
cluster_name = await _database(request).pool.fetchval(
"SELECT name FROM clusters ORDER BY created_at LIMIT 1"
)
return { return {
"username": username, "username": username,
"ticket": ticket, "ticket": ticket,
"CSRFPreventionToken": csrf_token(ticket, key), "CSRFPreventionToken": csrf_token(ticket, key),
"clustername": str(cluster_name or "pve-simulator"),
"cap": {"vms": {"VM.Audit": 1, "VM.PowerMgmt": 1}}, "cap": {"vms": {"VM.Audit": 1, "VM.PowerMgmt": 1}},
} }
async def nodes(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def nodes(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
from app.handlers.nodes import load_node_ops
rows = await _database(request).pool.fetch( rows = await _database(request).pool.fetch(
"SELECT name AS node, status FROM nodes ORDER BY name" "SELECT name AS node, status FROM nodes ORDER BY name"
) )
return [{"node": str(row["node"]), "status": str(row["status"])} for row in rows] result: list[dict[str, Any]] = []
for row in rows:
name = str(row["node"])
ops = await load_node_ops(request, name)
status_payload = ops.get("status")
status_dict = dict(status_payload) if isinstance(status_payload, dict) else {}
fingerprint = status_dict.get("ssl_fingerprint") or status_dict.get("fingerprint")
if fingerprint in (None, "", 0) or isinstance(fingerprint, dict | list):
from app.simulation.seed import _seed_fingerprint
fingerprint = _seed_fingerprint(name)
def _as_float(value: object, default: float) -> float:
if isinstance(value, bool) or value is None or isinstance(value, dict | list):
return default
try:
return float(str(value))
except (TypeError, ValueError):
return default
def _as_int(value: object, default: int) -> int:
if isinstance(value, bool) or value is None or isinstance(value, dict | list):
return default
try:
return int(float(str(value)))
except (TypeError, ValueError):
return default
item: dict[str, Any] = {
"node": name,
"status": str(row["status"]),
"type": "node",
"ssl_fingerprint": str(fingerprint),
"cpu": _as_float(status_dict.get("cpu"), 0.0),
"maxcpu": _as_int(status_dict.get("maxcpu"), 4),
"mem": _as_int(status_dict.get("mem"), _as_int(status_dict.get("memory"), 0)),
"maxmem": _as_int(status_dict.get("maxmem"), 8 * 1024**3),
"uptime": _as_int(status_dict.get("uptime"), 0),
"level": str(status_dict.get("level") or ""),
}
result.append(item)
return result
async def node_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
from app.handlers.nodes import load_node_ops
node = str(cast(dict[str, Any], inputs["values"])["node"]) node = str(cast(dict[str, Any], inputs["values"])["node"])
row = await _database(request).pool.fetchrow( row = await _database(request).pool.fetchrow(
"SELECT name, status FROM nodes WHERE name=$1", node "SELECT name, status FROM nodes WHERE name=$1", node
) )
if row is None: if row is None:
raise ApiError(404, "node does not exist") raise ApiError(404, "node does not exist")
ops = await load_node_ops(request, node)
status = ops.get("status")
payload = dict(status) if isinstance(status, dict) else {}
mem_used = int(payload.get("mem") or 0)
mem_total = int(payload.get("maxmem") or 8 * 1024**3)
memory = payload.get("memory")
if isinstance(memory, dict):
mem_used = int(memory.get("used") or mem_used)
mem_total = int(memory.get("total") or mem_total)
rootfs = payload.get("rootfs")
if not isinstance(rootfs, dict):
rootfs = {
"used": int(payload.get("rootfs_used") or mem_used // 8),
"total": int(payload.get("rootfs_total") or 100 * 1024**3),
"avail": int(payload.get("rootfs_avail") or 80 * 1024**3),
"free": int(payload.get("rootfs_free") or 80 * 1024**3),
}
cpuinfo = payload.get("cpuinfo")
if not isinstance(cpuinfo, dict):
cpuinfo = {
"cpus": int(payload.get("maxcpu") or 4),
"cores": int(payload.get("maxcpu") or 4),
"sockets": 1,
"model": str(payload.get("cpu_model") or "QEMU Virtual CPU"),
"flags": "",
}
boot_info = payload.get("boot-info")
if not isinstance(boot_info, dict):
boot_info = {"mode": "efi", "secureboot": 0}
current_kernel = payload.get("current-kernel")
if not isinstance(current_kernel, dict):
kversion = str(payload.get("kversion") or "6.8.12-1-pve")
current_kernel = {
"sysname": "Linux",
"release": kversion,
"version": f"#1 SMP PREEMPT_DYNAMIC {kversion}",
"machine": "x86_64",
}
return { return {
"status": str(row["status"]), "status": str(row["status"]),
"node": str(row["name"]), "uptime": int(payload.get("uptime") or 0),
"uptime": 0, "wait": float(payload.get("wait") or 0.0),
"cpu": 0.0, "idle": float(payload.get("idle") or 0.95),
"memory": {"used": 0, "total": 0}, "cpu": float(payload.get("cpu") or 0.0),
"loadavg": list(payload.get("loadavg") or ["0.00", "0.00", "0.00"]),
"memory": {
"used": mem_used,
"total": mem_total,
"free": max(mem_total - mem_used, 0),
"available": max(mem_total - mem_used, 0),
},
"rootfs": rootfs,
"swap": payload.get("swap")
if isinstance(payload.get("swap"), dict)
else {"used": 0, "total": 0, "free": 0},
"cpuinfo": cpuinfo,
"kversion": str(payload.get("kversion") or current_kernel.get("release") or ""),
"pveversion": str(payload.get("pveversion") or "pve-manager/9.2.3"),
"current-kernel": current_kernel,
"boot-info": boot_info,
} }
async def resources(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def resources(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await _database(request).pool.fetch( """Cluster-wide inventory in Proxmox ``/cluster/resources`` shape.
"""SELECT r.kind AS type, r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id Do not dump guest ``state`` / config blobs: QEMU ``cpu`` is a model
ORDER BY r.kind, r.external_id""" string (e.g. ``qemu64``), while this endpoint's ``cpu`` is utilization
) (float). Bridged clients (bpg / pulumi-proxmoxve) decode strictly.
result: list[dict[str, Any]] = [] """
for row in rows:
raw_state = row["state"] def _as_dict(raw: object) -> dict[str, Any]:
state = json.loads(raw_state) if isinstance(raw_state, str) else dict(raw_state) if isinstance(raw, str):
result.append( loaded = json.loads(raw)
{ return dict(loaded) if isinstance(loaded, dict) else {}
"type": str(row["type"]), return dict(raw) if isinstance(raw, dict) else {}
"id": f"{row['type']}/{row['external_id']}",
"node": str(row["node"]), def _num(value: object, default: float | int) -> float | int:
**state, if isinstance(value, bool) or value is None or isinstance(value, dict | list):
} return default
try:
if isinstance(default, float):
return float(str(value))
return int(float(str(value)))
except (TypeError, ValueError):
return default
def _memory_bytes(state: dict[str, Any]) -> int:
raw = state.get("maxmem", state.get("memory"))
if raw in (None, ""):
return 0
value = _num(raw, 0)
# QEMU config stores memory in MiB; cluster resources use bytes.
if isinstance(raw, str) or (isinstance(value, int) and 0 < value < 10_000_000):
return int(value) * 1024 * 1024
return int(value)
def _maxcpu(state: dict[str, Any]) -> int:
cores = int(_num(state.get("cores", state.get("cpus", 1)), 1))
sockets = int(_num(state.get("sockets", 1), 1))
return max(cores * sockets, 1)
def _cpu_util(state: dict[str, Any], *, running: bool) -> float:
if not running:
return 0.0
samples = state.get("rrddata")
if isinstance(samples, list) and samples:
last = samples[-1]
if isinstance(last, dict):
return float(_num(last.get("cpu"), 0.0))
return (
float(_num(state.get("cpu"), 0.0)) if not isinstance(state.get("cpu"), str) else 0.0
) )
type_filter = cast(dict[str, Any], inputs.get("values") or {}).get("type")
result: list[dict[str, Any]] = []
if type_filter in (None, "node"):
node_rows = await _database(request).pool.fetch(
"SELECT name AS node, status FROM nodes ORDER BY name"
)
from app.handlers.nodes import load_node_ops
for row in node_rows:
name = str(row["node"])
ops = await load_node_ops(request, name)
status_payload = ops.get("status")
status_dict = dict(status_payload) if isinstance(status_payload, dict) else {}
mem_raw = status_dict.get("mem", status_dict.get("memory"))
if isinstance(mem_raw, dict):
mem_val = int(_num(mem_raw.get("used"), 0))
maxmem_val = int(
_num(status_dict.get("maxmem", mem_raw.get("total")), 8 * 1024**3)
)
else:
mem_val = int(_num(mem_raw, 0))
maxmem_val = int(_num(status_dict.get("maxmem"), 8 * 1024**3))
result.append(
{
"type": "node",
"id": f"node/{name}",
"node": name,
"status": str(row["status"]),
"cpu": float(_num(status_dict.get("cpu"), 0.0)),
"maxcpu": int(_num(status_dict.get("maxcpu"), 4)),
"mem": mem_val,
"maxmem": maxmem_val,
"uptime": int(_num(status_dict.get("uptime"), 0)),
"level": str(status_dict.get("level") or ""),
}
)
kind_filter: tuple[str, ...] | None
if type_filter == "vm":
kind_filter = ("qemu", "lxc")
elif type_filter == "storage":
kind_filter = ("storage",)
elif type_filter in (None,):
kind_filter = ("qemu", "lxc", "storage", "pool")
elif type_filter in {"qemu", "lxc", "storage", "pool", "sdn"}:
kind_filter = (str(type_filter),)
else:
kind_filter = ()
ha_by_guest: dict[str, str] = {}
if kind_filter and ("qemu" in kind_filter or "lxc" in kind_filter):
ha_rows = await _database(request).pool.fetch(
"SELECT external_id, state FROM resources WHERE kind='ha'"
)
for ha_row in ha_rows:
sid = str(ha_row["external_id"])
ha_state = _as_dict(ha_row["state"])
ha_by_guest[sid] = str(ha_state.get("state") or "started")
pool_by_vmid: dict[str, str] = {}
if kind_filter and ("qemu" in kind_filter or "lxc" in kind_filter or "pool" in kind_filter):
membership = await _database(request).pool.fetch(
"""SELECT p.pool_id AS pool, r.external_id AS vmid
FROM pool_members pm
JOIN pools p ON p.id=pm.pool_id
JOIN resources r ON r.id=pm.resource_id"""
)
for row in membership:
pool_by_vmid[str(row["vmid"])] = str(row["pool"])
node_names = [
str(item["node"])
for item in await _database(request).pool.fetch(
"SELECT name AS node FROM nodes ORDER BY name"
)
]
if kind_filter:
rows = await _database(request).pool.fetch(
"""SELECT r.kind AS type, r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind = ANY($1::text[])
ORDER BY r.kind, r.external_id""",
list(kind_filter),
)
for row in rows:
kind = str(row["type"])
external_id = str(row["external_id"])
node = str(row["node"])
state = _as_dict(row["state"])
if kind in {"qemu", "lxc"}:
status = str(state.get("status") or "stopped")
running = status in {"running", "paused"}
vmid = int(external_id)
sid = f"{'vm' if kind == 'qemu' else 'ct'}:{external_id}"
item: dict[str, Any] = {
"type": kind,
"id": f"{kind}/{external_id}",
"node": node,
"vmid": vmid,
"name": str(state.get("name") or f"{kind}-{external_id}"),
"status": status,
"template": bool(state.get("template") in (True, "1", 1)),
"cpu": _cpu_util(state, running=running),
"maxcpu": _maxcpu(state),
"mem": int(_num(state.get("mem"), 0)) if running else 0,
"maxmem": _memory_bytes(state),
"disk": int(_num(state.get("disk"), 0)),
"maxdisk": int(_num(state.get("maxdisk"), 0)),
"uptime": int(_num(state.get("uptime"), 0)) if running else 0,
}
if sid in ha_by_guest:
item["hastate"] = ha_by_guest[sid]
if external_id in pool_by_vmid:
item["pool"] = pool_by_vmid[external_id]
result.append(item)
elif kind == "storage":
content = state.get("content")
if isinstance(content, list):
content_text = ",".join(str(part) for part in content)
else:
content_text = str(content or "")
shared = bool(_num(state.get("shared"), 0))
# Schema allows one storages row per storage_id; expand to every
# node like a real PVE dump (local + shared storages).
targets = node_names if (shared or len(node_names) > 1) else [node]
if not shared and external_id not in {"local", "local-lvm", "ceph"}:
targets = [node]
for target in targets:
result.append(
{
"type": "storage",
"id": f"storage/{target}/{external_id}",
"node": target,
"storage": external_id,
"status": str(state.get("status") or "available"),
"content": content_text,
"disk": int(_num(state.get("disk"), 0)),
"maxdisk": int(_num(state.get("maxdisk"), 1 * 1024**3)),
"shared": shared,
"plugintype": str(
state.get("plugintype") or state.get("type") or "dir"
),
}
)
elif kind == "pool":
result.append(
{
"type": "pool",
"id": f"pool/{external_id}",
"pool": external_id,
"comment": str(state.get("comment") or ""),
}
)
if type_filter in (None, "sdn"):
metadata = await cluster_metadata(request)
sdn = metadata.get("sdn")
zones = sdn.get("zones") if isinstance(sdn, dict) else None
if isinstance(zones, dict):
for zone_name, zone in sorted(zones.items()):
if not isinstance(zone, dict):
continue
result.append(
{
"type": "sdn",
"id": f"sdn/{zone_name}",
"sdn": zone_name,
"status": str(zone.get("status") or "ok"),
}
)
return result return result
async def node_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
node = str(cast(dict[str, Any], inputs["values"])["node"])
await require_node(request, node)
return subdirs(
"apt",
"ceph",
"disks",
"firewall",
"lxc",
"network",
"qemu",
"services",
"status",
"storage",
"tasks",
"version",
"vzdump",
)
async def node_version(request: Request, inputs: dict[str, Any]) -> dict[str, str]:
node = str(cast(dict[str, Any], inputs["values"])["node"])
await require_node(request, node)
return runtime_version_payload(request)
registry.register("/version", "GET", version) registry.register("/version", "GET", version)
registry.register("/access/ticket", "POST", login) registry.register("/access/ticket", "POST", login)
registry.register("/nodes", "GET", nodes) registry.register("/nodes", "GET", nodes)
registry.register("/nodes/{node}", "GET", node_index)
registry.register("/nodes/{node}/status", "GET", node_status) registry.register("/nodes/{node}/status", "GET", node_status)
registry.register("/nodes/{node}/version", "GET", node_version)
registry.register("/cluster/resources", "GET", resources) registry.register("/cluster/resources", "GET", resources)
register_access_handlers(registry)
register_cluster_handlers(registry)
register_notifications_handlers(registry)
register_mapping_handlers(registry)
register_acme_handlers(registry)
register_cluster_config_handlers(registry)
register_sdn_handlers(registry)
register_storage_handlers(registry)
register_pool_handlers(registry)
register_ceph_handlers(registry)
register_backup_handlers(registry)
register_ha_handlers(registry)
register_node_ops_handlers(registry)
register_firewall_handlers(registry)
register_qemu_handlers(registry) register_qemu_handlers(registry)
register_lxc_handlers(registry)
register_nodes_extra_handlers(registry)
register_cluster_extra_handlers(registry)
register_legacy_aliases(registry)
return registry return registry
+558
View File
@@ -0,0 +1,558 @@
"""Firewall handlers backed by cluster metadata."""
from __future__ import annotations
import json
from collections.abc import Callable
from typing import Any, cast
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import database, require_node, state, subdirs, values
from app.simulation.seed import CLUSTER_ID
ScopeFn = Callable[[dict[str, Any]], str]
async def _load_firewall(request: Request) -> dict[str, Any]:
row = await database(request).pool.fetchrow(
"SELECT metadata FROM clusters WHERE id=$1",
CLUSTER_ID,
)
metadata = state(row["metadata"]) if row is not None else {}
firewall = metadata.get("firewall")
return dict(firewall) if isinstance(firewall, dict) else {}
async def _save_firewall(request: Request, firewall: dict[str, Any]) -> None:
await database(request).pool.execute(
"""UPDATE clusters SET metadata = jsonb_set(
COALESCE(metadata, '{}'::jsonb), '{firewall}', $2::jsonb, true
), updated_at=now() WHERE id=$1""",
CLUSTER_ID,
json.dumps(firewall, sort_keys=True),
)
def _empty_scope() -> dict[str, Any]:
return {
"options": {},
"rules": [],
"aliases": {},
"ipset": {},
"groups": {},
"log": [],
}
def _get_scope(firewall: dict[str, Any], scope: str) -> dict[str, Any] | None:
scopes = firewall.get("scopes")
if not isinstance(scopes, dict):
return None
section = scopes.get(scope)
return cast(dict[str, Any], section) if isinstance(section, dict) else None
def _ensure_scope(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
"""Create an empty durable scope on mutation; never injects catalog defaults."""
scopes = firewall.setdefault("scopes", {})
if not isinstance(scopes, dict):
scopes = {}
firewall["scopes"] = scopes
section = scopes.get(scope)
if not isinstance(section, dict):
section = _empty_scope()
scopes[scope] = section
section.setdefault("options", {})
section.setdefault("rules", [])
section.setdefault("aliases", {})
section.setdefault("ipset", {})
section.setdefault("groups", {})
section.setdefault("log", [])
return cast(dict[str, Any], section)
def _scope_data(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
"""Mutation helper — ensure scope exists without template defaults."""
return _ensure_scope(firewall, scope)
def register_firewall_handlers(registry: HandlerRegistry) -> None:
def register_scope(
base: str,
scope_fn: ScopeFn,
*,
require_node_name: bool = False,
include_macros: bool = False,
include_groups: bool = False,
) -> None:
async def _ready(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
if require_node_name:
await require_node(request, str(payload["node"]))
return payload
async def index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await _ready(request, inputs)
names = ["aliases", "ipset", "log", "options", "refs", "rules"]
if include_groups:
names.insert(2, "groups")
if include_macros:
names.append("macros")
return subdirs(*names)
async def options_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _get_scope(firewall, scope_fn(payload))
if section is None:
return {}
options = section.get("options", {})
return dict(options) if isinstance(options, dict) else {}
async def options_put(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _ensure_scope(firewall, scope_fn(payload))
current = dict(section.get("options") or {})
for key, value in payload.items():
if key in {"node", "vmid", "delete", "digest"}:
continue
current[key] = value
section["options"] = current
await _save_firewall(request, firewall)
async def rules_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
rules = section.get("rules", [])
return list(rules) if isinstance(rules, list) else []
async def rules_create(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _scope_data(firewall, scope_fn(payload))
rules = section.setdefault("rules", [])
if not isinstance(rules, list):
rules = section["rules"] = []
rule = {
key: value for key, value in payload.items() if key not in {"node", "vmid", "pos"}
}
rule["pos"] = len(rules)
rules.append(rule)
await _save_firewall(request, firewall)
async def rule_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
pos = int(values(inputs)["pos"])
rules = await rules_list(request, inputs)
if pos < 0 or pos >= len(rules):
raise ApiError(404, "firewall rule does not exist")
return dict(rules[pos])
async def rule_update(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
pos = int(payload["pos"])
firewall = await _load_firewall(request)
rules = _scope_data(firewall, scope_fn(payload)).setdefault("rules", [])
if not isinstance(rules, list) or pos < 0 or pos >= len(rules):
raise ApiError(404, "firewall rule does not exist")
rules[pos] = {
**rules[pos],
**{k: v for k, v in payload.items() if k not in {"node", "vmid"}},
}
await _save_firewall(request, firewall)
async def rule_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
pos = int(payload["pos"])
firewall = await _load_firewall(request)
rules = _scope_data(firewall, scope_fn(payload)).setdefault("rules", [])
if not isinstance(rules, list) or pos < 0 or pos >= len(rules):
raise ApiError(404, "firewall rule does not exist")
del rules[pos]
for index, rule in enumerate(rules):
if isinstance(rule, dict):
rule["pos"] = index
await _save_firewall(request, firewall)
async def aliases_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _get_scope(firewall, scope_fn(payload))
aliases = section.get("aliases", {}) if section else {}
if not isinstance(aliases, dict):
return []
return [
{"name": name, **{k: v for k, v in data.items() if k != "name"}}
for name, data in sorted(aliases.items())
if isinstance(data, dict)
]
async def aliases_create(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
name = str(payload["name"])
firewall = await _load_firewall(request)
section = _scope_data(firewall, scope_fn(payload))
aliases = section.setdefault("aliases", {})
if name in aliases:
raise ApiError(400, f"alias '{name}' already exists")
aliases[name] = {
"name": name,
"cidr": str(payload.get("cidr") or ""),
"comment": str(payload.get("comment") or ""),
}
await _save_firewall(request, firewall)
async def aliases_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = await _ready(request, inputs)
name = str(payload["name"])
firewall = await _load_firewall(request)
alias = _scope_data(firewall, scope_fn(payload)).get("aliases", {}).get(name)
if not isinstance(alias, dict):
raise ApiError(404, "alias does not exist")
return dict(alias)
async def aliases_update(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
name = str(payload["name"])
firewall = await _load_firewall(request)
aliases = _scope_data(firewall, scope_fn(payload)).setdefault("aliases", {})
if name not in aliases or not isinstance(aliases[name], dict):
raise ApiError(404, "alias does not exist")
current = dict(aliases[name])
if payload.get("rename"):
new_name = str(payload["rename"])
if new_name in aliases and new_name != name:
raise ApiError(400, f"alias '{new_name}' already exists")
del aliases[name]
name = new_name
current["name"] = new_name
for key in ("cidr", "comment"):
if key in payload:
current[key] = payload[key]
aliases[name] = current
await _save_firewall(request, firewall)
async def aliases_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
name = str(payload["name"])
firewall = await _load_firewall(request)
aliases = _scope_data(firewall, scope_fn(payload)).setdefault("aliases", {})
if name not in aliases:
raise ApiError(404, "alias does not exist")
del aliases[name]
await _save_firewall(request, firewall)
async def ipset_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _get_scope(firewall, scope_fn(payload))
ipsets = section.get("ipset", {}) if section else {}
if not isinstance(ipsets, dict):
return []
return [
{"name": name, "comment": data.get("comment", "")}
for name, data in sorted(ipsets.items())
if isinstance(data, dict)
]
async def ipset_create(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
name = str(payload["name"])
firewall = await _load_firewall(request)
ipsets = _scope_data(firewall, scope_fn(payload)).setdefault("ipset", {})
if name in ipsets:
raise ApiError(400, f"ipset '{name}' already exists")
ipsets[name] = {
"name": name,
"comment": str(payload.get("comment") or ""),
"entries": {},
}
await _save_firewall(request, firewall)
async def ipset_get(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
name = str(payload["name"])
firewall = await _load_firewall(request)
ipset = _scope_data(firewall, scope_fn(payload)).get("ipset", {}).get(name)
if not isinstance(ipset, dict):
raise ApiError(404, "ipset does not exist")
entries = ipset.get("entries", {})
if not isinstance(entries, dict):
return []
return [
{"cidr": cidr, **{k: v for k, v in data.items() if k != "cidr"}}
for cidr, data in sorted(entries.items())
if isinstance(data, dict)
]
async def ipset_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
name = str(payload["name"])
firewall = await _load_firewall(request)
ipsets = _scope_data(firewall, scope_fn(payload)).setdefault("ipset", {})
if name not in ipsets:
raise ApiError(404, "ipset does not exist")
del ipsets[name]
await _save_firewall(request, firewall)
async def ipset_entry_create(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
name = str(payload["name"])
cidr = str(payload["cidr"])
firewall = await _load_firewall(request)
ipsets = _scope_data(firewall, scope_fn(payload)).setdefault("ipset", {})
if name not in ipsets or not isinstance(ipsets[name], dict):
raise ApiError(404, "ipset does not exist")
entries = ipsets[name].setdefault("entries", {})
if cidr in entries:
raise ApiError(400, f"ip '{cidr}' already exists in ipset")
entries[cidr] = {
"cidr": cidr,
"comment": str(payload.get("comment") or ""),
"nomatch": int(bool(payload.get("nomatch"))),
}
await _save_firewall(request, firewall)
async def ipset_entry_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = await _ready(request, inputs)
name = str(payload["name"])
cidr = str(payload["cidr"])
firewall = await _load_firewall(request)
ipset = _scope_data(firewall, scope_fn(payload)).get("ipset", {}).get(name)
if not isinstance(ipset, dict):
raise ApiError(404, "ipset does not exist")
entry = ipset.get("entries", {}).get(cidr)
if not isinstance(entry, dict):
raise ApiError(404, "ipset entry does not exist")
return dict(entry)
async def ipset_entry_update(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
name = str(payload["name"])
cidr = str(payload["cidr"])
firewall = await _load_firewall(request)
ipsets = _scope_data(firewall, scope_fn(payload)).setdefault("ipset", {})
if name not in ipsets or not isinstance(ipsets[name], dict):
raise ApiError(404, "ipset does not exist")
entries = ipsets[name].setdefault("entries", {})
if cidr not in entries:
raise ApiError(404, "ipset entry does not exist")
current = dict(entries[cidr])
if "comment" in payload:
current["comment"] = payload["comment"]
if "nomatch" in payload:
current["nomatch"] = int(bool(payload.get("nomatch")))
entries[cidr] = current
await _save_firewall(request, firewall)
async def ipset_entry_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
name = str(payload["name"])
cidr = str(payload["cidr"])
firewall = await _load_firewall(request)
ipsets = _scope_data(firewall, scope_fn(payload)).setdefault("ipset", {})
if name not in ipsets or not isinstance(ipsets[name], dict):
raise ApiError(404, "ipset does not exist")
entries = ipsets[name].setdefault("entries", {})
if cidr not in entries:
raise ApiError(404, "ipset entry does not exist")
del entries[cidr]
await _save_firewall(request, firewall)
async def refs_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
refs: list[dict[str, Any]] = []
for name in section.get("aliases", {}) or {}:
refs.append({"type": "alias", "name": name})
for name in section.get("ipset", {}) or {}:
refs.append({"type": "ipset", "name": name})
return refs
async def log_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
log = section.get("log", [])
return list(log) if isinstance(log, list) else []
async def macros_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
from app.handlers.common import cluster_metadata
metadata = await cluster_metadata(request)
macros = metadata.get("firewall_macros")
if isinstance(macros, list):
return [dict(item) for item in macros if isinstance(item, dict)]
firewall = await _load_firewall(request)
nested = firewall.get("macros")
if isinstance(nested, list):
return [dict(item) for item in nested if isinstance(item, dict)]
return []
async def groups_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _get_scope(firewall, scope_fn(payload))
groups = section.get("groups", {}) if section else {}
if not isinstance(groups, dict):
return []
return [
{"group": name, "comment": data.get("comment", "")}
for name, data in sorted(groups.items())
if isinstance(data, dict)
]
async def groups_create(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
group = str(payload["group"])
firewall = await _load_firewall(request)
groups = _scope_data(firewall, scope_fn(payload)).setdefault("groups", {})
if group in groups:
raise ApiError(400, f"security group '{group}' already exists")
groups[group] = {"comment": str(payload.get("comment") or ""), "rules": []}
await _save_firewall(request, firewall)
async def group_rules(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
group = str(payload["group"])
firewall = await _load_firewall(request)
groups = _scope_data(firewall, scope_fn(payload)).get("groups", {})
if group not in groups or not isinstance(groups[group], dict):
raise ApiError(404, "security group does not exist")
rules = groups[group].get("rules", [])
return list(rules) if isinstance(rules, list) else []
async def group_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
group = str(payload["group"])
firewall = await _load_firewall(request)
groups = _scope_data(firewall, scope_fn(payload)).setdefault("groups", {})
if group not in groups:
raise ApiError(404, "security group does not exist")
del groups[group]
await _save_firewall(request, firewall)
async def group_rule_create(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
group = str(payload["group"])
firewall = await _load_firewall(request)
groups = _scope_data(firewall, scope_fn(payload)).setdefault("groups", {})
if group not in groups or not isinstance(groups[group], dict):
raise ApiError(404, "security group does not exist")
rules = groups[group].setdefault("rules", [])
if not isinstance(rules, list):
rules = groups[group]["rules"] = []
rule = {
key: value
for key, value in payload.items()
if key not in {"node", "vmid", "group", "pos"}
}
rule["pos"] = len(rules)
rules.append(rule)
await _save_firewall(request, firewall)
async def group_rule_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
rules = await group_rules(request, inputs)
pos = int(values(inputs)["pos"])
if pos < 0 or pos >= len(rules):
raise ApiError(404, "firewall rule does not exist")
return dict(rules[pos])
async def group_rule_update(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
group = str(payload["group"])
pos = int(payload["pos"])
firewall = await _load_firewall(request)
groups = _scope_data(firewall, scope_fn(payload)).setdefault("groups", {})
if group not in groups or not isinstance(groups[group], dict):
raise ApiError(404, "security group does not exist")
rules = groups[group].setdefault("rules", [])
if not isinstance(rules, list) or pos < 0 or pos >= len(rules):
raise ApiError(404, "firewall rule does not exist")
rules[pos] = {
**rules[pos],
**{k: v for k, v in payload.items() if k not in {"node", "vmid", "group"}},
}
await _save_firewall(request, firewall)
async def group_rule_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
group = str(payload["group"])
pos = int(payload["pos"])
firewall = await _load_firewall(request)
groups = _scope_data(firewall, scope_fn(payload)).setdefault("groups", {})
if group not in groups or not isinstance(groups[group], dict):
raise ApiError(404, "security group does not exist")
rules = groups[group].setdefault("rules", [])
if not isinstance(rules, list) or pos < 0 or pos >= len(rules):
raise ApiError(404, "firewall rule does not exist")
del rules[pos]
await _save_firewall(request, firewall)
registry.register(base, "GET", index)
registry.register(f"{base}/options", "GET", options_get)
registry.register(f"{base}/options", "PUT", options_put)
registry.register(f"{base}/rules", "GET", rules_list)
registry.register(f"{base}/rules", "POST", rules_create)
registry.register(f"{base}/rules/{{pos}}", "GET", rule_get)
registry.register(f"{base}/rules/{{pos}}", "PUT", rule_update)
registry.register(f"{base}/rules/{{pos}}", "DELETE", rule_delete)
registry.register(f"{base}/aliases", "GET", aliases_list)
registry.register(f"{base}/aliases", "POST", aliases_create)
registry.register(f"{base}/aliases/{{name}}", "GET", aliases_get)
registry.register(f"{base}/aliases/{{name}}", "PUT", aliases_update)
registry.register(f"{base}/aliases/{{name}}", "DELETE", aliases_delete)
registry.register(f"{base}/ipset", "GET", ipset_list)
registry.register(f"{base}/ipset", "POST", ipset_create)
registry.register(f"{base}/ipset/{{name}}", "GET", ipset_get)
registry.register(f"{base}/ipset/{{name}}", "DELETE", ipset_delete)
registry.register(f"{base}/ipset/{{name}}", "POST", ipset_entry_create)
registry.register(f"{base}/ipset/{{name}}/{{cidr}}", "GET", ipset_entry_get)
registry.register(f"{base}/ipset/{{name}}/{{cidr}}", "PUT", ipset_entry_update)
registry.register(f"{base}/ipset/{{name}}/{{cidr}}", "DELETE", ipset_entry_delete)
registry.register(f"{base}/refs", "GET", refs_list)
registry.register(f"{base}/log", "GET", log_list)
if include_macros:
registry.register(f"{base}/macros", "GET", macros_list)
if include_groups:
registry.register(f"{base}/groups", "GET", groups_list)
registry.register(f"{base}/groups", "POST", groups_create)
registry.register(f"{base}/groups/{{group}}", "GET", group_rules)
registry.register(f"{base}/groups/{{group}}", "POST", group_rule_create)
registry.register(f"{base}/groups/{{group}}", "DELETE", group_delete)
registry.register(f"{base}/groups/{{group}}/{{pos}}", "GET", group_rule_get)
registry.register(f"{base}/groups/{{group}}/{{pos}}", "PUT", group_rule_update)
registry.register(f"{base}/groups/{{group}}/{{pos}}", "DELETE", group_rule_delete)
register_scope(
"/cluster/firewall",
lambda _payload: "cluster",
include_macros=True,
include_groups=True,
)
register_scope(
"/nodes/{node}/firewall",
lambda payload: f"node:{payload['node']}",
require_node_name=True,
)
register_scope(
"/nodes/{node}/qemu/{vmid}/firewall",
lambda payload: f"qemu:{payload['node']}:{payload['vmid']}",
require_node_name=True,
)
register_scope(
"/nodes/{node}/lxc/{vmid}/firewall",
lambda payload: f"lxc:{payload['node']}:{payload['vmid']}",
require_node_name=True,
)
+390
View File
@@ -0,0 +1,390 @@
"""High availability semantic handlers."""
from __future__ import annotations
import hashlib
import json
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import (
cluster_metadata,
database,
save_cluster_metadata,
state,
subdirs,
values,
)
from app.simulation.seed import CLUSTER_ID, stable_id
def _ha_groups(metadata: dict[str, Any]) -> dict[str, dict[str, Any]]:
groups = metadata.get("ha_groups", {})
if not isinstance(groups, dict):
return {}
return {str(key): dict(value) for key, value in groups.items() if isinstance(value, dict)}
def _ha_digest(payload: dict[str, Any]) -> str:
material = {key: value for key, value in payload.items() if key != "digest"}
return hashlib.sha1( # noqa: S324 - Proxmox HA digests use SHA-1
json.dumps(material, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
def register_ha_handlers(registry: HandlerRegistry) -> None:
async def ha_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("groups", "resources", "rules", "status")
async def ha_resources(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"""SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind='ha' ORDER BY r.external_id"""
)
result: list[dict[str, Any]] = []
for row in rows:
payload = state(row["state"])
sid = str(row["external_id"])
entry = {
"sid": sid,
"type": "vm" if sid.startswith("vm:") else "ct",
"state": payload.get("state", "started"),
"group": payload.get("group"),
"node": str(row["node"]),
"max_relocate": payload.get("max_relocate", 1),
"max_restart": payload.get("max_restart", 1),
}
entry["digest"] = _ha_digest(entry)
result.append(entry)
return result
async def ha_resource_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
sid = str(values(inputs)["sid"])
items = await ha_resources(request, inputs)
for item in items:
if item["sid"] == sid:
return item
raise ApiError(404, "HA resource does not exist")
async def ha_resource_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
sid = str(payload["sid"])
group = str(payload.get("group") or "")
exists = await database(request).pool.fetchval(
"""SELECT EXISTS(SELECT 1 FROM resources WHERE kind='ha' AND external_id=$1)""",
sid,
)
if exists:
raise ApiError(409, "HA resource already exists")
guest_kind, _, guest_id = sid.partition(":")
if guest_kind not in {"vm", "ct"} or not guest_id.isdigit():
raise ApiError(400, "invalid HA resource sid")
resource_kind = "qemu" if guest_kind == "vm" else "lxc"
guest = await database(request).pool.fetchrow(
"""SELECT r.id, n.name FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind=$1 AND r.external_id=$2""",
resource_kind,
guest_id,
)
if guest is None:
raise ApiError(404, "guest does not exist")
node = await database(request).pool.fetchrow(
"SELECT id FROM nodes WHERE name=$1",
str(guest["name"]),
)
if node is None:
raise ApiError(404, "node does not exist")
ha_state = {
"state": str(payload.get("state") or "started"),
"group": group or None,
"max_relocate": int(payload.get("max_relocate") or 1),
"max_restart": int(payload.get("max_restart") or 1),
}
await database(request).pool.execute(
"""INSERT INTO resources(id, node_id, cluster_id, kind, external_id, state, metadata)
VALUES($1, $2, $3, 'ha', $4, $5::jsonb, '{}'::jsonb)""",
stable_id(f"ha:{sid}"),
node["id"],
CLUSTER_ID,
sid,
json.dumps(ha_state, sort_keys=True),
)
async def ha_resource_update(request: Request, inputs: dict[str, Any]) -> None:
sid = str(values(inputs)["sid"])
payload = values(inputs)
row = await database(request).pool.fetchrow(
"SELECT id, state FROM resources WHERE kind='ha' AND external_id=$1",
sid,
)
if row is None:
raise ApiError(404, "HA resource does not exist")
current = state(row["state"])
updated = {
**current,
**{
key: value
for key, value in payload.items()
if key not in {"sid", "delete", "digest"}
},
}
await database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb, updated_at=now() WHERE id=$1",
row["id"],
json.dumps(updated, sort_keys=True),
)
async def ha_resource_delete(request: Request, inputs: dict[str, Any]) -> None:
sid = str(values(inputs)["sid"])
status = await database(request).pool.execute(
"DELETE FROM resources WHERE kind='ha' AND external_id=$1",
sid,
)
if status != "DELETE 1":
raise ApiError(404, "HA resource does not exist")
async def ha_groups(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
configured = _ha_groups(metadata)
return [
{
"group": group_id,
"nodes": str(payload.get("nodes", "")),
"nofailback": int(payload.get("nofailback", 0)),
"restricted": int(payload.get("restricted", 0)),
"type": "group",
"comment": payload.get("comment", ""),
}
for group_id, payload in sorted(configured.items())
]
async def ha_group_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
group = str(values(inputs)["group"])
for item in await ha_groups(request, inputs):
if item["group"] == group:
return item
raise ApiError(404, "HA group does not exist")
async def ha_group_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
group = str(payload["group"])
metadata = await cluster_metadata(request)
groups = _ha_groups(metadata)
if group in groups:
raise ApiError(409, "HA group already exists")
groups[group] = {
"nodes": str(payload.get("nodes") or ""),
"nofailback": int(payload.get("nofailback") or 0),
"restricted": int(payload.get("restricted") or 0),
"comment": str(payload.get("comment") or ""),
}
metadata["ha_groups"] = groups
await save_cluster_metadata(request, metadata)
async def ha_group_update(request: Request, inputs: dict[str, Any]) -> None:
group = str(values(inputs)["group"])
metadata = await cluster_metadata(request)
groups = _ha_groups(metadata)
if group not in groups:
raise ApiError(404, "HA group does not exist")
payload = values(inputs)
groups[group] = {
**groups[group],
**{
key: value
for key, value in payload.items()
if key not in {"group", "delete", "digest"}
},
}
metadata["ha_groups"] = groups
await save_cluster_metadata(request, metadata)
async def ha_group_delete(request: Request, inputs: dict[str, Any]) -> None:
group = str(values(inputs)["group"])
metadata = await cluster_metadata(request)
groups = _ha_groups(metadata)
if group not in groups:
raise ApiError(404, "HA group does not exist")
del groups[group]
metadata["ha_groups"] = groups
await save_cluster_metadata(request, metadata)
async def ha_status(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("current", "manager_status")
async def ha_status_current(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
master = await database(request).pool.fetchval(
"SELECT name FROM nodes WHERE status='online' ORDER BY name LIMIT 1"
)
metadata = await cluster_metadata(request)
ha_raw = metadata.get("ha")
ha: dict[str, Any] = dict(ha_raw) if isinstance(ha_raw, dict) else {}
armed = bool(ha.get("armed")) if "armed" in ha else False
status_raw = ha.get("status_current")
status: dict[str, Any] = dict(status_raw) if isinstance(status_raw, dict) else {}
quorate = bool(int(status.get("quorate", metadata.get("quorate", 1)) or 1))
master_node = str(status.get("master_node") or master or "")
rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind='ha' ORDER BY r.external_id"""
)
nodes = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
result: list[dict[str, Any]] = [
{
"id": "quorum",
"type": "quorum",
"node": master_node,
"status": "OK" if quorate else "offline",
"quorate": quorate,
},
{
"id": "master",
"type": "master",
"node": master_node,
"status": "active" if armed else "disabled",
},
{
"id": "fencing",
"type": "fencing",
"node": master_node,
"status": "armed" if armed else "disarmed",
"armed-state": "armed" if armed else "disarmed",
},
]
for node_row in nodes:
node_name = str(node_row["name"])
result.append(
{
"id": f"lrm:{node_name}",
"type": "lrm",
"node": node_name,
"status": "active" if quorate else "offline",
}
)
for row in rows:
payload = state(row["state"])
sid = str(row["external_id"])
crm = str(payload.get("state") or "started")
result.append(
{
"id": f"service:{sid}",
"type": "service",
"sid": sid,
"node": str(row["node"]),
"crm_state": crm,
"state": crm,
"status": crm,
"request_state": str(payload.get("request_state") or crm),
"max_relocate": int(payload.get("max_relocate") or 1),
"max_restart": int(payload.get("max_restart") or 1),
}
)
return result
async def ha_manager_status(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
ha_raw = metadata.get("ha")
ha: dict[str, Any] = dict(ha_raw) if isinstance(ha_raw, dict) else {}
manager_raw = ha.get("manager_status")
manager: dict[str, Any] = dict(manager_raw) if isinstance(manager_raw, dict) else {}
armed = bool(ha.get("armed")) if "armed" in ha else False
return {
"manager_status": manager.get("manager_status", "active" if armed else "disabled"),
"quorum": manager.get("quorum", ""),
"armed": 1 if armed else 0,
}
async def ha_rules(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
rules = metadata.get("ha_rules")
if isinstance(rules, list):
return [dict(item) for item in rules if isinstance(item, dict)]
return []
async def ha_relocate(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
sid = str(payload["sid"])
target = str(payload.get("node") or payload.get("target") or "")
if not target:
raise ApiError(400, "parameter verification failed - target node missing")
ha_row = await database(request).pool.fetchrow(
"SELECT id, state FROM resources WHERE kind='ha' AND external_id=$1",
sid,
)
if ha_row is None:
raise ApiError(404, "HA resource does not exist")
node = await database(request).pool.fetchrow(
"SELECT id, name FROM nodes WHERE name=$1",
target,
)
if node is None:
raise ApiError(404, "node does not exist")
guest_kind, _, guest_id = sid.partition(":")
resource_kind = "qemu" if guest_kind == "vm" else "lxc"
await database(request).pool.execute(
"UPDATE resources SET node_id=$2, updated_at=now() WHERE kind='ha' AND external_id=$1",
sid,
node["id"],
)
await database(request).pool.execute(
"""UPDATE resources SET node_id=$3, updated_at=now()
WHERE kind=$1 AND external_id=$2""",
resource_kind,
guest_id,
node["id"],
)
current = state(ha_row["state"])
current["node"] = target
current["state"] = current.get("state") or "started"
await database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb, updated_at=now() WHERE id=$1",
ha_row["id"],
json.dumps(current, sort_keys=True),
)
return {
"sid": sid,
"requested-node": target,
"comigrated-resources": [],
}
async def ha_migrate(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await ha_relocate(request, inputs)
async def ha_arm(request: Request, _inputs: dict[str, Any]) -> None:
metadata = await cluster_metadata(request)
ha = dict(metadata.get("ha") or {})
ha["armed"] = True
metadata["ha"] = ha
await save_cluster_metadata(request, metadata)
async def ha_disarm(request: Request, _inputs: dict[str, Any]) -> None:
metadata = await cluster_metadata(request)
ha = dict(metadata.get("ha") or {})
ha["armed"] = False
metadata["ha"] = ha
await save_cluster_metadata(request, metadata)
registry.register("/cluster/ha", "GET", ha_index)
registry.register("/cluster/ha/groups", "GET", ha_groups)
registry.register("/cluster/ha/groups", "POST", ha_group_create)
registry.register("/cluster/ha/groups/{group}", "GET", ha_group_get)
registry.register("/cluster/ha/groups/{group}", "PUT", ha_group_update)
registry.register("/cluster/ha/groups/{group}", "DELETE", ha_group_delete)
registry.register("/cluster/ha/resources", "GET", ha_resources)
registry.register("/cluster/ha/resources", "POST", ha_resource_create)
registry.register("/cluster/ha/resources/{sid}", "GET", ha_resource_get)
registry.register("/cluster/ha/resources/{sid}", "PUT", ha_resource_update)
registry.register("/cluster/ha/resources/{sid}", "DELETE", ha_resource_delete)
registry.register("/cluster/ha/status", "GET", ha_status)
registry.register("/cluster/ha/status/current", "GET", ha_status_current)
registry.register("/cluster/ha/status/manager_status", "GET", ha_manager_status)
registry.register("/cluster/ha/rules", "GET", ha_rules)
registry.register("/cluster/ha/resources/{sid}/migrate", "POST", ha_migrate)
registry.register("/cluster/ha/resources/{sid}/relocate", "POST", ha_relocate)
registry.register("/cluster/ha/status/arm-ha", "POST", ha_arm)
registry.register("/cluster/ha/status/disarm-ha", "POST", ha_disarm)
+59
View File
@@ -0,0 +1,59 @@
"""Legacy Proxmox path aliases for older contract snapshots."""
from __future__ import annotations
from app.api.registry import HandlerRegistry
def register_legacy_aliases(registry: HandlerRegistry) -> None:
"""Register older-path synonyms onto already-registered handlers when present."""
def alias(old_path: str, old_verb: str, new_path: str, new_verb: str | None = None) -> None:
verb = (new_verb or old_verb).upper()
handler = registry.get(new_path, verb)
if handler is None:
return
if registry.get(old_path, old_verb) is not None:
return
registry.register(old_path, old_verb.upper(), handler)
alias("/access/tfa", "POST", "/access/tfa/{userid}", "POST")
alias("/access/tfa", "PUT", "/access/tfa/{userid}/{id}", "PUT")
# /cluster/backupinfo is registered as a PVE 6 stub string in backup.py
alias(
"/cluster/backupinfo/not_backed_up",
"GET",
"/cluster/backup-info/not-backed-up",
"GET",
)
alias("/nodes/{node}/ceph/config", "GET", "/nodes/{node}/ceph/cfg/raw", "GET")
alias("/nodes/{node}/ceph/configdb", "GET", "/nodes/{node}/ceph/cfg/db", "GET")
alias("/nodes/{node}/ceph/disks", "GET", "/nodes/{node}/ceph/osd", "GET")
alias("/nodes/{node}/ceph/flags", "GET", "/cluster/ceph/flags", "GET")
alias("/nodes/{node}/ceph/flags/{flag}", "POST", "/cluster/ceph/flags/{flag}", "PUT")
alias("/nodes/{node}/ceph/flags/{flag}", "DELETE", "/cluster/ceph/flags/{flag}", "PUT")
alias("/nodes/{node}/ceph/pools", "GET", "/nodes/{node}/ceph/pool", "GET")
alias("/nodes/{node}/ceph/pools", "POST", "/nodes/{node}/ceph/pool", "POST")
alias("/nodes/{node}/ceph/pools/{name}", "GET", "/nodes/{node}/ceph/pool/{name}", "GET")
alias("/nodes/{node}/ceph/pools/{name}", "PUT", "/nodes/{node}/ceph/pool/{name}", "PUT")
alias(
"/nodes/{node}/ceph/pools/{name}",
"DELETE",
"/nodes/{node}/ceph/pool/{name}",
"DELETE",
)
alias("/nodes/{node}/cpu", "GET", "/nodes/{node}/capabilities/qemu/cpu", "GET")
alias(
"/nodes/{node}/hardware/pci/{pciid}",
"GET",
"/nodes/{node}/hardware/pci/{pci-id-or-mapping}",
"GET",
)
alias(
"/nodes/{node}/hardware/pci/{pciid}/mdev",
"GET",
"/nodes/{node}/hardware/pci/{pci-id-or-mapping}/mdev",
"GET",
)
alias("/nodes/{node}/scan/glusterfs", "GET", "/nodes/{node}/scan/nfs", "GET")
alias("/nodes/{node}/scan/usb", "GET", "/nodes/{node}/hardware/usb", "GET")
+682
View File
@@ -0,0 +1,682 @@
"""Persistent LXC semantic handlers."""
from __future__ import annotations
import json
from collections.abc import Mapping
from typing import Any, cast
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.db.pool import AsyncpgDatabase
from app.db.primitives import ConflictError
from app.handlers.common import (
disk_size_bytes,
replace_disk_size,
require_node,
resize_size_bytes,
subdirs,
)
from app.simulation.transitions import InvalidTransitionError, VmState, plan_transition
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
def _database(request: Request) -> AsyncpgDatabase:
return cast(AsyncpgDatabase, request.app.state.database)
def _values(inputs: dict[str, Any]) -> dict[str, Any]:
return cast(dict[str, Any], inputs["values"])
def _state(value: object) -> dict[str, Any]:
if isinstance(value, str):
return cast(dict[str, Any], json.loads(value))
return dict(cast(Mapping[str, Any], value))
# Runtime-only keys stored in ``resources.state``; never echo on GET /config.
_LXC_CONFIG_INTERNAL = frozenset(
{
"agent",
"cloudinit_dump",
"config",
"consoles",
"interfaces",
"migrate_preconditions",
"name",
"rrd",
"rrddata",
"status",
"volume_moves",
}
)
def _public_lxc_config(
config: dict[str, Any], state: dict[str, Any], *, vmid: int
) -> dict[str, Any]:
import hashlib
merged = {**state, **config}
payload: dict[str, Any] = {"vmid": vmid}
for key, value in merged.items():
if key in _LXC_CONFIG_INTERNAL:
continue
if isinstance(value, dict | list):
continue
payload[key] = value
digest_src = {key: value for key, value in payload.items() if key != "digest"}
payload["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(digest_src, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
return payload
def _lxc_index_item(vmid: int, vm_state: dict[str, Any], config: dict[str, Any]) -> dict[str, Any]:
status = str(vm_state.get("status", "stopped"))
running = status in {"running", "paused"}
memory_mb = int(config.get("memory", config.get("mem", 512)))
maxmem = memory_mb * 2**20
return {
"vmid": vmid,
"name": str(config.get("hostname", config.get("name", f"ct-{vmid}"))),
"status": status,
"cpus": int(config.get("cores", config.get("cpus", 1))),
"cpu": float(vm_state.get("cpu", 0.05 if running else 0.0)),
"maxmem": maxmem,
"mem": int(vm_state.get("mem", maxmem // 2 if running else 0)),
"maxdisk": int(vm_state.get("maxdisk", config.get("maxdisk", 8 * 1024**3))),
"disk": int(vm_state.get("disk", 0)),
"diskread": int(vm_state.get("diskread", 0)),
"diskwrite": int(vm_state.get("diskwrite", 0)),
"netin": int(vm_state.get("netin", 0)),
"netout": int(vm_state.get("netout", 0)),
"maxswap": int(config.get("swap", 512)) * 2**20,
"uptime": int(vm_state.get("uptime", 0)) if running else 0,
"lock": str(vm_state.get("lock", "")),
"tags": str(config.get("tags", "")),
"template": int(bool(vm_state.get("template", False))),
"pressurecpusome": float(vm_state.get("pressurecpusome", 0.0)),
"pressurecpufull": float(vm_state.get("pressurecpufull", 0.0)),
"pressureiosome": float(vm_state.get("pressureiosome", 0.0)),
"pressureiofull": float(vm_state.get("pressureiofull", 0.0)),
"pressurememorysome": float(vm_state.get("pressurememorysome", 0.0)),
"pressurememoryfull": float(vm_state.get("pressurememoryfull", 0.0)),
}
def register_lxc_handlers(registry: HandlerRegistry) -> None:
async def lxc_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(_values(inputs)["node"])
rows = await _database(request).pool.fetch(
"""SELECT r.external_id::integer AS vmid, r.state, c.config
FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN containers c ON c.resource_id=r.id
WHERE n.name=$1 AND r.kind='lxc' ORDER BY r.external_id::integer""",
node,
)
return [
_lxc_index_item(int(row["vmid"]), _state(row["state"]), _state(row["config"]))
for row in rows
]
async def lxc_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"])
row = await _lxc_resource(request, node, vmid)
return _public_lxc_config(_state(row["config"]), _state(row["state"]), vmid=int(vmid))
async def lxc_current(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
node, vmid = str(payload["node"]), int(payload["vmid"])
row = await _lxc_resource(request, node, str(vmid))
vm_state = _state(row["state"])
config = _state(row["config"])
item = _lxc_index_item(vmid, vm_state, config)
item["ha"] = {"managed": int(vm_state.get("ha_managed", 0))}
return item
async def lxc_status(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs)
await _lxc_resource(request, str(payload["node"]), str(payload["vmid"]))
return subdirs(
"current",
"reboot",
"resume",
"shutdown",
"start",
"stop",
"suspend",
)
async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid = str(values["node"]), str(values["vmid"])
database = _database(request)
row = await database.pool.fetchrow(
"""SELECT r.id, r.state FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='lxc' AND r.external_id=$2""",
node,
vmid,
)
if row is None:
raise ApiError(404, "container does not exist")
current = str(_state(row["state"]).get("status", "stopped"))
try:
plan_transition(VmState(current), operation)
except (InvalidTransitionError, ValueError) as error:
raise ApiError(409, f"cannot {operation} container while it is {current}") from error
upid = str(Upid.allocate(node, f"pct{operation}", vmid, str(request.state.principal)))
try:
task = await TaskRepository(database.pool).create(
upid=upid,
task_type=f"lxc-{operation}",
payload={"node": node, "vmid": vmid, "resource_id": str(row["id"])},
resource_key=f"lxc:{vmid}",
idempotency_key=request.headers.get("Idempotency-Key"),
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
async def start(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("start", request, inputs)
async def stop(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("stop", request, inputs)
async def shutdown(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("shutdown", request, inputs)
async def reboot(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("reboot", request, inputs)
async def suspend(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("suspend", request, inputs)
async def resume(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("resume", request, inputs)
async def create(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid = str(values["node"]), int(values["vmid"])
database = _database(request)
if not await database.pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", node
):
raise ApiError(404, "node does not exist")
if await database.pool.fetchval(
"""SELECT EXISTS(SELECT 1 FROM resources
WHERE external_id=$1 AND kind IN ('qemu','lxc'))""",
str(vmid),
):
raise ApiError(409, "VMID already exists")
config = {
key: value
for key, value in values.items()
if key not in {"node", "vmid", "force", "start", "ostemplate"}
}
if "ostemplate" in values:
config["ostemplate"] = values["ostemplate"]
return await _create_task(
request,
node=node,
vmid=str(vmid),
task_type="lxc-create",
payload={
"node": node,
"vmid": vmid,
"config": config,
"start": bool(values.get("start")),
},
)
async def update(request: Request, inputs: dict[str, Any]) -> None:
values = _values(inputs)
node, vmid = str(values["node"]), str(values["vmid"])
database = _database(request)
row = await database.pool.fetchrow(
"""SELECT r.id, r.version, r.state, c.config FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN containers c ON c.resource_id=r.id
WHERE n.name=$1 AND r.kind='lxc' AND r.external_id=$2""",
node,
vmid,
)
if row is None:
raise ApiError(404, "container does not exist")
control = {"node", "vmid", "digest", "delete", "revert", "skiplock"}
provided = frozenset(str(item) for item in inputs.get("provided", values))
changes = {
key: value for key, value in values.items() if key in provided and key not in control
}
delete = str(values.get("delete", "")) if "delete" in provided else ""
state = _state(row["state"])
config = _state(row["config"])
state.update(changes)
config.update(changes)
for key in delete.split(","):
if key:
state.pop(key, None)
config.pop(key, None)
status = await database.pool.execute(
"""UPDATE resources SET state=$3::jsonb, version=version+1,
updated_at=now() WHERE id=$1 AND version=$2""",
row["id"],
row["version"],
json.dumps(state, sort_keys=True),
)
if status != "UPDATE 1":
raise ApiError(409, "configuration changed concurrently")
await database.pool.execute(
"UPDATE containers SET config=$2::jsonb WHERE resource_id=$1",
row["id"],
json.dumps(config, sort_keys=True),
)
async def delete(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid = str(values["node"]), str(values["vmid"])
database = _database(request)
row = await database.pool.fetchrow(
"""SELECT r.id, r.state FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='lxc' AND r.external_id=$2""",
node,
vmid,
)
if row is None:
raise ApiError(404, "container does not exist")
if str(_state(row["state"]).get("status")) != "stopped":
raise ApiError(409, "cannot delete a running container")
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-delete",
payload={"node": node, "vmid": vmid, "resource_id": str(row["id"])},
)
async def snapshot_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
rows = await _database(request).pool.fetch(
"""SELECT name, parent_name, description, created_at FROM snapshots
WHERE resource_id=$1 ORDER BY created_at, name""",
resource["id"],
)
return [
{
"name": row["name"],
"parent": row["parent_name"],
"description": row["description"] or "",
"snaptime": int(row["created_at"].timestamp()),
}
for row in rows
]
async def snapshot_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
row = await _snapshot(request, values)
state = _state(row["state"])
return {
"name": row["name"],
"parent": row["parent_name"],
"description": row["description"] or "",
"snaptime": int(row["created_at"].timestamp()),
**state,
}
async def snapshot_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
row = await _snapshot(request, _values(inputs))
return {"description": row["description"] or "", **_state(row["state"])}
async def snapshot_update(request: Request, inputs: dict[str, Any]) -> None:
values = _values(inputs)
row = await _snapshot(request, values)
await _database(request).pool.execute(
"UPDATE snapshots SET description=$2 WHERE id=$1",
row["id"],
str(values.get("description", "")),
)
async def snapshot_task(operation: str, request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid, snapname = (
str(values["node"]),
str(values["vmid"]),
str(values["snapname"]),
)
resource = await _lxc_resource(request, node, vmid)
if operation == "snapshot-create":
exists = await _database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM snapshots WHERE resource_id=$1 AND name=$2)",
resource["id"],
snapname,
)
if exists:
raise ApiError(409, "snapshot already exists")
else:
await _snapshot(request, values)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type=f"lxc-{operation}",
payload={
"node": node,
"vmid": vmid,
"resource_id": str(resource["id"]),
"snapname": snapname,
"description": str(values.get("description", "")),
},
)
async def snapshot_create(request: Request, inputs: dict[str, Any]) -> str:
return await snapshot_task("snapshot-create", request, inputs)
async def snapshot_delete(request: Request, inputs: dict[str, Any]) -> str:
return await snapshot_task("snapshot-delete", request, inputs)
async def snapshot_rollback(request: Request, inputs: dict[str, Any]) -> str:
return await snapshot_task("snapshot-rollback", request, inputs)
async def clone(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid, newid = str(values["node"]), str(values["vmid"]), str(values["newid"])
source = await _lxc_resource(request, node, vmid)
if await _database(request).pool.fetchval(
"""SELECT EXISTS(SELECT 1 FROM resources
WHERE external_id=$1 AND kind IN ('qemu','lxc'))""",
newid,
):
raise ApiError(409, "VMID already exists")
target = str(values.get("target") or node)
if not await _database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", target
):
raise ApiError(404, "target node does not exist")
return await _create_task(
request,
node=target,
vmid=newid,
task_type="lxc-clone",
payload={
"source_resource_id": str(source["id"]),
"source_vmid": vmid,
"node": target,
"vmid": int(newid),
"name": values.get("hostname") or values.get("name"),
"full": bool(values.get("full", False)),
},
)
async def migrate_preconditions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
target = values.get("target")
if target in {None, ""}:
raise ApiError(400, "parameter 'target' is required")
target = str(target)
exists = await _database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", target
)
if not exists:
raise ApiError(404, "target node does not exist")
state = _state(resource["state"])
pre = state.get("migrate_preconditions")
payload = dict(pre) if isinstance(pre, dict) else {}
payload["running"] = str(state.get("status") or "") == "running"
payload.setdefault("local_disks", [])
payload.setdefault("local_resources", [])
return payload
async def migrate(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid = str(values["node"]), str(values["vmid"])
target = values.get("target")
if target in {None, ""}:
raise ApiError(400, "parameter 'target' is required")
target = str(target)
resource = await _lxc_resource(request, node, vmid)
if target == node:
raise ApiError(400, "target node is the same as source node")
await migrate_preconditions(request, inputs)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-migrate",
payload={
"resource_id": str(resource["id"]),
"node": node,
"target": target,
"vmid": vmid,
"online": bool(values.get("online", False)),
},
)
async def remote_migrate(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
target_endpoint = str(values.get("target-endpoint") or values.get("target_endpoint") or "")
target = str(values.get("target") or "")
if not target_endpoint:
raise ApiError(400, "parameter target-endpoint is required")
if not target:
raise ApiError(400, "parameter target is required")
node, vmid = str(values["node"]), str(values["vmid"])
resource = await _lxc_resource(request, node, vmid)
if target == node:
raise ApiError(400, "target node is the same as source node")
if not await _database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", target
):
raise ApiError(404, "target node does not exist")
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-remote-migrate",
payload={
"resource_id": str(resource["id"]),
"node": node,
"target": target,
"vmid": vmid,
"target-endpoint": target_endpoint,
"online": bool(values.get("online", False)),
},
)
async def pending(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
config = _state(resource["config"])
changes = cast(Mapping[str, Any], state.get("pending", {}))
return [
{"key": key, "value": str(config.get(key, "")), "pending": str(value)}
for key, value in sorted(changes.items())
]
async def lxc_feature(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
await _lxc_resource(request, str(payload["node"]), str(payload["vmid"]))
return {
"hasFeature": {
"snapshot": 1,
"clone": 1,
"copy": 1,
"template": 1,
"move_volume": 1,
}
}
async def lxc_resize(request: Request, inputs: dict[str, Any]) -> str:
payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"])
disk = str(payload.get("disk") or "rootfs")
resource = await _lxc_resource(request, node, vmid)
config = _state(resource["config"])
if disk not in config:
raise ApiError(400, f"disk {disk} does not exist")
try:
current = disk_size_bytes(str(config[disk]))
size = resize_size_bytes(str(payload["size"]), current)
except ValueError as error:
raise ApiError(400, str(error)) from error
config[disk] = replace_disk_size(str(config[disk]), size)
await _database(request).pool.execute(
"UPDATE containers SET config=$2::jsonb WHERE resource_id=$1",
resource["id"],
json.dumps(config, sort_keys=True),
)
await _database(request).pool.execute(
"""UPDATE resources SET state=state || $2::jsonb, version=version+1,
updated_at=now() WHERE id=$1""",
resource["id"],
json.dumps({disk: config[disk]}, sort_keys=True),
)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-resize",
payload={"resource_id": str(resource["id"]), "disk": disk, "size": size},
)
async def lxc_template(request: Request, inputs: dict[str, Any]) -> None:
payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"])
resource = await _lxc_resource(request, node, vmid)
state = _state(resource["state"])
if state.get("status") != "stopped":
raise ApiError(409, "container must be stopped to convert to template")
await _database(request).pool.execute(
"UPDATE containers SET template=true WHERE resource_id=$1",
resource["id"],
)
state["template"] = True
await _database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb WHERE id=$1",
resource["id"],
json.dumps(state, sort_keys=True),
)
async def lxc_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"])
await require_node(request, node)
await _lxc_resource(request, node, vmid)
return subdirs(
"clone",
"config",
"feature",
"firewall",
"migrate",
"pending",
"resize",
"snapshot",
"status",
"template",
)
registry.register("/nodes/{node}/lxc", "GET", lxc_list)
registry.register("/nodes/{node}/lxc", "POST", create)
registry.register("/nodes/{node}/lxc/{vmid}", "GET", lxc_index)
registry.register("/nodes/{node}/lxc/{vmid}", "DELETE", delete)
registry.register("/nodes/{node}/lxc/{vmid}/config", "GET", lxc_config)
registry.register("/nodes/{node}/lxc/{vmid}/config", "PUT", update)
registry.register("/nodes/{node}/lxc/{vmid}/status", "GET", lxc_status)
registry.register("/nodes/{node}/lxc/{vmid}/status/current", "GET", lxc_current)
registry.register("/nodes/{node}/lxc/{vmid}/status/start", "POST", start)
registry.register("/nodes/{node}/lxc/{vmid}/status/stop", "POST", stop)
registry.register("/nodes/{node}/lxc/{vmid}/status/shutdown", "POST", shutdown)
registry.register("/nodes/{node}/lxc/{vmid}/status/reboot", "POST", reboot)
registry.register("/nodes/{node}/lxc/{vmid}/status/suspend", "POST", suspend)
registry.register("/nodes/{node}/lxc/{vmid}/status/resume", "POST", resume)
registry.register("/nodes/{node}/lxc/{vmid}/snapshot", "GET", snapshot_list)
registry.register("/nodes/{node}/lxc/{vmid}/snapshot", "POST", snapshot_create)
registry.register("/nodes/{node}/lxc/{vmid}/snapshot/{snapname}", "GET", snapshot_get)
registry.register("/nodes/{node}/lxc/{vmid}/snapshot/{snapname}", "DELETE", snapshot_delete)
registry.register("/nodes/{node}/lxc/{vmid}/snapshot/{snapname}/config", "GET", snapshot_config)
registry.register("/nodes/{node}/lxc/{vmid}/snapshot/{snapname}/config", "PUT", snapshot_update)
registry.register(
"/nodes/{node}/lxc/{vmid}/snapshot/{snapname}/rollback", "POST", snapshot_rollback
)
registry.register("/nodes/{node}/lxc/{vmid}/clone", "POST", clone)
registry.register("/nodes/{node}/lxc/{vmid}/migrate", "GET", migrate_preconditions)
registry.register("/nodes/{node}/lxc/{vmid}/migrate", "POST", migrate)
registry.register("/nodes/{node}/lxc/{vmid}/remote_migrate", "POST", remote_migrate)
registry.register("/nodes/{node}/lxc/{vmid}/pending", "GET", pending)
registry.register("/nodes/{node}/lxc/{vmid}/feature", "GET", lxc_feature)
registry.register("/nodes/{node}/lxc/{vmid}/resize", "PUT", lxc_resize)
registry.register("/nodes/{node}/lxc/{vmid}/template", "POST", lxc_template)
from app.handlers.lxc_extra import register_lxc_extra_handlers
register_lxc_extra_handlers(registry)
async def _create_task(
request: Request,
*,
node: str,
vmid: str,
task_type: str,
payload: dict[str, Any],
) -> str:
database = _database(request)
worker_type = {
"lxc-create": "pctcreate",
"lxc-delete": "pctdestroy",
"lxc-snapshot-create": "pctsnapshot",
"lxc-snapshot-delete": "pctdelsnapshot",
"lxc-snapshot-rollback": "pctrollback",
"lxc-clone": "pctclone",
"lxc-migrate": "pctmigrate",
"lxc-remote-migrate": "pctremote",
"lxc-resize": "pctresize",
"lxc-move-volume": "pctmove",
}[task_type]
upid = str(Upid.allocate(node, worker_type, vmid, str(request.state.principal)))
try:
task = await TaskRepository(database.pool).create(
upid=upid,
task_type=task_type,
payload=payload,
resource_key=f"lxc:{vmid}",
idempotency_key=request.headers.get("Idempotency-Key"),
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
async def _lxc_resource(request: Request, node: str, vmid: str) -> Any:
row = await _database(request).pool.fetchrow(
"""SELECT r.id, r.state, c.config FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN containers c ON c.resource_id=r.id
WHERE n.name=$1 AND r.kind='lxc' AND r.external_id=$2""",
node,
vmid,
)
if row is None:
raise ApiError(404, "container does not exist")
return row
async def _snapshot(request: Request, values: dict[str, Any]) -> Any:
row = await _database(request).pool.fetchrow(
"""SELECT s.* FROM snapshots s
JOIN resources r ON r.id=s.resource_id JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='lxc' AND r.external_id=$2 AND s.name=$3""",
str(values["node"]),
str(values["vmid"]),
str(values["snapname"]),
)
if row is None:
raise ApiError(404, "snapshot does not exist")
return row
+153
View File
@@ -0,0 +1,153 @@
"""Remaining LXC console / RRD / volume helpers with durable state."""
from __future__ import annotations
import json
import secrets
from typing import Any, cast
from fastapi import Request
from app.api.registry import HandlerRegistry
from app.config import Settings
from app.handlers.lxc import _database, _lxc_resource, _state, _values
from app.security.auth import issue_ticket
def _settings(request: Request) -> Settings:
return cast(Settings, request.app.state.settings)
async def _save_state(request: Request, resource_id: Any, state: dict[str, Any]) -> None:
await _database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb, version=version+1, updated_at=now() WHERE id=$1",
resource_id,
json.dumps(state, sort_keys=True),
)
async def _save_config(request: Request, resource_id: Any, config: dict[str, Any]) -> None:
await _database(request).pool.execute(
"UPDATE containers SET config=$2::jsonb WHERE resource_id=$1",
resource_id,
json.dumps(config, sort_keys=True),
)
def register_lxc_extra_handlers(registry: HandlerRegistry) -> None:
async def interfaces(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
ifaces = state.get("interfaces")
return [dict(item) for item in ifaces] if isinstance(ifaces, list) else []
async def move_volume(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.lxc import _create_task
values = _values(inputs)
node, vmid = str(values["node"]), str(values["vmid"])
resource = await _lxc_resource(request, node, vmid)
volume = str(values.get("volume") or values.get("disk") or "rootfs")
storage = str(values.get("storage") or "local-lvm")
config = _state(resource["config"])
current = str(config.get(volume) or "")
if current:
# rewrite storage prefix when present
rest = current.split(":", 1)[1] if ":" in current else current
config[volume] = f"{storage}:{rest}"
await _save_config(request, resource["id"], config)
state = _state(resource["state"])
moves = state.setdefault("volume_moves", [])
if not isinstance(moves, list):
moves = state["volume_moves"] = []
moves.append({"volume": volume, "storage": storage})
await _save_state(request, resource["id"], state)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-move-volume",
payload={
"resource_id": str(resource["id"]),
"volume": volume,
"storage": storage,
},
)
async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
rrd_state = state.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
series = state.get("rrddata")
return [dict(item) for item in series] if isinstance(series, list) else []
async def _console(request: Request, inputs: dict[str, Any], kind: str) -> dict[str, Any]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
key = _settings(request).ticket_signing_key.get_secret_value().encode()
ticket = issue_ticket(str(request.state.principal), key)
port = 6900 + int(values["vmid"]) % 1000
state = _state(resource["state"])
consoles = state.setdefault("consoles", {})
payload = {
"type": kind,
"port": port,
"ticket": ticket,
"upid": (
f"UPID:{values['node']}:{secrets.token_hex(4)}:"
f"{kind}:{values['vmid']}:{request.state.principal}:"
),
"user": str(request.state.principal),
}
consoles[kind] = {k: v for k, v in payload.items() if k != "ticket"}
await _save_state(request, resource["id"], state)
return payload
async def vncproxy(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _console(request, inputs, "vnc")
async def spiceproxy(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _console(request, inputs, "spice")
async def termproxy(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _console(request, inputs, "term")
async def mtunnel(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _console(request, inputs, "mtunnel")
async def _ws(request: Request, inputs: dict[str, Any], kind: str) -> dict[str, Any]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
console = state.get("consoles", {}).get(kind) or {"port": 6900}
key = _settings(request).ticket_signing_key.get_secret_value().encode()
return {
"port": console.get("port", 6900),
"ticket": issue_ticket(str(request.state.principal), key),
}
async def vncwebsocket(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _ws(request, inputs, "vnc")
async def mtunnelwebsocket(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _ws(request, inputs, "mtunnel")
base = "/nodes/{node}/lxc/{vmid}"
registry.register(f"{base}/interfaces", "GET", interfaces)
registry.register(f"{base}/move_volume", "POST", move_volume)
registry.register(f"{base}/rrd", "GET", rrd)
registry.register(f"{base}/rrddata", "GET", rrddata)
registry.register(f"{base}/vncproxy", "POST", vncproxy)
registry.register(f"{base}/spiceproxy", "POST", spiceproxy)
registry.register(f"{base}/termproxy", "POST", termproxy)
registry.register(f"{base}/mtunnel", "POST", mtunnel)
registry.register(f"{base}/vncwebsocket", "GET", vncwebsocket)
registry.register(f"{base}/mtunnelwebsocket", "GET", mtunnelwebsocket)
+106
View File
@@ -0,0 +1,106 @@
"""Cluster resource mapping handlers (dir/pci/usb)."""
from __future__ import annotations
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs, values
def _mappings(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.get("mapping")
if not isinstance(current, dict):
return {"dir": {}, "pci": {}, "usb": {}}
return {
"dir": dict(current["dir"]) if isinstance(current.get("dir"), dict) else {},
"pci": dict(current["pci"]) if isinstance(current.get("pci"), dict) else {},
"usb": dict(current["usb"]) if isinstance(current.get("usb"), dict) else {},
}
def register_mapping_handlers(registry: HandlerRegistry) -> None:
async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("dir", "pci", "usb")
def register_kind(kind: str) -> None:
base = f"/cluster/mapping/{kind}"
async def list_items(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
store = _mappings(metadata)[kind]
check_node = values(inputs).get("check-node")
result = [{"id": key, **item} for key, item in sorted(store.items())]
if check_node:
for item in result:
item["checks"] = {str(check_node): "OK"}
return result
async def create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
item_id = str(payload["id"])
metadata = await cluster_metadata(request)
mapping = _mappings(metadata)
store = mapping[kind]
if item_id in store:
raise ApiError(400, f"{kind} mapping '{item_id}' already exists")
store[item_id] = {
key: value for key, value in payload.items() if key not in {"delete", "digest"}
}
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata)
async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
item_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
store = _mappings(metadata)[kind]
if item_id not in store:
raise ApiError(404, f"{kind} mapping does not exist")
return {"id": item_id, **store[item_id]}
async def update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
item_id = str(payload["id"])
metadata = await cluster_metadata(request)
mapping = _mappings(metadata)
store = mapping[kind]
if item_id not in store:
raise ApiError(404, f"{kind} mapping does not exist")
current = dict(store[item_id])
for key in [
item.strip() for item in str(payload.get("delete") or "").split(",") if item.strip()
]:
current.pop(key, None)
for key, value in payload.items():
if key in {"id", "delete", "digest"}:
continue
current[key] = value
current["id"] = item_id
store[item_id] = current
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata)
async def delete(request: Request, inputs: dict[str, Any]) -> None:
item_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
mapping = _mappings(metadata)
store = mapping[kind]
if item_id not in store:
raise ApiError(404, f"{kind} mapping does not exist")
del store[item_id]
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata)
registry.register(base, "GET", list_items)
registry.register(base, "POST", create)
registry.register(f"{base}/{{id}}", "GET", get)
registry.register(f"{base}/{{id}}", "PUT", update)
registry.register(f"{base}/{{id}}", "DELETE", delete)
registry.register("/cluster/mapping", "GET", index)
register_kind("dir")
register_kind("pci")
register_kind("usb")
+425
View File
@@ -0,0 +1,425 @@
"""Node-level operational handlers (apt, network, disks, services)."""
from __future__ import annotations
from typing import Any, cast
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import (
database,
node_metadata,
require_node,
save_node_metadata,
subdirs,
values,
)
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
async def load_node_ops(request: Request, node: str) -> dict[str, Any]:
metadata = await node_metadata(request, node)
ops = metadata.get("ops")
if isinstance(ops, dict):
return ops
return {}
async def save_node_ops(request: Request, node: str, ops: dict[str, Any]) -> None:
metadata = await node_metadata(request, node)
metadata["ops"] = ops
await save_node_metadata(request, node, metadata)
def register_node_ops_handlers(registry: HandlerRegistry) -> None:
async def apt_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"]))
return subdirs("changelog", "repositories", "update", "versions")
async def apt_versions(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
packages = ops.get("apt", {}).get("packages", [])
if not isinstance(packages, list):
return []
result: list[dict[str, Any]] = []
for item in packages:
if not isinstance(item, dict):
continue
entry = dict(item)
status = str(entry.pop("Status", entry.get("CurrentState", "Installed")))
if status.lower() == "upgradable":
entry["CurrentState"] = "Installed"
entry.setdefault("OldVersion", entry.get("Version", ""))
else:
entry["CurrentState"] = "Installed" if status.lower() == "installed" else status
entry.setdefault("Package", entry.get("Package") or "unknown")
entry.setdefault("Version", entry.get("Version") or "")
entry.setdefault("Arch", "amd64")
entry.setdefault("Origin", "Proxmox")
result.append(entry)
return result
async def apt_repositories(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
repositories = ops.get("apt", {}).get("repositories")
if isinstance(repositories, dict) and "files" in repositories:
return {
"digest": str(repositories.get("digest") or "seedapt"),
"files": list(repositories.get("files") or []),
"errors": list(repositories.get("errors") or []),
"infos": list(repositories.get("infos") or []),
"standard-repos": list(repositories.get("standard-repos") or []),
}
legacy = list(repositories) if isinstance(repositories, list) else []
return {
"digest": "seedapt",
"files": [
{
"path": "/etc/apt/sources.list.d/pve-enterprise.list",
"file-type": "list",
"digest": "seedapt-file0",
"repositories": legacy,
}
],
"errors": [],
"infos": [],
"standard-repos": [
{
"handle": "no-subscription",
"name": "Proxmox VE No-Subscription Repository",
"status": 1,
}
],
}
async def apt_changelog(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
name = str(values(inputs).get("name") or "pve-manager")
ops = await load_node_ops(request, node)
apt = ops.get("apt")
changelogs = apt.get("changelogs") if isinstance(apt, dict) else None
if not isinstance(changelogs, dict):
return ""
return str(changelogs.get(name) or "")
async def apt_update_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
apt = ops.get("apt")
update = apt.get("update") if isinstance(apt, dict) else None
return dict(update) if isinstance(update, dict) else {}
async def apt_update_start(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
apt = ops.setdefault("apt", {})
apt["update"] = {"status": "running", "exitstatus": ""}
await save_node_ops(request, node, ops)
return await _node_task(request, node=node, task_type="aptupdate", worker="aptupdate")
async def network_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
network = ops.get("network", [])
return [dict(item) for item in network] if isinstance(network, list) else []
async def network_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
iface = str(values(inputs)["iface"])
for item in await network_list(request, inputs):
if item.get("iface") == iface:
return item
raise ApiError(404, "interface does not exist")
async def network_mutate(request: Request, inputs: dict[str, Any]) -> str | None:
node = str(values(inputs)["node"])
payload = values(inputs)
ops = await load_node_ops(request, node)
network = list(ops.get("network") or [])
iface = payload.get("iface")
method = request.method.upper()
if method == "DELETE":
target = str(iface or "")
if not any(item.get("iface") == target for item in network):
raise ApiError(404, "interface does not exist")
ops["network"] = [item for item in network if item.get("iface") != target]
elif method == "POST":
name = str(iface or payload.get("iface") or "")
if not name:
raise ApiError(400, "parameter verification failed - 'iface' missing")
if any(item.get("iface") == name for item in network):
raise ApiError(400, f"interface '{name}' already exists")
entry = {
key: value
for key, value in payload.items()
if key not in {"node", "delete", "digest"}
}
entry["iface"] = name
entry.setdefault("type", "bridge")
entry.setdefault("active", 1)
network.append(entry)
ops["network"] = network
elif method == "PUT" and iface is not None:
name = str(iface)
found = False
updated: list[dict[str, Any]] = []
for item in network:
if item.get("iface") != name:
updated.append(item)
continue
found = True
merged = {
**item,
**{
key: value
for key, value in payload.items()
if key not in {"node", "iface", "delete", "digest"}
},
}
merged["iface"] = name
updated.append(merged)
if not found:
raise ApiError(404, "interface does not exist")
ops["network"] = updated
else:
# PUT /nodes/{node}/network without iface → reload (returns UPID string).
ops["network_applied"] = True
await save_node_ops(request, node, ops)
return await _node_task(
request, node=node, task_type="network-reload", worker="srvreload"
)
await save_node_ops(request, node, ops)
return None
async def disks_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"]))
return subdirs("directory", "list", "lvm", "lvmthin", "smart", "zfs")
async def _disks(request: Request, node: str) -> dict[str, Any]:
ops = await load_node_ops(request, node)
disks = ops.get("disks")
if not isinstance(disks, dict):
return {}
return cast(dict[str, Any], disks)
async def disks_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
disks = await _disks(request, node)
items = disks.get("list", [])
return [dict(item) for item in items] if isinstance(items, list) else []
async def disks_smart(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or "/dev/sda")
disks = await _disks(request, node)
smart = disks.get("smart")
if not isinstance(smart, dict) or disk not in smart:
return {}
return dict(smart[disk])
async def disks_collection(
request: Request, inputs: dict[str, Any], key: str
) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
disks = await _disks(request, node)
items = disks.get(key, [])
return [dict(item) for item in items] if isinstance(items, list) else []
async def disks_directory(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
return await disks_collection(request, inputs, "directory")
async def disks_lvm(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
return await disks_collection(request, inputs, "lvm")
async def disks_lvmthin(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
return await disks_collection(request, inputs, "lvmthin")
async def disks_zfs(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
return await disks_collection(request, inputs, "zfs")
async def disks_initgpt(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or values(inputs).get("device") or "")
if not disk:
raise ApiError(400, "parameter verification failed - 'disk' missing")
ops = await load_node_ops(request, node)
disks = ops.get("disks")
if not isinstance(disks, dict):
disks = {}
ops["disks"] = disks
items = list(disks.get("list") or [])
found = False
for item in items:
if item.get("devpath") == disk:
item["gpt"] = 1
found = True
break
if not found:
items.append(
{
"devpath": disk,
"size": 0,
"model": "SIM-DISK",
"serial": disk,
"gpt": 1,
}
)
disks["list"] = items
ops["disks"] = disks
await save_node_ops(request, node, ops)
return await _node_task(request, node=node, task_type="disk-initgpt", worker="diskinit")
async def disks_wipedisk(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or values(inputs).get("device") or "")
if not disk:
raise ApiError(400, "parameter verification failed - 'disk' missing")
ops = await load_node_ops(request, node)
disks = ops.get("disks")
if not isinstance(disks, dict):
raise ApiError(404, "disk does not exist")
items = list(disks.get("list") or [])
for item in items:
if item.get("devpath") == disk:
item["wiped"] = 1
item["gpt"] = 0
break
else:
raise ApiError(404, "disk does not exist")
disks["list"] = items
smart = disks.get("smart")
if isinstance(smart, dict):
smart.pop(disk, None)
ops["disks"] = disks
await save_node_ops(request, node, ops)
return await _node_task(request, node=node, task_type="disk-wipe", worker="diskwipe")
async def services_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
services = ops.get("services") or {}
result: list[dict[str, Any]] = []
for name in sorted(services):
payload = dict(services[name]) if isinstance(services[name], dict) else {}
state_name = str(payload.get("state", "stopped"))
enabled = bool(int(payload.get("enabled", 0) or 0))
result.append(
{
"service": name,
"name": f"{name}.service",
"desc": str(payload.get("desc") or name),
"state": state_name,
"active-state": "active" if state_name == "running" else "inactive",
"unit-state": "enabled" if enabled else "disabled",
}
)
return result
async def service_get(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
node = str(values(inputs)["node"])
service = str(values(inputs)["service"])
ops = await load_node_ops(request, node)
services = ops.setdefault("services", {})
if service not in services:
services[service] = {"state": "stopped", "enabled": 0}
await save_node_ops(request, node, ops)
return subdirs("state", "start", "stop", "restart", "reload")
async def service_state(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
service = str(values(inputs)["service"])
ops = await load_node_ops(request, node)
services = ops.setdefault("services", {})
if service not in services:
services[service] = {"state": "stopped", "enabled": 0}
await save_node_ops(request, node, ops)
payload = dict(services[service]) if isinstance(services[service], dict) else {}
state_name = str(payload.get("state", "stopped"))
enabled = bool(int(payload.get("enabled", 0) or 0))
return {
"service": service,
"state": state_name,
"active-state": "active" if state_name == "running" else "inactive",
"unit-state": "enabled" if enabled else "disabled",
}
async def service_action(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
service = str(values(inputs)["service"])
path = request.url.path.rstrip("/")
action = path.rsplit("/", 1)[-1]
ops = await load_node_ops(request, node)
services = ops.setdefault("services", {})
current = dict(services.get(service) or {"state": "stopped", "enabled": 0})
if action == "start":
current["state"] = "running"
current["enabled"] = 1
elif action == "stop":
current["state"] = "stopped"
elif action in {"restart", "reload"}:
current["state"] = "running"
current["enabled"] = 1
else:
raise ApiError(400, f"unknown service action: {action}")
services[service] = current
ops["services"] = services
await save_node_ops(request, node, ops)
return await _node_task(
request,
node=node,
task_type=f"service-{action}",
worker=f"srv{action}",
)
registry.register("/nodes/{node}/apt", "GET", apt_index)
registry.register("/nodes/{node}/apt/versions", "GET", apt_versions)
registry.register("/nodes/{node}/apt/repositories", "GET", apt_repositories)
registry.register("/nodes/{node}/apt/changelog", "GET", apt_changelog)
registry.register("/nodes/{node}/apt/update", "GET", apt_update_status)
registry.register("/nodes/{node}/apt/update", "POST", apt_update_start)
registry.register("/nodes/{node}/network", "GET", network_list)
registry.register("/nodes/{node}/network", "POST", network_mutate)
registry.register("/nodes/{node}/network", "PUT", network_mutate)
registry.register("/nodes/{node}/network/{iface}", "GET", network_get)
registry.register("/nodes/{node}/network/{iface}", "PUT", network_mutate)
registry.register("/nodes/{node}/network/{iface}", "DELETE", network_mutate)
registry.register("/nodes/{node}/disks", "GET", disks_index)
registry.register("/nodes/{node}/disks/list", "GET", disks_list)
registry.register("/nodes/{node}/disks/smart", "GET", disks_smart)
registry.register("/nodes/{node}/disks/directory", "GET", disks_directory)
registry.register("/nodes/{node}/disks/lvm", "GET", disks_lvm)
registry.register("/nodes/{node}/disks/lvmthin", "GET", disks_lvmthin)
registry.register("/nodes/{node}/disks/zfs", "GET", disks_zfs)
registry.register("/nodes/{node}/disks/initgpt", "POST", disks_initgpt)
registry.register("/nodes/{node}/disks/wipedisk", "PUT", disks_wipedisk)
registry.register("/nodes/{node}/services", "GET", services_index)
registry.register("/nodes/{node}/services/{service}", "GET", service_get)
registry.register("/nodes/{node}/services/{service}/state", "GET", service_state)
registry.register("/nodes/{node}/services/{service}/start", "POST", service_action)
registry.register("/nodes/{node}/services/{service}/stop", "POST", service_action)
registry.register("/nodes/{node}/services/{service}/restart", "POST", service_action)
registry.register("/nodes/{node}/services/{service}/reload", "POST", service_action)
async def _node_task(request: Request, *, node: str, task_type: str, worker: str) -> str:
import secrets
from app.api.errors import ApiError
from app.db.primitives import ConflictError
pool = database(request).pool
upid = str(Upid.allocate(node, worker, "0", str(request.state.principal)))
try:
task = await TaskRepository(pool).create(
upid=upid,
task_type=task_type,
payload={"node": node},
resource_key=f"node:{node}:{task_type}:{secrets.token_hex(4)}",
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
File diff suppressed because it is too large Load Diff
+283
View File
@@ -0,0 +1,283 @@
"""Cluster notifications endpoints and matchers persisted in metadata."""
from __future__ import annotations
import time
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs, values
_SECRET_KEYS = frozenset({"token", "password", "secret"})
def _public(endpoint: dict[str, Any]) -> dict[str, Any]:
return {key: value for key, value in endpoint.items() if key not in _SECRET_KEYS}
def _notifications(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.get("notifications")
if not isinstance(current, dict):
return {
"endpoints": {"gotify": {}, "sendmail": {}, "smtp": {}, "webhook": {}},
"matchers": {},
"tests": [],
}
endpoints = current.get("endpoints")
if not isinstance(endpoints, dict):
endpoints = {}
result = {
key: value
for key, value in current.items()
if key not in {"endpoints", "matchers", "tests"}
}
result["endpoints"] = {
"gotify": dict(endpoints["gotify"]) if isinstance(endpoints.get("gotify"), dict) else {},
"sendmail": (
dict(endpoints["sendmail"]) if isinstance(endpoints.get("sendmail"), dict) else {}
),
"smtp": dict(endpoints["smtp"]) if isinstance(endpoints.get("smtp"), dict) else {},
"webhook": dict(endpoints["webhook"]) if isinstance(endpoints.get("webhook"), dict) else {},
}
result["matchers"] = (
dict(current["matchers"]) if isinstance(current.get("matchers"), dict) else {}
)
result["tests"] = list(current["tests"]) if isinstance(current.get("tests"), list) else []
return result
def register_notifications_handlers(registry: HandlerRegistry) -> None:
async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs(
"endpoints",
"matcher-field-values",
"matcher-fields",
"matchers",
"targets",
)
async def endpoints_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("gotify", "sendmail", "smtp", "webhook")
def register_kind(kind: str, create_keys: tuple[str, ...]) -> None:
base = f"/cluster/notifications/endpoints/{kind}"
async def list_endpoints(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].get(kind) or {}
return [_public({"name": name, **item}) for name, item in sorted(store.items())]
async def create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name in store:
raise ApiError(400, f"{kind} endpoint '{name}' already exists")
entry = {key: payload[key] for key in create_keys if key in payload}
entry["name"] = name
entry.setdefault("disable", 0)
store[name] = entry
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].get(kind) or {}
if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist")
return _public({"name": name, **store[name]})
async def update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist")
current = dict(store[name])
delete_keys = [
item.strip() for item in str(payload.get("delete") or "").split(",") if item.strip()
]
for key in delete_keys:
current.pop(key, None)
for key, value in payload.items():
if key in {"name", "delete", "digest"}:
continue
current[key] = value
current["name"] = name
store[name] = current
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def delete(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist")
del store[name]
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
registry.register(base, "GET", list_endpoints)
registry.register(base, "POST", create)
registry.register(f"{base}/{{name}}", "GET", get)
registry.register(f"{base}/{{name}}", "PUT", update)
registry.register(f"{base}/{{name}}", "DELETE", delete)
async def matchers_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
store = _notifications(metadata)["matchers"]
return [{"name": name, **item} for name, item in sorted(store.items())]
async def matchers_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
notifications = _notifications(metadata)
store = notifications["matchers"]
if name in store:
raise ApiError(400, f"matcher '{name}' already exists")
store[name] = {
key: value for key, value in payload.items() if key not in {"delete", "digest"}
}
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def matchers_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["matchers"]
if name not in store:
raise ApiError(404, "matcher does not exist")
return {"name": name, **store[name]}
async def matchers_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
notifications = _notifications(metadata)
store = notifications["matchers"]
if name not in store:
raise ApiError(404, "matcher does not exist")
current = dict(store[name])
for key in [
item.strip() for item in str(payload.get("delete") or "").split(",") if item.strip()
]:
current.pop(key, None)
for key, value in payload.items():
if key in {"name", "delete", "digest"}:
continue
current[key] = value
current["name"] = name
store[name] = current
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def matchers_delete(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
notifications = _notifications(metadata)
store = notifications["matchers"]
if name not in store:
raise ApiError(404, "matcher does not exist")
del store[name]
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def matcher_fields(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
fields = _notifications(metadata).get("matcher_fields")
if isinstance(fields, list):
return [dict(item) for item in fields if isinstance(item, dict)]
return []
async def matcher_field_values(
request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
values_list = _notifications(metadata).get("matcher_field_values")
if isinstance(values_list, list):
return [dict(item) for item in values_list if isinstance(item, dict)]
return []
async def targets(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
notifications = _notifications(metadata)
result: list[dict[str, Any]] = []
for kind, store in (notifications.get("endpoints") or {}).items():
if not isinstance(store, dict):
continue
for name, item in store.items():
result.append(
{
"name": name,
"type": kind,
"comment": item.get("comment", ""),
"disable": int(bool(item.get("disable"))),
}
)
return result
async def target_test(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
notifications = _notifications(metadata)
found = False
for store in (notifications.get("endpoints") or {}).values():
if isinstance(store, dict) and name in store:
found = True
break
if not found:
raise ApiError(404, "notification target does not exist")
tests = list(notifications.get("tests") or [])
tests.append({"name": name, "tested_at": int(time.time()), "ok": True})
notifications["tests"] = tests
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
registry.register("/cluster/notifications", "GET", index)
registry.register("/cluster/notifications/endpoints", "GET", endpoints_index)
register_kind("gotify", ("comment", "disable", "name", "server", "token"))
register_kind(
"sendmail",
("author", "comment", "disable", "from-address", "mailto", "mailto-user", "name"),
)
register_kind(
"smtp",
(
"author",
"comment",
"disable",
"from-address",
"mailto",
"mailto-user",
"mode",
"name",
"password",
"port",
"server",
"username",
),
)
register_kind(
"webhook",
("body", "comment", "disable", "header", "method", "name", "secret", "url"),
)
registry.register("/cluster/notifications/matchers", "GET", matchers_list)
registry.register("/cluster/notifications/matchers", "POST", matchers_create)
registry.register("/cluster/notifications/matchers/{name}", "GET", matchers_get)
registry.register("/cluster/notifications/matchers/{name}", "PUT", matchers_update)
registry.register("/cluster/notifications/matchers/{name}", "DELETE", matchers_delete)
registry.register("/cluster/notifications/matcher-fields", "GET", matcher_fields)
registry.register("/cluster/notifications/matcher-field-values", "GET", matcher_field_values)
registry.register("/cluster/notifications/targets", "GET", targets)
registry.register("/cluster/notifications/targets/{name}/test", "POST", target_test)
+160
View File
@@ -0,0 +1,160 @@
"""Resource pool semantic handlers."""
from __future__ import annotations
import json
import uuid
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import database, require_value, state, values
from app.simulation.seed import CLUSTER_ID, stable_id
async def _pool_members(request: Request, pool_id: uuid.UUID) -> list[dict[str, Any]]:
rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.kind, r.state, n.name AS node
FROM pool_members pm
JOIN resources r ON r.id = pm.resource_id
JOIN nodes n ON n.id = r.node_id
WHERE pm.pool_id=$1
ORDER BY r.external_id::integer""",
pool_id,
)
members: list[dict[str, Any]] = []
for row in rows:
guest_state = state(row["state"])
kind = str(row["kind"])
vmid = int(row["external_id"])
members.append(
{
"id": f"{kind}/{vmid}",
"node": str(row["node"]),
"type": kind,
"vmid": vmid,
"name": str(guest_state.get("name") or f"{kind}-{vmid}"),
}
)
return members
def register_pool_handlers(registry: HandlerRegistry) -> None:
async def pool_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = values(inputs)
filter_poolid = payload.get("poolid")
rows = await database(request).pool.fetch(
"""SELECT id, pool_id, comment, metadata FROM pools
WHERE ($1::text IS NULL OR pool_id=$1)
ORDER BY pool_id""",
str(filter_poolid) if filter_poolid is not None else None,
)
result: list[dict[str, Any]] = []
for row in rows:
metadata = state(row["metadata"])
members = await _pool_members(request, row["id"])
if not members and isinstance(metadata.get("members"), list):
# Fallback for metadata-only membership before pool_members rows exist.
for item in metadata["members"]:
members.append(
{
"id": f"qemu/{item}",
"node": "",
"type": "qemu",
"vmid": int(item) if str(item).isdigit() else 0,
}
)
pool_item: dict[str, Any] = {
"poolid": str(row["pool_id"]),
"members": members,
}
if row["comment"] is not None:
pool_item["comment"] = str(row["comment"])
elif metadata.get("comment"):
pool_item["comment"] = str(metadata["comment"])
result.append(pool_item)
return result
async def pool_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
items = await pool_list(request, inputs)
if not items:
raise ApiError(404, "pool does not exist")
return items[0]
async def pool_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
poolid = str(payload["poolid"])
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM pools WHERE pool_id=$1)",
poolid,
)
if exists:
raise ApiError(409, "pool already exists")
await database(request).pool.execute(
"""INSERT INTO pools(id, cluster_id, pool_id, comment, metadata)
VALUES($1, $2, $3, $4, $5::jsonb)""",
stable_id(f"pool:{poolid}"),
CLUSTER_ID,
poolid,
payload.get("comment"),
json.dumps({"members": []}, sort_keys=True),
)
async def pool_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
poolid = str(payload["poolid"])
pool_row = await database(request).pool.fetchrow(
"SELECT id FROM pools WHERE pool_id=$1",
poolid,
)
if pool_row is None:
raise ApiError(404, "pool does not exist")
if payload.get("comment") is not None:
await database(request).pool.execute(
"UPDATE pools SET comment=$2 WHERE pool_id=$1",
poolid,
payload.get("comment"),
)
if "vms" in payload:
vmids = [item.strip() for item in str(payload["vms"]).split(",") if item.strip()]
for vmid in vmids:
resource = await database(request).pool.fetchrow(
"""SELECT id FROM resources
WHERE kind IN ('qemu', 'lxc') AND external_id=$1""",
vmid,
)
if resource is None:
continue
await database(request).pool.execute(
"""INSERT INTO pool_members(pool_id, resource_id)
VALUES($1, $2) ON CONFLICT DO NOTHING""",
pool_row["id"],
resource["id"],
)
if "delete" in payload:
vmids = [item.strip() for item in str(payload["delete"]).split(",") if item.strip()]
await database(request).pool.execute(
"""DELETE FROM pool_members pm USING resources r
WHERE pm.pool_id=$1 AND pm.resource_id=r.id AND r.external_id = ANY($2::text[])""",
pool_row["id"],
vmids,
)
async def pool_delete(request: Request, inputs: dict[str, Any]) -> None:
poolid = str(require_value(values(inputs), "poolid"))
status = await database(request).pool.execute(
"DELETE FROM pools WHERE pool_id=$1",
poolid,
)
if status != "DELETE 1":
raise ApiError(404, "pool does not exist")
registry.register("/pools", "GET", pool_list)
registry.register("/pools", "POST", pool_create)
registry.register("/pools", "PUT", pool_update)
registry.register("/pools", "DELETE", pool_delete)
registry.register("/pools/{poolid}", "GET", pool_get)
registry.register("/pools/{poolid}", "PUT", pool_update)
registry.register("/pools/{poolid}", "DELETE", pool_delete)
+908 -27
View File
@@ -3,6 +3,7 @@
from __future__ import annotations from __future__ import annotations
import json import json
import re
from collections.abc import Mapping from collections.abc import Mapping
from typing import Any, cast from typing import Any, cast
@@ -11,6 +12,9 @@ from fastapi import Request
from app.api.errors import ApiError from app.api.errors import ApiError
from app.api.registry import HandlerRegistry from app.api.registry import HandlerRegistry
from app.db.pool import AsyncpgDatabase from app.db.pool import AsyncpgDatabase
from app.db.primitives import ConflictError
from app.handlers.common import require_node, subdirs
from app.simulation.transitions import InvalidTransitionError, VmState, plan_transition
from app.tasks.repository import TaskRepository from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid from app.tasks.upid import Upid
@@ -29,31 +33,202 @@ def _state(value: object) -> dict[str, Any]:
return dict(cast(Mapping[str, Any], value)) return dict(cast(Mapping[str, Any], value))
# Runtime-only keys stored in ``resources.state``; never echo on GET /config.
_QEMU_CONFIG_INTERNAL = frozenset(
{
"agent",
"cloudinit_dump",
"config",
"interfaces",
"migrate_preconditions",
"rrd",
"rrddata",
"sendkey",
"status",
}
)
def _agent_config_string(value: object) -> str:
"""Proxmox QEMU config ``agent`` is a string (e.g. ``1`` / ``0``), not an object."""
if isinstance(value, dict):
return "1" if value.get("enabled", True) else "0"
if isinstance(value, bool):
return "1" if value else "0"
if value in (None, ""):
return "0"
text = str(value).strip()
return text or "0"
def _public_qemu_config(config: dict[str, Any], state: dict[str, Any]) -> dict[str, Any]:
"""Wire-format QEMU config for clients (bpg / pulumi-proxmoxve)."""
import hashlib
merged = {**state, **config}
payload: dict[str, Any] = {}
for key, value in merged.items():
if key in _QEMU_CONFIG_INTERNAL:
continue
if isinstance(value, dict | list):
continue
payload[key] = value
agent = config.get("agent", state.get("agent", "0"))
payload["agent"] = _agent_config_string(agent)
digest_src = {key: value for key, value in payload.items() if key != "digest"}
payload["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(digest_src, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
return payload
def _cpu_utilization(vm_state: Mapping[str, Any], *, running: bool, idle: float = 0.1) -> float:
"""Status ``cpu`` is utilization; config may store model strings like ``qemu64``."""
raw = vm_state.get("cpu")
if isinstance(raw, (int, float)):
return float(raw)
if isinstance(raw, str):
try:
return float(raw)
except ValueError:
pass
return idle if running else 0.0
def _qemu_index_item(vmid: int, vm_state: dict[str, Any], config: dict[str, Any]) -> dict[str, Any]:
status = str(vm_state.get("status", "stopped"))
running = status in {"running", "paused"}
memory_mb = int(config.get("memory", config.get("mem", 2048)))
maxmem = memory_mb * 2**20
mem = int(vm_state.get("mem", maxmem // 2 if running else 0))
return {
"vmid": vmid,
"name": str(config.get("name", f"vm-{vmid}")),
"status": status,
"qmpstatus": status if running else "stopped",
"cpus": int(config.get("cores", config.get("cpus", 1))),
"cpu": _cpu_utilization(vm_state, running=running),
"maxmem": maxmem,
"mem": mem,
"memhost": int(vm_state.get("memhost", mem)),
"maxdisk": int(vm_state.get("maxdisk", config.get("maxdisk", 32 * 1024**3))),
"disk": int(vm_state.get("disk", 0)),
"diskread": int(vm_state.get("diskread", 0)),
"diskwrite": int(vm_state.get("diskwrite", 0)),
"netin": int(vm_state.get("netin", 0)),
"netout": int(vm_state.get("netout", 0)),
"uptime": int(vm_state.get("uptime", 0)) if running else 0,
"pid": int(vm_state.get("pid", 12_345 if running else 0)),
"lock": str(vm_state.get("lock", "")),
"tags": str(config.get("tags", "")),
"template": int(bool(vm_state.get("template", False))),
"running-qemu": str(vm_state.get("running-qemu", "9.2.0")) if running else "",
"running-machine": str(vm_state.get("running-machine", "pc-i440fx-9.2")) if running else "",
"pressurecpusome": float(vm_state.get("pressurecpusome", 0.0)),
"pressurecpufull": float(vm_state.get("pressurecpufull", 0.0)),
"pressureiosome": float(vm_state.get("pressureiosome", 0.0)),
"pressureiofull": float(vm_state.get("pressureiofull", 0.0)),
"pressurememorysome": float(vm_state.get("pressurememorysome", 0.0)),
"pressurememoryfull": float(vm_state.get("pressurememoryfull", 0.0)),
}
def register_qemu_handlers(registry: HandlerRegistry) -> None: def register_qemu_handlers(registry: HandlerRegistry) -> None:
async def qemu_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def qemu_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(_values(inputs)["node"]) node = str(_values(inputs)["node"])
rows = await _database(request).pool.fetch( rows = await _database(request).pool.fetch(
"""SELECT r.external_id::integer AS vmid, r.state """SELECT r.external_id::integer AS vmid, r.state, v.config
FROM resources r JOIN nodes n ON n.id=r.node_id FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN virtual_machines v ON v.resource_id=r.id
WHERE n.name=$1 AND r.kind='qemu' ORDER BY r.external_id::integer""", WHERE n.name=$1 AND r.kind='qemu' ORDER BY r.external_id::integer""",
node, node,
) )
return [{"vmid": int(row["vmid"]), **_state(row["state"])} for row in rows] return [
_qemu_index_item(int(row["vmid"]), _state(row["state"]), _state(row["config"]))
for row in rows
]
async def qemu_status_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs)
await _qemu_resource(request, str(payload["node"]), str(payload["vmid"]))
return subdirs(
"current",
"reboot",
"reset",
"resume",
"shutdown",
"start",
"stop",
"suspend",
)
async def qemu_status_current(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
node, vmid = str(payload["node"]), int(payload["vmid"])
resource = await _qemu_resource(request, node, str(vmid))
vm_state = _state(resource["state"])
config = _state(resource["config"])
status = str(vm_state.get("status", "stopped"))
running = status in {"running", "paused"}
memory_mb = int(config.get("memory", config.get("mem", 2048)))
maxmem = memory_mb * 2**20
mem_used = int(vm_state.get("mem", maxmem // 2 if running else 0))
uptime = int(
vm_state.get(
"uptime",
int(
await _database(request).pool.fetchval(
"SELECT extract(epoch from now())::bigint"
)
)
% 86_400
if running
else 0,
)
)
agent_enabled = str(config.get("agent", "0")).startswith("1")
item = _qemu_index_item(vmid, vm_state, config)
item.update(
{
"uptime": uptime,
"mem": mem_used,
"balloon": int(vm_state.get("balloon", 0)),
"ballooninfo": {
"actual": mem_used,
"max_mem": maxmem,
"mem_swapped_in": 0,
"mem_swapped_out": 0,
},
"template": bool(vm_state.get("template", False)),
"ha": {"managed": int(vm_state.get("ha_managed", 0))},
"agent": bool(running and agent_enabled),
"clipboard": str(vm_state.get("clipboard", "")),
"spice": bool(vm_state.get("spice", False)),
"serial": bool(vm_state.get("serial", False)),
}
)
return item
async def qemu_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def qemu_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"]) node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"])
row = await _database(request).pool.fetchrow( row = await _database(request).pool.fetchrow(
"""SELECT r.state FROM resources r JOIN nodes n ON n.id=r.node_id """SELECT r.state, v.config FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN virtual_machines v ON v.resource_id=r.id
WHERE n.name=$1 AND r.kind='qemu' AND r.external_id=$2""", WHERE n.name=$1 AND r.kind='qemu' AND r.external_id=$2""",
node, node,
vmid, vmid,
) )
if row is None: if row is None:
raise ApiError(404, "virtual machine does not exist") raise ApiError(404, "virtual machine does not exist")
return {"vmid": int(vmid), **_state(row["state"])} return {
"vmid": int(vmid),
async def qemu_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: **_public_qemu_config(_state(row["config"]), _state(row["state"])),
return await qemu_config(request, inputs) }
async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str: async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
@@ -68,36 +243,501 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
if row is None: if row is None:
raise ApiError(404, "virtual machine does not exist") raise ApiError(404, "virtual machine does not exist")
current = str(_state(row["state"]).get("status", "stopped")) current = str(_state(row["state"]).get("status", "stopped"))
if (operation == "start" and current != "stopped") or ( try:
operation == "stop" and current != "running" plan_transition(VmState(current), operation)
): except (InvalidTransitionError, ValueError) as error:
raise ApiError(409, f"cannot {operation} VM while it is {current}") raise ApiError(409, f"cannot {operation} VM while it is {current}") from error
timestamp = int(await database.pool.fetchval("SELECT extract(epoch from now())::bigint")) upid = str(Upid.allocate(node, f"qm{operation}", vmid, str(request.state.principal)))
pid = int(await database.pool.fetchval("SELECT pg_backend_pid()")) try:
upid = str(Upid(node, pid, pid, timestamp, f"qm{operation}", vmid, "root@pam")) task = await TaskRepository(database.pool).create(
task = await TaskRepository(database.pool).create( upid=upid,
upid=upid, task_type=f"qemu-{operation}",
task_type=f"qemu-{operation}", payload={"node": node, "vmid": vmid, "resource_id": str(row["id"])},
payload={"node": node, "vmid": vmid, "resource_id": str(row["id"])}, resource_key=f"qemu:{vmid}",
resource_key=f"qemu:{vmid}", idempotency_key=request.headers.get("Idempotency-Key"),
idempotency_key=request.headers.get("Idempotency-Key"), )
) except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid return task.upid
async def create(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid = str(values["node"]), int(values["vmid"])
database = _database(request)
if not await database.pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", node
):
raise ApiError(404, "node does not exist")
if await database.pool.fetchval(
"""SELECT EXISTS(SELECT 1 FROM resources
WHERE external_id=$1 AND kind IN ('qemu','lxc'))""",
str(vmid),
):
raise ApiError(409, "VMID already exists")
config = {
key: value
for key, value in values.items()
if key not in {"node", "vmid", "force", "archive", "start"}
}
return await _create_task(
request,
node=node,
vmid=str(vmid),
task_type="qemu-create",
payload={"node": node, "vmid": vmid, "config": config},
)
async def update(request: Request, inputs: dict[str, Any], *, asynchronous: bool) -> str | None:
values = _values(inputs)
node, vmid = str(values["node"]), str(values["vmid"])
database = _database(request)
row = await database.pool.fetchrow(
"""SELECT r.id, r.version, r.state, v.config FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN virtual_machines v ON v.resource_id=r.id
WHERE n.name=$1 AND r.kind='qemu' AND r.external_id=$2""",
node,
vmid,
)
if row is None:
raise ApiError(404, "virtual machine does not exist")
control = {"node", "vmid", "digest", "delete", "revert", "skiplock", "background_delay"}
provided = frozenset(str(item) for item in inputs.get("provided", values))
changes = {
key: value for key, value in values.items() if key in provided and key not in control
}
delete = str(values.get("delete", "")) if "delete" in provided else ""
if asynchronous:
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-update",
payload={
"node": node,
"vmid": vmid,
"resource_id": str(row["id"]),
"changes": changes,
"delete": delete,
},
)
state = _state(row["state"])
config = _state(row["config"])
state.update(changes)
config.update(changes)
for key in delete.split(","):
if key:
state.pop(key, None)
config.pop(key, None)
status = await database.pool.execute(
"""UPDATE resources SET state=$3::jsonb, version=version+1,
updated_at=now() WHERE id=$1 AND version=$2""",
row["id"],
row["version"],
json.dumps(state, sort_keys=True),
)
if status != "UPDATE 1":
raise ApiError(409, "configuration changed concurrently")
await database.pool.execute(
"""UPDATE virtual_machines SET config=$2::jsonb
WHERE resource_id=$1""",
row["id"],
json.dumps(config, sort_keys=True),
)
return None
async def update_async(request: Request, inputs: dict[str, Any]) -> str:
result = await update(request, inputs, asynchronous=True)
if not isinstance(result, str):
raise RuntimeError("async QEMU update did not create a task")
return result
async def update_sync(request: Request, inputs: dict[str, Any]) -> None:
await update(request, inputs, asynchronous=False)
async def delete(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid = str(values["node"]), str(values["vmid"])
database = _database(request)
row = await database.pool.fetchrow(
"""SELECT r.id, r.state FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='qemu' AND r.external_id=$2""",
node,
vmid,
)
if row is None:
raise ApiError(404, "virtual machine does not exist")
if str(_state(row["state"]).get("status")) != "stopped":
raise ApiError(409, "cannot delete a running virtual machine")
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-delete",
payload={"node": node, "vmid": vmid, "resource_id": str(row["id"])},
)
async def start(request: Request, inputs: dict[str, Any]) -> str: async def start(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("start", request, inputs) return await mutate("start", request, inputs)
async def stop(request: Request, inputs: dict[str, Any]) -> str: async def stop(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("stop", request, inputs) return await mutate("stop", request, inputs)
async def task_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def shutdown(request: Request, inputs: dict[str, Any]) -> str:
tasks = await TaskRepository(_database(request).pool).list_for_node( return await mutate("shutdown", request, inputs)
str(_values(inputs)["node"])
async def reboot(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("reboot", request, inputs)
async def reset(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("reset", request, inputs)
async def suspend(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("suspend", request, inputs)
async def resume(request: Request, inputs: dict[str, Any]) -> str:
return await mutate("resume", request, inputs)
async def snapshot_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
rows = await _database(request).pool.fetch(
"""SELECT name, parent_name, description, created_at, state FROM snapshots
WHERE resource_id=$1 ORDER BY created_at, name""",
resource["id"],
) )
result: list[dict[str, Any]] = []
for row in rows:
raw_state = row["state"] if "state" in row else None
snap_state = _state(raw_state) if raw_state is not None else {}
result.append(
{
"name": row["name"],
"parent": row["parent_name"],
"description": row["description"] or "",
"snaptime": int(row["created_at"].timestamp()),
"vmstate": int(bool(snap_state.get("vmstate"))),
}
)
return result
async def snapshot_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
row = await _snapshot(request, values)
raw_state = row["state"] if "state" in row else None
state = _state(raw_state) if raw_state is not None else {}
return {
"name": row["name"],
"parent": row["parent_name"],
"description": row["description"] or "",
"snaptime": int(row["created_at"].timestamp()),
**state,
}
async def snapshot_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
row = await _snapshot(request, _values(inputs))
return {"description": row["description"] or "", **_state(row["state"])}
async def snapshot_update(request: Request, inputs: dict[str, Any]) -> None:
values = _values(inputs)
row = await _snapshot(request, values)
await _database(request).pool.execute(
"UPDATE snapshots SET description=$2 WHERE id=$1",
row["id"],
str(values.get("description", "")),
)
async def snapshot_task(operation: str, request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid, snapname = (
str(values["node"]),
str(values["vmid"]),
str(values["snapname"]),
)
resource = await _qemu_resource(request, node, vmid)
if operation == "snapshot-create":
exists = await _database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM snapshots WHERE resource_id=$1 AND name=$2)",
resource["id"],
snapname,
)
if exists:
raise ApiError(409, "snapshot already exists")
else:
await _snapshot(request, values)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type=f"qemu-{operation}",
payload={
"node": node,
"vmid": vmid,
"resource_id": str(resource["id"]),
"snapname": snapname,
"description": str(values.get("description", "")),
"vmstate": bool(values.get("vmstate", False)),
"start": bool(values.get("start", False)),
},
)
async def snapshot_create(request: Request, inputs: dict[str, Any]) -> str:
return await snapshot_task("snapshot-create", request, inputs)
async def snapshot_delete(request: Request, inputs: dict[str, Any]) -> str:
return await snapshot_task("snapshot-delete", request, inputs)
async def snapshot_rollback(request: Request, inputs: dict[str, Any]) -> str:
return await snapshot_task("snapshot-rollback", request, inputs)
async def clone(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid, newid = str(values["node"]), str(values["vmid"]), str(values["newid"])
source = await _qemu_resource(request, node, vmid)
if await _database(request).pool.fetchval(
"""SELECT EXISTS(SELECT 1 FROM resources
WHERE external_id=$1 AND kind IN ('qemu','lxc'))""",
newid,
):
raise ApiError(409, "VMID already exists")
target = str(values.get("target") or node)
if not await _database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", target
):
raise ApiError(404, "target node does not exist")
return await _create_task(
request,
node=target,
vmid=newid,
task_type="qemu-clone",
payload={
"source_resource_id": str(source["id"]),
"source_vmid": vmid,
"node": target,
"vmid": int(newid),
"name": values.get("name"),
"description": values.get("description"),
"full": bool(values.get("full", False)),
},
)
async def migrate_preconditions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
target = values.get("target")
if target in {None, ""}:
raise ApiError(400, "parameter 'target' is required")
target = str(target)
exists = await _database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", target
)
if not exists:
raise ApiError(404, "target node does not exist")
state = _state(resource["state"])
pre = state.get("migrate_preconditions")
payload = dict(pre) if isinstance(pre, dict) else {}
payload["running"] = str(state.get("status") or "") == "running"
payload.setdefault("local_disks", [])
payload.setdefault("local_resources", [])
return payload
async def migrate(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid = str(values["node"]), str(values["vmid"])
target = values.get("target")
if target in {None, ""}:
raise ApiError(400, "parameter 'target' is required")
target = str(target)
resource = await _qemu_resource(request, node, vmid)
if target == node:
raise ApiError(400, "target node is the same as source node")
await migrate_preconditions(request, inputs)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-migrate",
payload={
"resource_id": str(resource["id"]),
"node": node,
"target": target,
"vmid": vmid,
"online": bool(values.get("online", False)),
},
)
async def remote_migrate(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
target_endpoint = str(values.get("target-endpoint") or values.get("target_endpoint") or "")
target = str(values.get("target") or "")
if not target_endpoint:
raise ApiError(400, "parameter target-endpoint is required")
if not target:
raise ApiError(400, "parameter target is required")
node, vmid = str(values["node"]), str(values["vmid"])
resource = await _qemu_resource(request, node, vmid)
if target == node:
raise ApiError(400, "target node is the same as source node")
if not await _database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", target
):
raise ApiError(404, "target node does not exist")
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-remote-migrate",
payload={
"resource_id": str(resource["id"]),
"node": node,
"target": target,
"vmid": vmid,
"target-endpoint": target_endpoint,
"online": bool(values.get("online", False)),
},
)
async def resize(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid, disk = str(values["node"]), str(values["vmid"]), str(values["disk"])
resource = await _qemu_resource(request, node, vmid)
config = _state(resource["config"])
if disk not in config:
raise ApiError(400, f"disk {disk} does not exist")
current = _disk_size_bytes(str(config[disk]))
size = _resize_bytes(str(values["size"]), current)
config[disk] = _replace_disk_size(str(config[disk]), size)
status = await _database(request).pool.execute(
"""UPDATE virtual_machines SET config=$2::jsonb
WHERE resource_id=$1""",
resource["id"],
json.dumps(config, sort_keys=True),
)
if status != "UPDATE 1":
raise ApiError(409, "configuration changed concurrently")
await _database(request).pool.execute(
"""UPDATE resources SET state=state || $2::jsonb,version=version+1,
updated_at=now() WHERE id=$1""",
resource["id"],
json.dumps({disk: config[disk]}, sort_keys=True),
)
await _database(request).pool.execute(
"""INSERT INTO vm_disks(id,resource_id,device,storage_id,size_bytes)
VALUES(gen_random_uuid(),$1,$2,$3,$4)
ON CONFLICT(resource_id,device) DO UPDATE SET size_bytes=EXCLUDED.size_bytes""",
resource["id"],
disk,
str(config[disk]).split(":", 1)[0],
size,
)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-resize",
payload={"resource_id": str(resource["id"]), "disk": disk, "size": size},
)
async def move_disk(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid, disk = str(values["node"]), str(values["vmid"]), str(values["disk"])
resource = await _qemu_resource(request, node, vmid)
if disk not in _state(resource["config"]):
raise ApiError(400, f"disk {disk} does not exist")
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-move-disk",
payload={
"resource_id": str(resource["id"]),
"disk": disk,
"storage": str(values.get("storage") or "local-lvm"),
"target_vmid": int(values.get("target-vmid") or vmid),
"target_disk": str(values.get("target-disk") or disk),
"delete": bool(values.get("delete", True)),
},
)
async def pending(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
config = _state(resource["config"])
changes = cast(Mapping[str, Any], state.get("pending", {}))
return [ return [
{"upid": task.upid, "status": task.status, "type": task.task_type} for task in tasks {"key": key, "value": str(config.get(key, "")), "pending": str(value)}
for key, value in sorted(changes.items())
] ]
async def agent_result(
command: str, request: Request, inputs: dict[str, Any]
) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
if command == "get-time":
seconds = int(
await _database(request).pool.fetchval("SELECT extract(epoch from now())::bigint")
)
return {"result": {"seconds": seconds, "nanoseconds": 0}}
state = _state(resource["state"])
agent = state.get("agent")
results = agent.get("results") if isinstance(agent, dict) else None
if not isinstance(results, dict) or command not in results:
raise ApiError(404, f"agent command '{command}' has no stored result")
return {"result": results[command]}
async def agent_info(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_result("info", request, inputs)
async def agent_osinfo(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_result("get-osinfo", request, inputs)
async def agent_hostname(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
config = _state(resource["config"])
name = state.get("name") or config.get("name")
if name:
return {"result": {"host-name": str(name)}}
return await agent_result("get-host-name", request, inputs)
async def agent_network(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_result("network-get-interfaces", request, inputs)
async def agent_time(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_result("get-time", request, inputs)
async def agent_ping(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_result("ping", request, inputs)
async def task_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
from app.tasks.upid import Upid
node = str(_values(inputs)["node"])
tasks = await TaskRepository(_database(request).pool).list_for_node(node)
result: list[dict[str, Any]] = []
for task in tasks:
try:
parsed = Upid.parse(task.upid)
except ValueError:
continue
finished = task.status in {"success", "error", "cancelled"}
entry: dict[str, Any] = {
"upid": task.upid,
"id": parsed.task_id,
"node": parsed.node,
"pid": parsed.pid,
"pstart": parsed.process_start,
"starttime": parsed.start_time,
"type": parsed.task_type,
"user": parsed.user,
}
if finished:
entry["status"] = "OK" if task.status == "success" else str(task.status).upper()
entry["endtime"] = parsed.start_time + max(task.progress, 1)
result.append(entry)
return result
async def task_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def task_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
task = await TaskRepository(_database(request).pool).get_by_upid( task = await TaskRepository(_database(request).pool).get_by_upid(
str(_values(inputs)["upid"]) str(_values(inputs)["upid"])
@@ -123,11 +763,252 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
for index, message in enumerate(await repository.logs(task.id)) for index, message in enumerate(await repository.logs(task.id))
] ]
async def qemu_feature(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
await _qemu_resource(request, str(payload["node"]), str(payload["vmid"]))
return {
"hasFeature": {
"snapshot": 1,
"clone": 1,
"copy": 1,
"template": 1,
"move_disk": 1,
"agent": 1,
}
}
async def qemu_template(request: Request, inputs: dict[str, Any]) -> str:
payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"])
resource = await _qemu_resource(request, node, vmid)
state = _state(resource["state"])
if state.get("status") != "stopped":
raise ApiError(409, "virtual machine must be stopped to convert to template")
await _database(request).pool.execute(
"UPDATE virtual_machines SET template=true WHERE resource_id=$1",
resource["id"],
)
state["template"] = True
await _database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb WHERE id=$1",
resource["id"],
json.dumps(state, sort_keys=True),
)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-template",
payload={"resource_id": str(resource["id"])},
)
async def qemu_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"])
await require_node(request, node)
await _qemu_resource(request, node, vmid)
return subdirs(
"agent",
"clone",
"config",
"feature",
"firewall",
"migrate",
"move_disk",
"pending",
"resize",
"snapshot",
"status",
"template",
)
async def task_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs)
node, upid = str(payload["node"]), str(payload["upid"])
await require_node(request, node)
task = await TaskRepository(_database(request).pool).get_by_upid(upid)
if task is None:
raise ApiError(404, "task does not exist")
return subdirs("log", "status")
async def task_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = _values(inputs)
upid = str(payload["upid"])
repository = TaskRepository(_database(request).pool)
task = await repository.get_by_upid(upid)
if task is None:
raise ApiError(404, "task does not exist")
if task.status in {"success", "error", "cancelled"}:
return
await repository.request_cancel(task.id)
registry.register("/nodes/{node}/qemu", "GET", qemu_list) registry.register("/nodes/{node}/qemu", "GET", qemu_list)
registry.register("/nodes/{node}/qemu", "POST", create)
registry.register("/nodes/{node}/qemu/{vmid}", "GET", qemu_index)
registry.register("/nodes/{node}/qemu/{vmid}", "DELETE", delete)
registry.register("/nodes/{node}/qemu/{vmid}/config", "GET", qemu_config) registry.register("/nodes/{node}/qemu/{vmid}/config", "GET", qemu_config)
registry.register("/nodes/{node}/qemu/{vmid}/status/current", "GET", qemu_status) registry.register("/nodes/{node}/qemu/{vmid}/config", "POST", update_async)
registry.register("/nodes/{node}/qemu/{vmid}/config", "PUT", update_sync)
registry.register("/nodes/{node}/qemu/{vmid}/status", "GET", qemu_status_index)
registry.register("/nodes/{node}/qemu/{vmid}/status/current", "GET", qemu_status_current)
registry.register("/nodes/{node}/qemu/{vmid}/status/start", "POST", start) registry.register("/nodes/{node}/qemu/{vmid}/status/start", "POST", start)
registry.register("/nodes/{node}/qemu/{vmid}/status/stop", "POST", stop) registry.register("/nodes/{node}/qemu/{vmid}/status/stop", "POST", stop)
registry.register("/nodes/{node}/qemu/{vmid}/status/shutdown", "POST", shutdown)
registry.register("/nodes/{node}/qemu/{vmid}/status/reboot", "POST", reboot)
registry.register("/nodes/{node}/qemu/{vmid}/status/reset", "POST", reset)
registry.register("/nodes/{node}/qemu/{vmid}/status/suspend", "POST", suspend)
registry.register("/nodes/{node}/qemu/{vmid}/status/resume", "POST", resume)
registry.register("/nodes/{node}/qemu/{vmid}/snapshot", "GET", snapshot_list)
registry.register("/nodes/{node}/qemu/{vmid}/snapshot", "POST", snapshot_create)
registry.register("/nodes/{node}/qemu/{vmid}/snapshot/{snapname}", "GET", snapshot_get)
registry.register("/nodes/{node}/qemu/{vmid}/snapshot/{snapname}", "DELETE", snapshot_delete)
registry.register(
"/nodes/{node}/qemu/{vmid}/snapshot/{snapname}/config", "GET", snapshot_config
)
registry.register(
"/nodes/{node}/qemu/{vmid}/snapshot/{snapname}/config", "PUT", snapshot_update
)
registry.register(
"/nodes/{node}/qemu/{vmid}/snapshot/{snapname}/rollback", "POST", snapshot_rollback
)
registry.register("/nodes/{node}/qemu/{vmid}/clone", "POST", clone)
registry.register("/nodes/{node}/qemu/{vmid}/migrate", "GET", migrate_preconditions)
registry.register("/nodes/{node}/qemu/{vmid}/migrate", "POST", migrate)
registry.register("/nodes/{node}/qemu/{vmid}/remote_migrate", "POST", remote_migrate)
registry.register("/nodes/{node}/qemu/{vmid}/resize", "PUT", resize)
registry.register("/nodes/{node}/qemu/{vmid}/move_disk", "POST", move_disk)
registry.register("/nodes/{node}/qemu/{vmid}/pending", "GET", pending)
registry.register("/nodes/{node}/qemu/{vmid}/agent/info", "GET", agent_info)
registry.register("/nodes/{node}/qemu/{vmid}/agent/get-osinfo", "GET", agent_osinfo)
registry.register("/nodes/{node}/qemu/{vmid}/agent/get-host-name", "GET", agent_hostname)
registry.register(
"/nodes/{node}/qemu/{vmid}/agent/network-get-interfaces", "GET", agent_network
)
registry.register("/nodes/{node}/qemu/{vmid}/agent/get-time", "GET", agent_time)
registry.register("/nodes/{node}/qemu/{vmid}/agent/ping", "POST", agent_ping)
registry.register("/nodes/{node}/qemu/{vmid}/feature", "GET", qemu_feature)
registry.register("/nodes/{node}/qemu/{vmid}/template", "POST", qemu_template)
registry.register("/nodes/{node}/tasks", "GET", task_list) registry.register("/nodes/{node}/tasks", "GET", task_list)
registry.register("/nodes/{node}/tasks/{upid}", "GET", task_index)
registry.register("/nodes/{node}/tasks/{upid}", "DELETE", task_delete)
registry.register("/nodes/{node}/tasks/{upid}/status", "GET", task_status) registry.register("/nodes/{node}/tasks/{upid}/status", "GET", task_status)
registry.register("/nodes/{node}/tasks/{upid}/log", "GET", task_log) registry.register("/nodes/{node}/tasks/{upid}/log", "GET", task_log)
from app.handlers.qemu_extra import register_qemu_extra_handlers
register_qemu_extra_handlers(registry)
async def _create_task(
request: Request,
*,
node: str,
vmid: str,
task_type: str,
payload: dict[str, Any],
) -> str:
database = _database(request)
worker_type = {
"qemu-create": "qmcreate",
"qemu-delete": "qmdestroy",
"qemu-update": "qmconfig",
"qemu-snapshot-create": "qmsnapshot",
"qemu-snapshot-delete": "qmdelsnapshot",
"qemu-snapshot-rollback": "qmrollback",
"qemu-clone": "qmclone",
"qemu-migrate": "qmigrate",
"qemu-remote-migrate": "qmremote",
"qemu-move-disk": "qmmove",
"qemu-resize": "qmresize",
"qemu-template": "qmtemplate",
}[task_type]
upid = str(Upid.allocate(node, worker_type, vmid, str(request.state.principal)))
try:
task = await TaskRepository(database.pool).create(
upid=upid,
task_type=task_type,
payload=payload,
resource_key=f"qemu:{vmid}",
idempotency_key=request.headers.get("Idempotency-Key"),
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
async def _qemu_resource(request: Request, node: str, vmid: str) -> Any:
row = await _database(request).pool.fetchrow(
"""SELECT r.id, r.state, v.config FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN virtual_machines v ON v.resource_id=r.id
WHERE n.name=$1 AND r.kind='qemu' AND r.external_id=$2""",
node,
vmid,
)
if row is None:
raise ApiError(404, "virtual machine does not exist")
return row
async def _snapshot(request: Request, values: dict[str, Any]) -> Any:
row = await _database(request).pool.fetchrow(
"""SELECT s.* FROM snapshots s
JOIN resources r ON r.id=s.resource_id JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='qemu' AND r.external_id=$2 AND s.name=$3""",
str(values["node"]),
str(values["vmid"]),
str(values["snapname"]),
)
if row is None:
raise ApiError(404, "snapshot does not exist")
return row
async def _agent_resource(request: Request, values: dict[str, Any]) -> Any:
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
config = _state(resource["config"])
state = _state(resource["state"])
agent = state.get("agent")
if not isinstance(agent, dict):
agent = config.get("agent")
if isinstance(agent, dict):
enabled = bool(agent.get("enabled", True))
else:
enabled = str(config.get("agent", "0")).split(",", 1)[0].lower() in {"1", "true", "yes"}
if not enabled:
raise ApiError(409, "QEMU guest agent is not enabled")
if state.get("status") != "running":
raise ApiError(409, "QEMU guest agent is not running")
return resource
_SIZE_RE = re.compile(r"^(?P<value>\d+)(?P<unit>[KMGT]?)$", re.IGNORECASE)
def _size_bytes(value: str) -> int:
match = _SIZE_RE.fullmatch(value.strip())
if match is None:
raise ApiError(400, f"invalid disk size: {value}")
units = {"": 1, "K": 2**10, "M": 2**20, "G": 2**30, "T": 2**40}
return int(match.group("value")) * units[match.group("unit").upper()]
def _disk_size_bytes(value: str) -> int:
for part in value.split(","):
if part.startswith("size="):
return _size_bytes(part.removeprefix("size="))
return 0
def _resize_bytes(value: str, current: int) -> int:
if value.startswith("+"):
return current + _size_bytes(value[1:])
result = _size_bytes(value)
if result < current:
raise ApiError(400, "shrinking disks is not supported")
return result
def _replace_disk_size(value: str, size: int) -> str:
parts = [part for part in value.split(",") if not part.startswith("size=")]
parts.append(f"size={size // 2**30}G" if size % 2**30 == 0 else f"size={size}")
return ",".join(parts)
+419
View File
@@ -0,0 +1,419 @@
"""Additional QEMU guest/agent/console endpoints with durable guest state."""
from __future__ import annotations
import json
import secrets
from collections.abc import Mapping
from typing import Any, cast
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.config import Settings
from app.handlers.qemu import _agent_resource, _database, _qemu_resource, _state, _values
from app.security.auth import issue_ticket
def _settings(request: Request) -> Settings:
return cast(Settings, request.app.state.settings)
async def _save_guest_state(request: Request, resource_id: Any, state: dict[str, Any]) -> None:
await _database(request).pool.execute(
"UPDATE resources SET state=$2::jsonb, version=version+1, updated_at=now() WHERE id=$1",
resource_id,
json.dumps(state, sort_keys=True),
)
async def _save_guest_config(request: Request, resource_id: Any, config: dict[str, Any]) -> None:
await _database(request).pool.execute(
"UPDATE virtual_machines SET config=$2::jsonb WHERE resource_id=$1",
resource_id,
json.dumps(config, sort_keys=True),
)
def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
async def agent_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await _agent_resource(request, _values(inputs))
return [
{"name": name}
for name in (
"exec",
"exec-status",
"file-read",
"file-write",
"fsfreeze-freeze",
"fsfreeze-status",
"fsfreeze-thaw",
"fstrim",
"get-fsinfo",
"get-memory-block-info",
"get-memory-blocks",
"get-timezone",
"get-users",
"get-vcpus",
"info",
"ping",
"set-user-password",
"shutdown",
"suspend-disk",
"suspend-hybrid",
"suspend-ram",
)
]
async def agent_post(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
command = str(payload.get("command") or "ping")
resource = await _agent_resource(request, payload)
state = _state(resource["state"])
agent = state.setdefault("agent", {})
agent["last_command"] = command
await _save_guest_state(request, resource["id"], state)
return {"result": {"command": command, "ok": 1}}
async def _agent_blob(command: str, request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
agent = state.get("agent")
blobs = agent.get("results") if isinstance(agent, dict) else None
if not isinstance(blobs, dict) or command not in blobs:
raise ApiError(404, f"agent command '{command}' has no stored result")
return {"result": blobs[command]}
async def agent_users(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _agent_blob("get-users", request, inputs)
async def agent_fsinfo(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _agent_blob("get-fsinfo", request, inputs)
async def agent_memory_block_info(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _agent_blob("get-memory-block-info", request, inputs)
async def agent_memory_blocks(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _agent_blob("get-memory-blocks", request, inputs)
async def agent_timezone(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _agent_blob("get-timezone", request, inputs)
async def agent_vcpus(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _agent_blob("get-vcpus", request, inputs)
async def agent_exec(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
resource = await _agent_resource(request, payload)
state = _state(resource["state"])
agent = state.setdefault("agent", {})
execs = agent.setdefault("exec", {})
pid = int(agent.get("next_pid", 1000)) + 1
agent["next_pid"] = pid
execs[str(pid)] = {
"exited": 1,
"exitcode": 0,
"out-data": "",
"err-data": "",
"command": payload.get("command"),
}
await _save_guest_state(request, resource["id"], state)
return {"pid": pid}
async def agent_exec_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
resource = await _agent_resource(request, payload)
pid = str(payload.get("pid") or "")
state = _state(resource["state"])
result = state.get("agent", {}).get("exec", {}).get(pid)
if not isinstance(result, dict):
raise ApiError(404, "exec process does not exist")
return {"result": result}
async def agent_file_read(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
resource = await _agent_resource(request, payload)
path = str(payload.get("file") or payload.get("path") or "/etc/hostname")
state = _state(resource["state"])
agent = state.get("agent")
files = agent.get("files") if isinstance(agent, dict) else None
if not isinstance(files, dict) or path not in files:
raise ApiError(404, "file does not exist")
content = str(files[path])
return {"result": {"content": content, "truncated": True}}
async def agent_file_write(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
resource = await _agent_resource(request, payload)
default_path = "guest-agent-out"
path = str(payload.get("file") or payload.get("path") or default_path)
content = str(payload.get("content") or "")
state = _state(resource["state"])
files = state.setdefault("agent", {}).setdefault("files", {})
files[path] = content
await _save_guest_state(request, resource["id"], state)
return {"result": None}
async def agent_fsfreeze(
request: Request, inputs: dict[str, Any], status: str
) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
agent = state.setdefault("agent", {})
agent["fsfreeze"] = status
agent.setdefault("results", {})["fsfreeze-status"] = status
await _save_guest_state(request, resource["id"], state)
return {"result": status}
async def agent_fsfreeze_freeze(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_fsfreeze(request, inputs, "frozen")
async def agent_fsfreeze_thaw(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_fsfreeze(request, inputs, "thawed")
async def agent_fsfreeze_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _agent_blob("fsfreeze-status", request, inputs)
async def agent_fstrim(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
agent = dict(state.get("agent") or {})
agent["last_fstrim"] = True
state["agent"] = agent
await _save_guest_state(request, resource["id"], state)
results = agent.get("results") if isinstance(agent.get("results"), dict) else {}
fstrim = results.get("fstrim") if isinstance(results, dict) else None
if isinstance(fstrim, dict):
return {"result": fstrim}
return {"result": {}}
async def agent_set_password(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
resource = await _agent_resource(request, payload)
username = str(payload.get("username") or "root")
state = _state(resource["state"])
passwords = state.setdefault("agent", {}).setdefault("passwords", {})
passwords[username] = True # store only presence, not secret
await _save_guest_state(request, resource["id"], state)
return {"result": None}
async def agent_shutdown(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
state["status"] = "stopped"
await _save_guest_state(request, resource["id"], state)
return {"result": None}
async def agent_suspend(request: Request, inputs: dict[str, Any], mode: str) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
state["status"] = "paused"
state.setdefault("agent", {})["suspend"] = mode
await _save_guest_state(request, resource["id"], state)
return {"result": None}
async def agent_suspend_disk(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_suspend(request, inputs, "disk")
async def agent_suspend_ram(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_suspend(request, inputs, "ram")
async def agent_suspend_hybrid(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await agent_suspend(request, inputs, "hybrid")
async def cloudinit_get(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
config = _state(resource["config"])
state = _state(resource["state"])
pending = cast(Mapping[str, Any], state.get("pending", {}))
keys = sorted(
{
key
for key in set(config) | set(pending)
if str(key).startswith(("ci", "ipconfig", "sshkeys", "nameserver", "searchdomain"))
}
)
return [
{
"key": key,
"value": str(config.get(key, "")),
"pending": str(pending[key]) if key in pending else None,
}
for key in keys
]
async def cloudinit_update(request: Request, inputs: dict[str, Any]) -> None:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
state["cloudinit_generation"] = int(state.get("cloudinit_generation") or 0) + 1
await _save_guest_state(request, resource["id"], state)
async def cloudinit_dump(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
dump = state.get("cloudinit_dump")
return str(dump) if dump is not None else ""
async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
rrd_state = state.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
series = state.get("rrddata")
return [dict(item) for item in series] if isinstance(series, list) else []
async def monitor(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
command = str(values.get("command") or "info status")
state = _state(resource["state"])
history = state.setdefault("monitor", [])
if not isinstance(history, list):
history = state["monitor"] = []
output = f"OK {command}"
history.append({"command": command, "output": output})
await _save_guest_state(request, resource["id"], state)
return output
async def sendkey(request: Request, inputs: dict[str, Any]) -> None:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
key = str(values.get("key") or "")
if not key:
raise ApiError(400, "parameter verification failed - 'key' missing")
state = _state(resource["state"])
keys = state.setdefault("sendkey", [])
if not isinstance(keys, list):
keys = state["sendkey"] = []
keys.append(key)
await _save_guest_state(request, resource["id"], state)
async def unlink(request: Request, inputs: dict[str, Any]) -> None:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
idlist = [
item.strip()
for item in str(values.get("idlist") or values.get("ids") or "").split(",")
if item.strip()
]
if not idlist:
raise ApiError(400, "parameter verification failed - 'idlist' missing")
config = _state(resource["config"])
for disk in idlist:
config.pop(disk, None)
await _save_guest_config(request, resource["id"], config)
state = _state(resource["state"])
state["config"] = config
await _save_guest_state(request, resource["id"], state)
async def _console_proxy(request: Request, inputs: dict[str, Any], kind: str) -> dict[str, Any]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
key = _settings(request).ticket_signing_key.get_secret_value().encode()
ticket = issue_ticket(str(request.state.principal), key)
port = 5900 + int(values["vmid"]) % 1000
state = _state(resource["state"])
consoles = state.setdefault("consoles", {})
payload = {
"type": kind,
"port": port,
"ticket": ticket,
"upid": (
f"UPID:{values['node']}:{secrets.token_hex(4)}:"
f"{kind}:{values['vmid']}:{request.state.principal}:"
),
"user": str(request.state.principal),
"cert": "",
}
if values.get("generate-password") or values.get("websocket"):
payload["password"] = secrets.token_urlsafe(8)
consoles[kind] = {k: v for k, v in payload.items() if k != "ticket"}
await _save_guest_state(request, resource["id"], state)
return payload
async def vncproxy(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _console_proxy(request, inputs, "vnc")
async def spiceproxy(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _console_proxy(request, inputs, "spice")
async def termproxy(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _console_proxy(request, inputs, "term")
async def mtunnel(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await _console_proxy(request, inputs, "mtunnel")
async def websocket_ticket(
request: Request, inputs: dict[str, Any], kind: str
) -> dict[str, Any]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
console = state.get("consoles", {}).get(kind) or {"port": 5900}
key = _settings(request).ticket_signing_key.get_secret_value().encode()
return {
"port": console.get("port", 5900),
"ticket": issue_ticket(str(request.state.principal), key),
}
async def vncwebsocket(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await websocket_ticket(request, inputs, "vnc")
async def mtunnelwebsocket(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await websocket_ticket(request, inputs, "mtunnel")
async def dbus_vmstate(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
state["dbus_vmstate"] = True
await _save_guest_state(request, resource["id"], state)
return {"result": "OK"}
base = "/nodes/{node}/qemu/{vmid}"
registry.register(f"{base}/agent", "GET", agent_index)
registry.register(f"{base}/agent", "POST", agent_post)
registry.register(f"{base}/agent/exec", "POST", agent_exec)
registry.register(f"{base}/agent/exec-status", "GET", agent_exec_status)
registry.register(f"{base}/agent/file-read", "GET", agent_file_read)
registry.register(f"{base}/agent/file-write", "POST", agent_file_write)
registry.register(f"{base}/agent/fsfreeze-freeze", "POST", agent_fsfreeze_freeze)
registry.register(f"{base}/agent/fsfreeze-status", "POST", agent_fsfreeze_status)
registry.register(f"{base}/agent/fsfreeze-thaw", "POST", agent_fsfreeze_thaw)
registry.register(f"{base}/agent/fstrim", "POST", agent_fstrim)
registry.register(f"{base}/agent/get-fsinfo", "GET", agent_fsinfo)
registry.register(f"{base}/agent/get-memory-block-info", "GET", agent_memory_block_info)
registry.register(f"{base}/agent/get-memory-blocks", "GET", agent_memory_blocks)
registry.register(f"{base}/agent/get-timezone", "GET", agent_timezone)
registry.register(f"{base}/agent/get-users", "GET", agent_users)
registry.register(f"{base}/agent/get-vcpus", "GET", agent_vcpus)
registry.register(f"{base}/agent/set-user-password", "POST", agent_set_password)
registry.register(f"{base}/agent/shutdown", "POST", agent_shutdown)
registry.register(f"{base}/agent/suspend-disk", "POST", agent_suspend_disk)
registry.register(f"{base}/agent/suspend-hybrid", "POST", agent_suspend_hybrid)
registry.register(f"{base}/agent/suspend-ram", "POST", agent_suspend_ram)
registry.register(f"{base}/cloudinit", "GET", cloudinit_get)
registry.register(f"{base}/cloudinit", "PUT", cloudinit_update)
registry.register(f"{base}/cloudinit/dump", "GET", cloudinit_dump)
registry.register(f"{base}/rrd", "GET", rrd)
registry.register(f"{base}/rrddata", "GET", rrddata)
registry.register(f"{base}/monitor", "POST", monitor)
registry.register(f"{base}/sendkey", "PUT", sendkey)
registry.register(f"{base}/unlink", "PUT", unlink)
registry.register(f"{base}/vncproxy", "POST", vncproxy)
registry.register(f"{base}/spiceproxy", "POST", spiceproxy)
registry.register(f"{base}/termproxy", "POST", termproxy)
registry.register(f"{base}/mtunnel", "POST", mtunnel)
registry.register(f"{base}/vncwebsocket", "GET", vncwebsocket)
registry.register(f"{base}/mtunnelwebsocket", "GET", mtunnelwebsocket)
registry.register(f"{base}/dbus-vmstate", "POST", dbus_vmstate)
+1085
View File
File diff suppressed because it is too large Load Diff
+659
View File
@@ -0,0 +1,659 @@
"""Storage semantic handlers."""
from __future__ import annotations
import json
from typing import Any
from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import (
database,
require_node,
require_value,
state,
storage_payload,
subdirs,
values,
)
from app.simulation.seed import CLUSTER_ID, stable_id
def register_storage_handlers(registry: HandlerRegistry) -> None:
async def _storage_row(request: Request, node: str | None, storage_id: str) -> Any:
row = await database(request).pool.fetchrow(
"""SELECT s.storage_id, s.storage_type, s.shared, s.capacity_bytes, s.used_bytes,
s.config, n.name AS node_name
FROM storages s
JOIN resources r ON r.id = s.resource_id
JOIN nodes n ON n.id = r.node_id
WHERE s.storage_id=$1 AND ($2::text IS NULL OR n.name=$2)""",
storage_id,
node,
)
if row is None:
raise ApiError(404, "storage does not exist")
return row
async def storage_ids(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
rows = await database(_request).pool.fetch(
"SELECT DISTINCT storage_id FROM storages ORDER BY storage_id"
)
return [{"storage": str(row["storage_id"])} for row in rows]
async def storage_create(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
storage_id = str(payload["storage"])
storage_type = str(payload.get("type") or "dir")
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM storages WHERE storage_id=$1)",
storage_id,
)
if exists:
raise ApiError(409, "storage ID already exists")
node = await database(request).pool.fetchrow(
"SELECT id, name FROM nodes ORDER BY name LIMIT 1"
)
if node is None:
raise ApiError(503, "no nodes available")
resource_id = stable_id(f"storage:{storage_id}")
config = {
key: value
for key, value in payload.items()
if key not in {"storage", "type", "nodes", "delete"}
}
if "content" in payload:
config["content"] = [
item.strip() for item in str(payload["content"]).split(",") if item.strip()
]
async with database(request).pool.acquire() as connection:
async with connection.transaction():
await connection.execute(
"""INSERT INTO resources(id, node_id, kind, external_id, state, cluster_id)
VALUES($1, $2, 'storage', $3, $4::jsonb, $5)""",
resource_id,
node["id"],
storage_id,
json.dumps({**config, "status": "available"}, sort_keys=True),
CLUSTER_ID,
)
await connection.execute(
"""INSERT INTO storages(
resource_id, cluster_id, storage_id, storage_type, shared, config
) VALUES($1, $2, $3, $4, $5, $6::jsonb)""",
resource_id,
CLUSTER_ID,
storage_id,
storage_type,
bool(payload.get("shared", False)),
json.dumps(config, sort_keys=True),
)
return {"storage": storage_id, "type": storage_type, "config": config}
async def storage_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
storage_id = str(values(inputs)["storage"])
row = await _storage_row(request, None, storage_id)
config = state(row["config"])
return {
"storage": storage_id,
"type": str(row["storage_type"]),
"shared": int(bool(row["shared"])),
**config,
}
async def storage_update(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
storage_id = str(values(inputs)["storage"])
row = await database(request).pool.fetchrow(
"""SELECT s.resource_id, s.config FROM storages s WHERE s.storage_id=$1""",
storage_id,
)
if row is None:
raise ApiError(404, "storage does not exist")
current = state(row["config"])
provided = values(inputs)
updated = {
**current,
**{
key: value
for key, value in provided.items()
if key not in {"storage", "delete", "digest"}
},
}
await database(request).pool.execute(
"UPDATE storages SET config=$2::jsonb WHERE storage_id=$1",
storage_id,
json.dumps(updated, sort_keys=True),
)
return updated
async def storage_delete(request: Request, inputs: dict[str, Any]) -> None:
storage_id = str(values(inputs)["storage"])
status = await database(request).pool.execute(
"""DELETE FROM resources r USING storages s
WHERE s.resource_id=r.id AND s.storage_id=$1""",
storage_id,
)
if status != "DELETE 1":
raise ApiError(404, "storage does not exist")
async def node_storage_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
rows = await database(request).pool.fetch(
"""SELECT s.storage_id, s.storage_type, s.shared,
s.capacity_bytes, s.used_bytes, s.config
FROM storages s
JOIN resources r ON r.id = s.resource_id
JOIN nodes n ON n.id = r.node_id
WHERE n.name=$1 OR s.shared = true
OR s.storage_id IN ('local', 'local-lvm')
ORDER BY s.storage_id""",
node,
)
return [storage_payload(row) for row in rows]
async def node_storage_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
return subdirs("content", "status", "upload")
async def node_storage_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
row = await _storage_row(request, None, storage_id)
return storage_payload(row)
async def node_storage_content(
request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
contents = await database(request).pool.fetch(
"""SELECT volume_id, content_type, size_bytes, metadata, created_at
FROM storage_contents WHERE storage_resource_id=$1 ORDER BY created_at DESC""",
resource_id,
)
backups = await database(request).pool.fetch(
"""SELECT b.volume_id, b.size_bytes, b.metadata, b.created_at, r.external_id AS vmid
FROM backups b
LEFT JOIN resources r ON r.id = b.resource_id
WHERE b.storage_resource_id=$1
ORDER BY b.created_at DESC""",
resource_id,
)
result: list[dict[str, Any]] = []
for item in contents:
metadata = state(item["metadata"])
result.append(
{
"volid": str(item["volume_id"]),
"content": str(item["content_type"]),
"size": int(item["size_bytes"]),
"format": metadata.get("format", "raw"),
"ctime": int(item["created_at"].timestamp()),
}
)
for item in backups:
metadata = state(item["metadata"])
result.append(
{
"volid": str(item["volume_id"]),
"content": "backup",
"size": int(item["size_bytes"]),
"format": "vma.zst",
"vmid": int(item["vmid"]) if item["vmid"] is not None else None,
"notes": metadata.get("notes-template"),
"ctime": int(item["created_at"].timestamp()),
}
)
return result
async def _content_item(
request: Request, storage_resource_id: object, volume_id: str
) -> dict[str, Any]:
row = await database(request).pool.fetchrow(
"""SELECT volume_id, content_type, size_bytes, metadata, created_at
FROM storage_contents WHERE storage_resource_id=$1 AND volume_id=$2""",
storage_resource_id,
volume_id,
)
if row is not None:
metadata = state(row["metadata"])
return {
"volid": str(row["volume_id"]),
"content": str(row["content_type"]),
"size": int(row["size_bytes"]),
"format": metadata.get("format", "raw"),
"ctime": int(row["created_at"].timestamp()),
}
backup = await database(request).pool.fetchrow(
"""SELECT b.volume_id, b.size_bytes, b.metadata, b.created_at, r.external_id AS vmid
FROM backups b
LEFT JOIN resources r ON r.id = b.resource_id
WHERE b.storage_resource_id=$1 AND b.volume_id=$2""",
storage_resource_id,
volume_id,
)
if backup is None:
raise ApiError(404, "volume does not exist")
metadata = state(backup["metadata"])
return {
"volid": str(backup["volume_id"]),
"content": "backup",
"size": int(backup["size_bytes"]),
"format": "vma.zst",
"vmid": int(backup["vmid"]) if backup["vmid"] is not None else None,
"notes": metadata.get("notes-template"),
"ctime": int(backup["created_at"].timestamp()),
}
async def node_storage_content_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
volume_id = str(values(inputs)["volume"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
return await _content_item(request, resource_id, volume_id)
async def node_storage_content_delete(request: Request, inputs: dict[str, Any]) -> None:
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
volume_id = str(values(inputs)["volume"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
status = await database(request).pool.execute(
"DELETE FROM storage_contents WHERE storage_resource_id=$1 AND volume_id=$2",
resource_id,
volume_id,
)
if status == "DELETE 1":
return
status = await database(request).pool.execute(
"DELETE FROM backups WHERE storage_resource_id=$1 AND volume_id=$2",
resource_id,
volume_id,
)
if status != "DELETE 1":
raise ApiError(404, "volume does not exist")
async def node_storage_allocate(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import _node_task
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
payload = values(inputs)
filename = str(payload.get("filename") or f"vm-{payload.get('vmid', '0')}-disk-0")
content_type = str(payload.get("content") or "images")
size_raw = str(payload.get("size") or "1G")
# Accept either integer bytes or Proxmox size strings like 32G.
try:
size = int(size_raw)
except ValueError:
from app.handlers.common import parse_size_bytes
try:
size = parse_size_bytes(size_raw)
except ValueError as error:
raise ApiError(400, f"invalid size '{size_raw}'") from error
format_name = str(payload.get("format") or "raw")
volume_id = f"{storage_id}:{content_type}/{filename}"
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
await database(request).pool.execute(
"""INSERT INTO storage_contents(
id, storage_resource_id, volume_id, content_type, size_bytes, metadata
) VALUES(gen_random_uuid(), $1, $2, $3, $4, $5::jsonb)
ON CONFLICT (storage_resource_id, volume_id) DO UPDATE
SET size_bytes=EXCLUDED.size_bytes,
content_type=EXCLUDED.content_type,
metadata=EXCLUDED.metadata""",
resource_id,
volume_id,
content_type,
size,
json.dumps(
{"filename": filename, "format": format_name, "source": "allocate"},
sort_keys=True,
),
)
return await _node_task(request, node=node, task_type="storage-allocate", worker="imgcopy")
async def node_storage_upload(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import _node_task
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
payload = values(inputs)
filename = str(payload.get("filename") or "upload.bin")
content_type = str(payload.get("content") or "iso")
raw_size = payload.get("size") or 0
try:
size = int(raw_size)
except (TypeError, ValueError) as error:
raise ApiError(400, "invalid size") from error
volume_id = f"{storage_id}:{content_type}/{filename}"
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
await database(request).pool.execute(
"""INSERT INTO storage_contents(
id, storage_resource_id, volume_id, content_type, size_bytes, metadata
) VALUES(gen_random_uuid(), $1, $2, $3, $4, $5::jsonb)
ON CONFLICT (storage_resource_id, volume_id) DO UPDATE
SET size_bytes=EXCLUDED.size_bytes,
content_type=EXCLUDED.content_type,
metadata=EXCLUDED.metadata""",
resource_id,
volume_id,
content_type,
size,
json.dumps({"filename": filename, "source": "upload"}, sort_keys=True),
)
return await _node_task(request, node=node, task_type="storage-upload", worker="download")
async def node_storage_prunebackups(
request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]] | str:
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
if request.method == "DELETE":
from app.handlers.nodes import _node_task
keep = int(values(inputs).get("keep-last") or values(inputs).get("keep_last") or 1)
await database(request).pool.execute(
"""DELETE FROM backups
WHERE storage_resource_id=$1 AND id IN (
SELECT id FROM backups
WHERE storage_resource_id=$1
ORDER BY created_at DESC
OFFSET $2
)""",
resource_id,
keep,
)
return await _node_task(
request, node=node, task_type="prune-backups", worker="prunebackups"
)
rows = await database(request).pool.fetch(
"""SELECT volume_id, size_bytes, created_at FROM backups
WHERE storage_resource_id=$1 ORDER BY created_at DESC""",
resource_id,
)
return [
{
"volid": str(row["volume_id"]),
"size": int(row["size_bytes"]),
"ctime": int(row["created_at"].timestamp()),
}
for row in rows
]
async def content_copy(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
storage_id = str(payload["storage"])
volume = str(payload["volume"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
row = await database(request).pool.fetchrow(
"""SELECT volume_id, content_type, size_bytes, metadata
FROM storage_contents WHERE storage_resource_id=$1 AND volume_id=$2""",
resource_id,
volume,
)
if row is None:
raise ApiError(404, "volume does not exist")
target = str(payload.get("target") or f"{volume}-copy")
await database(request).pool.execute(
"""INSERT INTO storage_contents(
id, storage_resource_id, volume_id, content_type, size_bytes, metadata
) VALUES(gen_random_uuid(), $1, $2, $3, $4, $5::jsonb)
ON CONFLICT (storage_resource_id, volume_id) DO UPDATE
SET size_bytes=EXCLUDED.size_bytes, metadata=EXCLUDED.metadata""",
resource_id,
target,
row["content_type"],
row["size_bytes"],
json.dumps({**state(row["metadata"]), "copied_from": volume}, sort_keys=True),
)
return f"UPID:{node}:copy:{target}"
async def content_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
node = str(payload["node"])
storage_id = str(payload["storage"])
volume = str(payload["volume"])
await require_node(request, node)
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
row = await database(request).pool.fetchrow(
"""SELECT metadata FROM storage_contents
WHERE storage_resource_id=$1 AND volume_id=$2""",
resource_id,
volume,
)
if row is None:
raise ApiError(404, "volume does not exist")
meta = state(row["metadata"])
if "notes" in payload:
meta["notes"] = payload["notes"]
if "protected" in payload:
meta["protected"] = int(bool(payload["protected"]))
await database(request).pool.execute(
"""UPDATE storage_contents SET metadata=$3::jsonb
WHERE storage_resource_id=$1 AND volume_id=$2""",
resource_id,
volume,
json.dumps(meta, sort_keys=True),
)
async def download_url(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
storage_id = str(payload["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
filename = str(payload.get("filename") or "download.bin")
content_type = str(payload.get("content") or "iso")
volume_id = f"{storage_id}:{content_type}/{filename}"
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
await database(request).pool.execute(
"""INSERT INTO storage_contents(
id, storage_resource_id, volume_id, content_type, size_bytes, metadata
) VALUES(gen_random_uuid(), $1, $2, $3, 0, $4::jsonb)
ON CONFLICT (storage_resource_id, volume_id) DO UPDATE
SET metadata=EXCLUDED.metadata""",
resource_id,
volume_id,
content_type,
json.dumps(
{
"filename": filename,
"url": payload.get("url"),
"source": "download-url",
},
sort_keys=True,
),
)
return f"UPID:{node}:download:{filename}"
async def oci_pull(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload["node"])
storage_id = str(payload["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
reference = str(payload.get("reference") or "image:latest")
filename = str(payload.get("filename") or reference.replace("/", "_"))
volume_id = f"{storage_id}:import/{filename}"
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
await database(request).pool.execute(
"""INSERT INTO storage_contents(
id, storage_resource_id, volume_id, content_type, size_bytes, metadata
) VALUES(gen_random_uuid(), $1, $2, 'import', 0, $3::jsonb)
ON CONFLICT (storage_resource_id, volume_id) DO UPDATE
SET metadata=EXCLUDED.metadata""",
resource_id,
volume_id,
json.dumps({"reference": reference, "source": "oci"}, sort_keys=True),
)
return f"UPID:{node}:oci-pull:{filename}"
async def file_restore_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
row = await _storage_row(request, None, str(payload["storage"]))
filepath = str(payload.get("filepath") or "/")
config = state(row["config"])
restore = config.get("file_restore")
if not isinstance(restore, dict):
return []
items = restore.get(filepath) or restore.get(filepath.rstrip("/") or "/")
return [dict(item) for item in items] if isinstance(items, list) else []
async def file_restore_download(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
downloads = config.get("file_restore_downloads")
filepath = str(payload.get("filepath") or "/")
if isinstance(downloads, dict) and filepath in downloads:
entry = downloads[filepath]
return dict(entry) if isinstance(entry, dict) else {"filepath": filepath}
return {
"download-url": f"/api2/json/nodes/{payload['node']}/storage/"
f"{payload['storage']}/file-restore/download",
"filepath": filepath,
"volume": payload.get("volume"),
}
async def storage_identity(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
identity = config.get("identity")
if isinstance(identity, dict):
return {
"storage": str(row["storage_id"]),
"type": str(identity.get("type") or row["storage_type"]),
"fingerprint": str(identity.get("fingerprint") or ""),
}
return {
"storage": str(row["storage_id"]),
"type": str(row["storage_type"]),
"fingerprint": "",
}
async def import_metadata(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
storage_id = str(payload["storage"])
volume = str(require_value(payload, "volume"))
row = await _storage_row(request, None, storage_id)
config = state(row["config"])
by_volume = config.get("import_metadata_by_volume")
if isinstance(by_volume, dict) and volume in by_volume:
meta = by_volume[volume]
return dict(meta) if isinstance(meta, dict) else {}
meta = config.get("import_metadata")
if isinstance(meta, dict):
return {"source": volume, **meta}
return {}
async def storage_rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
rrd_state = config.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def storage_rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
series = config.get("rrddata")
return [dict(item) for item in series] if isinstance(series, list) else []
registry.register("/storage", "GET", storage_ids)
registry.register("/storage", "POST", storage_create)
registry.register("/storage/{storage}", "GET", storage_get)
registry.register("/storage/{storage}", "PUT", storage_update)
registry.register("/storage/{storage}", "DELETE", storage_delete)
registry.register("/nodes/{node}/storage", "GET", node_storage_list)
registry.register("/nodes/{node}/storage/{storage}", "GET", node_storage_index)
registry.register("/nodes/{node}/storage/{storage}/status", "GET", node_storage_status)
registry.register("/nodes/{node}/storage/{storage}/content", "GET", node_storage_content)
registry.register("/nodes/{node}/storage/{storage}/content", "POST", node_storage_allocate)
registry.register(
"/nodes/{node}/storage/{storage}/content/{volume}", "GET", node_storage_content_get
)
registry.register(
"/nodes/{node}/storage/{storage}/content/{volume}", "DELETE", node_storage_content_delete
)
registry.register("/nodes/{node}/storage/{storage}/upload", "POST", node_storage_upload)
registry.register(
"/nodes/{node}/storage/{storage}/prunebackups", "GET", node_storage_prunebackups
)
registry.register(
"/nodes/{node}/storage/{storage}/prunebackups", "DELETE", node_storage_prunebackups
)
registry.register("/nodes/{node}/storage/{storage}/content/{volume}", "POST", content_copy)
registry.register("/nodes/{node}/storage/{storage}/content/{volume}", "PUT", content_update)
registry.register("/nodes/{node}/storage/{storage}/download-url", "POST", download_url)
registry.register("/nodes/{node}/storage/{storage}/oci-registry-pull", "POST", oci_pull)
registry.register("/nodes/{node}/storage/{storage}/file-restore/list", "GET", file_restore_list)
registry.register(
"/nodes/{node}/storage/{storage}/file-restore/download", "GET", file_restore_download
)
registry.register("/nodes/{node}/storage/{storage}/identity", "GET", storage_identity)
registry.register("/nodes/{node}/storage/{storage}/import-metadata", "GET", import_metadata)
registry.register("/nodes/{node}/storage/{storage}/rrd", "GET", storage_rrd)
registry.register("/nodes/{node}/storage/{storage}/rrddata", "GET", storage_rrddata)
+83 -36
View File
@@ -2,25 +2,31 @@
from __future__ import annotations from __future__ import annotations
import asyncio
from typing import cast from typing import cast
from fastapi import FastAPI, Response from fastapi import FastAPI
from app.api.errors import ApiError, api_error_handler, unhandled_exception_handler from app.api.errors import ApiError, api_error_handler, unhandled_exception_handler
from app.api.middleware import RequestContextMiddleware from app.api.middleware import RequestContextMiddleware
from app.api.registry import HandlerRegistry, register_contract_routes from app.api.openapi import openapi_tag_metadata
from app.compatibility import CompatibilityDimension, build_report, load_evidence_manifest from app.api.registry import HandlerRegistry
from app.config import Settings, get_settings from app.config import Settings, get_settings
from app.contracts.model import Snapshot from app.contracts.model import Snapshot
from app.contracts.runtime import apply_runtime_contract, contract_store_root
from app.db.pool import AsyncpgDatabase, Database from app.db.pool import AsyncpgDatabase, Database
from app.handlers.core import build_core_handlers from app.handlers.core import build_core_handlers
from app.lifespan import DatabaseFactory, WorkerFactory, create_lifespan, default_database_factory from app.lifespan import DatabaseFactory, WorkerFactory, create_lifespan, default_database_factory
from app.logging import configure_logging from app.logging import configure_logging
from app.observability.health import router as health_router from app.observability.health import router as health_router
from app.simulation.clock import AcceleratedClock from app.simulation.clock import AcceleratedClock
from app.tasks.backup import backup_handler
from app.tasks.lxc import lxc_handler
from app.tasks.qemu import qemu_handler from app.tasks.qemu import qemu_handler
from app.tasks.repository import TaskRepository from app.tasks.repository import TaskRepository
from app.tasks.worker import TaskWorker from app.tasks.worker import TaskWorker
from app.version import get_app_version
from app.web.routes import router as web_router
def create_app( def create_app(
@@ -39,11 +45,72 @@ def create_app(
def task_worker(database: Database) -> TaskWorker: def task_worker(database: Database) -> TaskWorker:
adapter = cast(AsyncpgDatabase, database) adapter = cast(AsyncpgDatabase, database)
repository = TaskRepository(adapter.pool) repository = TaskRepository(adapter.pool)
handler = qemu_handler(repository, AcceleratedClock(resolved.simulation_time_scale)) clock = AcceleratedClock(resolved.simulation_time_scale)
qemu = qemu_handler(repository, clock)
lxc = lxc_handler(repository, clock)
backup = backup_handler(repository, clock)
return TaskWorker( return TaskWorker(
repository, repository,
"simulator-worker", "simulator-worker",
{"qemu-start": handler, "qemu-stop": handler}, {
"qemu-clone": qemu,
"qemu-create": qemu,
"qemu-delete": qemu,
"qemu-reboot": qemu,
"qemu-reset": qemu,
"qemu-resume": qemu,
"qemu-shutdown": qemu,
"qemu-migrate": qemu,
"qemu-move-disk": qemu,
"qemu-snapshot-create": qemu,
"qemu-snapshot-delete": qemu,
"qemu-snapshot-rollback": qemu,
"qemu-start": qemu,
"qemu-stop": qemu,
"qemu-suspend": qemu,
"qemu-update": qemu,
"qemu-resize": qemu,
"qemu-template": qemu,
"lxc-clone": lxc,
"lxc-create": lxc,
"lxc-delete": lxc,
"lxc-migrate": lxc,
"lxc-reboot": lxc,
"lxc-resume": lxc,
"lxc-resize": lxc,
"lxc-shutdown": lxc,
"lxc-snapshot-create": lxc,
"lxc-snapshot-delete": lxc,
"lxc-snapshot-rollback": lxc,
"lxc-start": lxc,
"lxc-stop": lxc,
"lxc-suspend": lxc,
"vzdump": backup,
"aptupdate": backup,
"network-reload": backup,
"disk-initgpt": backup,
"disk-wipe": backup,
"disk-directory-create": backup,
"disk-directory-delete": backup,
"disk-lvm-create": backup,
"disk-lvm-delete": backup,
"disk-lvmthin-create": backup,
"disk-lvmthin-delete": backup,
"disk-zfs-create": backup,
"disk-zfs-delete": backup,
"service-start": backup,
"service-stop": backup,
"service-restart": backup,
"service-reload": backup,
"storage-allocate": backup,
"storage-upload": backup,
"prune-backups": backup,
"acme": backup,
"sdn-apply": backup,
"ceph-flags": backup,
"cluster-create": backup,
"cluster-join": backup,
},
concurrency=resolved.task_worker_concurrency, concurrency=resolved.task_worker_concurrency,
lease_seconds=resolved.task_lease_seconds, lease_seconds=resolved.task_lease_seconds,
) )
@@ -51,50 +118,30 @@ def create_app(
resolved_workers = (task_worker,) resolved_workers = (task_worker,)
app = FastAPI( app = FastAPI(
title=resolved.app_name, title=resolved.app_name,
version="0.1.0", version=get_app_version(),
openapi_tags=openapi_tag_metadata(),
lifespan=create_lifespan(resolved, database_factory, resolved_workers or ()), lifespan=create_lifespan(resolved, database_factory, resolved_workers or ()),
) )
app.state.settings = resolved app.state.settings = resolved
app.state.contract_swap_lock = asyncio.Lock()
app.add_middleware(RequestContextMiddleware, header_name=resolved.request_id_header) app.add_middleware(RequestContextMiddleware, header_name=resolved.request_id_header)
app.add_exception_handler(Exception, unhandled_exception_handler) app.add_exception_handler(Exception, unhandled_exception_handler)
app.add_exception_handler(ApiError, api_error_handler) app.add_exception_handler(ApiError, api_error_handler)
app.include_router(web_router)
app.include_router(health_router) app.include_router(health_router)
if resolved.contract_snapshot is not None: if resolved.contract_snapshot is not None:
snapshot = Snapshot.model_validate_json(resolved.contract_snapshot.read_bytes()) snapshot = Snapshot.model_validate_json(resolved.contract_snapshot.read_bytes())
resolved_handlers = handlers or build_core_handlers(resolved) resolved_handlers = handlers or build_core_handlers(resolved)
register_contract_routes( apply_runtime_contract(
app, app,
snapshot, snapshot,
resolved_handlers, handlers=resolved_handlers,
resolved.contract_fallback, store_root=contract_store_root(resolved),
fallback=resolved.contract_fallback,
settings=resolved,
require_evidence_match=True,
register_admin=True,
) )
declared = frozenset(
(path.path, method.verb) for path in snapshot.paths for method in path.methods
)
dimensions = {CompatibilityDimension.ROUTE_METHOD: declared}
if resolved.compatibility_evidence is not None:
evidence = load_evidence_manifest(resolved.compatibility_evidence)
if evidence.source_version != snapshot.source_version:
raise ValueError("compatibility evidence version does not match contract")
dimensions.update(evidence.dimension_map())
dimensions[CompatibilityDimension.ROUTE_METHOD] = declared
report = build_report(
snapshot,
implemented=resolved_handlers.keys() & declared,
dimensions=dimensions,
)
@app.get("/admin/compatibility", include_in_schema=False)
async def compatibility_report() -> dict[str, object]:
return report.as_json()
@app.get("/admin/compatibility.md", include_in_schema=False)
async def compatibility_report_markdown() -> Response:
return Response(report.as_markdown(), media_type="text/markdown")
@app.get("/admin/compatibility.html", include_in_schema=False)
async def compatibility_report_html() -> Response:
return Response(report.as_html(), media_type="text/html")
return app return app
+1 -1
View File
@@ -10,7 +10,7 @@ from pydantic import BaseModel
from app.db.pool import Database from app.db.pool import Database
from app.dependencies import get_database from app.dependencies import get_database
router = APIRouter(prefix="/health", tags=["health"]) router = APIRouter(prefix="/health", tags=["Simulator"])
class HealthResponse(BaseModel): class HealthResponse(BaseModel):
+7 -2
View File
@@ -57,17 +57,19 @@ def authorize(
entries: tuple[AclEntry, ...], entries: tuple[AclEntry, ...],
*, *,
token_privileges: frozenset[str] | None = None, token_privileges: frozenset[str] | None = None,
require_all: bool = True,
) -> bool: ) -> bool:
privileges = effective_privileges(principal, path, entries) privileges = effective_privileges(principal, path, entries)
if token_privileges is not None: if token_privileges is not None:
privileges &= token_privileges privileges &= token_privileges
return required <= privileges return required <= privileges if require_all else bool(required & privileges)
@dataclass(frozen=True, slots=True) @dataclass(frozen=True, slots=True)
class CapabilityRequirement: class CapabilityRequirement:
path: str path: str
privileges: frozenset[str] privileges: frozenset[str]
require_all: bool = True
def requirement_from_contract( def requirement_from_contract(
@@ -84,4 +86,7 @@ def requirement_from_contract(
raw_privileges = check[2] raw_privileges = check[2]
if not isinstance(raw_privileges, list): if not isinstance(raw_privileges, list):
return None return None
return CapabilityRequirement(raw_path, frozenset(str(item) for item in raw_privileges)) require_all = not (len(check) >= 4 and check[3] == "any")
return CapabilityRequirement(
raw_path, frozenset(str(item) for item in raw_privileges), require_all
)
+25
View File
@@ -0,0 +1,25 @@
"""Canonical Ceph cluster flag names (PVE /cluster/ceph/flags)."""
from __future__ import annotations
# Keep in sync with Proxmox `ceph_flags` API (majors 6-9).
CEPH_FLAG_DESCRIPTIONS: dict[str, str] = {
"nobackfill": "Pause backfill operations",
"nodeep-scrub": "Disable deep scrubbing",
"nodown": "Do not mark OSDs down",
"noin": "Do not mark OSDs in",
"noout": "Do not mark OSDs out",
"norebalance": "Pause rebalance",
"norecover": "Pause recovery",
"noscrub": "Disable scrubbing",
"notieragent": "Pause tiering agent",
"noup": "Do not mark OSDs up",
"pause": "Pause I/O",
}
def default_ceph_flags(*, enabled: frozenset[str] | None = None) -> dict[str, int]:
"""Return the full flag map; unset flags are ``0`` (PVE GET → ``false``)."""
active = enabled or frozenset()
return {name: (1 if name in active else 0) for name in CEPH_FLAG_DESCRIPTIONS}
+370
View File
@@ -0,0 +1,370 @@
"""Enterprise-scale demo cluster profile for realistic emulator workloads."""
from __future__ import annotations
import uuid
from collections import defaultdict
from collections.abc import Sequence
from app.simulation.seed import (
SeedNode,
SeedProfile,
SeedResource,
SeedTask,
_node,
_resource,
stable_id,
)
DEMO_NODE_COUNT = 20
DEMO_QEMU_COUNT = 850
DEMO_LXC_COUNT = 150
DEMO_CEPH_OSD_COUNT = 300
CEPH_TOTAL_BYTES = 5 * 1024**5
QEMU_VMID_START = 100
LXC_VMID_START = 10_000
QEMU_PREFIXES = (
"web",
"api",
"db",
"cache",
"mq",
"batch",
"ml",
"monitor",
"log",
"ci",
"k8s",
"vpn",
"ldap",
"git",
"proxy",
)
LXC_PREFIXES = (
"svc-nginx",
"svc-haproxy",
"svc-dns",
"svc-vault",
"svc-redis",
"mon-agent",
"backup-agent",
"ceph-mgr",
"lb-vip",
"proxy-squid",
"jump-host",
"ntp",
"syslog",
"metrics",
"bastion",
)
TIERS = ("prod", "staging", "dev", "qa", "dr")
POOLS = (
("production", 280),
("staging", 160),
("development", 130),
("qa", 100),
("gpu-workloads", 80),
("legacy", 100),
)
TASK_TYPES = (
"vzdump",
"qmstart",
"qmstop",
"qmmigrate",
"qmreboot",
"qmclone",
"aptupdate",
"startall",
"stopall",
"cephosd",
"pct-start",
"pct-stop",
)
def _even_node_slots(node_count: int, total: int, *, phase: int = 0) -> tuple[int, ...]:
"""Return `total` node indices distributed as evenly as possible."""
if total <= 0:
return ()
base, remainder = divmod(total, node_count)
slots: list[int] = []
for node_index in range(node_count):
slots.extend([node_index] * (base + (1 if node_index < remainder else 0)))
if phase:
phase %= len(slots)
slots = slots[phase:] + slots[:phase]
return tuple(slots)
def _even_sample(resources: Sequence[SeedResource], count: int) -> list[str]:
"""Pick `count` resource IDs spread evenly across the provided sequence."""
if count <= 0 or not resources:
return []
if count >= len(resources):
return [resource.external_id for resource in resources]
step = len(resources) / count
return [resources[int(index * step)].external_id for index in range(count)]
def _guest_name(prefixes: tuple[str, ...], index: int) -> str:
prefix = prefixes[index % len(prefixes)]
tier = TIERS[index % len(TIERS)]
return f"{tier}-{prefix}-{index:04d}"
def _qemu_state(vmid: int, index: int) -> dict[str, object]:
statuses = ("running", "running", "running", "running", "stopped", "paused")
cpus = (1, 2, 2, 4, 4, 8, 8, 16, 32)[index % 9]
memory_mb = (512, 1024, 2048, 4096, 8192, 16_384, 32_768, 65_536)[index % 8]
pool_name = POOLS[index % len(POOLS)][0]
return {
"name": _guest_name(QEMU_PREFIXES, index),
"status": statuses[index % len(statuses)],
"cpus": cpus,
"cores": cpus,
"memory": memory_mb,
"maxmem": memory_mb,
"pool": pool_name,
"tags": f"{TIERS[index % len(TIERS)]};{pool_name}",
"agent": index % 3 != 0,
"template": index % 97 == 0,
"onboot": index % 5 != 0,
"vmid": vmid,
}
def _lxc_state(vmid: int, index: int) -> dict[str, object]:
statuses = ("running", "running", "stopped", "stopped")
memory_mb = (256, 512, 1024, 2048, 4096)[index % 5]
pool_name = POOLS[(index + 2) % len(POOLS)][0]
return {
"name": _guest_name(LXC_PREFIXES, index),
"status": statuses[index % len(statuses)],
"cpus": (1, 1, 2, 2, 4)[index % 5],
"memory": memory_mb,
"maxmem": memory_mb,
"pool": pool_name,
"tags": f"container;{pool_name}",
"unprivileged": index % 4 != 0,
"template": index % 41 == 0,
"vmid": vmid,
}
def _demo_task(index: int, node: SeedNode, task_type: str, resource_id: str) -> SeedTask:
return SeedTask(
stable_id(f"demo-task:{index}:{task_type}:{resource_id}"),
f"UPID:{node.name}:{index:07X}:{index:07X}:67{index:06X}:"
f"{task_type}:{resource_id}:root@pam:",
task_type,
{"resource_id": resource_id, "node": node.name, "seeded": True},
)
def demo_cluster_profile() -> SeedProfile:
nodes = tuple(
_node(f"pve{index:02d}", "offline" if index == 19 else "online")
for index in range(1, DEMO_NODE_COUNT + 1)
)
node_count = len(nodes)
resources: list[SeedResource] = []
qemu_slots = _even_node_slots(node_count, DEMO_QEMU_COUNT, phase=0)
lxc_slots = _even_node_slots(node_count, DEMO_LXC_COUNT, phase=node_count // 2)
osd_slots = _even_node_slots(node_count, DEMO_CEPH_OSD_COUNT, phase=node_count // 4)
qemu_resources: list[SeedResource] = []
for offset, node_index in enumerate(qemu_slots):
vmid = QEMU_VMID_START + offset
resource = _resource(nodes[node_index], "qemu", str(vmid), _qemu_state(vmid, offset))
qemu_resources.append(resource)
resources.append(resource)
lxc_resources: list[SeedResource] = []
for offset, node_index in enumerate(lxc_slots):
vmid = LXC_VMID_START + offset
resource = _resource(nodes[node_index], "lxc", str(vmid), _lxc_state(vmid, offset))
lxc_resources.append(resource)
resources.append(resource)
guests_by_node: dict[uuid.UUID, list[SeedResource]] = defaultdict(list)
for guest in (*qemu_resources, *lxc_resources):
guests_by_node[guest.node_id].append(guest)
for node in nodes:
resources.append(
_resource(
node,
"storage",
f"local-{node.name}",
{
"content": ["iso", "vztmpl", "backup"],
"status": "available",
"storage_type": "dir",
},
)
)
resources.append(
_resource(
node,
"storage",
f"local-lvm-{node.name}",
{
"content": ["images", "rootdir"],
"status": "available",
"storage_type": "lvmthin",
"shared": False,
},
)
)
resources.append(
_resource(
node,
"storage",
f"backup-{node.name}",
{
"content": ["backup"],
"status": "available",
"storage_type": "dir",
"shared": False,
"total_bytes": 4 * 1024**4,
"used_bytes": int(2.2 * 1024**4),
},
)
)
if int(node.name[3:]) % 2 == 0:
resources.append(
_resource(
node,
"storage",
f"local-zfs-{node.name}",
{
"content": ["images", "rootdir"],
"status": "available",
"storage_type": "zfspool",
"shared": False,
},
)
)
used_bytes = int(CEPH_TOTAL_BYTES * 0.62)
resources.append(
_resource(
nodes[0],
"storage",
"ceph-prod",
{
"content": ["images", "rootdir", "backup"],
"shared": True,
"status": "available",
"storage_type": "ceph",
"ceph_pool": "rbd",
"total_bytes": CEPH_TOTAL_BYTES,
"used_bytes": used_bytes,
"osd_count": DEMO_CEPH_OSD_COUNT,
},
)
)
resources.append(
_resource(
nodes[node_count // 2],
"storage",
"nfs-backup",
{
"content": ["backup", "iso"],
"shared": True,
"status": "available",
"storage_type": "nfs",
"total_bytes": 80 * 1024**4,
"used_bytes": 52 * 1024**4,
},
)
)
for osd_index, node_index in enumerate(osd_slots):
node = nodes[node_index]
osd_id = osd_index
weight = round(0.8 + (osd_index % 17) * 0.05, 2)
size_bytes = CEPH_TOTAL_BYTES // DEMO_CEPH_OSD_COUNT
resources.append(
_resource(
node,
"ceph-osd",
f"osd.{osd_id}",
{
"osd_id": osd_id,
"status": "up" if osd_index != 42 else "down",
"in": osd_index != 42,
"weight": weight,
"size_bytes": size_bytes,
"used_bytes": int(size_bytes * (0.55 + (osd_index % 10) * 0.03)),
"device_class": "ssd" if osd_index % 4 else "hdd",
},
)
)
qemu_by_node = [
sorted(guests_by_node[node.id], key=lambda resource: int(resource.external_id))
for node in nodes
]
pool_guest_cursor = 0
for pool_index, (pool_id, member_count) in enumerate(POOLS):
pool_guests: list[SeedResource] = []
per_node, extra = divmod(member_count, node_count)
for node_index, node_guests in enumerate(qemu_by_node):
take = per_node + (1 if node_index < extra else 0)
start = (pool_guest_cursor + node_index) % len(node_guests) if node_guests else 0
for offset in range(take):
if not node_guests:
break
pool_guests.append(node_guests[(start + offset) % len(node_guests)])
pool_guest_cursor += member_count
pool_guests.sort(key=lambda resource: int(resource.external_id))
resources.append(
_resource(
nodes[pool_index % node_count],
"pool",
pool_id,
{
"members": _even_sample(pool_guests, min(40, len(pool_guests))),
"member_count": len(pool_guests),
"comment": f"Simulated {pool_id} pool",
},
)
)
ha_guests = [
qemu_resources[int(index * len(qemu_resources) / min(120, len(qemu_resources)))]
for index in range(min(120, len(qemu_resources)))
]
for ha_index, guest in enumerate(ha_guests):
node = next(node for node in nodes if node.id == guest.node_id)
resources.append(
_resource(
node,
"ha",
f"vm:{guest.external_id}",
{
"state": "started" if ha_index % 5 else "stopped",
"group": "critical-services",
"max_relocate": 2,
"max_restart": 3,
},
)
)
tasks: list[SeedTask] = []
guest_cycle = sorted(
(*qemu_resources, *lxc_resources),
key=lambda resource: (resource.node_id, int(resource.external_id)),
)
for index in range(1, 251):
guest = guest_cycle[(index - 1) % len(guest_cycle)]
node = next(node for node in nodes if node.id == guest.node_id)
task_type = TASK_TYPES[index % len(TASK_TYPES)]
tasks.append(_demo_task(index, node, task_type, guest.external_id))
return SeedProfile("demo-cluster", nodes, tuple(resources), tuple(tasks))
+2531 -93
View File
File diff suppressed because it is too large Load Diff
+4
View File
@@ -29,6 +29,10 @@ class InvalidTransitionError(ValueError):
TRANSITIONS: dict[tuple[VmState, str], tuple[VmState, VmState]] = { TRANSITIONS: dict[tuple[VmState, str], tuple[VmState, VmState]] = {
(VmState.STOPPED, "start"): (VmState.STARTING, VmState.RUNNING), (VmState.STOPPED, "start"): (VmState.STARTING, VmState.RUNNING),
(VmState.RUNNING, "stop"): (VmState.STOPPING, VmState.STOPPED), (VmState.RUNNING, "stop"): (VmState.STOPPING, VmState.STOPPED),
(VmState.RUNNING, "shutdown"): (VmState.STOPPING, VmState.STOPPED),
(VmState.RUNNING, "reboot"): (VmState.STOPPING, VmState.RUNNING),
(VmState.RUNNING, "reset"): (VmState.STOPPING, VmState.RUNNING),
(VmState.RUNNING, "suspend"): (VmState.PAUSING, VmState.PAUSED),
(VmState.RUNNING, "pause"): (VmState.PAUSING, VmState.PAUSED), (VmState.RUNNING, "pause"): (VmState.PAUSING, VmState.PAUSED),
(VmState.PAUSED, "resume"): (VmState.RESUMING, VmState.RUNNING), (VmState.PAUSED, "resume"): (VmState.RESUMING, VmState.RUNNING),
(VmState.RUNNING, "migrate"): (VmState.MIGRATING, VmState.RUNNING), (VmState.RUNNING, "migrate"): (VmState.MIGRATING, VmState.RUNNING),
+282
View File
@@ -0,0 +1,282 @@
"""Probe every declared contract method across majors 6-9.
Order: GET, then PUT, then POST, then DELETE. Critical buckets
(``unimplemented_501``, ``unsupported_message``, ``server_5xx``,
``exception``) must stay empty — this module backs the CI surface gate.
"""
from __future__ import annotations
import asyncio
import json
import os
import re
import sys
from collections import Counter, defaultdict
from pathlib import Path
from typing import Any
from urllib.parse import quote
import asyncpg # type: ignore[import-untyped]
from httpx import ASGITransport, AsyncClient
from pydantic import SecretStr
from app.config import Settings
from app.contracts.examples import path_param_example, schema_example
from app.contracts.model import Method, Snapshot
from app.db.migrations import migrate
from app.db.pool import AsyncpgDatabase
from app.main import create_app
from app.simulation.seed import apply_seed, lab_profile
from app.web.contract_catalog import get_major_releases
_BUNDLED_9 = Path(
"contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json"
)
_EVIDENCE_9 = Path("evidence/pve-9.2.3.json")
_PATH_RE = re.compile(r"\{([^{}]+)\}")
_FORBIDDEN = re.compile(
r"not supported in the emulator|not implemented in the simulator|"
r"handler pending for this contract method|is not supported in the (emulator|simulator)",
re.I,
)
_EXTRA_PATH: dict[str, object] = {
"groupid": "admins",
"roleid": "Administrator",
"zone": "localnet",
"vnet": "vnet0",
"subnet": "10.0.0.0-24",
"controller": "evpn1",
"dns": "dns1",
"ipam": "pve",
"flag": "noout",
"osdid": "0",
"monid": "0",
"id": "example",
"cputype": "custom1",
"pci-id-or-mapping": "0000:00:1f.0",
"rule": "rule1",
"sid": "vm:100",
"pos": "0",
"cidr": "10.0.0.0/24",
"tokenid": "automation",
"fabric_id": "fab1",
"node_id": "pve01",
"url_seq": "1",
"route-map-id": "rm1",
"order": "10",
"userid": "root@pam",
"realm": "pam",
"name": "example",
"plugin": "example",
"target": "example",
}
def _path_value(name: str) -> str:
value = path_param_example(name)
if value is None:
value = _EXTRA_PATH.get(name, "example")
return str(value)
def render_path(template: str) -> str:
def replace(match: re.Match[str]) -> str:
return quote(str(_path_value(match.group(1))), safe="@._-")
return _PATH_RE.sub(replace, template)
def body_for(method: Method, path_template: str) -> dict[str, Any]:
path_names = set(_PATH_RE.findall(path_template))
payload: dict[str, Any] = {}
for parameter in method.parameters:
if parameter.name in path_names:
continue
if parameter.definition.optional:
continue
payload[parameter.name] = schema_example(parameter.definition, name=parameter.name)
return payload
def classify(status: int, text: str) -> str:
if _FORBIDDEN.search(text or ""):
return "unsupported_message"
if status == 501:
return "unimplemented_501"
if 200 <= status < 300:
return "success_2xx"
if status in {401, 403}:
return "auth_401_403"
if status in {400, 404, 405, 409, 412, 422, 423}:
return "client_4xx"
if status >= 500:
return "server_5xx"
return f"other_{status}"
async def prepare_db(url: str) -> None:
connection = await asyncpg.connect(url)
try:
await migrate(connection)
await apply_seed(connection, lab_profile())
finally:
await connection.close()
async def login(client: AsyncClient) -> str:
response = await client.post(
"/api2/json/access/ticket",
content="username=root%40pam&password=secret",
headers={"content-type": "application/x-www-form-urlencoded"},
)
response.raise_for_status()
data = response.json()["data"]
client.cookies.set("PVEAuthCookie", data["ticket"])
return str(data["CSRFPreventionToken"])
async def probe_major(
client: AsyncClient,
csrf: str,
major: int,
snapshot: Snapshot,
) -> dict[str, Any]:
apply = await client.post("/ui/api/contract/apply", params={"major": major})
apply.raise_for_status()
applied = apply.json()
report = (await client.get("/admin/compatibility")).json()
by_verb: dict[str, Counter[str]] = defaultdict(Counter)
failures: list[dict[str, Any]] = []
samples_ok: dict[str, int] = Counter()
methods = [(path.path, method) for path in snapshot.paths for method in path.methods]
order = {"GET": 0, "PUT": 1, "POST": 2, "DELETE": 3}
methods.sort(key=lambda item: (order.get(item[1].verb.upper(), 9), item[0]))
for path_template, method in methods:
verb = method.verb.upper()
url = f"/api2/json{render_path(path_template)}"
headers = {"CSRFPreventionToken": csrf} if verb != "GET" else {}
payload = body_for(method, path_template)
try:
if verb == "GET":
response = await client.get(url, headers=headers, params=payload or None)
elif verb == "PUT":
response = await client.put(url, data=payload or {}, headers=headers)
elif verb == "POST":
response = await client.post(url, data=payload or {}, headers=headers)
elif verb == "DELETE":
response = await client.request("DELETE", url, data=payload or {}, headers=headers)
else:
continue
except Exception as exc:
by_verb[verb]["exception"] += 1
failures.append(
{
"verb": verb,
"path": path_template,
"error": str(exc)[:200],
"bucket": "exception",
}
)
continue
text = response.text
bucket = classify(response.status_code, text)
by_verb[verb][bucket] += 1
samples_ok[verb] += int(bucket == "success_2xx")
if bucket in {"unimplemented_501", "unsupported_message", "server_5xx", "exception"}:
failures.append(
{
"verb": verb,
"path": path_template,
"status": response.status_code,
"bucket": bucket,
"body": text[:240],
}
)
levels = report.get("levels") or {}
dims = report.get("dimensions") or {}
return {
"major": major,
"version": snapshot.source_version,
"apply": applied,
"declared": report.get("total_declared"),
"implemented": (levels.get("implemented") or {}).get("count"),
"verified": (levels.get("verified") or {}).get("count"),
"dimensions_min": min((item.get("count") or 0) for item in dims.values()) if dims else 0,
"by_verb": {verb: dict(counter) for verb, counter in by_verb.items()},
"success_by_verb": dict(samples_ok),
"failure_count": len(failures),
"failures": failures[:40],
}
async def run_probe(*, database_url: str | None = None) -> list[dict[str, Any]]:
"""Run the full surface probe and return per-major result dicts."""
url = database_url or os.environ.get("TEST_DATABASE_URL") or os.environ.get("DATABASE_URL")
if not url:
raise RuntimeError("TEST_DATABASE_URL / DATABASE_URL required")
await prepare_db(url)
settings = Settings(
database_url=SecretStr(url),
contract_snapshot=_BUNDLED_9,
compatibility_evidence=_EVIDENCE_9,
ticket_signing_key=SecretStr("development-only-signing-key-change-me"),
)
app = create_app(settings=settings, database_factory=lambda s: AsyncpgDatabase(s))
releases = {release.major: release for release in get_major_releases()}
results: list[dict[str, Any]] = []
async with app.router.lifespan_context(app):
async with AsyncClient(
transport=ASGITransport(app=app),
base_url="http://test",
timeout=30.0,
) as client:
csrf = await login(client)
for major in (6, 7, 8, 9):
release = releases[major]
if release.bundled_revision is None:
raise RuntimeError(f"missing bundled revision for major {major}")
snapshot = Snapshot.model_validate_json(
(Path("contracts") / release.bundled_revision / "snapshot.json").read_bytes()
)
# Keep a single DB seed for the whole run to avoid deadlocks with
# the live app pool during DELETE FROM cascades.
results.append(await probe_major(client, csrf, major, snapshot))
return results
async def main() -> int:
try:
results = await run_probe()
except RuntimeError as error:
print(str(error), file=sys.stderr)
return 2
out = Path("evidence/_api_surface_probe.json")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(results, indent=2, ensure_ascii=False), encoding="utf-8")
print(json.dumps({"ok": True, "report": str(out), "majors": len(results)}))
for item in results:
print(
f"PVE {item['version']}: declared={item['declared']} "
f"impl={item['implemented']} ver={item['verified']} "
f"fail={item['failure_count']}"
)
for verb in ("GET", "PUT", "POST", "DELETE"):
buckets = item["by_verb"].get(verb) or {}
if not buckets:
continue
total = sum(buckets.values())
print(f" {verb}: total={total} {buckets}")
critical = sum(int(item["failure_count"]) for item in results)
return 1 if critical else 0
if __name__ == "__main__":
raise SystemExit(asyncio.run(main()))
+91
View File
@@ -0,0 +1,91 @@
"""Worker semantics for backup/vzdump tasks."""
from __future__ import annotations
import json
from typing import Any
from app.simulation.clock import Clock
from app.simulation.seed import stable_id
from app.tasks.repository import Task, TaskRepository
from app.tasks.worker import TaskHandler
def backup_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
async def execute(task: Task) -> dict[str, Any]:
if task.task_type == "aptupdate":
node = str(task.payload.get("node", "unknown"))
await repository.append_log(task.id, f"starting apt update on {node}")
await clock.sleep(1.0)
async with repository.pool.acquire() as connection:
metadata = await connection.fetchval(
"SELECT metadata FROM nodes WHERE name=$1",
node,
)
if metadata is not None:
payload = json.loads(metadata) if isinstance(metadata, str) else dict(metadata)
ops = payload.setdefault("ops", {})
apt = ops.setdefault("apt", {})
packages = list(apt.get("packages") or [])
for package in packages:
if isinstance(package, dict) and package.get("Status") == "upgradable":
package["Status"] = "installed"
if package.get("Version"):
package["OldVersion"] = package["Version"]
apt["packages"] = packages
apt["update"] = {"status": "stopped", "exitstatus": "OK"}
payload["ops"] = ops
await connection.execute(
"UPDATE nodes SET metadata=$2::jsonb, updated_at=now() WHERE name=$1",
node,
json.dumps(payload, sort_keys=True),
)
await repository.append_log(task.id, "apt update finished")
return {"status": "OK"}
# Node/cluster ops whose durable side effects were applied by the API handler.
if task.task_type != "vzdump":
await repository.append_log(task.id, f"{task.task_type} completed")
return {"status": "OK"}
node = str(task.payload["node"])
vmids = [str(item) for item in task.payload.get("vmids", [])]
storage_id = str(task.payload.get("storage") or "nfs-backup")
await repository.append_log(task.id, f"starting vzdump on {node} for {len(vmids)} guests")
async with repository.pool.acquire() as connection:
storage_resource_id = await connection.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
if storage_resource_id is None:
raise ValueError(f"storage {storage_id} does not exist")
created = 0
for index, vmid in enumerate(vmids):
resource_id = await connection.fetchval(
"""SELECT r.id FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='qemu' AND r.external_id=$2""",
node,
vmid,
)
volume_id = f"backup/vzdump-qemu-{vmid}-{task.id.hex[:8]}-{index:04d}.vma.zst"
await connection.execute(
"""INSERT INTO backups(
id, resource_id, storage_resource_id, volume_id, size_bytes, metadata
) VALUES($1, $2, $3, $4, $5, $6::jsonb)
ON CONFLICT (storage_resource_id, volume_id) DO NOTHING""",
stable_id(f"backup-task:{task.id}:{vmid}"),
resource_id,
storage_resource_id,
volume_id,
(8 + index) * 1024**3,
json.dumps(
{"mode": task.payload.get("mode", "snapshot"), "type": "vzdump"},
sort_keys=True,
),
)
created += 1
await repository.append_log(task.id, f"backup archive created: {volume_id}")
await repository.append_log(task.id, f"vzdump finished ({created} archives)")
return {"created": created}
return execute
+248
View File
@@ -0,0 +1,248 @@
"""Worker semantics for asynchronous LXC transitions."""
from __future__ import annotations
import json
import uuid
from collections.abc import Mapping
from typing import Any, cast
from app.simulation.clock import Clock
from app.simulation.transitions import VmState, plan_transition
from app.tasks.repository import Task, TaskRepository
from app.tasks.worker import TaskHandler
def lxc_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
async def execute(task: Task) -> dict[str, Any]:
operation = task.task_type.removeprefix("lxc-")
if operation == "create":
return await _create(repository, task, clock)
if operation == "clone":
return await _clone(repository, task)
resource_id = uuid.UUID(str(task.payload["resource_id"]))
if operation == "delete":
return await _delete(repository, task, resource_id)
if operation.startswith("snapshot-"):
return await _snapshot(
repository, task, resource_id, operation.removeprefix("snapshot-")
)
if operation == "migrate" or operation == "remote-migrate":
return await _migrate(repository, task, resource_id, clock)
if operation == "resize":
await repository.append_log(task.id, "container resize completed")
return {"status": "OK"}
async with repository.pool.acquire() as connection:
row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id)
if row is None:
raise ValueError("resource disappeared")
state = _object(row["state"])
transition = plan_transition(VmState(str(state["status"])), operation)
state["status"] = transition.intermediate
await connection.execute(
"UPDATE resources SET state=$2::jsonb WHERE id=$1",
resource_id,
json.dumps(state),
)
await repository.append_log(task.id, f"container {operation} started")
await clock.sleep(1.0)
async with repository.pool.acquire() as connection:
state["status"] = transition.after
await connection.execute(
"UPDATE resources SET state=$2::jsonb WHERE id=$1",
resource_id,
json.dumps(state),
)
await repository.append_log(task.id, f"container {operation} completed")
return {"status": str(transition.after)}
return execute
async def _create(repository: TaskRepository, task: Task, clock: Clock) -> dict[str, Any]:
node, vmid = str(task.payload["node"]), int(task.payload["vmid"])
config = dict(task.payload.get("config", {}))
start = bool(task.payload.get("start", False))
resource_id = uuid.uuid4()
status = "running" if start else "stopped"
state = {"status": status, **config}
async with repository.pool.acquire() as connection:
async with connection.transaction():
node_row = await connection.fetchrow(
"SELECT id, cluster_id FROM nodes WHERE name=$1", node
)
if node_row is None:
raise ValueError("node disappeared")
await connection.execute(
"""INSERT INTO resources(
id, node_id, cluster_id, kind, external_id, state, metadata
) VALUES($1, $2, $3, 'lxc', $4, $5::jsonb, '{}'::jsonb)""",
resource_id,
node_row["id"],
node_row["cluster_id"],
str(vmid),
json.dumps(state, sort_keys=True),
)
await connection.execute(
"""INSERT INTO containers(resource_id, cluster_id, vmid, config)
VALUES($1, $2, $3, $4::jsonb)""",
resource_id,
node_row["cluster_id"],
vmid,
json.dumps(config, sort_keys=True),
)
if start:
await clock.sleep(0.5)
await repository.append_log(task.id, f"container {vmid} created")
return {"vmid": vmid, "status": status}
async def _delete(repository: TaskRepository, task: Task, resource_id: uuid.UUID) -> dict[str, Any]:
async with repository.pool.acquire() as connection:
status = await connection.execute("DELETE FROM resources WHERE id=$1", resource_id)
if status != "DELETE 1":
raise ValueError("resource disappeared")
await repository.append_log(task.id, "container deleted")
return {"deleted": True}
async def _snapshot(
repository: TaskRepository,
task: Task,
resource_id: uuid.UUID,
operation: str,
) -> dict[str, Any]:
name = str(task.payload["snapname"])
async with repository.pool.acquire() as connection:
async with connection.transaction():
if operation == "create":
row = await connection.fetchrow(
"""SELECT r.state, c.config FROM resources r
JOIN containers c ON c.resource_id=r.id WHERE r.id=$1""",
resource_id,
)
if row is None:
raise ValueError("resource disappeared")
captured = {
"resource_state": _object(row["state"]),
"config": _object(row["config"]),
}
await connection.execute(
"""INSERT INTO snapshots(id, resource_id, name, description, state)
VALUES($1, $2, $3, $4, $5::jsonb)""",
uuid.uuid4(),
resource_id,
name,
str(task.payload.get("description", "")),
json.dumps(captured, sort_keys=True),
)
elif operation == "delete":
status = await connection.execute(
"DELETE FROM snapshots WHERE resource_id=$1 AND name=$2",
resource_id,
name,
)
if status != "DELETE 1":
raise ValueError("snapshot disappeared")
elif operation == "rollback":
row = await connection.fetchrow(
"SELECT state FROM snapshots WHERE resource_id=$1 AND name=$2",
resource_id,
name,
)
if row is None:
raise ValueError("snapshot disappeared")
captured = _object(row["state"])
state = dict(cast(Mapping[str, Any], captured["resource_state"]))
await connection.execute(
"""UPDATE resources SET state=$2::jsonb, version=version+1,
updated_at=now() WHERE id=$1""",
resource_id,
json.dumps(state, sort_keys=True),
)
await connection.execute(
"UPDATE containers SET config=$2::jsonb WHERE resource_id=$1",
resource_id,
json.dumps(captured["config"], sort_keys=True),
)
else:
raise ValueError(f"unsupported snapshot operation: {operation}")
await repository.append_log(task.id, f"snapshot {name} {operation} completed")
return {"snapshot": name, "operation": operation}
async def _clone(repository: TaskRepository, task: Task) -> dict[str, Any]:
source_id = uuid.UUID(str(task.payload["source_resource_id"]))
target_id = uuid.uuid4()
node, vmid = str(task.payload["node"]), int(task.payload["vmid"])
async with repository.pool.acquire() as connection:
async with connection.transaction():
source = await connection.fetchrow(
"""SELECT r.state, c.config FROM resources r
JOIN containers c ON c.resource_id=r.id WHERE r.id=$1""",
source_id,
)
target = await connection.fetchrow(
"SELECT id, cluster_id FROM nodes WHERE name=$1", node
)
if source is None or target is None:
raise ValueError("clone source or target disappeared")
config = _object(source["config"])
if task.payload.get("name") is not None:
config["hostname"] = task.payload["name"]
state = {**_object(source["state"]), **config, "status": "stopped"}
await connection.execute(
"""INSERT INTO resources(id,node_id,cluster_id,kind,external_id,state,metadata)
VALUES($1,$2,$3,'lxc',$4,$5::jsonb,'{}'::jsonb)""",
target_id,
target["id"],
target["cluster_id"],
str(vmid),
json.dumps(state),
)
await connection.execute(
"""INSERT INTO containers(resource_id,cluster_id,vmid,config)
VALUES($1,$2,$3,$4::jsonb)""",
target_id,
target["cluster_id"],
vmid,
json.dumps(config),
)
await repository.append_log(task.id, f"container cloned to {vmid}")
return {"vmid": vmid, "node": node}
async def _migrate(
repository: TaskRepository, task: Task, resource_id: uuid.UUID, clock: Clock
) -> dict[str, Any]:
target = str(task.payload["target"])
async with repository.pool.acquire() as connection:
row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id)
if row is None:
raise ValueError("resource disappeared")
state = _object(row["state"])
transition = plan_transition(VmState(str(state["status"])), "migrate")
state["status"] = transition.intermediate
await connection.execute(
"UPDATE resources SET state=$2::jsonb WHERE id=$1", resource_id, json.dumps(state)
)
await repository.append_log(task.id, f"migration to {target} started")
await clock.sleep(1.0)
async with repository.pool.acquire() as connection:
node = await connection.fetchrow("SELECT id FROM nodes WHERE name=$1", target)
if node is None:
raise ValueError("target node disappeared")
state["status"] = transition.after
await connection.execute(
"""UPDATE resources SET node_id=$2,state=$3::jsonb,version=version+1,
updated_at=now() WHERE id=$1""",
resource_id,
node["id"],
json.dumps(state),
)
await repository.append_log(task.id, f"migration to {target} completed")
return {"node": target, "status": str(transition.after)}
def _object(value: object) -> dict[str, Any]:
return json.loads(value) if isinstance(value, str) else dict(cast(Mapping[str, Any], value))
+292 -1
View File
@@ -4,6 +4,8 @@ from __future__ import annotations
import json import json
import uuid import uuid
from collections.abc import Mapping
from typing import Any, cast
from app.simulation.clock import Clock from app.simulation.clock import Clock
from app.simulation.transitions import VmState, plan_transition from app.simulation.transitions import VmState, plan_transition
@@ -12,9 +14,29 @@ from app.tasks.worker import TaskHandler
def qemu_handler(repository: TaskRepository, clock: Clock) -> TaskHandler: def qemu_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
async def execute(task: Task) -> dict[str, str]: async def execute(task: Task) -> dict[str, Any]:
operation = task.task_type.removeprefix("qemu-") operation = task.task_type.removeprefix("qemu-")
if operation == "create":
return await _create(repository, task)
if operation == "clone":
return await _clone(repository, task)
resource_id = uuid.UUID(str(task.payload["resource_id"])) resource_id = uuid.UUID(str(task.payload["resource_id"]))
if operation == "update":
return await _update(repository, task, resource_id)
if operation == "delete":
return await _delete(repository, task, resource_id)
if operation.startswith("snapshot-"):
return await _snapshot(
repository, task, resource_id, operation.removeprefix("snapshot-")
)
if operation == "migrate" or operation == "remote-migrate":
return await _migrate(repository, task, resource_id, clock)
if operation == "move-disk":
return await _move_disk(repository, task, resource_id)
if operation in {"resize", "template"}:
# Side effects already persisted by the API handler; complete the worker.
await repository.append_log(task.id, f"VM {operation} completed")
return {"status": "OK"}
async with repository.pool.acquire() as connection: async with repository.pool.acquire() as connection:
row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id) row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id)
if row is None: if row is None:
@@ -41,3 +63,272 @@ def qemu_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
return {"status": str(transition.after)} return {"status": str(transition.after)}
return execute return execute
async def _create(repository: TaskRepository, task: Task) -> dict[str, Any]:
from app.simulation.seed import enrich_guest_state
node, vmid = str(task.payload["node"]), int(task.payload["vmid"])
config = dict(task.payload.get("config", {}))
resource_id = uuid.uuid4()
state = enrich_guest_state({"status": "stopped", **config}, kind="qemu", vmid=str(vmid))
async with repository.pool.acquire() as connection:
async with connection.transaction():
node_row = await connection.fetchrow(
"SELECT id, cluster_id FROM nodes WHERE name=$1", node
)
if node_row is None:
raise ValueError("node disappeared")
await connection.execute(
"""INSERT INTO resources(
id, node_id, cluster_id, kind, external_id, state, metadata
) VALUES($1, $2, $3, 'qemu', $4, $5::jsonb, '{}'::jsonb)""",
resource_id,
node_row["id"],
node_row["cluster_id"],
str(vmid),
json.dumps(state, sort_keys=True),
)
await connection.execute(
"""INSERT INTO virtual_machines(resource_id, cluster_id, vmid, config)
VALUES($1, $2, $3, $4::jsonb)""",
resource_id,
node_row["cluster_id"],
vmid,
json.dumps(config, sort_keys=True),
)
await repository.append_log(task.id, f"VM {vmid} created")
return {"vmid": vmid, "status": "stopped"}
async def _update(repository: TaskRepository, task: Task, resource_id: uuid.UUID) -> dict[str, Any]:
changes = dict(task.payload.get("changes", {}))
delete_keys = tuple(str(task.payload.get("delete", "")).split(","))
async with repository.pool.acquire() as connection:
async with connection.transaction():
row = await connection.fetchrow(
"""SELECT r.state, v.config FROM resources r
JOIN virtual_machines v ON v.resource_id=r.id WHERE r.id=$1""",
resource_id,
)
if row is None:
raise ValueError("resource disappeared")
state = _object(row["state"])
config = _object(row["config"])
config.update(changes)
for key in delete_keys:
if key:
config.pop(key, None)
state.pop(key, None)
state.update(changes)
await connection.execute(
"""UPDATE resources SET state=$2::jsonb, version=version+1,
updated_at=now() WHERE id=$1""",
resource_id,
json.dumps(state, sort_keys=True),
)
await connection.execute(
"UPDATE virtual_machines SET config=$2::jsonb WHERE resource_id=$1",
resource_id,
json.dumps(config, sort_keys=True),
)
await repository.append_log(task.id, "VM configuration updated")
return {"updated": sorted(changes), "deleted": sorted(key for key in delete_keys if key)}
async def _delete(repository: TaskRepository, task: Task, resource_id: uuid.UUID) -> dict[str, Any]:
async with repository.pool.acquire() as connection:
status = await connection.execute("DELETE FROM resources WHERE id=$1", resource_id)
if status != "DELETE 1":
raise ValueError("resource disappeared")
await repository.append_log(task.id, "VM deleted")
return {"deleted": True}
async def _snapshot(
repository: TaskRepository,
task: Task,
resource_id: uuid.UUID,
operation: str,
) -> dict[str, Any]:
name = str(task.payload["snapname"])
async with repository.pool.acquire() as connection:
async with connection.transaction():
if operation == "create":
row = await connection.fetchrow(
"""SELECT r.state, v.config FROM resources r
JOIN virtual_machines v ON v.resource_id=r.id WHERE r.id=$1""",
resource_id,
)
if row is None:
raise ValueError("resource disappeared")
captured = {
"resource_state": _object(row["state"]),
"config": _object(row["config"]),
"vmstate": bool(task.payload.get("vmstate", False)),
}
await connection.execute(
"""INSERT INTO snapshots(id, resource_id, name, description, state)
VALUES($1, $2, $3, $4, $5::jsonb)""",
uuid.uuid4(),
resource_id,
name,
str(task.payload.get("description", "")),
json.dumps(captured, sort_keys=True),
)
elif operation == "delete":
status = await connection.execute(
"DELETE FROM snapshots WHERE resource_id=$1 AND name=$2",
resource_id,
name,
)
if status != "DELETE 1":
raise ValueError("snapshot disappeared")
elif operation == "rollback":
row = await connection.fetchrow(
"SELECT state FROM snapshots WHERE resource_id=$1 AND name=$2",
resource_id,
name,
)
if row is None:
raise ValueError("snapshot disappeared")
captured = _object(row["state"])
state = dict(cast(Mapping[str, Any], captured["resource_state"]))
if bool(task.payload.get("start", False)):
state["status"] = "running"
await connection.execute(
"""UPDATE resources SET state=$2::jsonb, version=version+1,
updated_at=now() WHERE id=$1""",
resource_id,
json.dumps(state, sort_keys=True),
)
await connection.execute(
"UPDATE virtual_machines SET config=$2::jsonb WHERE resource_id=$1",
resource_id,
json.dumps(captured["config"], sort_keys=True),
)
else:
raise ValueError(f"unsupported snapshot operation: {operation}")
await repository.append_log(task.id, f"snapshot {name} {operation} completed")
return {"snapshot": name, "operation": operation}
async def _clone(repository: TaskRepository, task: Task) -> dict[str, Any]:
source_id = uuid.UUID(str(task.payload["source_resource_id"]))
target_id = uuid.uuid4()
node, vmid = str(task.payload["node"]), int(task.payload["vmid"])
async with repository.pool.acquire() as connection:
async with connection.transaction():
source = await connection.fetchrow(
"""SELECT r.state, v.config FROM resources r
JOIN virtual_machines v ON v.resource_id=r.id WHERE r.id=$1""",
source_id,
)
target = await connection.fetchrow(
"SELECT id, cluster_id FROM nodes WHERE name=$1", node
)
if source is None or target is None:
raise ValueError("clone source or target disappeared")
config = _object(source["config"])
if task.payload.get("name") is not None:
config["name"] = task.payload["name"]
state = {**_object(source["state"]), **config, "status": "stopped"}
await connection.execute(
"""INSERT INTO resources(id,node_id,cluster_id,kind,external_id,state,metadata)
VALUES($1,$2,$3,'qemu',$4,$5::jsonb,'{}'::jsonb)""",
target_id,
target["id"],
target["cluster_id"],
str(vmid),
json.dumps(state),
)
await connection.execute(
"""INSERT INTO virtual_machines(resource_id,cluster_id,vmid,config)
VALUES($1,$2,$3,$4::jsonb)""",
target_id,
target["cluster_id"],
vmid,
json.dumps(config),
)
await repository.append_log(task.id, f"VM cloned to {vmid}")
return {"vmid": vmid, "node": node}
async def _migrate(
repository: TaskRepository, task: Task, resource_id: uuid.UUID, clock: Clock
) -> dict[str, Any]:
target = str(task.payload["target"])
async with repository.pool.acquire() as connection:
row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id)
if row is None:
raise ValueError("resource disappeared")
state = _object(row["state"])
transition = plan_transition(VmState(str(state["status"])), "migrate")
state["status"] = transition.intermediate
await connection.execute(
"UPDATE resources SET state=$2::jsonb WHERE id=$1", resource_id, json.dumps(state)
)
await repository.append_log(task.id, f"migration to {target} started")
await clock.sleep(1.0)
async with repository.pool.acquire() as connection:
node = await connection.fetchrow("SELECT id FROM nodes WHERE name=$1", target)
if node is None:
raise ValueError("target node disappeared")
state["status"] = transition.after
await connection.execute(
"""UPDATE resources SET node_id=$2,state=$3::jsonb,version=version+1,
updated_at=now() WHERE id=$1""",
resource_id,
node["id"],
json.dumps(state),
)
await repository.append_log(task.id, f"migration to {target} completed")
return {"node": target, "status": str(transition.after)}
async def _move_disk(
repository: TaskRepository, task: Task, resource_id: uuid.UUID
) -> dict[str, Any]:
disk = str(task.payload["disk"])
target_disk = str(task.payload["target_disk"])
storage = str(task.payload["storage"])
async with repository.pool.acquire() as connection:
async with connection.transaction():
row = await connection.fetchrow(
"SELECT config FROM virtual_machines WHERE resource_id=$1", resource_id
)
if row is None:
raise ValueError("resource disappeared")
config = _object(row["config"])
if disk not in config:
raise ValueError("disk disappeared")
original = str(config[disk])
suffix = original.split(":", 1)[1] if ":" in original else original
config[target_disk] = f"{storage}:{suffix}"
if bool(task.payload.get("delete", True)) and target_disk != disk:
config.pop(disk, None)
await connection.execute(
"UPDATE virtual_machines SET config=$2::jsonb WHERE resource_id=$1",
resource_id,
json.dumps(config, sort_keys=True),
)
await connection.execute(
"""UPDATE resources SET state=state || $2::jsonb, version=version+1,
updated_at=now() WHERE id=$1""",
resource_id,
json.dumps({target_disk: config[target_disk]}, sort_keys=True),
)
await connection.execute(
"""UPDATE vm_disks SET device=$2,storage_id=$3
WHERE resource_id=$1 AND device=$4""",
resource_id,
target_disk,
storage,
disk,
)
await repository.append_log(task.id, f"disk {disk} moved to {storage}")
return {"disk": target_disk, "storage": storage}
def _object(value: object) -> dict[str, Any]:
return json.loads(value) if isinstance(value, str) else dict(cast(Mapping[str, Any], value))
+16
View File
@@ -3,6 +3,8 @@
from __future__ import annotations from __future__ import annotations
import re import re
import secrets
import time
from dataclasses import dataclass from dataclasses import dataclass
UPID_RE = re.compile( UPID_RE = re.compile(
@@ -57,3 +59,17 @@ class Upid:
task_id=values["task_id"], task_id=values["task_id"],
user=values["user"], user=values["user"],
) )
@classmethod
def allocate(cls, node: str, task_type: str, task_id: str, user: str) -> Upid:
"""Build a collision-resistant UPID for a new task."""
return cls(
node=node,
pid=secrets.randbits(32),
process_start=secrets.randbits(32),
start_time=int(time.time()) & 0xFFFFFFFF,
task_type=task_type,
task_id=str(task_id),
user=user,
)
+130
View File
@@ -0,0 +1,130 @@
"""SemVer source of truth: VERSION file (synced to pyproject + Helm)."""
from __future__ import annotations
import re
from functools import lru_cache
from pathlib import Path
_VERSION_FILE = Path(__file__).resolve().parents[1] / "VERSION"
_PROJECT_ROOT = _VERSION_FILE.parent
_PYPROJECT_FILE = _PROJECT_ROOT / "pyproject.toml"
_CHART_FILE = _PROJECT_ROOT / "helm" / "proxmox-api-simulator" / "Chart.yaml"
_VALUES_FILE = _PROJECT_ROOT / "helm" / "proxmox-api-simulator" / "values.yaml"
_DEFAULT_VERSION = "0.1.0"
_VERSION_RE = re.compile(r"^(\d+)\.(\d+)\.(\d+)$")
def _read_version_file(path: Path) -> str | None:
try:
text = path.read_text(encoding="utf-8").strip()
except OSError:
return None
if _VERSION_RE.match(text):
return text
return None
@lru_cache
def get_app_version() -> str:
for path in (
Path("/app/VERSION"),
_VERSION_FILE,
Path.cwd() / "VERSION",
):
found = _read_version_file(path)
if found:
return found
try:
from importlib.metadata import version
meta = version("proxmox-api-simulator")
if _VERSION_RE.match(meta):
return meta
except Exception:
pass
return _DEFAULT_VERSION
def get_app_version_label() -> str:
clear_version_cache()
return f"v{get_app_version()}"
def parse_version(value: str) -> tuple[int, int, int]:
match = _VERSION_RE.match(value.strip())
if not match:
raise ValueError(f"Invalid semantic version: {value!r}")
return int(match.group(1)), int(match.group(2)), int(match.group(3))
def format_version(major: int, minor: int, patch: int) -> str:
return f"{major}.{minor}.{patch}"
def clear_version_cache() -> None:
get_app_version.cache_clear()
def write_project_version(version: str) -> None:
parse_version(version)
_VERSION_FILE.write_text(f"{version}\n", encoding="utf-8")
pyproject = _PYPROJECT_FILE.read_text(encoding="utf-8")
pyproject, count = re.subn(
r'^version = ".*"$',
f'version = "{version}"',
pyproject,
count=1,
flags=re.MULTILINE,
)
if count != 1:
raise RuntimeError("Failed to update pyproject.toml version")
_PYPROJECT_FILE.write_text(pyproject, encoding="utf-8")
chart = _CHART_FILE.read_text(encoding="utf-8")
chart, chart_count = re.subn(
r"^version: .*$",
f"version: {version}",
chart,
count=1,
flags=re.MULTILINE,
)
chart, app_count = re.subn(
r'^appVersion: ".*"$',
f'appVersion: "{version}"',
chart,
count=1,
flags=re.MULTILINE,
)
if chart_count != 1 or app_count != 1:
raise RuntimeError("Failed to update Chart.yaml version")
_CHART_FILE.write_text(chart, encoding="utf-8")
values = _VALUES_FILE.read_text(encoding="utf-8")
values, values_count = re.subn(
r'^(\s*tag:\s*).*$',
rf'\1"{version}"',
values,
count=1,
flags=re.MULTILINE,
)
if values_count != 1:
raise RuntimeError("Failed to update helm values.yaml image.tag")
_VALUES_FILE.write_text(values, encoding="utf-8")
clear_version_cache()
def bump_patch_version() -> str:
major, minor, patch = parse_version(get_app_version())
version = format_version(major, minor, patch + 1)
write_project_version(version)
return version
def bump_minor_version() -> str:
major, minor, patch = parse_version(get_app_version())
version = format_version(major, minor + 1, 0)
write_project_version(version)
return version
View File
+14
View File
@@ -0,0 +1,14 @@
"""Static web console assets."""
from __future__ import annotations
from pathlib import Path
_WEB_ROOT = Path(__file__).parent
_CONSOLE_HTML = _WEB_ROOT / "index.html"
def console_html() -> str:
"""Return the latest console markup from disk."""
return _CONSOLE_HTML.read_text(encoding="utf-8")
+77
View File
@@ -0,0 +1,77 @@
"""Catalog-scoped compatibility summaries for the web console."""
from __future__ import annotations
from app.compatibility import CompatibilityDimension, CompatibilityReport, build_report
from app.config import Settings
from app.contracts.model import Snapshot
from app.web.contract_catalog import major_release
MethodKey = tuple[str, str]
def compatibility_payload(
snapshot: Snapshot,
major: int,
*,
implemented_methods: frozenset[MethodKey] | None,
runtime_report: CompatibilityReport | None,
runtime_version: str | None,
settings: Settings | None,
) -> dict[str, object]:
"""Build a compatibility summary for the selected catalog major."""
declared = frozenset(
(contract_path.path, method.verb.upper())
for contract_path in snapshot.paths
for method in contract_path.methods
)
implemented = (implemented_methods or frozenset()) & declared
if runtime_report is not None and runtime_report.source_version == snapshot.source_version:
payload = runtime_report.as_json()
evidence_scope = "full"
else:
dimensions: dict[CompatibilityDimension, frozenset[MethodKey]] = {
CompatibilityDimension.ROUTE_METHOD: declared,
}
if runtime_report is not None:
for dimension, methods in runtime_report.dimensions.items():
if dimension == CompatibilityDimension.ROUTE_METHOD:
continue
dimensions[dimension] = methods & declared
else:
for dimension in CompatibilityDimension:
if dimension != CompatibilityDimension.ROUTE_METHOD:
dimensions[dimension] = frozenset()
empty: frozenset[MethodKey] = frozenset()
if runtime_report is None:
observed = empty
verified = empty
incompatible = empty
regressions = empty
else:
observed = runtime_report.observed & declared
verified = runtime_report.verified & declared
incompatible = runtime_report.incompatible & declared
regressions = runtime_report.regressions & declared
catalog_report = build_report(
snapshot,
implemented=implemented,
observed=observed,
verified=verified,
dimensions=dimensions,
incompatible=incompatible,
regressions=regressions,
)
payload = catalog_report.as_json()
evidence_scope = "catalog"
release = major_release(major, settings)
payload["major"] = major
payload["catalog_version"] = snapshot.source_version
payload["latest_version"] = release.latest_version
payload["runtime_version"] = runtime_version
payload["evidence_scope"] = evidence_scope
return payload
+293
View File
@@ -0,0 +1,293 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Proxmox API Emulator</title>
<style>
:root {
color-scheme: dark;
--bg: #0b1020;
--panel: #121a2f;
--panel-2: #18233d;
--border: #2a3555;
--text: #e8eefc;
--muted: #93a0c0;
--accent: #5b8cff;
--accent-2: #3dd6c6;
--danger: #ff6b7a;
--ok: #4ade80;
--shadow: 0 18px 50px rgba(0, 0, 0, 0.35);
--radius: 16px;
--mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
--sans: Inter, ui-sans-serif, system-ui, -apple-system, Segoe UI, sans-serif;
}
* { box-sizing: border-box; }
body {
margin: 0;
min-height: 100vh;
font-family: var(--sans);
background:
radial-gradient(circle at top left, rgba(91, 140, 255, 0.18), transparent 28%),
radial-gradient(circle at top right, rgba(61, 214, 198, 0.12), transparent 24%),
var(--bg);
color: var(--text);
}
.wrap { max-width: 1180px; margin: 0 auto; padding: 32px 20px 48px; }
header {
display: flex; justify-content: space-between; align-items: flex-start;
gap: 16px; margin-bottom: 28px;
}
h1 { margin: 0 0 8px; font-size: clamp(1.6rem, 2vw, 2.2rem); }
.subtitle { color: var(--muted); max-width: 56ch; line-height: 1.5; }
.links { display: flex; gap: 10px; flex-wrap: wrap; }
.links a {
color: var(--text); text-decoration: none; border: 1px solid var(--border);
background: rgba(255,255,255,0.03); padding: 8px 12px; border-radius: 999px;
font-size: 0.92rem;
}
.grid {
display: grid; grid-template-columns: repeat(12, 1fr); gap: 18px;
}
.card {
background: linear-gradient(180deg, rgba(255,255,255,0.03), transparent), var(--panel);
border: 1px solid var(--border); border-radius: var(--radius); padding: 18px;
box-shadow: var(--shadow);
}
.span-4 { grid-column: span 4; }
.span-5 { grid-column: span 5; }
.span-7 { grid-column: span 7; }
.span-8 { grid-column: span 8; }
.span-12 { grid-column: span 12; }
@media (max-width: 900px) {
.span-4, .span-5, .span-7, .span-8 { grid-column: span 12; }
}
h2 { margin: 0 0 14px; font-size: 1rem; letter-spacing: 0.02em; }
label { display: block; font-size: 0.85rem; color: var(--muted); margin-bottom: 6px; }
input, select, textarea, button {
width: 100%; font: inherit; border-radius: 12px; border: 1px solid var(--border);
background: var(--panel-2); color: var(--text); padding: 11px 12px;
}
textarea { min-height: 120px; font-family: var(--mono); font-size: 0.88rem; resize: vertical; }
button {
cursor: pointer; background: linear-gradient(135deg, var(--accent), #4068d8);
border: none; font-weight: 600; margin-top: 10px;
}
button.secondary {
background: transparent; border: 1px solid var(--border); font-weight: 500;
}
.row { display: grid; grid-template-columns: 1fr 1fr; gap: 10px; }
.stats { display: grid; grid-template-columns: repeat(3, 1fr); gap: 10px; }
.stat {
background: var(--panel-2); border: 1px solid var(--border); border-radius: 14px;
padding: 14px;
}
.stat .label { color: var(--muted); font-size: 0.82rem; margin-bottom: 6px; }
.stat .value { font-size: 1.25rem; font-weight: 700; }
.pill {
display: inline-flex; align-items: center; gap: 8px; padding: 6px 10px;
border-radius: 999px; background: rgba(74, 222, 128, 0.12); color: var(--ok);
border: 1px solid rgba(74, 222, 128, 0.25); font-size: 0.85rem;
}
.pill.warn { color: #fbbf24; background: rgba(251, 191, 36, 0.12); border-color: rgba(251,191,36,0.25); }
pre {
margin: 0; padding: 14px; border-radius: 14px; background: #070b14;
border: 1px solid var(--border); overflow: auto; font-family: var(--mono);
font-size: 0.84rem; line-height: 1.45; min-height: 180px;
}
.actions { display: flex; gap: 10px; flex-wrap: wrap; }
.actions button { width: auto; flex: 1 1 140px; margin-top: 0; }
.hint { color: var(--muted); font-size: 0.85rem; margin-top: 8px; line-height: 1.4; }
.error { color: var(--danger); }
</style>
</head>
<body>
<div class="wrap">
<header>
<div>
<h1>Proxmox API Emulator</h1>
<p class="subtitle">
Stateful API console for the emulator. Authenticate, inspect cluster state,
send requests, and follow UPID tasks against the same `/api2/json` surface
used by proxmoxer and other clients.
</p>
</div>
<div class="links">
<a href="/docs" target="_blank" rel="noreferrer">OpenAPI</a>
<a href="/admin/compatibility.html" target="_blank" rel="noreferrer">Compatibility</a>
<a href="/health/ready" target="_blank" rel="noreferrer">Readiness</a>
</div>
</header>
<div class="grid">
<section class="card span-4">
<h2>Authentication</h2>
<label for="username">Username</label>
<input id="username" value="root@pam" autocomplete="username">
<label for="password">Password</label>
<input id="password" type="password" value="secret" autocomplete="current-password">
<button id="login-btn">Get ticket</button>
<p class="hint">Development credentials from the deterministic seed profile.</p>
<label for="csrf">CSRFPreventionToken</label>
<input id="csrf" readonly>
<div id="auth-status" class="pill warn" style="margin-top:12px;">Not authenticated</div>
</section>
<section class="card span-8">
<h2>Cluster snapshot</h2>
<div class="stats">
<div class="stat"><div class="label">PVE version</div><div class="value" id="stat-version">—</div></div>
<div class="stat"><div class="label">Nodes</div><div class="value" id="stat-nodes">—</div></div>
<div class="stat"><div class="label">Resources</div><div class="value" id="stat-resources">—</div></div>
</div>
<div class="actions" style="margin-top:14px;">
<button class="secondary" data-action="refresh">Refresh overview</button>
<button class="secondary" data-action="nodes">GET /nodes</button>
<button class="secondary" data-action="resources">GET /cluster/resources</button>
</div>
</section>
<section class="card span-5">
<h2>Quick request</h2>
<div class="row">
<div>
<label for="method">Method</label>
<select id="method">
<option>GET</option>
<option>POST</option>
<option>PUT</option>
<option>DELETE</option>
</select>
</div>
<div>
<label for="path">Path</label>
<input id="path" value="/api2/json/nodes/pve01/qemu">
</div>
</div>
<label for="body">Body (JSON or form key=value)</label>
<textarea id="body"></textarea>
<button id="send-btn">Send request</button>
<p class="hint">Mutating requests automatically attach the CSRF header when a ticket is present.</p>
</section>
<section class="card span-7">
<h2>Response</h2>
<pre id="output">Waiting for a request…</pre>
</section>
<section class="card span-12">
<h2>Task monitor</h2>
<div class="row">
<div>
<label for="node">Node</label>
<input id="node" value="pve01">
</div>
<div>
<label for="upid">UPID</label>
<input id="upid" placeholder="UPID returned by a mutation">
</div>
</div>
<div class="actions" style="margin-top:10px;">
<button class="secondary" id="task-status-btn">Task status</button>
<button class="secondary" id="task-log-btn">Task log</button>
</div>
</section>
</div>
</div>
<script>
const output = document.getElementById("output");
const csrfInput = document.getElementById("csrf");
const authStatus = document.getElementById("auth-status");
function show(data, status) {
output.textContent = JSON.stringify({ status, data }, null, 2);
}
async function apiFetch(path, options = {}) {
const headers = new Headers(options.headers || {});
if (csrfInput.value && options.method && options.method !== "GET") {
headers.set("CSRFPreventionToken", csrfInput.value);
}
const response = await fetch(path, { ...options, headers, credentials: "include" });
let data;
try { data = await response.json(); } catch { data = { raw: await response.text() }; }
show(data, response.status);
return { response, data };
}
async function refreshOverview() {
const version = await apiFetch("/api2/json/version");
document.getElementById("stat-version").textContent =
version.data?.data?.version || "unknown";
const nodes = await apiFetch("/api2/json/nodes");
const resources = await apiFetch("/api2/json/cluster/resources");
document.getElementById("stat-nodes").textContent =
Array.isArray(nodes.data?.data) ? nodes.data.data.length : "0";
document.getElementById("stat-resources").textContent =
Array.isArray(resources.data?.data) ? resources.data.data.length : "0";
}
document.getElementById("login-btn").addEventListener("click", async () => {
const username = document.getElementById("username").value;
const password = document.getElementById("password").value;
const body = new URLSearchParams({ username, password });
const result = await apiFetch("/api2/json/access/ticket", {
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body,
});
const ticket = result.data?.data;
if (ticket?.CSRFPreventionToken) {
csrfInput.value = ticket.CSRFPreventionToken;
authStatus.textContent = `Authenticated as ${ticket.username}`;
authStatus.classList.remove("warn");
} else {
authStatus.textContent = "Authentication failed";
authStatus.classList.add("warn");
}
await refreshOverview();
});
document.getElementById("send-btn").addEventListener("click", async () => {
const method = document.getElementById("method").value;
const path = document.getElementById("path").value;
const rawBody = document.getElementById("body").value.trim();
const options = { method };
if (rawBody && method !== "GET" && method !== "DELETE") {
if (rawBody.startsWith("{")) {
options.headers = { "Content-Type": "application/json" };
options.body = rawBody;
} else {
options.headers = { "Content-Type": "application/x-www-form-urlencoded" };
options.body = rawBody;
}
}
await apiFetch(path, options);
});
document.querySelector('[data-action="refresh"]').addEventListener("click", refreshOverview);
document.querySelector('[data-action="nodes"]').addEventListener("click", () =>
apiFetch("/api2/json/nodes"));
document.querySelector('[data-action="resources"]').addEventListener("click", () =>
apiFetch("/api2/json/cluster/resources"));
document.getElementById("task-status-btn").addEventListener("click", async () => {
const node = document.getElementById("node").value;
const upid = encodeURIComponent(document.getElementById("upid").value);
await apiFetch(`/api2/json/nodes/${node}/tasks/${upid}/status`);
});
document.getElementById("task-log-btn").addEventListener("click", async () => {
const node = document.getElementById("node").value;
const upid = encodeURIComponent(document.getElementById("upid").value);
await apiFetch(`/api2/json/nodes/${node}/tasks/${upid}/log`);
});
refreshOverview().catch((error) => {
output.textContent = String(error);
output.classList.add("error");
});
</script>
</body>
</html>
+464
View File
@@ -0,0 +1,464 @@
"""Lazy-loaded Proxmox API contract catalog grouped by major PVE release."""
from __future__ import annotations
import asyncio
import re
from dataclasses import dataclass
from datetime import UTC, datetime
from functools import lru_cache
from pathlib import Path
from app.api.openapi import contract_openapi_tag
from app.config import Settings
from app.contracts.examples import path_param_example, schema_example, wire_param_name
from app.contracts.importer import RemoteSourceImporter
from app.contracts.model import Method, Parameter, Snapshot
from app.contracts.normalize import normalize_snapshot
from app.contracts.source import ApiViewerParser
from app.contracts.store import RevisionStore
_PATH_PARAM = re.compile(r"\{([^{}]+)\}")
@dataclass(frozen=True, slots=True)
class MajorReleaseMeta:
major: int
latest_version: str
bundled_revision: str | None = None
@dataclass(frozen=True, slots=True)
class MajorRelease:
major: int
latest_version: str
artifact_url: str
bundled_revision: str | None = None
_MAJOR_METADATA: tuple[MajorReleaseMeta, ...] = (
MajorReleaseMeta(
major=6,
latest_version="6.4-15",
bundled_revision="96cd7121e75cdb3efd58f79ca988f6b235a2f28e6f7eae276ae243f65d8a6724",
),
MajorReleaseMeta(
major=7,
latest_version="7.4-16",
bundled_revision="2cf632fa6ea4939ca9cb7998ade688150db25b0684600f53ac0ca95730f1d99f",
),
MajorReleaseMeta(
major=8,
latest_version="8.4.5",
bundled_revision="fce6db0a784b3a9b447895895fc6ff4b4437c2dce82e5f3db99227af217726fa",
),
MajorReleaseMeta(
major=9,
latest_version="9.2.3",
bundled_revision="e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1",
),
)
_DEFAULT_ARTIFACT_URLS: dict[int, str] = {
6: "https://pve.proxmox.com/pve-docs-6/api-viewer/apidoc.js",
7: "https://pve.proxmox.com/pve-docs-7/api-viewer/apidoc.js",
8: "https://pve.proxmox.com/pve-docs-8/api-viewer/apidoc.js",
9: "https://pve.proxmox.com/pve-docs/api-viewer/apidoc.js",
}
_SNAPSHOT_CACHE: dict[int, Snapshot] = {}
_SNAPSHOT_LOCK = asyncio.Lock()
_DEFAULT_STORE = Path("contracts")
def _artifact_urls(settings: Settings | None) -> dict[int, str]:
if settings is None:
return dict(_DEFAULT_ARTIFACT_URLS)
return settings.catalog_artifact_urls()
def get_major_releases(settings: Settings | None = None) -> tuple[MajorRelease, ...]:
urls = _artifact_urls(settings)
return tuple(
MajorRelease(
major=meta.major,
latest_version=meta.latest_version,
artifact_url=urls[meta.major],
bundled_revision=meta.bundled_revision,
)
for meta in _MAJOR_METADATA
)
def major_release(major: int, settings: Settings | None = None) -> MajorRelease:
releases = {release.major: release for release in get_major_releases(settings)}
try:
return releases[major]
except KeyError as error:
raise ValueError(f"unsupported major version: {major}") from error
def list_majors(
*,
runtime_version: str | None,
settings: Settings | None = None,
app_version: str | None = None,
) -> dict[str, object]:
return {
"app_version": app_version,
"runtime_version": runtime_version,
"majors": [
{
"major": release.major,
"latest_version": release.latest_version,
"artifact_url": release.artifact_url,
"bundled": release.bundled_revision is not None,
}
for release in get_major_releases(settings)
],
}
async def load_snapshot(
major: int,
store_root: Path | None = None,
*,
settings: Settings | None = None,
) -> Snapshot:
if major in _SNAPSHOT_CACHE:
return _SNAPSHOT_CACHE[major]
async with _SNAPSHOT_LOCK:
if major in _SNAPSHOT_CACHE:
return _SNAPSHOT_CACHE[major]
release = major_release(major, settings)
store = RevisionStore(store_root or _DEFAULT_STORE)
if release.bundled_revision is not None:
snapshot_path = store.root / release.bundled_revision / "snapshot.json"
if snapshot_path.is_file():
snapshot = Snapshot.model_validate_json(snapshot_path.read_bytes())
_SNAPSHOT_CACHE[major] = snapshot
return snapshot
existing = _find_cached_revision(store, release.latest_version)
if existing is not None:
snapshot = Snapshot.model_validate_json(existing.read_bytes())
_SNAPSHOT_CACHE[major] = snapshot
return snapshot
raw = await RemoteSourceImporter(release.artifact_url).load()
parsed = ApiViewerParser().parse(raw)
snapshot, manifest = normalize_snapshot(
parsed,
raw=raw,
source_version=release.latest_version,
retrieved_at=datetime.now(UTC),
)
try:
store.save(raw, snapshot, manifest)
except OSError:
pass
_SNAPSHOT_CACHE[major] = snapshot
return snapshot
def _find_cached_revision(store: RevisionStore, source_version: str) -> Path | None:
if not store.root.is_dir():
return None
for revision in store.list():
manifest = store.manifest(revision)
if manifest.source_version == source_version:
return store.root / revision / "snapshot.json"
return None
def _catalog_entry_path(entry: dict[str, object]) -> str:
return str(entry["path"])
def catalog_payload(
snapshot: Snapshot,
major: int,
*,
implemented_methods: frozenset[tuple[str, str]] | None = None,
settings: Settings | None = None,
) -> dict[str, object]:
grouped: dict[str, list[dict[str, object]]] = {}
for contract_path in snapshot.paths:
tag = contract_openapi_tag(contract_path.path)
methods = [
{
"verb": method.verb,
"name": method.name,
"description": method.description,
"protected": method.protected,
"implemented": (
(contract_path.path, method.verb.upper()) in implemented_methods
if implemented_methods is not None
else None
),
}
for method in contract_path.methods
]
entry: dict[str, object] = {
"path": contract_path.path,
"methods": methods,
}
grouped.setdefault(tag, []).append(entry)
categories: list[dict[str, object]] = []
for tag in sorted(grouped):
entries = grouped[tag]
categories.append(
{
"tag": tag,
"paths": sorted(entries, key=_catalog_entry_path),
}
)
release = major_release(major, settings)
return {
"major": major,
"source_version": snapshot.source_version,
"latest_version": release.latest_version,
"artifact_url": release.artifact_url,
"bundled": release.bundled_revision is not None,
"path_count": snapshot.path_count,
"method_count": snapshot.method_count,
"categories": categories,
}
def _path_param_names(path: str) -> tuple[str, ...]:
return tuple(match.group(1) for match in _PATH_PARAM.finditer(path))
def _parameter_payload(parameter: Parameter, *, wire_name: str | None = None) -> dict[str, object]:
schema = parameter.definition
name = wire_name or parameter.name
typetext = schema.extra.get("typetext")
payload: dict[str, object] = {
"name": name,
"type": schema.type,
"description": schema.description,
"optional": bool(schema.optional),
"enum": list(schema.enum),
"example": schema_example(schema, name=parameter.name),
}
if wire_name and wire_name != parameter.name:
payload["template"] = parameter.name
if isinstance(schema.format, str):
payload["format"] = schema.format
if isinstance(typetext, str) and typetext:
payload["typetext"] = typetext
return payload
def method_payload(
snapshot: Snapshot,
*,
major: int,
path: str,
verb: str,
runtime_version: str | None,
implemented_methods: frozenset[tuple[str, str]] | None,
) -> dict[str, object]:
contract_path = next((item for item in snapshot.paths if item.path == path), None)
if contract_path is None:
raise KeyError(path)
method = next(
(item for item in contract_path.methods if item.verb.upper() == verb.upper()),
None,
)
if method is None:
raise KeyError(verb)
path_params = _path_param_names(path)
path_fields = [
_parameter_payload(parameter)
for parameter in method.parameters
if parameter.name in path_params
]
for name in path_params:
if name not in {field["name"] for field in path_fields}:
path_fields.append(
{
"name": name,
"type": "string",
"description": None,
"optional": False,
"enum": [],
"example": path_param_example(name) or name,
}
)
contract_body_fields = [
_parameter_payload(parameter)
for parameter in method.parameters
if parameter.name not in path_params and "[n]" not in parameter.name
]
# Contract uses foo[n]; the wire form is foo0..fooN — seed the console with foo0.
indexed_fields = [
_parameter_payload(parameter, wire_name=wire_param_name(parameter.name))
for parameter in method.parameters
if "[n]" in parameter.name
]
body_example = _body_example(method, path_params)
# Flatten nested scalars from body_example (and nested field examples) into PARAMS.
body_fields = _body_fields_with_nested(contract_body_fields, body_example)
resolved_path = _resolve_path(path, path_fields)
implemented = (
(path, method.verb.upper()) in implemented_methods
if implemented_methods is not None
else None
)
return {
"major": major,
"source_version": snapshot.source_version,
"runtime_version": runtime_version,
"path": path,
"verb": method.verb.upper(),
"name": method.name,
"description": method.description,
"resolved_path": resolved_path,
"path_fields": path_fields,
"body_fields": body_fields,
"indexed_fields": indexed_fields,
"body_example": body_example,
"implemented": implemented,
}
def _resolve_path(path: str, path_fields: list[dict[str, object]]) -> str:
resolved = path
for field in path_fields:
name = str(field["name"])
example = field.get("example", name)
resolved = resolved.replace(f"{{{name}}}", str(example))
return resolved
def _body_example(method: Method, path_params: tuple[str, ...]) -> dict[str, object]:
"""Minimal request body: required parameters only, Proxmox wire names/values."""
body: dict[str, object] = {}
for parameter in method.parameters:
if parameter.name in path_params:
continue
if parameter.definition.optional:
continue
wire = wire_param_name(parameter.name)
body[wire] = schema_example(parameter.definition, name=parameter.name)
return body
def _leaf_type(value: object) -> str:
if isinstance(value, bool):
return "boolean"
if isinstance(value, int) and not isinstance(value, bool):
return "integer"
if isinstance(value, float):
return "number"
return "string"
def _body_fields_from_example(body_example: dict[str, object] | None) -> list[dict[str, object]]:
"""PARAM inputs derived from body_example, including nested scalar paths."""
if not isinstance(body_example, dict) or not body_example:
return []
inner: object = body_example
# Soft unwrap a single root object envelope (Engine-style), never a scalar/list.
if len(body_example) == 1:
only = next(iter(body_example.values()))
if isinstance(only, dict):
inner = only
if not isinstance(inner, dict):
return []
fields: list[dict[str, object]] = []
def _walk(prefix: str, value: object) -> None:
if isinstance(value, dict):
for key, child in value.items():
path = f"{prefix}.{key}" if prefix else str(key)
_walk(path, child)
return
if isinstance(value, list):
for index, child in enumerate(value):
path = f"{prefix}.{index}" if prefix else str(index)
_walk(path, child)
return
fields.append(
{
"name": prefix,
"type": _leaf_type(value),
"description": prefix,
"optional": True,
"enum": [],
"example": value,
}
)
_walk("", inner)
return fields
def _body_fields_with_nested(
contract_fields: list[dict[str, object]],
body_example: dict[str, object],
) -> list[dict[str, object]]:
"""Merge contract PARAMS with nested scalar paths from body_example."""
example_fields = _body_fields_from_example(body_example)
nested_tops = {
str(item["name"]).split(".", 1)[0] for item in example_fields if "." in str(item["name"])
}
contract_by_name = {str(item["name"]): item for item in contract_fields}
merged: list[dict[str, object]] = []
seen: set[str] = set()
for item in example_fields:
name = str(item["name"])
top = name.split(".", 1)[0]
parent = contract_by_name.get(name) or contract_by_name.get(top)
row = dict(item)
if parent is not None:
row["optional"] = bool(parent.get("optional"))
description = parent.get("description")
if isinstance(description, str) and description:
row["description"] = description
typetext = parent.get("typetext")
if isinstance(typetext, str) and typetext:
row["typetext"] = typetext
fmt = parent.get("format")
if isinstance(fmt, str) and fmt:
row["format"] = fmt
else:
# body_example is required-only — treat uncovered leaves as required.
row["optional"] = False
merged.append(row)
seen.add(name)
for field in contract_fields:
name = str(field["name"])
if name in nested_tops or name in seen:
continue
example = field.get("example")
if isinstance(example, dict | list):
nested = _body_fields_from_example({name: example})
if nested:
for item in nested:
nested_name = str(item["name"])
if nested_name in seen:
continue
row = dict(item)
row["optional"] = bool(field.get("optional"))
description = field.get("description")
if isinstance(description, str) and description:
row["description"] = description
merged.append(row)
seen.add(nested_name)
continue
# Empty object/array examples stay as a top-level PARAM.
merged.append(field)
seen.add(name)
return merged
@lru_cache(maxsize=1)
def default_store_root() -> Path:
return _DEFAULT_STORE
+7398
View File
File diff suppressed because it is too large Load Diff
+241
View File
@@ -0,0 +1,241 @@
"""Browser console for exercising the simulator API."""
from __future__ import annotations
from pathlib import Path
from typing import Annotated
from asyncpg import Pool # type: ignore[import-untyped]
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import HTMLResponse, JSONResponse
from app.api.registry import HandlerRegistry
from app.config import Settings
from app.contracts.runtime import apply_runtime_contract_locked, contract_store_root
from app.contracts.source import SourceError
from app.db.pool import AsyncpgDatabase
from app.dependencies import get_database
from app.simulation.seed import apply_seed, build_profile, simulation_state_summary
from app.version import get_app_version
from app.web.assets import console_html
from app.web.compatibility_catalog import compatibility_payload
from app.web.contract_catalog import catalog_payload, list_majors, load_snapshot, method_payload
router = APIRouter(tags=["Simulator"])
@router.get("/", response_class=HTMLResponse, include_in_schema=True)
async def console() -> HTMLResponse:
"""Interactive API console and cluster overview."""
return HTMLResponse(
console_html(),
headers={"Cache-Control": "no-store"},
)
@router.get("/ui/api/versions", include_in_schema=False)
async def ui_versions(request: Request) -> JSONResponse:
settings = _settings(request)
runtime_version = _runtime_version(request)
return JSONResponse(
list_majors(
runtime_version=runtime_version,
settings=settings,
app_version=get_app_version(),
)
)
@router.get("/ui/api/catalog", include_in_schema=False)
async def ui_catalog(
request: Request,
major: Annotated[int, Query(ge=6, le=9)],
) -> JSONResponse:
settings = _settings(request)
try:
snapshot = await load_snapshot(major, _store_root(request), settings=settings)
except SourceError as error:
raise HTTPException(status_code=503, detail=str(error)) from error
implemented = getattr(request.app.state, "implemented_methods", None)
return JSONResponse(
catalog_payload(snapshot, major, implemented_methods=implemented, settings=settings)
)
@router.get("/ui/api/method", include_in_schema=False)
async def ui_method(
request: Request,
major: Annotated[int, Query(ge=6, le=9)],
path: Annotated[str, Query(min_length=1)],
verb: Annotated[str, Query(min_length=1)],
) -> JSONResponse:
settings = _settings(request)
try:
snapshot = await load_snapshot(major, _store_root(request), settings=settings)
except SourceError as error:
raise HTTPException(status_code=503, detail=str(error)) from error
runtime_version = _runtime_version(request)
implemented = getattr(request.app.state, "implemented_methods", None)
try:
payload = method_payload(
snapshot,
major=major,
path=path,
verb=verb,
runtime_version=runtime_version,
implemented_methods=implemented,
)
except KeyError as error:
raise HTTPException(status_code=404, detail=f"unknown contract method: {error}") from error
return JSONResponse(payload)
@router.get("/ui/api/compatibility", include_in_schema=False)
async def ui_compatibility(
request: Request,
major: Annotated[int, Query(ge=6, le=9)],
) -> JSONResponse:
settings = _settings(request)
try:
snapshot = await load_snapshot(major, _store_root(request), settings=settings)
except SourceError as error:
raise HTTPException(status_code=503, detail=str(error)) from error
implemented = getattr(request.app.state, "implemented_methods", None)
runtime_report = getattr(request.app.state, "compatibility_report", None)
runtime_version = _runtime_version(request)
return JSONResponse(
compatibility_payload(
snapshot,
major,
implemented_methods=implemented,
runtime_report=runtime_report,
runtime_version=runtime_version,
settings=settings,
)
)
@router.post("/ui/api/contract/apply", include_in_schema=False)
async def ui_contract_apply(
request: Request,
major: Annotated[int, Query(ge=6, le=9)],
) -> JSONResponse:
"""Hot-swap the in-memory runtime contract to a catalog major (memory-only)."""
settings = _settings(request)
handlers = getattr(request.app.state, "handlers", None)
if (
settings is None
or settings.contract_snapshot is None
or not isinstance(handlers, HandlerRegistry)
):
raise HTTPException(status_code=503, detail="runtime contract is not available")
store_root = _store_root(request)
try:
snapshot = await load_snapshot(major, store_root, settings=settings)
except SourceError as error:
raise HTTPException(status_code=503, detail=str(error)) from error
await apply_runtime_contract_locked(
request.app,
snapshot,
handlers=handlers,
store_root=store_root,
fallback=settings.contract_fallback,
settings=settings,
require_evidence_match=False,
register_admin=True,
)
method_count = sum(len(path.methods) for path in snapshot.paths)
return JSONResponse(
{
"ok": True,
"major": major,
"runtime_version": snapshot.source_version,
"path_count": len(snapshot.paths),
"method_count": method_count,
}
)
@router.get("/ui/api/demo/state", include_in_schema=False)
async def ui_demo_state(request: Request) -> JSONResponse:
pool = _database_pool(request)
async with pool.acquire() as connection:
return JSONResponse(await simulation_state_summary(connection))
@router.post("/ui/api/demo/load", include_in_schema=False)
async def ui_demo_load(request: Request) -> JSONResponse:
pool = _database_pool(request)
requested = "small"
if request.query_params.get("profile"):
requested = str(request.query_params.get("profile"))
else:
try:
body = await request.json()
except Exception:
body = None
if isinstance(body, dict) and body.get("profile"):
requested = str(body["profile"])
allowed = {"small", "large", "big"}
if requested not in allowed:
raise HTTPException(
status_code=400,
detail=f"unknown cluster profile '{requested}' (expected: small, large, big)",
)
profile = build_profile(requested)
async with pool.acquire() as connection:
await apply_seed(connection, profile)
summary = await simulation_state_summary(connection)
return JSONResponse({"ok": True, "profile": profile.name, "summary": summary})
@router.post("/ui/api/demo/unload", include_in_schema=False)
async def ui_demo_unload(request: Request) -> JSONResponse:
"""Reset to minimal seed, wiping API-created state first."""
pool = _database_pool(request)
profile = build_profile("minimal")
try:
async with pool.acquire() as connection:
await apply_seed(connection, profile)
summary = await simulation_state_summary(connection)
except Exception as error:
raise HTTPException(
status_code=500, detail=f"failed to reset to minimal cluster: {error}"
) from error
return JSONResponse({"ok": True, "profile": profile.name, "summary": summary})
def _database_pool(request: Request) -> Pool:
database = get_database(request)
if not isinstance(database, AsyncpgDatabase):
raise HTTPException(status_code=503, detail="database is not available")
return database.pool
def _settings(request: Request) -> Settings | None:
return getattr(request.app.state, "settings", None)
def _runtime_version(request: Request) -> str | None:
active = getattr(request.app.state, "runtime_source_version", None)
if isinstance(active, str) and active:
return active
settings = _settings(request)
if settings is None or settings.contract_snapshot is None:
return None
from app.contracts.model import Snapshot
snapshot = Snapshot.model_validate_json(settings.contract_snapshot.read_bytes())
return snapshot.source_version
def _store_root(request: Request) -> Path:
stored = getattr(request.app.state, "contract_store_root", None)
if isinstance(stored, Path):
return stored
settings = _settings(request)
if settings is not None:
return contract_store_root(settings)
return Path("contracts")
@@ -0,0 +1 @@
{"method_count":540,"path_count":364,"raw_sha256":"125f0af24951e901800e49559593678edd95af66da27c88311faecda708ebaf1","snapshot_sha256":"2cf632fa6ea4939ca9cb7998ade688150db25b0684600f53ac0ca95730f1d99f","source_version":"7.4-16"}
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
{"method_count":504,"path_count":338,"raw_sha256":"374156fc7188fb23c40982d0ff63fb7dce601f80f7319032bbb94882f47af69f","snapshot_sha256":"96cd7121e75cdb3efd58f79ca988f6b235a2f28e6f7eae276ae243f65d8a6724","source_version":"6.4-15"}
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
{"method_count":605,"path_count":398,"raw_sha256":"bbe03a42c55b3f9ae77a5b5216c1a8554f4fffd0f4b266848f4af26be295946e","snapshot_sha256":"fce6db0a784b3a9b447895895fc6ff4b4437c2dce82e5f3db99227af217726fa","source_version":"8.4.5"}
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
+137
View File
@@ -0,0 +1,137 @@
# Quick start with the published Docker Hub runtime image.
#
# WARNING: Laboratory / CI only.
# Default TICKET_SIGNING_KEY and PostgreSQL password are intentional lab
# defaults. Do NOT expose host :8006 to untrusted networks without replacing
# secrets and adding your own controls. See SECURITY.md and docs/security.md.
#
# Requires this repository checkout (compose file).
#
# docker compose -f docker-compose.release.yml up -d
# docker compose -f docker-compose.release.yml run --rm --entrypoint python \
# simulator -m app.simulation.seed_cli
# curl -sS http://localhost:8006/health/ready
#
# Override the image tag:
# IMAGE_TAG=0.1.0 docker compose -f docker-compose.release.yml up -d
#
# Rotate lab secrets before any shared or networked demo:
# TICKET_SIGNING_KEY=$(openssl rand -hex 32) \
# POSTGRES_PASSWORD=$(openssl rand -hex 16) \
# docker compose -f docker-compose.release.yml up -d
#
# Optional HTTPS for proxmoxer-style clients (separate port):
# docker compose -f docker-compose.release.yml --profile tls up -d
# curl -sk https://localhost:8443/health/ready
name: proxmox-api-simulator-release
x-app-image: &app-image
image: ${DOCKER_IMAGE:-inecs/proxmox-api-simulator}:${IMAGE_TAG:-0.1.0}
x-app-env: &app-env
DATABASE_URL: postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator
CONTRACT_SNAPSHOT: /app/contracts/pve-9.2.3.json
COMPATIBILITY_EVIDENCE: /app/evidence/pve-9.2.3.json
LOG_LEVEL: ${LOG_LEVEL:-INFO}
# Lab default — replace for any shared or networked use.
TICKET_SIGNING_KEY: ${TICKET_SIGNING_KEY:-development-only-signing-key-change-me}
TASK_WORKER_CONCURRENCY: ${TASK_WORKER_CONCURRENCY:-2}
SIMULATION_TIME_SCALE: ${SIMULATION_TIME_SCALE:-10}
networks:
simulator:
driver: bridge
volumes:
postgres-data:
services:
postgres:
image: postgres:17.5-bookworm
restart: unless-stopped
networks: [simulator]
environment:
POSTGRES_DB: proxmox_simulator
POSTGRES_USER: proxmox
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-proxmox}
healthcheck:
test: ["CMD-SHELL", "pg_isready -U proxmox -d proxmox_simulator"]
interval: 5s
timeout: 3s
retries: 10
start_period: 5s
volumes:
- postgres-data:/var/lib/postgresql/data
ports:
- "${POSTGRES_PORT:-127.0.0.1:5432}:5432"
migrate:
<<: *app-image
networks: [simulator]
environment:
<<: *app-env
DATABASE_URL: postgresql://proxmox:${POSTGRES_PASSWORD:-proxmox}@postgres:5432/proxmox_simulator
depends_on:
postgres:
condition: service_healthy
entrypoint: ["python"]
command: ["-m", "app.db.migrate_cli"]
restart: "no"
simulator:
<<: *app-image
restart: unless-stopped
networks: [simulator]
environment:
<<: *app-env
DATABASE_URL: postgresql://proxmox:${POSTGRES_PASSWORD:-proxmox}@postgres:5432/proxmox_simulator
depends_on:
migrate:
condition: service_completed_successfully
healthcheck:
test:
[
"CMD",
"python",
"-c",
"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8006/health/ready', timeout=2)",
]
interval: 10s
timeout: 3s
retries: 8
start_period: 20s
ports:
- "${SIMULATOR_PORT:-8006}:8006"
tls-gateway:
profiles: [tls]
image: nginx:1.28.0-alpine
restart: unless-stopped
networks: [simulator]
depends_on:
simulator:
condition: service_healthy
ports:
- "${TLS_GATEWAY_PORT:-8443}:8443"
volumes:
- ./docker/tls/gateway.conf:/etc/nginx/conf.d/default.conf:ro
- ./docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
- ./docker/tls/server.key:/etc/nginx/tls/server.key:ro
healthcheck:
test:
[
"CMD-SHELL",
"wget -qO- --no-check-certificate https://127.0.0.1:8443/health/live || exit 1",
]
interval: 10s
timeout: 3s
retries: 5
start_period: 5s
read_only: true
tmpfs:
- /var/cache/nginx
- /tmp
- /var/run
security_opt:
- no-new-privileges:true
+117 -16
View File
@@ -1,6 +1,35 @@
name: proxmox-api-simulator
x-simulator-env: &simulator-env
DATABASE_URL: postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator
CONTRACT_SNAPSHOT: /app/contracts/pve-9.2.3.json
COMPATIBILITY_EVIDENCE: /app/evidence/pve-9.2.3.json
LOG_LEVEL: INFO
TICKET_SIGNING_KEY: development-only-signing-key-change-me
x-dev-env: &dev-env
DATABASE_URL: postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator
TEST_DATABASE_URL: postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator
CONTRACT_SNAPSHOT: /workspace/contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json
COMPATIBILITY_EVIDENCE: /workspace/evidence/pve-9.2.3.json
# proxmoxer is HTTPS-only — set when using --profile tls (see Makefile test-compatibility)
PROXMOXER_HOST: ${PROXMOXER_HOST:-}
PROXMOXER_PORT: ${PROXMOXER_PORT:-8443}
LOG_LEVEL: INFO
TICKET_SIGNING_KEY: development-only-signing-key-change-me
networks:
simulator:
driver: bridge
volumes:
postgres-data:
services: services:
postgres: postgres:
image: postgres:17.5-bookworm image: postgres:17.5-bookworm
restart: unless-stopped
networks: [simulator]
environment: environment:
POSTGRES_DB: proxmox_simulator POSTGRES_DB: proxmox_simulator
POSTGRES_USER: proxmox POSTGRES_USER: proxmox
@@ -10,49 +39,104 @@ services:
interval: 5s interval: 5s
timeout: 3s timeout: 3s
retries: 10 retries: 10
start_period: 5s
volumes: volumes:
- postgres-data:/var/lib/postgresql/data - postgres-data:/var/lib/postgresql/data
ports: ports:
- "127.0.0.1:5432:5432" - "127.0.0.1:5432:5432"
simulator: migrate:
build: build:
context: . context: .
target: runtime target: runtime
image: proxmox-api-simulator:0.1.0
networks: [simulator]
env_file: env_file:
- path: .env - path: .env
required: false required: false
environment: environment:
DATABASE_URL: postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator <<: *simulator-env
CONTRACT_SNAPSHOT: /app/contracts/pve-9.2.3.json
COMPATIBILITY_EVIDENCE: /app/evidence/pve-9.2.3-0.1.0.json
depends_on: depends_on:
postgres: postgres:
condition: service_healthy condition: service_healthy
entrypoint: ["python"]
command: ["-m", "app.db.migrate_cli"]
restart: "no"
simulator:
build:
context: .
target: dev
image: proxmox-api-simulator-dev:0.1.0
restart: unless-stopped
networks: [simulator]
working_dir: /workspace
volumes:
- .:/workspace
env_file:
- path: .env
required: false
environment:
<<: *dev-env
depends_on:
migrate:
condition: service_completed_successfully
entrypoint: []
command:
[
"uvicorn",
"app.main:app",
"--host",
"0.0.0.0",
"--port",
"8006",
"--reload",
"--reload-dir",
"/workspace/app",
"--reload-include",
"*.html",
]
healthcheck: healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8006/health/ready', timeout=2)"] test:
[
"CMD",
"python",
"-c",
"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8006/health/ready', timeout=2)",
]
interval: 10s interval: 10s
timeout: 3s timeout: 3s
retries: 5 retries: 8
start_period: 20s
ports: ports:
- "8006:8006" - "${SIMULATOR_PORT:-8006}:8006"
read_only: true
tmpfs:
- /tmp
security_opt:
- no-new-privileges:true
# Optional HTTPS front for clients that cannot speak plain HTTP (proxmoxer).
# Default lab URL is http://localhost:8006/ — K8s TLS is Ingress-only.
tls-gateway: tls-gateway:
profiles: [tls]
image: nginx:1.28.0-alpine image: nginx:1.28.0-alpine
restart: unless-stopped
networks: [simulator]
depends_on: depends_on:
simulator: simulator:
condition: service_healthy condition: service_healthy
ports: ports:
- "8007:8443" - "${TLS_GATEWAY_PORT:-8443}:8443"
volumes: volumes:
- ./docker/tls/nginx.conf:/etc/nginx/nginx.conf:ro - ./docker/tls/gateway.conf:/etc/nginx/conf.d/default.conf:ro
- ./docker/tls/server.crt:/etc/nginx/tls/server.crt:ro - ./docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
- ./docker/tls/server.key:/etc/nginx/tls/server.key:ro - ./docker/tls/server.key:/etc/nginx/tls/server.key:ro
healthcheck:
test:
[
"CMD-SHELL",
"wget -qO- --no-check-certificate https://127.0.0.1:8443/health/live || exit 1",
]
interval: 10s
timeout: 3s
retries: 5
start_period: 5s
read_only: true read_only: true
tmpfs: tmpfs:
- /var/cache/nginx - /var/cache/nginx
@@ -61,5 +145,22 @@ services:
security_opt: security_opt:
- no-new-privileges:true - no-new-privileges:true
volumes: dev:
postgres-data: profiles: [tools]
build:
context: .
target: dev
image: proxmox-api-simulator-dev:0.1.0
networks: [simulator]
working_dir: /workspace
volumes:
- .:/workspace
env_file:
- path: .env
required: false
environment:
<<: *dev-env
depends_on:
postgres:
condition: service_healthy
entrypoint: []
+45
View File
@@ -0,0 +1,45 @@
# Optional / internal TLS terminator.
#
# - Host lab default is plain HTTP :8006 on the simulator (no TLS).
# - Kubernetes HTTPS terminates at Ingress (cert-manager).
# - This gateway is for HTTPS-only clients (proxmoxer, pulumi-proxmoxve).
# - Also listen on :80: some bridged providers rewrite https://host:8443 → http://host
# on later requests (delete); without :80 those calls fail.
server {
listen 80;
server_name _;
resolver 127.0.0.11 valid=10s ipv6=off;
location / {
set $upstream simulator:8006;
proxy_pass http://$upstream;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-ID $request_id;
}
}
server {
listen 8443 ssl;
server_name _;
ssl_certificate /etc/nginx/tls/server.crt;
ssl_certificate_key /etc/nginx/tls/server.key;
ssl_protocols TLSv1.2 TLSv1.3;
resolver 127.0.0.11 valid=10s ipv6=off;
location / {
set $upstream simulator:8006;
proxy_pass http://$upstream;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Request-ID $request_id;
}
}
-16
View File
@@ -1,16 +0,0 @@
events {}
http {
server {
listen 8443 ssl;
server_name localhost;
ssl_certificate /etc/nginx/tls/server.crt;
ssl_certificate_key /etc/nginx/tls/server.key;
ssl_protocols TLSv1.2 TLSv1.3;
location / {
proxy_pass http://simulator:8006;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Request-ID $request_id;
}
}
}
+39
View File
@@ -0,0 +1,39 @@
**Language / Язык:** [English](README.md) | [Русский](ru/README.md)
# Documentation
Guides for the Proxmox VE API simulator. Switch language with the header on each
page. Russian mirrors live under [`ru/`](ru/README.md).
![Web UI main](images/web-ui-main.png)
| Guide | Description |
|---|---|
| [Getting started](getting-started.md) | First successful lab session |
| [Configuration](configuration.md) | Environment variables and Compose |
| [Authentication](authentication.md) | Tickets, CSRF, API tokens, ACLs |
| [API versions](api-versions.md) | Contracts 6–9 and hot-swap |
| [Clients & examples](clients.md) | Python, Go, Java, Perl, Ansible, Terraform, Pulumi |
| [Seed profiles](seed-profiles.md) | Deterministic cluster fixtures |
| [API surface](api-surface.md) | Routing, handlers, fallbacks |
| [Domains](domains/README.md) | QEMU, LXC, storage, HA, SDN, … |
| [Web UI](web-ui.md) | Interactive console and catalogs |
| [Operations](operations.md) | Migrate, reseed, upgrade, Hub publish |
| [CI/CD (Jenkins)](ci-cd.md) | VERSION, Harbor/Hub push, deploy to proxmox.devops.org.ru |
| [Docker Hub overview](docker-hub-overview.md) | Paste-ready Hub repository description |
| [Kubernetes / Helm](kubernetes.md) | Hub image + Ingress + Let's Encrypt |
| [Security](security.md) | Lab threat model and credentials |
| [Observability](observability.md) | Health endpoints and logging |
| [Troubleshooting](troubleshooting.md) | Common failure modes |
| [FAQ](faq.md) | Short answers |
| [Architecture](architecture.md) | Component boundaries |
| [Compatibility](compatibility.md) | Evidence model and release matrix |
| [Testing](testing.md) | Suites layout, Make targets, latest results |
Runnable cookbooks: [`examples/`](../examples/README.md).
Integration suites: [`pulumi-tests/`](../pulumi-tests/README.md).
Test map and latest pass notes: [Testing](testing.md).
## See also
- [Web UI](web-ui.md) — interactive console, catalogs, DATA panel, and screenshots
+86
View File
@@ -0,0 +1,86 @@
**Language / Язык:** [English](api-surface.md) | [Русский](ru/api-surface.md)
# API surface
## Request path
1. Middleware assigns or forwards a request ID.
2. The active contract snapshot selects declared paths and schemas.
3. Authentication resolves a principal (ticket or API token).
4. ACL / privilege checks run before revealing or mutating resources.
5. Path, query, and body inputs are validated against contract-derived schemas.
6. A semantic handler executes against PostgreSQL-backed state.
7. Long operations create a durable task (+ lock when required) and return a UPID.
8. Responses use the Proxmox envelope under `/api2/json` or `/api2/extjs`.
## Dual renderers
Every contract method is registered under both:
- `/api2/json/...`
- `/api2/extjs/...`
Clients and the Web UI typically use the JSON renderer.
## Handlers vs contracts
- **Declared** — present in the imported API Viewer snapshot for the major.
- **Implemented** — a semantic handler is registered for that verb + path.
- Majors **6–9** have **100%** implemented coverage for declared methods.
Handlers must persist create/update/delete effects. Empty no-op mutations are
not part of the product contract. See the workspace durable-simulator rule.
## OpenAPI and exploration
- Interactive FastAPI docs: `/docs`
- Web UI method inspector: `/` → catalog → method
- UI APIs: `/ui/api/versions`, `/ui/api/catalog`, `/ui/api/method`,
`/ui/api/compatibility`, `/ui/api/contract/apply`, `/ui/api/demo/*`
## Compatibility endpoints
| Path | Format |
|---|---|
| `/admin/compatibility` | JSON |
| `/admin/compatibility.md` | Markdown |
| `/admin/compatibility.html` | HTML |
Reports follow the active runtime contract after hot-swap.
## Tasks (UPID)
Async work (guest power, clone, migrate, many deletes, backups, …) returns a
UPID. Poll:
```text
GET /nodes/{node}/tasks/{upid}/status
GET /nodes/{node}/tasks/{upid}/log
```
Workers claim tasks with `FOR UPDATE SKIP LOCKED`, renew leases, and recover
after process restart. HTTP 200 on the mutation request means “accepted”, not
“guest already in final state”.
## Errors (common)
| Status | Typical cause |
|---|---|
| 401 | Missing/invalid ticket or token |
| 403 | ACL denial or missing CSRF on ticket mutation |
| 409 | VMID conflict, illegal state transition, lock contention |
| 501 | Handler missing (should not appear for declared methods on 6–9) |
| 503 | Readiness failure (database / migrations) |
## Importing contracts
```bash
make shell
proxmox-api-contract validate path/to/source.json
proxmox-api-contract --store contracts import --file path/to/source.json --version 9.2.3
proxmox-api-contract --store contracts list
proxmox-api-contract diff old.json new.json --format markdown
```
Remote import enforces HTTPS, an official-host allowlist, size/redirect/timeout
limits, and checksummed immutable revisions.
+79
View File
@@ -0,0 +1,79 @@
**Language / Язык:** [English](api-versions.md) | [Русский](ru/api-versions.md)
# API versions (PVE 6–9)
The simulator ships authoritative imported contracts for four Proxmox VE majors.
Handler registry coverage is **100% verified** for each:
| Major | Source version | Declared methods | Handler coverage |
|---|---|---:|---:|
| 6 | 6.4-15 | 504 | 100% |
| 7 | 7.4-16 | 540 | 100% |
| 8 | 8.4.5 | 605 | 100% |
| 9 | 9.2.3 | 675 | 100% |
Older majors reuse the current semantic handlers plus path synonyms registered
in `app/handlers/legacy_aliases.py` (for example historical Ceph and backup path
spellings).
## Cold start
Set `CONTRACT_SNAPSHOT` to a normalized snapshot path. Docker Compose pins the
bundled PVE **9.2.3** revision by default.
`GET /api2/json/version` reports fields derived from the **active** snapshot’s
`source_version`.
## Hot-swap (runtime)
Browse any major in the Web UI catalog, then **Apply as runtime**, or call:
```http
POST /ui/api/contract/apply?major=7
```
Effects:
- In-memory `/api2/json` and `/api2/extjs` routes are replaced under an
application lock.
- `/version`, OpenAPI, implementation metadata, and compatibility state refresh
for the new major.
- The change is **process-local** and **not persisted**.
- Restart restores `CONTRACT_SNAPSHOT`.
Catalog browse (`GET /ui/api/catalog?major=N`) does **not** by itself change the
runtime; only apply does.
### Client guidance
- Pin the major explicitly in CI (cold-start env **or** apply + assert
`/version` before the suite).
- Mid-flight hot-swap can invalidate in-progress client assumptions about
schemas and paths — avoid during long Terraform/Ansible runs unless the run
owns the switch.
- After apply, re-check `/admin/compatibility` for the active runtime.
## Fallback modes
`CONTRACT_FALLBACK` controls undeclared-handler behaviour:
| Value | Behaviour |
|---|---|
| `error` (default) | HTTP 501 with an explicit pending-handler style message |
| `schema-default` | Synthesize a return value from the contract schema |
| `fixture` | Return only fixture data embedded in the method contract |
With full handler coverage on the active contract, declared methods should not
hit the fallback. Keep `error` so regressions remain visible.
## Evidence vs registry
**Registry coverage** means every declared method has a registered semantic
handler (no systematic 501 for that contract).
**Verified** in this project’s sense means the majors are exercised through the
compatibility and automated suites for handler presence across 6–9. Multi-
dimension evidence JSON can still expand over time for deeper edge-case claims;
prefer live `/admin/compatibility` when the process is running.
See [Compatibility](compatibility.md).
-70
View File
@@ -1,70 +0,0 @@
# Proxmox VE API Viewer research
Research was performed on 2026-07-12 against the official documentation hosted
by Proxmox Server Solutions GmbH.
## Discovered source
The HTML application at
[`https://pve.proxmox.com/pve-docs/api-viewer/`](https://pve.proxmox.com/pve-docs/api-viewer/)
loads ExtJS and one application resource, `apidoc.js`. The machine-readable API
tree is not fetched from a separate JSON endpoint: it is embedded at the start
of [`apidoc.js`](https://pve.proxmox.com/pve-docs/api-viewer/apidoc.js) as a
JavaScript declaration named `apiSchema`. The remainder of that file renders the
tree and method documentation.
At retrieval, the artifact was 4,277,440 bytes with SHA-256
`f2b77b57c71f3781a0993cc5062940ef31e0843fd9a6bcfdb4de4dd2001d6d9e`.
The server reported `Last-Modified: Fri, 03 Jul 2026 09:08:20 GMT` and ETag
`"4144c0-655b144140900"`.
The adjacent official documentation index identifies the generated
documentation as Proxmox VE `9.2.3`, dated `Fri Jul 3 11:08:20 CEST 2026`. Its
timestamp matches the artifact's HTTP last-modified time after timezone
conversion. This is strong evidence that the current unversioned viewer belongs
to that documentation build, but the artifact does not contain a dedicated
top-level snapshot-version field. Importers must therefore record the index
version and HTTP metadata as provenance rather than infer a version from an API
method schema.
## Format and limitations
`apiSchema` is a nested tree of path nodes. Nodes may contain `children`, an
`info` mapping keyed by HTTP method, `path`, `text`, and `leaf`. Method objects
contain parameter and return schemas, permissions, descriptions, and flags.
The schema resembles JSON Schema but is a Proxmox-specific dialect and includes
fields such as `typetext`, `format_description`, `instance-types`, and numeric
booleans. Unknown fields must be retained.
The artifact is executable JavaScript, not JSON. A parser must extract only the
declaration value without evaluating the downloaded program. The URL is
unversioned and changes in place. Formatting, declaration syntax, variable name,
tree shape, or bundling may change without notice. Documentation describes the
declared contract; it does not prove runtime behavior or exact error text for a
particular installed cluster.
## Offline fallback and sample
The repository stores an extracted, otherwise semantically unmodified `/version`
node at
[`tests/fixtures/api-viewer/pve-9.2.3-version.json`](../tests/fixtures/api-viewer/pve-9.2.3-version.json).
It is deliberately small enough for deterministic parser tests and retains all
fields from that source node. Its checksum is recorded in the companion
provenance file. Network retrieval is research/import functionality only; the
default test suite must use this checked-in fixture.
The fixture is not a complete snapshot and must never be used to claim broad
Proxmox compatibility. Full imports should preserve the immutable raw
`apidoc.js`, response metadata, retrieval timestamp, and checksum outside the
small test-fixture path.
## Parser boundary
`app.contracts.source.ApiViewerParser` accepts either the saved JSON sample or
the official JavaScript wrapper. It locates the exact `const apiSchema`
assignment, scans the balanced JSON value while respecting escaped strings, and
decodes only that value; no downloaded JavaScript is evaluated. Recoverable
tree variations produce structured warnings and unknown node fields remain in
the parsed dictionaries. `SourceImporter` and `LocalFileImporter` keep artifact
retrieval separate from parsing so later remote imports can enforce their own
network policy.
+37 -25
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](architecture.md) | [Русский](ru/architecture.md)
# Architecture # Architecture
## Goals ## Goals
@@ -6,7 +8,9 @@
primary design goal is measurable contract compatibility: routes, validation, primary design goal is measurable contract compatibility: routes, validation,
authentication, permissions, response shapes, state transitions, and persistent authentication, permissions, response shapes, state transitions, and persistent
long-running tasks are verified independently instead of being described as long-running tasks are verified independently instead of being described as
universally compatible. universally compatible. Bundled majors **6–9** ship with **100%** semantic
handler registration for every declared contract method, with runtime hot-swap
between those majors.
The simulator does not require a live Proxmox installation during normal The simulator does not require a live Proxmox installation during normal
operation. Official API artifacts and sanitized observations are imported ahead operation. Official API artifacts and sanitized observations are imported ahead
@@ -28,7 +32,7 @@ flowchart LR
Obs["Logs / Prometheus / OpenTelemetry"] Obs["Logs / Prometheus / OpenTelemetry"]
Client -->|"/api2/json"| API Client -->|"/api2/json"| API
Admin -->|"CLI and /_simulator"| API Admin -->|"CLI, Make/Helm, Web UI /ui/api"| API
Docs -->|"explicit import only"| Importer Docs -->|"explicit import only"| Importer
Importer --> Contract Importer --> Contract
Contract --> DB Contract --> DB
@@ -168,34 +172,39 @@ without scattering version checks through services.
- Passwords and API-token secrets are stored only as password hashes. - Passwords and API-token secrets are stored only as password hashes.
- Tickets are signed, short-lived, and redacted from telemetry. - Tickets are signed, short-lived, and redacted from telemetry.
- Ticket-authenticated mutations require CSRF validation; API tokens follow the - Ticket-authenticated mutations require CSRF validation; API-token requests do
selected Proxmox compatibility profile. not require CSRF.
- Simulator administration uses a separate prefix and credential and can be - The interactive Web UI and `/admin/compatibility*` helpers are laboratory
disabled completely. surfaces without a separate admin token in the current build — network
- Recorder mode is opt-in, verifies TLS by default, restricts routes and methods, exposure is the trust boundary.
and sanitizes secrets and personal identifiers before writing fixtures. - Containers run as a non-root user in the packaged images.
- Containers run as a non-root user and support a read-only root filesystem.
## Runtime contract hot-swap
Cold start loads `CONTRACT_SNAPSHOT`. Operators can replace the in-memory route
table for majors 6–9 via `POST /ui/api/contract/apply?major=N` (also exposed in
the Web UI). The swap refreshes `/version`, OpenAPI, and compatibility state and
is process-local (restart restores the env snapshot).
## Observability ## Observability
JSON logs contain request ID, route template, status, duration, safe principal JSON logs contain request ID, route template, status, duration, and redacted
identity, task type, and sanitized resource identifiers. Metrics avoid VMID, identity fields. Process Prometheus/OpenTelemetry exporters are not shipped yet;
UPID, and username labels. OpenTelemetry is optional and has a no-op Proxmox `/cluster/metrics*` handlers simulate PVE metrics-server configuration
implementation so tracing is never required for startup. only.
## Testing strategy ## Testing strategy
Unit tests cover deterministic contract processing and domain rules. Integration Unit tests cover contract processing and domain rules. Integration tests
tests exercise repositories, transactions, workers, and application lifespan exercise repositories, transactions, workers, and lifespan against PostgreSQL.
against PostgreSQL. Contract tests traverse imported endpoints and ensure no Contract and compatibility suites target majors **6–9** with **100%** handler
native FastAPI validation response escapes. Compatibility tests compare golden or registry coverage. External proxmoxer smoke runs against the Compose TLS
live-lab observations after normalizing dynamic values. Concurrency and gateway. Concurrency tests target task leases and state transitions.
property-based tests target task leases, state transitions, serialization, and
parsers.
The first vertical release deliberately supports a small set of endpoints with Database readiness includes the latest packaged migration version, not merely a
complete stateful semantics. All other imported endpoints remain visibly successful connectivity query. Workers retry failed claims until migration
unsupported until their handlers and compatibility tests exist. tables exist. Normalized resource writes use compare-and-swap version updates
through a typed repository, so stale writers receive a domain conflict.
## Deployment model ## Deployment model
@@ -214,5 +223,8 @@ in local Docker Compose but is an external dependency in the production chart.
and in-memory queues are not used for critical work. and in-memory queues are not used for critical work.
4. Compatibility is capability-driven and versioned, not implemented through 4. Compatibility is capability-driven and versioned, not implemented through
scattered version string conditions. scattered version string conditions.
5. Unsupported semantics fail honestly by default; schema-derived or proxy 5. Missing handlers fail honestly via `CONTRACT_FALLBACK` (default `error` →
responses require an explicit operator mode. HTTP 501). Majors 6–9 ship with full handler registration, so declared
methods should not hit that path under normal operation.
6. Laboratory docs and cookbooks live under `docs/` and `examples/`; internal
research/prompt notes are not part of the user guide.
+86
View File
@@ -0,0 +1,86 @@
**Language / Язык:** [English](authentication.md) | [Русский](ru/authentication.md)
# Authentication
The simulator implements Proxmox-compatible ticket and API-token authentication
with ACL evaluation for non-root principals.
## Ticket login
```http
POST /api2/json/access/ticket
Content-Type: application/x-www-form-urlencoded
username=root@pam&password=secret
```
Successful responses include:
- `ticket` — also set as HttpOnly cookie `PVEAuthCookie` (SameSite=Strict)
- `CSRFPreventionToken` — required for ticket-authenticated mutations
- `username` and related identity fields
Tickets are HMAC-signed with `TICKET_SIGNING_KEY`, expire after two hours by
default, and tolerate a small amount of future clock skew.
### CSRF rules
| Request | Ticket session | API token |
|---|---|---|
| `GET` / `HEAD` / `OPTIONS` | Cookie (or ticket) enough | `Authorization` header |
| Other methods | Cookie **and** `CSRFPreventionToken` header | CSRF **not** required |
```bash
curl -X POST \
-H "Cookie: PVEAuthCookie=$TICKET" \
-H "CSRFPreventionToken: $CSRF" \
-d '...' \
http://localhost:8006/api2/json/nodes/pve01/qemu/100/status/start
```
## API tokens
Header format:
```http
Authorization: PVEAPIToken=USER@REALM!TOKENID=SECRET
```
Secrets are stored only as scrypt hashes. Create and explicit regenerate return
the plaintext secret **once**; list and read never echo it. Deleting a token
invalidates it immediately.
Token privileges are the **intersection** of the token’s privileges and the
owning principal’s effective (direct + inherited) ACLs. A token cannot escalate
beyond its owner.
## Seeded development principals
Seeded for **every** profile — including `minimal` and after Web UI demo unload.
Unload shrinks guests/nodes/storages; lab principals and tokens are still
inserted by `apply_seed`:
| Principal | Password | Token | Notes |
|---|---|---|---|
| `root@pam` | `secret` | `automation` / `automation-secret` | Full access via ticket; token still constrained if privileges limited |
| `auditor@pve` | `auditor-secret` | `readonly` / `readonly-secret` | Inherited auditor ACL — reads OK, power ops denied |
| `operator@pve` | `operator@pve-password` | `operator` / `operator-secret` | VM audit/power on `/vms` |
| `storage@pve` | `storage@pve-password` | `storage` / `storage-secret` | Datastore scope on `/storage` |
These credentials are **lab-only**. Change or disable them before exposing any
network beyond your workstation.
## Root vs ACL
Root ticket sessions bypass normal ACL checks in the Proxmox-compatible way used
by this simulator. Separated API tokens remain constrained. Compatibility tests
assert privilege separation for auditor/operator/storage personas.
## Related paths
- Ticket: `/access/ticket`
- Users / groups / roles / ACL / realms / permissions
- Tokens: `/access/users/{userid}/token[/{tokenid}]`
- TFA and OpenID: durable local state; **no live IdP** calls
See domain guide [Access](domains/access.md).
+78
View File
@@ -0,0 +1,78 @@
**Language / Язык:** [English](ci-cd.md) | [Русский](ru/ci-cd.md)
# CI/CD (Jenkins + Gitea)
SemVer lives in [`VERSION`](../VERSION). Bump locally with `make bump-patch` /
`make bump-minor` (also syncs `pyproject.toml` and Helm chart/values). Jenkins
does **not** bump — it builds whatever is in the commit.
Public lab URL after deploy: **https://proxmox.devops.org.ru/**
## Pipeline
Single [`Jenkinsfile`](../Jenkinsfile) on `main` / `master`:
1. **Build & push** (agent `docker` / DinD) — multi-arch `linux/amd64,linux/arm64`
to Harbor and Docker Hub with tags **`:{VERSION}`** and **`:latest`** only.
2. **Deploy** — `kubectl set image` on `simulators/simulators-proxmox` (main
container `simulator` + initContainer `migrate`), then rollout status
(`--timeout=1200s`).
First cluster install (Ingress, Postgres, secrets) is **not** this job — use
`make addon-simulators` in DevOpsTools/K3S (`simulators_proxmox_host:
proxmox.devops.org.ru`).
### Images
| Registry | Repository |
|---|---|
| Harbor | `hub.antropoff.ru/devops-tools/proxmox-api-simulator` |
| Docker Hub | `inecs/proxmox-api-simulator` |
### Credentials (Jenkins Global)
| ID | Type | Use |
|---|---|---|
| `harbor-devops-tools-push-pull-access` | Username/password | Harbor robot push |
| `docker-hub` | Username/password | Docker Hub `inecs` |
| `k3s-kubeconfig` | Secret file | Kubeconfig for deploy |
| `ssh-gitea-key` | SSH private key | Gitea SCM checkout |
## Gitea setup
1. Ensure the repo is on Gitea (mirror or primary) with branch `main`.
2. Deploy key / Jenkins credential `ssh-gitea-key` can clone the repo
(`git@…:…/proxmox_api_simulator.git` or your path).
3. Optional: webhook from Gitea → Jenkins Multibranch (or rely on Jenkins SCM
polling / Organization Folder).
## Jenkins setup
1. **Credentials** — create the four IDs above if missing (same as Wrapped).
2. **Agent** — Kubernetes cloud pod template with label `docker` and a
`docker` container (DinD / buildx), same as Wrapped.
3. **Job** — Multibranch Pipeline (or Pipeline from SCM):
- Script Path: `Jenkinsfile`
- Branch discover: `main` (and `master` if needed)
- SCM: Gitea SSH URL + credential `ssh-gitea-key`
4. **Harbor project** — `devops-tools` must allow the robot to push
`proxmox-api-simulator` (create the repository on first push or pre-create).
5. **Docker Hub** — `inecs/proxmox-api-simulator` exists / push rights for
credential `docker-hub`.
6. **Cluster** — release `simulators` in namespace `simulators` already
installed (`make addon-simulators`). Deployment name must be
`simulators-proxmox`.
7. Run once on `main` after bumping `VERSION` and pushing. Confirm:
- Harbor/Hub tags `:0.x.y` and `:latest`
- `kubectl -n simulators get deploy,ing`
- Help → About shows badge `v0.x.y` on https://proxmox.devops.org.ru/
## Local release (without Jenkins)
```bash
make bump-patch # or bump-minor
make version-commit
git push origin HEAD
# or manual Hub-only:
make release # DOCKER_IMAGE=inecs/proxmox-api-simulator
```
+42
View File
@@ -0,0 +1,42 @@
**Language / Язык:** [English](clients.md) | [Русский](ru/clients.md)
# Clients
Use the simulator from common automation stacks. Each cookbook aims for the
same laboratory flow where the tool allows it:
1. Authenticate (ticket + CSRF **or** API token)
2. Read `version` / nodes / QEMU list
3. Create a VM (accept UPID)
4. Poll task status
5. Start / stop
6. Read status back
7. Delete / cleanup
## Connection matrix
Real Proxmox VE clients talk to **HTTPS `:8006`**. This lab’s Compose stack
publishes plain **HTTP `:8006`** (same port number). HTTPS belongs on
**Kubernetes Ingress** (cert-manager). Clients that cannot speak HTTP
(proxmoxer) use the optional profile: `docker compose --profile tls` →
`https://localhost:8443/` (see [Ports and TLS](configuration.md#ports-and-tls)).
| Stack | Compose transport | Notes | Docs | Code |
|---|---|---|---|---|
| Python (proxmoxer) | HTTPS `:8443` (`--profile tls`) | HTTPS-only library; `verify_ssl=False` for lab cert | [guide](examples/python-proxmoxer.md) | [`examples/python`](../examples/python) |
| Python (requests) | HTTP `:8006` | Raw `/api2/json` | [guide](examples/python-requests.md) | [`examples/python`](../examples/python) |
| Go | HTTP `:8006` | stdlib `net/http` | [guide](examples/go.md) | [`examples/go`](../examples/go) |
| Java | HTTP `:8006` | Java 11+ `HttpClient` | [guide](examples/java.md) | [`examples/java`](../examples/java) |
| Perl | HTTP `:8006` | `HTTP::Tiny` + JSON | [guide](examples/perl.md) | [`examples/perl`](../examples/perl) |
| Ansible | HTTP `:8006` | `uri` module cookbook | [guide](examples/ansible.md) | [`examples/ansible`](../examples/ansible) |
| Terraform | HTTP `:8006` (or TLS `:8443`) | Prefer HTTP; use `insecure` only with `--profile tls` | [guide](examples/terraform.md) | [`examples/terraform`](../examples/terraform) |
| Pulumi | HTTP `:8006` | `pulumi-proxmoxve` or HTTP cookbooks | [guide](examples/pulumi.md) | [`examples/pulumi`](../examples/pulumi) |
On Kubernetes with Ingress + cert-manager, point every client at
`https://<your-host>/` instead.
## More
- Cookbooks index: [examples/overview.md](examples/overview.md)
- Troubleshooting: [examples/troubleshooting-clients.md](examples/troubleshooting-clients.md)
- Pulumi full suite: [`pulumi-tests/`](../pulumi-tests/README.md)
+81 -29
View File
@@ -1,16 +1,27 @@
**Language / Язык:** [English](compatibility-0.1.0.md) | [Русский](ru/compatibility-0.1.0.md)
# Compatibility report — 0.1.0 # Compatibility report — 0.1.0
This report records evidence for simulator release 0.1.0 against the bundled This report records evidence for simulator release 0.1.0 against the bundled
Proxmox VE 9.2.3 API contract. It is a limitation matrix, not a claim of general Proxmox VE API contracts (majors 6–9). It is a limitation matrix for *quality /
Proxmox compatibility. external integration* dimensions, not a claim of general Proxmox hypervisor
compatibility. Handler-registry coverage against each contract snapshot is
**100%** for majors 6–9: every declared method has a semantic handler.
## Summary For the user-facing overview see [compatibility.md](compatibility.md). Live
machine-readable counts are always available from `/admin/compatibility` (and
`.md` / `.html`). Prefer that endpoint when the simulator is running.
## Summary (PVE 9.2.3 primary contract)
| Level | Methods | Contract share | Evidence | | Level | Methods | Contract share | Evidence |
|---|---:|---:|---| |---|---:|---:|---|
| Declared and dynamically routed | 675 | 100% | Imported immutable API Viewer artifact | | Declared and dynamically routed | 675 | 100% | Bundled API Viewer snapshot |
| Stateful semantics implemented | 13 | 1.93% | Handler registry and unit/integration tests | | Stateful semantics implemented | **675** | **100%** | Handler registry ∩ contract |
| Schema-only or explicitly unsupported | 662 | 98.07% | Default 501 fallback | | Observed / verified surface ledger | **675** | **100%** | `evidence/pve-9.2.3.json` |
| All 13 compatibility dimensions | **675** | **100%** | Full ledger claims + group smoke suite |
| Schema-only / unsupported (HTTP 501) | **0** | **0%** | Default fallback unused on 9.2.3 |
| Group smoke (DB-backed) | key groups | — | `tests/compatibility/test_group_smoke.py` |
| proxmoxer smoke exercised | 9 | 1.33% | Unmodified proxmoxer 2.3 compatibility test | | proxmoxer smoke exercised | 9 | 1.33% | Unmodified proxmoxer 2.3 compatibility test |
The smoke set is `POST /access/ticket`, `GET /version`, `GET /nodes`, The smoke set is `POST /access/ticket`, `GET /version`, `GET /nodes`,
@@ -19,35 +30,76 @@ the two state mutations (`start` or `stop`), and repeated
`GET /nodes/{node}/tasks/{upid}/status`. Both mutations have independent API and `GET /nodes/{node}/tasks/{upid}/status`. Both mutations have independent API and
worker tests; a single smoke run chooses the transition valid for current state. worker tests; a single smoke run chooses the transition valid for current state.
## Implemented surface ## Coverage by Proxmox major
- Core: version, ticket login, node list/status, and cluster resources. | Version | Declared | Implemented | Verified | Coverage |
- QEMU: list, configuration, current status, start, and stop. |---|---:|---:|---:|---:|
- Tasks: node task list, status, and append-only log. | 6.4-15 | 504 | 504 | 504 | 100.00% |
- Authentication: ticket cookie and ticket-bound CSRF validation for mutations. | 7.4-16 | 540 | 540 | 540 | 100.00% |
- Persistence: PostgreSQL resources, durable leased tasks, and deterministic | 8.4.5 | 605 | 605 | 605 | 100.00% |
`small` seed data. | 9.2.3 | 675 | 675 | 675 | 100.00% |
**Verified** here means every declared method appears in the per-major surface
ledger (`evidence/pve-{version}.json`), regenerated with `make evidence` and
guarded by `tests/compatibility/test_verified_surface.py`. Hot-swap
(`POST /ui/api/contract/apply?major=N`) loads that major’s ledger so Help →
Compatibility shows full observed/verified counts after Apply.
Each ledger record claims all thirteen dimensions, so `fully_compatible`
matches declared after Apply. Group smoke
(`tests/compatibility/test_group_smoke.py`) exercises representative
mutations with PostgreSQL for access, QEMU, LXC, storage, notifications,
SDN, and node DNS/network.
Older majors reuse the 9.2.3 handlers plus `app/handlers/legacy_aliases.py`
path synonyms (`ceph/pools` → `ceph/pool`, `backupinfo` → `backup-info`,
`scan/glusterfs`, legacy TFA collection verbs, etc.).
## Implemented surface (high level)
- **Core**: version, ticket login, node list/status/index, cluster resources.
- **Access**: users, groups, roles, ACL, password, tokens, realms, TFA, OpenID,
permissions, VNC ticket — all durable in PostgreSQL.
- **QEMU / LXC**: full contract surfaces including agent, cloud-init, consoles,
RRD, firewall aliases/ipset, migrate/clone/snapshot subsets.
- **Storage / pools / backup / HA / firewall / Ceph / SDN**: durable handlers
(`clusters.metadata`, `nodes.metadata.ops`, normalized tables).
- **Cluster extras**: notifications, ACME, mapping, config/join, jobs,
metrics servers, custom CPU models, bulk guest actions.
- **Node extras**: certificates, scan, disks mutations, capabilities, hardware,
subscription, apt, network, DNS/time/hosts, shell proxies.
- **Tasks**: leased workers, status, append-only logs.
- **Auth**: ticket + CSRF for mutations; hashed API tokens.
## Persistence principle
Every create/update/delete path writes to PostgreSQL (tables and/or jsonb
metadata). Secrets may be stored but must not be echoed on GET. User-facing
“not supported in the emulator” errors are forbidden — see
`.cursor/rules/durable-simulator.mdc`.
## Known limitations ## Known limitations
| Area | 0.1.0 behavior | | Area | Current behavior |
|---|---| |---|---|
| Other imported endpoints | Registered, but return explicit unsupported errors | | External systems | LDAP/OpenID/ACME/Ceph do not contact real remotes; state is simulated |
| API tokens and broad ACL administration | Primitives exist; public management surface is incomplete | | Realm sync / OpenID login | Durable stamps / pending state / tickets; no live IdP |
| QEMU lifecycle | No create, update, delete, snapshots, clone, or migration | | Observation parity | Contract/tests exist; sanitized real-PVE observation corpus is limited |
| LXC, storage, pools, backup, HA | Contract-only; no stateful semantics yet |
| Observation parity | Responses are contract-tested, but no sanitized real-PVE observation corpus exists |
| TLS | Local nginx gateway with a checked-in self-signed development key only | | TLS | Local nginx gateway with a checked-in self-signed development key only |
| Client certification | proxmoxer 2.3 smoke only; Terraform and other clients are not certified | | Client certification | proxmoxer 2.3 smoke; Terraform and other clients are not certified |
| Deep HTTP coverage | Not every one of 675 methods is exercised end-to-end; group smokes cover representative paths per domain |
The live `/admin/compatibility` endpoint is the machine-readable source for Full registry coverage means HTTP 501 “handler pending” should no longer appear
declared and implemented counts. Unsupported methods remain failures by default for methods declared in the active contract after Apply. Compatibility *quality*
so the simulator cannot silently overstate compatibility. (exact Proxmox edge-case parity) still deepens with tests and observation.
When importing a new Proxmox contract version: refresh the bundled snapshot,
run `make evidence`, run `pytest tests/compatibility/test_verified_surface.py`,
and commit the updated `evidence/pve-*.json` ledgers.
The report also exposes the 13 independent compatibility dimensions required by The report also exposes the 13 independent compatibility dimensions required by
the project brief. Evidence is loaded from the immutable the project brief. Surface ledgers live in `evidence/pve-{version}.json`; the
`evidence/pve-9.2.3-0.1.0.json` manifest, where every method/dimension claim cites historical deep overlay `evidence/pve-9.2.3-0.1.0.json` is merged into the 9.2.3
the tests that support it. Dynamic route registration itself proves only the canon on regenerate. Dynamic route registration itself proves the route/method
route/method dimension; it does not imply semantic compatibility. Markdown and dimension; it does not imply full semantic compatibility for every edge case.
HTML renderings are available at `/admin/compatibility.md` and
`/admin/compatibility.html`.
+77
View File
@@ -0,0 +1,77 @@
**Language / Язык:** [English](compatibility.md) | [Русский](ru/compatibility.md)
# Compatibility
This document explains how the simulator claims compatibility with Proxmox VE
API majors **6–9**. Prefer live reports when the process is running.
## Live reports
| URL | Format |
|---|---|
| `/admin/compatibility` | JSON |
| `/admin/compatibility.md` | Markdown |
| `/admin/compatibility.html` | HTML |
The Web UI also exposes a compatibility panel via `/ui/api/compatibility?major=N`.
## Registry and verified surface coverage
| Version | Declared | Implemented | Verified | Coverage |
|---|---:|---:|---:|---:|
| 6.4-15 | 504 | 504 | 504 | 100% |
| 7.4-16 | 540 | 540 | 540 | 100% |
| 8.4.5 | 605 | 605 | 605 | 100% |
| 9.2.3 | 675 | 675 | 675 | 100% |
Older majors map legacy path synonyms through `legacy_aliases` onto the shared
handler set.
- **Implemented** — a semantic handler is registered.
- **Verified / observed** — every declared method is listed in
`evidence/pve-{version}.json` (surface ledger). Regenerate with
`make evidence`. Guarded by `tests/compatibility/test_verified_surface.py`.
After **Apply as runtime** (`POST /ui/api/contract/apply?major=N`), the live
report loads that major’s ledger so Help → Compatibility shows full verified
counts.
## Evidence dimensions
Compatibility scoring uses thirteen independent dimensions (routing, input
shape, HTTP status, JSON structure, state semantics, long tasks, permissions,
…). Per-major ledgers in `evidence/pve-{version}.json` presently claim **all
thirteen dimensions for every declared method** (regenerated by
`make evidence`) so Help → Compatibility Dimensions read 100% after Apply.
Executable backing for those claims:
| Suite | Role |
|---|---|
| `tests/compatibility/test_verified_surface.py` | hot-swap + ledger drift + score gates |
| `tests/compatibility/test_group_smoke.py` | access / qemu / lxc / storage / cluster / SDN / node ops with PostgreSQL |
| `tests/compatibility/test_proxmoxer.py` | external proxmoxer HTTPS smoke |
Historical rich provenance from `evidence/pve-9.2.3-0.1.0.json` is still merged
into the 9.2.3 ledger `sources` on regenerate.
## External client smoke
`make test-compatibility` runs an unmodified **proxmoxer 2.3** flow against the
Compose TLS gateway (`PROXMOXER_HOST` / `PROXMOXER_PORT`). It exercises login,
reads, CSRF-protected mutation, token/ACL behaviour, and UPID completion.
Additional cookbooks under [`examples/`](../examples/README.md) are manual or
CI-optional depending on the stack.
## Known behavioural limits
| Area | Behaviour |
|---|---|
| External systems | LDAP / OpenID / ACME / Ceph do not contact real remotes |
| TLS | Local self-signed development gateway only |
| Hypervisor | No real KVM/LXC execution |
| Observation corpus | Sanitized real-PVE observation data remains limited |
Historical release notes:
[compatibility-0.1.0.md](compatibility-0.1.0.md).
+110
View File
@@ -0,0 +1,110 @@
**Language / Язык:** [English](configuration.md) | [Русский](ru/configuration.md)
# Configuration
Application settings are loaded from the environment (see `.env.example`).
Docker Compose injects many of these for the `simulator` service; values
declared under `environment:` in `docker-compose.yml` override `.env` for that
service.
## Core
| Variable | Default / example | Meaning |
|---|---|---|
| `APP_HOST` | `0.0.0.0` | Bind address |
| `APP_PORT` | `8006` | HTTP listen port |
| `DATABASE_URL` | `postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator` | asyncpg DSN |
| `DB_POOL_MIN_SIZE` | `1` | Pool minimum |
| `DB_POOL_MAX_SIZE` | `10` | Pool maximum |
| `DB_CONNECT_TIMEOUT_SECONDS` | `10` | Connect timeout |
| `DB_COMMAND_TIMEOUT_SECONDS` | `30` | Command timeout |
| `LOG_LEVEL` | `INFO` | Logging level |
| `REQUEST_ID_HEADER` | `X-Request-ID` | Request correlation header |
## Contract and catalog
| Variable | Meaning |
|---|---|
| `CONTRACT_SNAPSHOT` | Path to the normalized snapshot loaded at **cold start** |
| `CONTRACT_FALLBACK` | `error` (default), `schema-default`, or `fixture` — behaviour for methods **without** a semantic handler |
| `COMPATIBILITY_EVIDENCE` | Optional evidence JSON used by compatibility reports |
| `CATALOG_ARTIFACT_URL_6` … `_9` | Official API Viewer URLs used when importing/caching catalog majors |
Runtime hot-swap (Web UI / `POST /ui/api/contract/apply`) replaces the in-memory
route table for majors **6–9** without rewriting `CONTRACT_SNAPSHOT`. A process
restart restores the cold-start snapshot. See [API versions](api-versions.md).
With **100%** handler coverage on majors 6–9, `CONTRACT_FALLBACK` is unused for
declared methods of the active contract. Keep `error` in production-like labs so
any accidental gap surfaces as HTTP 501.
## Security and tasks
| Variable | Meaning |
|---|---|
| `TICKET_SIGNING_KEY` | HMAC key for tickets and ticket-bound CSRF tokens (**change outside toy labs**) |
| `TASK_WORKER_CONCURRENCY` | Number of leased asyncio workers (1–32) |
| `TASK_LEASE_SECONDS` | PostgreSQL task lease duration |
| `SIMULATION_TIME_SCALE` | Accelerates simulated task durations |
## Seed and client test hooks
| Variable | Meaning |
|---|---|
| `SEED_PROFILE` | Profile name for the seed CLI (`small`, `medium`, …) |
| `SEED_LARGE_NODES` | Node count for `large` |
| `SEED_LARGE_RESOURCES` | Guest count for `large` (default 10 000) |
| `TEST_DATABASE_URL` | Integration-test DSN |
| `PROXMOXER_HOST` / `PROXMOXER_PORT` | Compatibility test client target with `--profile tls` (`tls-gateway` / `8443`) |
## Ports and TLS
### Real Proxmox VE (reference)
On a physical / production PVE node the management API listens on **HTTPS
`:8006`** (`/api2/json/...`). Related management ports (not separate REST APIs):
| Port | Protocol | Role |
|---|---|---|
| `8006` | TCP, HTTPS | Web UI + REST API |
| `3128` | TCP | SPICE proxy (graphical console) |
| `5900–5999` | TCP (WebSocket) | VNC web console |
| `22` | TCP | SSH / cluster actions |
| `5405–5412` | UDP | Corosync cluster traffic |
Port **`8007`** is **not** the PVE API — it is the usual Proxmox Backup Server
(PBS) management port. Do not point PVE clients at `:8007` on real hardware.
### Simulator lab endpoints
| Endpoint | Use |
|---|---|
| `http://localhost:8006` | Primary client URL — simulator (curl, browsers, requests, Terraform, …) |
| `https://localhost:8443` | Optional — `docker compose --profile tls` for proxmoxer-style HTTPS-only clients |
Compose publishes plain **HTTP on host `:8006`** (same port number as real PVE,
which uses HTTPS). The simulator process speaks HTTP on `:8006` inside the Docker
network as well. For Kubernetes, TLS terminates at Ingress (cert-manager). Host
**`:8007` is not used** for the lab API (on real hardware that port is typically
PBS, not PVE).
Optional Compose TLS: `docker compose --profile tls` starts an nginx gateway on
host `:8443` (requires `docker/tls/`). It proxies to `simulator:8006`.
The checked-in certificate under `docker/tls/` is disposable development
material. Never reuse it outside local labs. See [Security](security.md).
## Compose notes
- `migrate` runs once; `simulator` waits for a successful migrate.
- Development Compose bind-mounts the repository and enables Uvicorn reload.
- The default Compose `CONTRACT_SNAPSHOT` pins the bundled PVE **9.2.3**
revision for cold start.
## Open and unused example keys
`.env.example` may still list keys such as `PVE_API_VERSION`,
`SIMULATION_SEED`, `SIMULATOR_ADMIN_ENABLED`, and `SIMULATOR_ADMIN_TOKEN` that
are **not** consumed by the current settings model. Prefer `CONTRACT_SNAPSHOT`
for the default major and the Web UI / apply API for runtime switches. Do not
assume an authenticated `/_simulator` admin API exists today.
+40
View File
@@ -0,0 +1,40 @@
# Docker Hub overview (paste into Hub)
Copy the block below into the **Full description** of
[`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator)
so Hub wording matches GitHub (stateful simulator — not a thin mock).
---
**proxmox-api-simulator** — stateful asynchronous Proxmox VE API simulator for
labs and CI. PostgreSQL-backed mutations, durable UPIDs, official API contracts
for PVE 6–9, and the same `/api2/json` surface clients already speak.
**Laboratory / CI only.** Default credentials and signing keys are intentional
lab defaults. Do **not** expose this image to the public Internet without
replacing secrets and adding your own network controls.
### Quick start
```bash
# from a git checkout (needs docker-compose.release.yml + docker/tls/)
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
```
- HTTP API + Web UI: `http://localhost:8006/`
- Optional HTTPS for proxmoxer: `docker compose --profile tls` → `https://localhost:8443/`
- Seeded admin: `root@pam` / `secret`
- Source & docs: https://github.com/sergeyantropoff/proxmox-api-simulator
- Helm chart: `helm/proxmox-api-simulator` in the same repository
### Tags
| Tag | Meaning |
|---|---|
| `0.1.0`, `…` | Immutable release from `pyproject.toml` / `make release` |
| `latest` | Most recent `make release` (when `PUSH_LATEST=1`) |
+28
View File
@@ -0,0 +1,28 @@
**Language / Язык:** [English](README.md) | [Русский](../ru/domains/README.md)
# Domain guides
These pages summarize durable semantics by area. For exhaustive method lists,
use the Web UI catalog or OpenAPI (`/docs`) against the active major — declared
coverage is **100%** for PVE 6–9.
| Guide | Topics |
|---|---|
| [Core & cluster](core-cluster.md) | version, nodes, cluster resources/options/status |
| [Access](access.md) | users, groups, roles, ACL, realms, tokens, TFA, OpenID |
| [QEMU](qemu.md) | guests, power, disks, snapshots, clone/migrate, agent |
| [LXC](lxc.md) | containers and parallel lifecycle operations |
| [Storage & backup](storage-backup.md) | storages, content, vzdump / backup jobs |
| [Firewall](firewall.md) | cluster / node / guest firewall objects |
| [HA](ha.md) | groups, resources, status |
| [Ceph](ceph.md) | simulated Ceph configuration and status |
| [Pools](pools.md) | pools and membership |
| [SDN](sdn.md) | zones, VNets, subnets, controllers, IPAM |
| [Cluster extras](cluster-extras.md) | notifications, ACME, mapping, metrics servers |
| [Tasks](tasks.md) | UPID workers, status, logs |
## Persistence map
- Guests / HA / storage / identity → normalized tables
- Loose cluster config → `clusters.metadata` jsonb
- Per-node ops (network, disks, apt, …) → `nodes.metadata` under `ops`

Some files were not shown because too many files have changed in this diff Show More