Compare commits

...

9 Commits

Author SHA1 Message Date
Sergey Antropoff 98ba5004ad Fix Deploy stage for Alpine ash: drop bash arrays.
Jenkins docker agents run sh without array syntax; set simulator/migrate images with POSIX conditionals.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-28 02:12:16 +03:00
Sergey Antropoff c64cd4ca8b Raise deploy rollout timeout to 1200s and set images in one step.
Avoid double ReplicaSet churn and ImagePullBackOff races during Hub pulls.
2026-07-28 00:56:46 +03:00
Sergey Antropoff f81d6186ab Add SemVer VERSION, Jenkins CI/CD, and release 0.1.1.
Ship Harbor/Hub push with latest+version tags, deploy to proxmox.devops.org.ru, and show the package version badge in Help → About.
2026-07-28 00:24:05 +03:00
Sergey Antropoff f981ca1bf1 Return clear node-missing JSON and reset Web UI auth on 401.
Keep Ingress from rewriting API 404/405 into branded HTML; document curl/auth and a prompt for sibling sims.
2026-07-22 06:52:45 +03:00
Sergey Antropoff baa1f58ad0 Fix CI/pulumi blockers and document test suites with latest PASS results.
Stabilize node status, cluster-wide Ceph OSD ids, and QEMU cpu utilization so
status/list no longer 500 on model strings; align pulumi defaults to pve1;
add bilingual testing docs with the 2026-07-18 ci-all + pulumi-tests pass.
2026-07-18 09:38:28 +03:00
Sergey Antropoff 6f42d4c318 Simplify bilingual README and docs index around the Web UI.
Drop CI/Docker/License badges, keep a single main-window screenshot, and
link to the Web UI guide at the bottom of the English and Russian pages.
2026-07-18 09:01:44 +03:00
Sergey Antropoff 176c8fd71c Ignore generated pulumi-tests report artifacts.
Keep HTML/JSON/JUnit output from make pulumi-tests out of the repo.
2026-07-18 08:51:52 +03:00
Sergey Antropoff 0773f721ea Align sized cluster seeds and GET dumps with PVE wire shapes; restyle DATA panel.
- Scale small/large/big seeds (3×50 / 10×1000 / 20×2000) with proportional
  backups, snapshots, HA, replication, Ceph capacity, and OSD totals
  (10 / 100 / 500) plus matching node disks and crush/pg metadata
- Enrich handler responses for apt, certificates, qemu/lxc status, storage,
  SDN, metrics export, and related cluster/node dumps
- Flatten nested body_example fields into PARAMS and sync the request body
  via dotted paths (oVirt-style)
- Restyle DATA controls as size cards with full-width Reset to minimal /
  Refresh stats; unload reloads the minimal cluster
2026-07-18 08:46:11 +03:00
Sergey Antropoff 48df10b17e Prepare 0.1.0 for lab release: durable handlers, HTTP Compose, CI, and pulumi-tests.
- Harden DB-backed handlers and seed profiles; align client wire shapes for
  cluster resources, QEMU config, and node SSL fields
- Serve plain HTTP on Compose :8006; keep TLS optional (--profile tls) and
  terminate HTTPS at Kubernetes Ingress
- Add pulumi-tests (full contract surface majors 6–9 + BPG lifecycle) and
  make pulumi-tests
- Ship bilingual docs, CHANGELOG, SECURITY, CONTRIBUTING, and GitHub Actions
  (make ci + Compose/Helm validation)
2026-07-18 04:18:05 +03:00
225 changed files with 13188 additions and 1861 deletions
+3 -1
View File
@@ -23,5 +23,7 @@ SIMULATION_SEED=42
SIMULATION_TIME_SCALE=10
SIMULATOR_ADMIN_ENABLED=false
SIMULATOR_ADMIN_TOKEN=replace-with-a-long-random-secret
PROXMOXER_HOST=tls-gateway
PROXMOXER_HOST=
PROXMOXER_PORT=8443
# For proxmoxer (HTTPS-only): docker compose --profile tls up -d
# then PROXMOXER_HOST=tls-gateway PROXMOXER_PORT=8443
+60
View File
@@ -0,0 +1,60 @@
name: CI
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Compose / Helm validate
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Validate Compose files
run: |
docker compose -f docker-compose.yml config --quiet
docker compose -f docker-compose.release.yml config --quiet
- uses: azure/setup-helm@v4
with:
version: v3.16.4
- name: Lint and render Helm chart
run: |
make helm-lint
make helm-template >/dev/null
quality:
name: make ci
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- name: Prepare lab env
run: test -f .env || cp .env.example .env
- name: Build runtime and development images
run: make install
- name: Ruff + mypy + offline pytest + API surface probe
run: make ci
- name: Upload coverage XML
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-xml
path: coverage.xml
if-no-files-found: ignore
+4 -1
View File
@@ -67,9 +67,12 @@ hypothesis/
*.sublime-project
*.sublime-workspace
# Docker / local runtime
# Docker / local runtime (`make up-local` writes port remaps here)
*.log
docker-compose.override.yml
# Lab probe output (regenerated by scripts/probe_api_surface.py)
evidence/_api_surface_probe.json
# Pulumi suite reports (HTML/JSON/JUnit from make pulumi-tests)
pulumi-tests/reports/*
+30
View File
@@ -0,0 +1,30 @@
# Changelog
All notable changes to this project are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.1.0] - 2026-07-17
### Added
- Stateful Proxmox VE API simulator backed by PostgreSQL with durable UPID task
workers.
- Imported official API contracts for PVE majors **6.4-15**, **7.4-16**,
**8.4.5**, and **9.2.3** with full handler registration for declared methods.
- Interactive Web UI (catalog, runtime contract Apply, task monitor).
- Development Compose stack with plain HTTP on host `:8006` (real PVE port; real
PVE uses HTTPS). Optional TLS gateway via `--profile tls` on `:8443`.
- Published runtime image workflow (`make release`) for
[`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator).
- `docker-compose.release.yml` + Helm chart with optional Ingress / cert-manager.
- Bilingual documentation (English + Russian) and client cookbooks.
- GitHub Actions CI (`make ci`, Compose/Helm validation).
### Security
- Explicit lab-only threat model: default secrets and open `/ui` / `/admin`
helpers are for local/CI use. See [SECURITY.md](SECURITY.md).
[0.1.0]: https://github.com/sergeyantropoff/proxmox-api-simulator/releases/tag/v0.1.0
+69
View File
@@ -0,0 +1,69 @@
**Language / Язык:** [English](CONTRIBUTING.md) | [Русский](CONTRIBUTING.ru.md)
# Contributing
Thanks for helping improve the Proxmox VE API laboratory simulator.
## Prerequisites
- Docker + Docker Compose
- `make`
- No local Python toolchain required for day-to-day work (tools run in Compose)
## Local loop
```bash
cp -n .env.example .env
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
```
Primary HTTP endpoint (same port as real PVE): `http://localhost:8006/`
Optional HTTPS for proxmoxer: `docker compose --profile tls` →
`https://localhost:8443/` (self-signed — use `curl -sk` or accept the browser
warning).
## Quality gates (must pass before a PR)
```bash
make ci # ruff format/check + mypy + offline pytest + surface probe
make ci-all # also remaining integration + proxmoxer compatibility
make helm-lint # chart lint (+ ingress example values)
make pulumi-tests # Pulumi surface (majors 6–9) + pulumi-proxmoxve lifecycle
```
GitHub Actions runs `make ci` plus Compose/Helm validation on every push and PR
to `main`. Run `make ci-all`, `make helm-lint`, and `make pulumi-tests` locally
before larger API or client-facing changes.
Suite layout, Make targets, and the latest recorded local results:
[docs/testing.md](docs/testing.md).
## Project rules worth remembering
1. Mutations must **persist to PostgreSQL** (tables and/or jsonb metadata).
2. Do **not** add user-facing “not supported in the simulator” style errors —
see `.cursor/rules/durable-simulator.mdc`.
3. Prefer matching Proxmox request/response shapes from the contract snapshot.
4. Keep EN and RU docs in sync when you change operator-facing behaviour.
## Docs
- Index: [docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md)
- Security / lab threat model: [SECURITY.md](SECURITY.md)
## Releases
Maintainers publish the runtime image with:
```bash
docker login
make release # pushes inecs/proxmox-api-simulator:<version> (+ :latest)
```
After a public release, paste the overview from
[docs/docker-hub-overview.md](docs/docker-hub-overview.md) into the Docker Hub
repository description if it drifted.
+66
View File
@@ -0,0 +1,66 @@
**Language / Язык:** [English](CONTRIBUTING.md) | [Русский](CONTRIBUTING.ru.md)
# Участие в разработке
Спасибо за помощь с лабораторным симулятором Proxmox VE API.
## Требования
- Docker + Docker Compose
- `make`
- Локальный Python для повседневной работы не нужен (инструменты в Compose)
## Локальный цикл
```bash
cp -n .env.example .env
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
```
Основной HTTPS-эндпоинт (как у реального PVE): `http://localhost:8006/`
(self-signed — `curl -sk` или принять предупреждение в браузере).
## Quality gates (обязательны перед PR)
```bash
make ci # ruff format/check + mypy + offline pytest + surface probe
make ci-all # ещё integration + proxmoxer compatibility
make helm-lint # lint chart (+ ingress example values)
make pulumi-tests # Pulumi surface (majors 6–9) + pulumi-proxmoxve lifecycle
```
GitHub Actions на каждый push/PR в `main` запускает `make ci` и проверку
Compose/Helm. Перед крупными API/клиентскими изменениями локально прогоняйте
`make ci-all`, `make helm-lint` и `make pulumi-tests`.
Карта наборов, Make-цели и последние записанные локальные результаты:
[docs/ru/testing.md](docs/ru/testing.md).
## Важные правила проекта
1. Мутации должны **persist в PostgreSQL** (таблицы и/или jsonb metadata).
2. Не добавляйте user-facing сообщения вроде «not supported in the simulator» —
см. `.cursor/rules/durable-simulator.mdc`.
3. Сохраняйте формы запросов/ответов Proxmox из снимка контракта.
4. При изменении операторского поведения синхронизируйте EN и RU docs.
## Документация
- Индекс: [docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md)
- Безопасность / модель угроз лаборатории: [SECURITY.md](SECURITY.md)
## Релизы
Maintainers публикуют runtime-образ так:
```bash
docker login
make release # пушит inecs/proxmox-api-simulator:<version> (+ :latest)
```
После публичного релиза при необходимости вставьте overview из
[docs/docker-hub-overview.md](docs/docker-hub-overview.md) в описание репозитория
Docker Hub.
+4 -1
View File
@@ -16,7 +16,9 @@ FROM python:3.13-slim AS runtime
ARG APP_VERSION=0.1.0
LABEL org.opencontainers.image.title="proxmox-api-simulator" \
org.opencontainers.image.version="$APP_VERSION" \
org.opencontainers.image.source="https://github.com/example/proxmox-api-simulator"
org.opencontainers.image.description="Stateful Proxmox VE API simulator for labs and CI" \
org.opencontainers.image.source="https://github.com/sergeyantropoff/proxmox-api-simulator" \
org.opencontainers.image.licenses="Apache-2.0"
ENV PATH="/opt/venv/bin:$PATH" \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
@@ -25,6 +27,7 @@ ENV PATH="/opt/venv/bin:$PATH" \
RUN groupadd --system --gid 10001 simulator \
&& useradd --system --uid 10001 --gid simulator --home-dir /app --no-create-home simulator
COPY --from=builder /opt/venv /opt/venv
COPY VERSION /app/VERSION
COPY contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json /app/contracts/pve-9.2.3.json
COPY evidence/ /app/evidence/
WORKDIR /app
Vendored
+205
View File
@@ -0,0 +1,205 @@
// Proxmox API Simulator — CI/CD: multi-arch push (Harbor + Docker Hub) + deploy
//
// Один файл: Build (DinD) → Deploy (kubectl set image в релиз simulators).
// Первый install lab — через DevOpsTools/K3S: make addon-simulators
// (Ingress https://proxmox.devops.org.ru уже из addon values).
//
// Версия образа = VERSION из коммита. Локальный bump: make bump-patch / make push.
// Теги: :{VERSION} и :latest (без sha).
//
// Credentials (Global):
// harbor-devops-tools-push-pull-access — Harbor devops-tools (robot)
// docker-hub — Docker Hub (inecs)
// k3s-kubeconfig — kubeconfig к K3S
// ssh-gitea-key — SCM checkout Gitea (в job)
pipeline {
agent none
options {
buildDiscarder(logRotator(numToKeepStr: '20'))
disableConcurrentBuilds()
timeout(time: 60, unit: 'MINUTES')
timestamps()
}
environment {
HARBOR_REGISTRY = 'hub.antropoff.ru'
HARBOR_IMAGE = 'hub.antropoff.ru/devops-tools/proxmox-api-simulator'
DOCKERHUB_IMAGE = 'inecs/proxmox-api-simulator'
BUILDX_BUILDER = "jenkins-proxmox-api-simulator-${env.BUILD_NUMBER}"
HELM_NAMESPACE = 'simulators'
DEPLOYMENT_NAME = 'simulators-proxmox'
INGRESS_HOST = 'proxmox.devops.org.ru'
TZ = 'Europe/Moscow'
}
stages {
stage('Build & push') {
when {
anyOf {
branch 'main'
branch 'master'
}
}
agent { label 'docker' }
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Version') {
steps {
script {
env.IMAGE_VERSION = readFile('VERSION').trim()
if (!env.IMAGE_VERSION) {
error('VERSION file is empty')
}
echo "IMAGE_VERSION from VERSION → ${env.IMAGE_VERSION}"
}
}
}
stage('Buildx push') {
steps {
withCredentials([
usernamePassword(
credentialsId: 'harbor-devops-tools-push-pull-access',
usernameVariable: 'HARBOR_USER',
passwordVariable: 'HARBOR_PASS'
),
usernamePassword(
credentialsId: 'docker-hub',
usernameVariable: 'DOCKERHUB_USER',
passwordVariable: 'DOCKERHUB_PASS'
)
]) {
container('docker') {
sh '''
set -eux
test -n "${IMAGE_VERSION}"
echo "Building tags: ${IMAGE_VERSION}, latest (amd64+arm64)"
echo "$HARBOR_PASS" | docker login "$HARBOR_REGISTRY" -u "$HARBOR_USER" --password-stdin
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USER" --password-stdin
docker buildx rm "$BUILDX_BUILDER" 2>/dev/null || true
docker buildx create --name "$BUILDX_BUILDER" --driver docker-container --use
docker buildx inspect --bootstrap >/dev/null
docker buildx build \
--platform linux/amd64,linux/arm64 \
--target runtime \
--build-arg "APP_VERSION=${IMAGE_VERSION}" \
--provenance=false --sbom=false --push \
-t "${HARBOR_IMAGE}:${IMAGE_VERSION}" \
-t "${HARBOR_IMAGE}:latest" \
-t "${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" \
-t "${DOCKERHUB_IMAGE}:latest" \
-f Dockerfile .
echo "--- Harbor ---"
docker buildx imagetools inspect "${HARBOR_IMAGE}:${IMAGE_VERSION}" | sed -n '1,40p'
echo "--- Docker Hub ---"
docker buildx imagetools inspect "${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" | sed -n '1,40p'
docker buildx rm "$BUILDX_BUILDER" || true
'''
}
}
}
}
stage('Deploy') {
steps {
withCredentials([
file(credentialsId: 'k3s-kubeconfig', variable: 'KUBECONFIG')
]) {
container('docker') {
sh '''
set -eux
if ! command -v kubectl >/dev/null 2>&1; then
apk add --no-cache curl >/dev/null
ARCH="$(uname -m)"
case "$ARCH" in
x86_64) ARCH=amd64 ;;
aarch64|arm64) ARCH=arm64 ;;
esac
KVER="$(curl -fsSL https://dl.k8s.io/release/stable.txt)"
curl -fsSLo /usr/local/bin/kubectl \
"https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
chmod +x /usr/local/bin/kubectl
fi
kubectl version --client || true
if ! kubectl -n "$HELM_NAMESPACE" get deploy "$DEPLOYMENT_NAME" >/dev/null 2>&1; then
echo "Deployment ${HELM_NAMESPACE}/${DEPLOYMENT_NAME} not found." >&2
echo "Сначала: make addon-simulators (Ingress ${INGRESS_HOST})." >&2
exit 1
fi
echo "Rolling ${DOCKERHUB_IMAGE}:${IMAGE_VERSION} → ${HELM_NAMESPACE}/${DEPLOYMENT_NAME}"
echo "Public URL: https://${INGRESS_HOST}/"
# POSIX sh (Alpine ash): no bash arrays. One set-image when migrate exists.
INITS="$(kubectl -n "$HELM_NAMESPACE" get deploy "$DEPLOYMENT_NAME" \
-o jsonpath='{.spec.template.spec.initContainers[*].name}' || true)"
if printf '%s\n' $INITS | grep -qx migrate; then
kubectl -n "$HELM_NAMESPACE" set image \
"deployment/${DEPLOYMENT_NAME}" \
"simulator=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" \
"migrate=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}"
else
kubectl -n "$HELM_NAMESPACE" set image \
"deployment/${DEPLOYMENT_NAME}" \
"simulator=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}"
fi
kubectl -n "$HELM_NAMESPACE" rollout status \
"deployment/${DEPLOYMENT_NAME}" --timeout=1200s
kubectl -n "$HELM_NAMESPACE" get deploy,po,ing -o wide
'''
}
}
}
}
}
post {
always {
script {
try {
container('docker') {
sh 'docker buildx rm "$BUILDX_BUILDER" 2>/dev/null || true'
}
} catch (Ignored) {
// pod may already be gone
}
}
}
success {
echo "✓ ${DOCKERHUB_IMAGE}:{${IMAGE_VERSION},latest} → https://${INGRESS_HOST}/"
}
failure {
echo "✗ Build/push/deploy failed — см. лог"
}
}
}
}
post {
success {
echo "✓ Version: ${IMAGE_VERSION}"
echo "✓ Harbor: ${HARBOR_IMAGE}:{${IMAGE_VERSION},latest}"
echo "✓ Hub: ${DOCKERHUB_IMAGE}:{${IMAGE_VERSION},latest}"
echo "✓ Live: https://${INGRESS_HOST}/"
}
failure {
echo "✗ Pipeline failed"
}
}
}
+82 -12
View File
@@ -4,19 +4,22 @@ SERVICE_SIM := simulator
PYTEST_OFFLINE := -m "not integration and not compatibility"
# Docker Hub release image (runtime target only — not the local bind-mount "dev" image).
# SemVer source of truth: VERSION (synced to pyproject + Helm via bump targets).
DOCKERHUB_USER ?= inecs
IMAGE_NAME ?= proxmox-api-simulator
VERSION ?= $(shell sed -n 's/^version = "\(.*\)"/\1/p' pyproject.toml)
VERSION ?= $(shell cat VERSION 2>/dev/null || echo 0.1.0)
DOCKER_IMAGE ?= $(DOCKERHUB_USER)/$(IMAGE_NAME)
PUSH_LATEST ?= 1
COMPOSE_RELEASE ?= $(COMPOSE) -f docker-compose.release.yml
HELM_CHART ?= ./helm/proxmox-api-simulator
.PHONY: help install format lint typecheck test test-unit test-integration test-contract test-compatibility test-surface evidence coverage run dev up down restart logs docker-build docker-up docker-down docker-logs docker-restart db-up db-down db-migrate db-reset api-import api-diff seed clean ci ci-all shell release release-build release-up release-down release-seed helm-deps helm-template
.PHONY: help install format lint typecheck test test-unit test-integration test-contract test-compatibility test-surface evidence coverage run dev up up-local down down-local restart logs docker-build docker-up docker-down docker-logs docker-restart db-up db-down db-migrate db-reset api-import api-diff seed clean ci ci-all shell release release-build release-up release-down release-seed helm-deps helm-template helm-lint pulumi-tests push bump-patch bump-minor version-commit
help: ## Show available commands
@awk 'BEGIN {FS = ":.*## "}; /^[a-zA-Z0-9_-]+:.*## / {printf "%-22s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
@echo
@echo "Version: $$(cat VERSION 2>/dev/null || echo 0.1.0) (make bump-patch | bump-minor)"
install: ## Build runtime and development images
@test -f .env || cp .env.example .env
@@ -45,11 +48,16 @@ test-integration: ## Run tests that require PostgreSQL
test-contract: ## Run offline API contract tests
$(COMPOSE) run --rm --no-deps $(SERVICE_DEV) pytest -m contract
test-compatibility: ## Run proxmoxer smoke flow against the Compose stack
test-compatibility: ## Run proxmoxer smoke flow (needs --profile tls; proxmoxer is HTTPS-only)
@test -f .env || cp .env.example .env
$(COMPOSE) up -d --build --wait
$(COMPOSE) run --rm --entrypoint python $(SERVICE_SIM) -m app.simulation.seed_cli
$(COMPOSE) run --rm $(SERVICE_DEV) pytest -m compatibility
$(COMPOSE) --profile tls up -d --build --wait
# Medium profile provides pve1/pve2/pve3 required by the proxmoxer migration smoke.
$(COMPOSE) run --rm -e SEED_PROFILE=medium --entrypoint python $(SERVICE_SIM) \
-m app.simulation.seed_cli
$(COMPOSE) run --rm \
-e PROXMOXER_HOST=tls-gateway \
-e PROXMOXER_PORT=8443 \
$(SERVICE_DEV) pytest -m compatibility
test-surface: ## Probe every declared method on majors 6-9 (0x501 / 0xexception)
@test -f .env || cp .env.example .env
@@ -66,13 +74,21 @@ run: ## Run the application in the foreground
@test -f .env || cp .env.example .env
$(COMPOSE) up --build
up: ## Start PostgreSQL, simulator, and TLS gateway
up: ## Start PostgreSQL and simulator (plain HTTP :8006)
@test -f .env || cp .env.example .env
$(COMPOSE) up -d --build --wait
up-local: ## Start locally; remap busy ports via gitignored docker-compose.override.yml
@COMPOSE="$(COMPOSE)" ./scripts/up-local.sh
down: ## Stop local services
$(COMPOSE) down
down-local: ## Stop local stack and remove gitignored docker-compose.override.yml
@$(COMPOSE) down
@rm -f docker-compose.override.yml
@echo "Stopped local stack and removed docker-compose.override.yml"
restart: ## Rebuild and restart the stack
@test -f .env || cp .env.example .env
$(COMPOSE) up -d --build --force-recreate --wait
@@ -90,8 +106,8 @@ docker-build: ## Build runtime and development images
docker-up: up ## Alias for up
docker-restart: ## Rebuild and recreate simulator and TLS gateway only
$(COMPOSE) up -d --build --force-recreate simulator tls-gateway
docker-restart: ## Rebuild and recreate the simulator
$(COMPOSE) up -d --build --force-recreate simulator
docker-down: down ## Alias for down
@@ -122,7 +138,11 @@ api-diff: ## Compare API snapshots
seed: ## Seed simulation data
@test -f .env || cp .env.example .env
SEED_PROFILE="$${PROFILE:-small}" $(COMPOSE) run --rm --entrypoint python $(SERVICE_SIM) -m app.simulation.seed_cli
$(COMPOSE) run --rm \
-e SEED_PROFILE="$${PROFILE:-small}" \
-e SEED_LARGE_NODES="$${SEED_LARGE_NODES:-10}" \
-e SEED_LARGE_RESOURCES="$${SEED_LARGE_RESOURCES:-1000}" \
--entrypoint python $(SERVICE_SIM) -m app.simulation.seed_cli
shell: ## Open an interactive shell in the development container
$(COMPOSE) run --rm --no-deps $(SERVICE_DEV) bash
@@ -170,14 +190,64 @@ release-down: ## Stop the published Hub stack
$(COMPOSE_RELEASE) down
release-seed: ## Seed the published Hub stack (PROFILE=small by default)
SEED_PROFILE="$${PROFILE:-small}" IMAGE_TAG="$${IMAGE_TAG:-$(VERSION)}" DOCKER_IMAGE="$(DOCKER_IMAGE)" \
$(COMPOSE_RELEASE) run --rm --entrypoint python simulator -m app.simulation.seed_cli
IMAGE_TAG="$${IMAGE_TAG:-$(VERSION)}" DOCKER_IMAGE="$(DOCKER_IMAGE)" \
$(COMPOSE_RELEASE) run --rm \
-e SEED_PROFILE="$${PROFILE:-small}" \
-e SEED_LARGE_NODES="$${SEED_LARGE_NODES:-10}" \
-e SEED_LARGE_RESOURCES="$${SEED_LARGE_RESOURCES:-1000}" \
--entrypoint python simulator -m app.simulation.seed_cli
helm-deps: ## No-op placeholder (chart has no OCI dependencies)
@echo "Chart $(HELM_CHART) vendors PostgreSQL templates; no helm dependency update required."
helm-lint: ## Lint the Helm chart (requires helm)
helm lint $(HELM_CHART)
helm lint $(HELM_CHART) -f $(HELM_CHART)/values-ingress-example.yaml \
--set certManager.email=docs@example.com \
--set secret.ticketSigningKey=docs-only-signing-key
helm-template: ## Render Helm manifests locally (requires helm)
helm template pve-sim $(HELM_CHART) \
-f $(HELM_CHART)/values-ingress-example.yaml \
--set certManager.email=docs@example.com \
--set secret.ticketSigningKey=docs-only-signing-key
pulumi-tests: ## Full Pulumi suite (surface majors 6–9 + lifecycle, HTML report)
$(MAKE) -C pulumi-tests up
$(MAKE) -C pulumi-tests test
bump-patch: ## VERSION +0.0.1 (pyproject + Helm)
@PYTHONPATH=. python3 scripts/bump_version.py patch
@echo "VERSION → $$(cat VERSION)"
bump-minor: ## VERSION +0.1.0 (pyproject + Helm)
@PYTHONPATH=. python3 scripts/bump_version.py minor
@echo "VERSION → $$(cat VERSION)"
version-commit: ## Stage version files and commit "Bump version to …"
@git add VERSION pyproject.toml helm/proxmox-api-simulator/Chart.yaml helm/proxmox-api-simulator/values.yaml
@if git diff --cached --quiet; then \
echo "Nothing to commit for version files."; \
else \
git commit -m "Bump version to $$(cat VERSION)."; \
fi
push: ## bump-patch + commit version + push origin (GitHub + antropoff.ru)
@$(MAKE) bump-patch
@$(MAKE) version-commit
@set -e; \
git add .; \
echo "=== staged ==="; \
git status --short; \
echo; \
if git diff --cached --quiet; then \
echo "Nothing else to commit — pushing current branch."; \
else \
echo "Enter commit message for remaining changes, then Ctrl-D:"; \
msg=$$(cat </dev/tty); \
if [ -z "$$msg" ]; then echo "Empty commit message, aborting." >&2; exit 1; fi; \
git commit -m "$$msg"; \
fi; \
echo "Pushing to both remotes:"; \
git remote get-url --push --all origin | sed 's/^/ - /'; \
git push origin HEAD
+47 -15
View File
@@ -1,9 +1,16 @@
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
# proxmox-api-simulator
Stateful asynchronous [Proxmox VE](https://www.proxmox.com/) API simulator for
testing API clients and infrastructure tooling without a real hypervisor
cluster.
> **Laboratory / CI only.** Default credentials, signing keys, and open UI/admin
> helpers are intentional lab defaults. Do **not** expose this stack to the
> public Internet without replacing secrets and adding your own controls.
> See [SECURITY.md](SECURITY.md) and [Security](docs/security.md).
The simulator is backed by PostgreSQL, driven by imported official API
contracts, and exposes the same `/api2/json` and `/api2/extjs` surfaces as
Proxmox VE. Semantic handlers persist mutations; long-running work returns
@@ -37,13 +44,16 @@ Image: [`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-ap
### Docker Compose
Needs a checkout that includes `docker-compose.release.yml`. Do not publish host
`:8006` beyond a trusted lab without rotating `TICKET_SIGNING_KEY` / DB password.
```bash
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl http://localhost:8006/health/ready
curl http://localhost:8006/api2/json/version
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
```
Or: `make release-up && make release-seed PROFILE=small`
@@ -57,10 +67,14 @@ helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
--set certManager.email=you@example.com \
--set ingress.hosts[0].host=pve-sim.example.com \
--set ingress.tls[0].hosts[0]=pve-sim.example.com \
--set secret.ticketSigningKey="$(openssl rand -hex 32)"
--set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set postgresql.auth.password="$(openssl rand -hex 16)"
```
Requires an Ingress controller and cert-manager. Details:
Requires an Ingress controller and cert-manager. The chart Service speaks
**HTTP** `:8006`; TLS terminates at Ingress. Compose also serves plain HTTP on
`:8006`; optional HTTPS for proxmoxer-style clients is
`docker compose --profile tls` on host `:8443`. Details:
[Kubernetes / Helm](docs/kubernetes.md).
- HTTP API and Web UI (Compose): [http://localhost:8006/](http://localhost:8006/)
@@ -76,29 +90,34 @@ make install
make up
make seed PROFILE=small
curl http://localhost:8006/health/ready
curl http://localhost:8006/api2/json/version
curl -X POST -d 'username=root@pam&password=secret' \
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
curl -sS -X POST -d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket
```
- HTTP API and Web UI: [http://localhost:8006/](http://localhost:8006/)
- HTTPS gateway (self-signed, development only): `https://localhost:8007`
— checked-in `docker/tls/server.key` is a **lab-only** localhost cert; do not
reuse it outside local Compose.
(real PVE uses **HTTPS** on `:8006`; this lab uses plain HTTP on the same
port. Optional TLS for proxmoxer: `docker compose --profile tls` →
`https://localhost:8443/`)
- Port map and TLS notes: [Ports and TLS](docs/configuration.md#ports-and-tls).
- FastAPI schema docs: [http://localhost:8006/docs](http://localhost:8006/docs)
### Web UI
Interactive console with light/dark themes, endpoint catalog for PVE 6–9, and
runtime contract hot-swap. More detail: [Web UI](docs/web-ui.md).
Interactive console with light/dark themes, endpoint catalog for PVE 6–9,
runtime contract hot-swap, and a UPID task monitor.
![Web UI light theme](docs/images/web-ui-light.png)
![Web UI main](docs/images/web-ui-main.png)
![Web UI dark theme](docs/images/web-ui-dark.png)
More screens and details: [Web UI](docs/web-ui.md).
## Documentation
Documentation is bilingual. Use the **Language / Язык** switcher at the top of
each page, or open the Russian root [README.ru.md](README.ru.md). Index:
[docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md).
| Guide | Description |
|---|---|
| [Getting started](docs/getting-started.md) | First successful lab session |
@@ -120,6 +139,8 @@ runtime contract hot-swap. More detail: [Web UI](docs/web-ui.md).
| [Compatibility](docs/compatibility.md) | Evidence model and release matrix |
Runnable cookbooks live under [`examples/`](examples/README.md).
Pulumi integration suite (contract surface majors 6–9 + lifecycle, HTML report):
[`pulumi-tests/`](pulumi-tests/README.md) (`make pulumi-tests`).
## proxmoxer (HTTPS gateway)
@@ -128,7 +149,7 @@ from proxmoxer import ProxmoxAPI
proxmox = ProxmoxAPI(
"localhost",
port=8007,
port=8006,
user="root@pam",
password="secret",
verify_ssl=False, # local self-signed development certificate only
@@ -169,9 +190,20 @@ make release-build # build/tag only, no push
make release-up && make release-seed # run the published stack locally
```
## Contributing / security / changelog
- [CONTRIBUTING.md](CONTRIBUTING.md) · [CONTRIBUTING.ru.md](CONTRIBUTING.ru.md)
- [Testing](docs/testing.md) · [Тесты](docs/ru/testing.md) — suites, Make targets, latest results
- [SECURITY.md](SECURITY.md)
- [CHANGELOG.md](CHANGELOG.md)
## What this is not
- Not a hypervisor: no KVM/LXC execution on bare metal or nested hosts.
- Not a drop-in multi-tenant production Proxmox replacement.
- Remote IdP / LDAP / live Ceph / live ACME directories are simulated locally;
they do not call real external systems.
## See also
- [Web UI](docs/web-ui.md) — interactive console, catalogs, DATA panel, and screenshots
+210
View File
@@ -0,0 +1,210 @@
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
# proxmox-api-simulator
Stateful-асинхронный симулятор API [Proxmox VE](https://www.proxmox.com/) для
тестирования API-клиентов и инфраструктурных инструментов без реального
гипервизорного кластера.
> **Только лаборатория / CI.** Учётные данные, signing keys и открытые UI/admin
> helpers по умолчанию — намеренные лабораторные значения. **Не** выставляйте
> стек в публичный Интернет без замены секретов и своих сетевых ограничений.
> См. [SECURITY.md](SECURITY.md) и [Безопасность](docs/ru/security.md).
Симулятор работает на PostgreSQL, опирается на импортированные официальные
контракты API и предоставляет те же поверхности `/api2/json` и `/api2/extjs`,
что и Proxmox VE. Семантические обработчики сохраняют мутации; длительные
операции возвращают устойчивые UPID, которые выполняют воркеры с арендой задач.
## Проверенное покрытие API
Реестр обработчиков и верифицированные ledger поверхности — **100%** для каждого
включённого major:
| Контракт | Declared | Implemented | Verified |
|---|---:|---:|---:|
| PVE 6.4-15 | 504 | 504 | 504 |
| PVE 7.4-16 | 540 | 540 | 540 |
| PVE 8.4.5 | 605 | 605 | 605 |
| PVE 9.2.3 | 675 | 675 | 675 |
Переключение активного контракта в runtime — из Web UI (**Apply as runtime**) или
`POST /ui/api/contract/apply?major=N` — каждый Apply загружает
`evidence/pve-{version}.json`, чтобы observed/verified следовали выбранному
major. После импорта нового контракта перегенерируйте ledger: `make evidence`.
Живые отчёты: `/admin/compatibility` (также `.md` / `.html`). См.
[Совместимость](docs/ru/compatibility.md) и [Версии API](docs/ru/api-versions.md).
> Это измеримое покрытие контракта и обработчиков лабораторного симулятора —
> не утверждение, что каждый краевой случай Proxmox или удалённая интеграция
> ведёт себя идентично продакшен-железу.
## Быстрый старт (опубликованный образ)
Образ: [`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator)
### Docker Compose
Нужен checkout с `docker-compose.release.yml`. Не публикуйте хост `:8006` за
пределы доверенной лаборатории без ротации `TICKET_SIGNING_KEY` / пароля БД.
```bash
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
```
Или: `make release-up && make release-seed PROFILE=small`
### Helm (Kubernetes + Ingress + Let's Encrypt)
```bash
helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
-n proxmox-sim --create-namespace \
-f ./helm/proxmox-api-simulator/values-ingress-example.yaml \
--set certManager.email=you@example.com \
--set ingress.hosts[0].host=pve-sim.example.com \
--set ingress.tls[0].hosts[0]=pve-sim.example.com \
--set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set postgresql.auth.password="$(openssl rand -hex 16)"
```
Нужны Ingress-контроллер и cert-manager. Service чарта говорит по **HTTP**
`:8006`; TLS — на Ingress. Compose тоже отдаёт plain HTTP на `:8006`; опциональный
HTTPS для proxmoxer — `docker compose --profile tls` на хосте `:8443`.
Подробности: [Kubernetes / Helm](docs/ru/kubernetes.md).
- HTTP API и Web UI (Compose): [http://localhost:8006/](http://localhost:8006/)
- Схема FastAPI: [http://localhost:8006/docs](http://localhost:8006/docs)
- Админ по умолчанию после seed: `root@pam` / `secret`
## Быстрый старт (разработка из репозитория)
Сборка и запуск development-стека с bind-mount из этого репозитория:
```bash
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
curl -sS -X POST -d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket
```
- HTTP API и Web UI: [http://localhost:8006/](http://localhost:8006/)
(реальный PVE — **HTTPS** на `:8006`; лаборатория — plain HTTP на том же
порту. Опциональный TLS для proxmoxer: `docker compose --profile tls` →
`https://localhost:8443/`)
- Карта портов и TLS: [Порты и TLS](docs/ru/configuration.md#порты-и-tls).
- Схема FastAPI: [http://localhost:8006/docs](http://localhost:8006/docs)
### Web UI
Интерактивная консоль со светлой/тёмной темой, каталогом эндпоинтов PVE 6–9,
горячей сменой runtime-контракта и монитором задач UPID.
![Web UI, главный экран](docs/images/web-ui-main.png)
Больше экранов и подробностей: [Web UI](docs/ru/web-ui.md).
## Документация
Документация двуязычная. Переключатель **Language / Язык** — в шапке каждой
страницы; английский корень — [README.md](README.md). Индекс:
[docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md).
| Руководство | Описание |
|---|---|
| [Начало работы](docs/ru/getting-started.md) | Первая успешная лабораторная сессия |
| [Конфигурация](docs/ru/configuration.md) | Переменные окружения и Compose |
| [Аутентификация](docs/ru/authentication.md) | Тикеты, CSRF, API-токены, ACL |
| [Версии API](docs/ru/api-versions.md) | Контракты 6–9 и hot-swap |
| [Клиенты и примеры](docs/ru/clients.md) | Python, Go, Java, Perl, Ansible, Terraform, Pulumi |
| [Профили seed](docs/ru/seed-profiles.md) | Детерминированные фикстуры кластера |
| [Поверхность API](docs/ru/api-surface.md) | Маршрутизация, обработчики, fallback |
| [Домены](docs/ru/domains/README.md) | QEMU, LXC, storage, HA, SDN, … |
| [Web UI](docs/ru/web-ui.md) | Интерактивная консоль и каталоги |
| [Эксплуатация](docs/ru/operations.md) | Миграции, reseed, обновления |
| [Kubernetes / Helm](docs/ru/kubernetes.md) | Образ Hub + Ingress + Let's Encrypt |
| [Безопасность](docs/ru/security.md) | Модель угроз лаборатории и учётные данные |
| [Наблюдаемость](docs/ru/observability.md) | Health-эндпоинты и логирование |
| [Устранение неполадок](docs/ru/troubleshooting.md) | Типичные сбои |
| [FAQ](docs/ru/faq.md) | Краткие ответы |
| [Архитектура](docs/ru/architecture.md) | Границы компонентов |
| [Совместимость](docs/ru/compatibility.md) | Модель evidence и матрица релиза |
Индекс гайдов: [`docs/ru/README.md`](docs/ru/README.md).
Запускаемые cookbook: [`examples/`](examples/README.ru.md).
Интеграционный набор Pulumi (surface majors 6–9 + lifecycle, HTML-отчёт):
[`pulumi-tests/`](pulumi-tests/README.ru.md) (`make pulumi-tests`).
## proxmoxer (HTTPS-шлюз)
```python
from proxmoxer import ProxmoxAPI
proxmox = ProxmoxAPI(
"localhost",
port=8006,
user="root@pam",
password="secret",
verify_ssl=False, # только локальный self-signed сертификат разработки
)
print(proxmox.version.get())
print(proxmox.nodes("pve01").qemu.get())
```
Пример API-токена: пользователь `root@pam`, `token_name="automation"`,
`token_value="automation-secret"`. Запросам с токеном CSRF не нужен; мутациям
по тикету — нужен.
## Частые цели Make
```bash
make up / make down / make logs / make dev
make test # unit + contract (включая verified surface)
make test-integration # с PostgreSQL
make test-surface # все глаголы × majors 6-9 (0x501 / 0xexception)
make test-compatibility # proxmoxer против Compose
make evidence # перегенерация evidence/pve-*.json
make seed PROFILE=small
make db-migrate
make shell
make ci # ruff + mypy + offline pytest + surface probe
make release # сборка + push runtime-образа в Docker Hub
make release-up # pull/start docker-compose.release.yml
make release-seed PROFILE=small
```
Релиз в Docker Hub (нужен `docker login` владельца Hub; см.
[Эксплуатация](docs/ru/operations.md)):
```bash
make release # inecs/proxmox-api-simulator:<версия pyproject> + :latest
make release VERSION=0.2.0 # переопределить тег
make release-build # только build/tag, без push
make release-up && make release-seed # запустить опубликованный стек локально
```
## Участие / безопасность / changelog
- [CONTRIBUTING.md](CONTRIBUTING.md) · [CONTRIBUTING.ru.md](CONTRIBUTING.ru.md)
- [Testing](docs/testing.md) · [Тесты](docs/ru/testing.md) — наборы, Make-цели, последние результаты
- [SECURITY.md](SECURITY.md)
- [CHANGELOG.md](CHANGELOG.md)
## Чем это не является
- Не гипервизор: нет выполнения KVM/LXC на железе или nested-хостах.
- Не drop-in замена мультиарендного продакшен-Proxmox.
- Удалённые IdP / LDAP / live Ceph / live ACME эндпоинты симулируются локально;
к реальным внешним системам они не обращаются.
## См. также
- [Web UI](docs/ru/web-ui.md) — интерактивная консоль, каталоги, панель DATA и скриншоты
+52
View File
@@ -0,0 +1,52 @@
**Language / Язык:** [English](SECURITY.md) | [Русский](docs/ru/security.md)
# Security policy
## Supported versions
| Version | Supported |
|---|---|
| `0.1.x` (latest) | Yes — security reports accepted |
| older / untagged | Best-effort only |
## Threat model (read this first)
This repository is a **local / CI laboratory simulator**, not a hardened
multi-tenant public Proxmox deployment.
Default Compose and Helm values intentionally ship convenient lab secrets,
seeded passwords, open Web UI helper routes (`/ui/api/*`), and compatibility
endpoints (`/admin/compatibility*`). Treat network reachability as the trust
boundary.
**Do not** expose host port `8006` (or a public Ingress) to untrusted networks
without replacing every default secret and adding controls you own.
Full lab notes: [docs/security.md](docs/security.md) ·
[docs/ru/security.md](docs/ru/security.md).
## Reporting a vulnerability
Please **do not** open a public GitHub issue for sensitive reports.
Email the maintainer privately (account that owns the GitHub repository /
Docker Hub `inecs` namespace), or use GitHub
[ privately reported vulnerabilities](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability)
if enabled on the repository.
Include:
- Affected version / image tag (`inecs/proxmox-api-simulator:…`)
- Reproduction steps against a **local** lab (not third-party instances)
- Impact assessment (auth bypass, secret leak, RCE, etc.)
You should receive an acknowledgement within a few business days.
## Lab secrets that must be rotated outside toy labs
| Secret | Where |
|---|---|
| `TICKET_SIGNING_KEY` | Compose / Helm |
| PostgreSQL password | Compose / Helm |
| Seeded `root@pam` / API tokens | After `seed` |
| `docker/tls/server.key` | Checked-in self-signed material — never reuse outside local Compose |
+1
View File
@@ -0,0 +1 @@
0.1.1
+66 -2
View File
@@ -17,7 +17,7 @@ from app.api.errors import ApiError, ContractValidationError
from app.api.openapi import contract_openapi_tags
from app.config import Settings
from app.contracts.examples import schema_example
from app.contracts.model import Method, Schema, Snapshot
from app.contracts.model import Method, Parameter, Schema, Snapshot
from app.db.pool import AsyncpgDatabase
from app.security.acl import AclEntry, CapabilityRequirement, authorize, requirement_from_contract
from app.security.auth import parse_api_token, verify_csrf, verify_secret, verify_ticket
@@ -137,6 +137,64 @@ def register_legacy_handler_routes(
return seen
_PATH_PARAM_RE = re.compile(r"\{([^{}]+)\}")
def _unbound_method(path: str, verb: str) -> Method:
"""Minimal contract method so handler-only paths can be mounted."""
parameters = tuple(
Parameter(name=name, definition=Schema(type="string", optional=True))
for name in _PATH_PARAM_RE.findall(path)
)
return Method(
verb=verb.upper(),
name=f"unbound-{verb.lower()}-{path}",
description="Handler-backed route not present in the active contract snapshot",
parameters=parameters,
returns=Schema(),
permissions=None,
checksum="0" * 64,
)
def register_unbound_handler_routes(
app: FastAPI,
handlers: HandlerRegistry,
fallback: FallbackMode = "error",
*,
existing: set[tuple[str, str, str]] | None = None,
) -> set[tuple[str, str, str]]:
"""Mount semantic handlers that no cached contract currently declares.
Keeps DB-backed handlers reachable (no FastAPI 404) when Proxmox schema
omits a path that the simulator still implements.
"""
seen = existing if existing is not None else set()
for path, verb in sorted(handlers.keys()):
for renderer in ("json", "extjs"):
route = f"/api2/{renderer}{path}"
key = (route, verb, renderer)
if key in seen:
continue
method = _unbound_method(path, verb)
seen.add(key)
app.add_api_route(
route,
_endpoint(path, method, renderer, handlers, fallback),
methods=[verb],
name=f"unbound:{renderer}:{verb}:{path}",
tags=cast(list[str | Enum], contract_openapi_tags(path, renderer)),
openapi_extra={
"x-proxmox-method-checksum": method.checksum,
"x-proxmox-implementation": "implemented",
"x-proxmox-unbound-handler": True,
},
)
return seen
def _endpoint(
semantic_path: str,
method: Method,
@@ -302,8 +360,14 @@ async def _parse_inputs(request: Request, method: Method) -> dict[str, Any]:
continue
if name not in supplied:
if definition.optional:
# Proxmox schema ``default`` is often UI documentation text (e.g.
# bwlimit default = "restore limit from datacenter…"), not a value
# to materialize. Only inject defaults that coerce to the type.
if definition.default is not None:
parsed[name] = definition.default
try:
parsed[name] = _coerce(definition.default, definition)
except (TypeError, ValueError):
pass
continue
errors[name] = "property is missing and it is not optional"
continue
+177 -5
View File
@@ -1,7 +1,15 @@
"""Generate example values from Proxmox contract schemas."""
"""Generate example values from Proxmox contract schemas.
Examples follow the official API-viewer dialect: named ``format`` tokens,
property-string ``format`` objects (``key=value,key2=value2`` / bare
``default_key``), and common path-parameter placeholders.
"""
from __future__ import annotations
from collections.abc import Mapping
from typing import Any
from app.contracts.model import Schema
_PATH_PARAM_EXAMPLES: dict[str, object] = {
@@ -22,6 +30,53 @@ _PATH_PARAM_EXAMPLES: dict[str, object] = {
"key": "cpu",
"digest": "00000000",
"name": "example",
"clustername": "example",
}
_FORMAT_EXAMPLES: dict[str, object] = {
"pve-node": "pve01",
"pve-node-list": "pve01,pve02",
"pve-vmid": 100,
"pve-vmid-list": "100,101",
"pve-storage-id": "local",
"pve-storage-server": "192.168.0.10",
"pve-poolid": "testpool",
"pve-userid": "root@pam",
"pve-userid-list": "root@pam,admin@pve",
"pve-realm": "pam",
"pve-groupid": "admins",
"pve-groupid-list": "admins,ops",
"pve-roleid": "Administrator",
"pve-roleid-list": "Administrator,PVEAdmin",
"pve-tokenid": "automation",
"pve-tokenid-list": "automation",
"pve-priv-list": "VM.Allocate,VM.Config.Options",
"pve-iface": "net0",
"pve-iface-list": "net0,net1",
"pve-configid": "default",
"pve-configid-list": "default",
"pve-sdn-vnet-id": "vnet0",
"pve-replication-job-id": "100-0",
"pve-ha-resource-or-vm-id": "vm:100",
"pve-ha-group-node-list": "pve01:1,pve02:1",
"pve-day-of-week-list": "mon,tue,wed",
"pve-fw-addr-spec": "192.168.0.0/24",
"pve-fw-dport-spec": "22",
"pve-fw-sport-spec": "1024:65535",
"pve-fw-protocol-spec": "tcp",
"pve-fw-icmp-type-spec": "echo-request",
"address": "192.168.0.1",
"ip": "192.168.0.1",
"IPorCIDR": "192.168.0.0/24",
"IPorCIDRorAlias": "192.168.0.0/24",
"dns-name": "example.local",
"email-opt": "admin@example.local",
"email-list": "admin@example.local",
"email-or-username-list": "admin@example.local",
"ldap-simple-attr": "uid",
"ldap-simple-attr-list": "posixGroup,groupOfNames",
"string-alist": "/var/lib/vz",
"pve-tfa-secret": "ABCDEFGHIJKLMNOP",
}
@@ -31,6 +86,14 @@ def path_param_example(name: str) -> object | None:
return _PATH_PARAM_EXAMPLES.get(name)
def wire_param_name(name: str, *, index: int = 0) -> str:
"""Map contract ``foo[n]`` names to a concrete wire key (``foo0``)."""
if "[n]" in name:
return name.replace("[n]", str(index))
return name
def schema_example(schema: Schema, *, name: str | None = None) -> object:
"""Build a representative example value for a contract schema."""
@@ -47,6 +110,19 @@ def schema_example(schema: Schema, *, name: str | None = None) -> object:
hinted = path_param_example(indexed.rstrip("0123456789"))
if hinted is not None:
return hinted
base = indexed.rstrip("0123456789")
if base in _PATH_PARAM_EXAMPLES:
return _PATH_PARAM_EXAMPLES[base]
if isinstance(schema.format, Mapping):
return property_string_example(schema.format)
if isinstance(schema.format, str):
formatted = _named_format_example(schema.format, name=name)
if formatted is not None:
return formatted
if schema.format == "email":
return "user@example.com"
if schema.format == "uri":
return "https://example.com"
if schema.type == "array":
if schema.items is not None:
return [schema_example(schema.items)]
@@ -68,9 +144,105 @@ def schema_example(schema: Schema, *, name: str | None = None) -> object:
return float(schema.minimum)
return 1.0
if schema.type == "string" or schema.type is None:
if schema.format == "email":
return "user@example.com"
if schema.format == "uri":
return "https://example.com"
return "example"
return None
def property_string_example(fmt: Mapping[str, Any]) -> str:
"""Render a Proxmox property-string example from a ``format`` object.
Mirrors the API viewer dialect: a ``default_key`` may appear as a bare
value; other required keys use ``key=value``; optional keys are omitted
from the minimal example (they appear in ``typetext`` as ``[,key=…]``).
"""
bare: str | None = None
keyed: list[str] = []
for key, raw in sorted(fmt.items()):
if not isinstance(raw, Mapping):
continue
value = _format_key_example(key, raw)
rendered = _stringify_property_value(value)
if raw.get("default_key"):
bare = rendered
continue
if _is_optional_flag(raw.get("optional")):
continue
keyed.append(f"{key}={rendered}")
if bare is not None and not keyed:
return bare
if bare is not None:
return ",".join([bare, *keyed])
if keyed:
return ",".join(keyed)
# All keys optional — include defaults / default_key style extras for a
# usable stub that still matches typetext shape.
fallback: list[str] = []
for key, raw in sorted(fmt.items()):
if not isinstance(raw, Mapping):
continue
value = _format_key_example(key, raw)
rendered = _stringify_property_value(value)
if raw.get("default_key"):
return rendered
fallback.append(f"{key}={rendered}")
return ",".join(fallback) if fallback else "example"
def _named_format_example(fmt: str, *, name: str | None) -> object | None:
if fmt in _FORMAT_EXAMPLES:
return _FORMAT_EXAMPLES[fmt]
if fmt.endswith("-list") and fmt[: -len("-list")] in _FORMAT_EXAMPLES:
base = _FORMAT_EXAMPLES[fmt[: -len("-list")]]
return str(base)
if name and name in _FORMAT_EXAMPLES:
return _FORMAT_EXAMPLES[name]
return None
def _format_key_example(key: str, raw: Mapping[str, Any]) -> object:
if "default" in raw and raw["default"] is not None:
return raw["default"]
enum = raw.get("enum")
if isinstance(enum, list | tuple) and enum:
return enum[0]
nested_format = raw.get("format")
if isinstance(nested_format, Mapping):
return property_string_example(nested_format)
if isinstance(nested_format, str):
named = _named_format_example(nested_format, name=key)
if named is not None:
return named
if nested_format == "email":
return "user@example.com"
if nested_format == "uri":
return "https://example.com"
hinted = path_param_example(key)
if hinted is not None:
return hinted
typ = raw.get("type")
if typ == "boolean":
return 1
if typ == "integer":
minimum = raw.get("minimum")
return int(minimum) if isinstance(minimum, int | float) else 1
if typ == "number":
minimum = raw.get("minimum")
return float(minimum) if isinstance(minimum, int | float) else 1.0
pattern = raw.get("pattern")
if isinstance(pattern, str) and "second|minute|hour|day" in pattern:
return "1/second"
return "example"
def _stringify_property_value(value: object) -> str:
if isinstance(value, bool):
return "1" if value else "0"
return str(value)
def _is_optional_flag(value: object) -> bool:
if value is True or value == 1 or value == "1":
return True
return False
+3 -1
View File
@@ -14,6 +14,7 @@ from app.api.registry import (
HandlerRegistry,
register_contract_routes,
register_legacy_handler_routes,
register_unbound_handler_routes,
)
from app.compatibility import (
CompatibilityDimension,
@@ -106,7 +107,7 @@ def apply_runtime_contract(
clear_contract_routes(app)
registered = register_contract_routes(app, snapshot, handlers, fallback)
register_legacy_handler_routes(
registered = register_legacy_handler_routes(
app,
handlers,
store_root,
@@ -114,6 +115,7 @@ def apply_runtime_contract(
primary_version=snapshot.source_version,
existing=registered,
)
register_unbound_handler_routes(app, handlers, fallback, existing=registered)
report = build_compatibility_for_snapshot(
snapshot,
handlers,
+85 -14
View File
@@ -125,7 +125,7 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
return [
{
"userid": str(row["name"]),
"enable": 1 if row["enabled"] else 0,
"enable": bool(row["enabled"]),
"realm-type": str(row["realm_kind"]),
}
for row in rows
@@ -168,10 +168,25 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
)
if row is None:
raise ApiError(404, "user does not exist")
groups = await database(request).pool.fetch(
"""SELECT g.group_id FROM identity_group_members m
JOIN identity_groups g ON g.id = m.group_id
JOIN principals p ON p.id = m.principal_id
WHERE p.name=$1 ORDER BY g.group_id""",
userid,
)
tokens = await database(request).pool.fetch(
"""SELECT t.token_id FROM api_tokens t
JOIN principals p ON p.id = t.principal_id
WHERE p.name=$1 ORDER BY t.token_id""",
userid,
)
return {
"userid": str(row["name"]),
"enable": 1 if row["enabled"] else 0,
"enable": bool(row["enabled"]),
"realm-type": str(row["realm_kind"]),
"groups": [str(item["group_id"]) for item in groups],
"tokens": [{"tokenid": str(item["token_id"]), "privsep": 1} for item in tokens],
}
async def user_update(request: Request, inputs: dict[str, Any]) -> None:
@@ -397,10 +412,18 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
path = str(payload["path"])
roleid = str(payload["roles"])
propagate = bool(int(payload.get("propagate", 1)))
delete = bool(int(payload.get("delete", 0)))
users = [item.strip() for item in str(payload.get("users", "")).split(",") if item.strip()]
groups = [
item.strip() for item in str(payload.get("groups", "")).split(",") if item.strip()
]
tokens = [
item.strip() for item in str(payload.get("tokens", "")).split(",") if item.strip()
]
await database(request).pool.execute(
"""INSERT INTO roles(name) VALUES($1) ON CONFLICT DO NOTHING""",
roleid,
)
for userid in users:
principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1",
@@ -408,10 +431,15 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
)
if principal_id is None:
raise ApiError(404, f"user {userid} does not exist")
await database(request).pool.execute(
"""INSERT INTO roles(name) VALUES($1) ON CONFLICT DO NOTHING""",
roleid,
)
if delete:
await database(request).pool.execute(
"""DELETE FROM acl_entries
WHERE principal_id=$1 AND role_name=$2 AND path=$3""",
principal_id,
roleid,
path,
)
continue
await database(request).pool.execute(
"""INSERT INTO acl_entries(principal_id, role_name, path, propagate)
VALUES($1, $2, $3, $4)
@@ -429,10 +457,15 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
)
if group_id is None:
raise ApiError(404, f"group {groupid} does not exist")
await database(request).pool.execute(
"""INSERT INTO roles(name) VALUES($1) ON CONFLICT DO NOTHING""",
roleid,
)
if delete:
await database(request).pool.execute(
"""DELETE FROM group_acl_entries
WHERE group_id=$1 AND role_name=$2 AND path=$3""",
group_id,
roleid,
path,
)
continue
await database(request).pool.execute(
"""INSERT INTO group_acl_entries(group_id, role_name, path, propagate)
VALUES($1, $2, $3, $4)
@@ -443,6 +476,46 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
path,
propagate,
)
for token_ref in tokens:
# Wire form: userid!tokenid
userid, _, tokenid = token_ref.partition("!")
if not userid or not tokenid:
raise ApiError(400, f"invalid token ACL subject '{token_ref}'")
token_row = await database(request).pool.fetchrow(
"""SELECT t.id FROM api_tokens t
JOIN principals p ON p.id = t.principal_id
WHERE p.name=$1 AND t.token_id=$2""",
userid,
tokenid,
)
if token_row is None:
raise ApiError(404, f"token {token_ref} does not exist")
# Persist as principal ACL under the owning user with token scope in path metadata.
# Keep durable by tagging path; GET ACL still lists principal entries.
scoped_path = f"{path}#token:{tokenid}" if path != "/" else f"/#token:{tokenid}"
principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1",
userid,
)
if delete:
await database(request).pool.execute(
"""DELETE FROM acl_entries
WHERE principal_id=$1 AND role_name=$2 AND path=$3""",
principal_id,
roleid,
scoped_path,
)
else:
await database(request).pool.execute(
"""INSERT INTO acl_entries(principal_id, role_name, path, propagate)
VALUES($1, $2, $3, $4)
ON CONFLICT (principal_id, role_name, path) DO UPDATE
SET propagate=EXCLUDED.propagate""",
principal_id,
roleid,
scoped_path,
propagate,
)
async def token_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
userid = str(values(inputs)["userid"])
@@ -562,10 +635,8 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
)
if row is None:
raise ApiError(404, "role does not exist")
return {
"roleid": str(row["name"]),
"privs": ",".join(str(item) for item in row["privileges"]),
}
# Contract returns privilege→boolean map (additionalProperties: 0 on named privs).
return {str(priv): 1 for priv in row["privileges"]}
async def role_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
+60 -51
View File
@@ -2,6 +2,8 @@
from __future__ import annotations
import hashlib
import json
from typing import Any
from fastapi import Request
@@ -10,41 +12,34 @@ from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs, values
_DEFAULT_DIRECTORIES = [
{
"name": "Let's Encrypt V2",
"url": "https://acme-v02.api.letsencrypt.org/directory",
},
{
"name": "Let's Encrypt V2 Staging",
"url": "https://acme-staging-v02.api.letsencrypt.org/directory",
},
]
_CHALLENGE_SCHEMA = [
{
"id": "dns",
"name": "DNS plugin",
"type": "dns",
"fields": [{"name": "api", "type": "string"}],
}
]
def _acme(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault(
"acme",
{"accounts": {}, "plugins": {}, "meta": {}},
)
current = metadata.get("acme")
if not isinstance(current, dict):
current = {"accounts": {}, "plugins": {}, "meta": {}}
current = {}
metadata["acme"] = current
current.setdefault("accounts", {})
current.setdefault("plugins", {})
current.setdefault("meta", {})
if not isinstance(current.get("accounts"), dict):
current["accounts"] = {}
if not isinstance(current.get("plugins"), dict):
current["plugins"] = {}
if not isinstance(current.get("meta"), dict):
current["meta"] = {}
if not isinstance(current.get("directories"), list):
current["directories"] = []
if not isinstance(current.get("challenge_schema"), list):
current["challenge_schema"] = []
return current
def _default_directory(acme: dict[str, Any]) -> str:
directories = acme.get("directories")
if isinstance(directories, list):
for item in directories:
if isinstance(item, dict) and item.get("url"):
return str(item["url"])
return "https://acme-v02.api.letsencrypt.org/directory"
def register_acme_handlers(registry: HandlerRegistry) -> None:
async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs(
@@ -68,24 +63,26 @@ def register_acme_handlers(registry: HandlerRegistry) -> None:
for name, item in sorted(accounts.items())
]
async def account_create(request: Request, inputs: dict[str, Any]) -> None:
async def account_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
name = str(payload.get("name") or "default")
metadata = await cluster_metadata(request)
accounts = _acme(metadata)["accounts"]
acme = _acme(metadata)
accounts = acme["accounts"]
if name in accounts:
raise ApiError(400, f"ACME account '{name}' already exists")
directory = str(payload.get("directory") or _default_directory(acme))
accounts[name] = {
"name": name,
"contact": payload.get("contact"),
"directory": payload.get("directory") or _DEFAULT_DIRECTORIES[0]["url"],
"directory": directory,
"tos_url": payload.get("tos_url"),
"eab-kid": payload.get("eab-kid"),
# eab-hmac-key stored but never returned
"eab-hmac-key": payload.get("eab-hmac-key"),
"location": f"https://acme.example.local/acct/{name}",
}
await save_cluster_metadata(request, metadata)
return name
async def account_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
name = str(values(inputs)["name"])
@@ -94,8 +91,10 @@ def register_acme_handlers(registry: HandlerRegistry) -> None:
if not isinstance(account, dict):
raise ApiError(404, "ACME account does not exist")
return {
"name": name,
"contact": account.get("contact"),
"account": {
"name": name,
"contact": account.get("contact"),
},
"directory": account.get("directory"),
"tos": account.get("tos_url"),
"location": account.get("location"),
@@ -129,7 +128,13 @@ def register_acme_handlers(registry: HandlerRegistry) -> None:
for plugin_id, item in sorted(plugins.items()):
if plugin_type and item.get("type") != plugin_type:
continue
result.append({"plugin": plugin_id, **{k: v for k, v in item.items() if k != "data"}})
entry = {"plugin": plugin_id, **{k: v for k, v in item.items() if k != "data"}}
if "digest" not in entry:
material = {k: v for k, v in entry.items() if k != "digest"}
entry["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(material, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
result.append(entry)
return result
async def plugins_create(request: Request, inputs: dict[str, Any]) -> None:
@@ -181,29 +186,33 @@ def register_acme_handlers(registry: HandlerRegistry) -> None:
del plugins[plugin_id]
await save_cluster_metadata(request, metadata)
async def directories(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(_DEFAULT_DIRECTORIES)
async def directories(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
directories = _acme(metadata).get("directories")
if isinstance(directories, list):
return [dict(item) for item in directories if isinstance(item, dict)]
return []
async def challenge_schema(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(_CHALLENGE_SCHEMA)
async def challenge_schema(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
schema = _acme(metadata).get("challenge_schema")
if isinstance(schema, list):
return [dict(item) for item in schema if isinstance(item, dict)]
return []
async def meta(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
directory = str(values(inputs).get("directory") or _DEFAULT_DIRECTORIES[0]["url"])
metadata = await cluster_metadata(request)
meta_store = _acme(metadata).setdefault("meta", {})
payload = meta_store.setdefault(
directory,
{
"termsOfService": f"{directory.rstrip('/')}/tos",
"caaIdentities": ["letsencrypt.org"],
},
)
await save_cluster_metadata(request, metadata)
acme = _acme(metadata)
directory = str(values(inputs).get("directory") or _default_directory(acme))
meta_raw = acme.get("meta")
meta_store: dict[str, Any] = dict(meta_raw) if isinstance(meta_raw, dict) else {}
payload = meta_store.get(directory)
if not isinstance(payload, dict):
return {}
return dict(payload)
async def tos(request: Request, inputs: dict[str, Any]) -> str:
directory = str(values(inputs).get("directory") or _DEFAULT_DIRECTORIES[0]["url"])
result = await meta(request, {"values": {"directory": directory}, "provided": frozenset()})
result = await meta(request, inputs)
return str(result.get("termsOfService") or "")
registry.register("/cluster/acme", "GET", index)
+200 -145
View File
@@ -1,8 +1,8 @@
"""Cluster backup and vzdump handlers."""
"""Cluster backup jobs and node vzdump handlers."""
from __future__ import annotations
import json
import secrets
from typing import Any
from fastapi import Request
@@ -10,170 +10,224 @@ from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.db.primitives import ConflictError
from app.handlers.common import database, require_node, state, values
from app.handlers.common import (
cluster_metadata,
database,
require_node,
save_cluster_metadata,
state,
subdirs,
values,
)
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
def _backup_jobs(metadata: dict[str, Any]) -> dict[str, dict[str, Any]]:
jobs = metadata.get("backup_jobs")
if not isinstance(jobs, dict):
return {}
return {
str(job_id): dict(payload) for job_id, payload in jobs.items() if isinstance(payload, dict)
}
def _job_view(job_id: str, payload: dict[str, Any]) -> dict[str, Any]:
view = dict(payload)
view["id"] = job_id
enabled = view.get("enabled", 1)
view["enabled"] = bool(int(enabled)) if not isinstance(enabled, bool) else enabled
return view
def register_backup_handlers(registry: HandlerRegistry) -> None:
async def backup_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"""SELECT b.id, b.volume_id, b.size_bytes, b.metadata, b.created_at,
r.external_id AS vmid, n.name AS node, s.storage_id
FROM backups b
LEFT JOIN resources r ON r.id = b.resource_id
LEFT JOIN nodes n ON n.id = r.node_id
JOIN storages s ON s.resource_id = b.storage_resource_id
ORDER BY b.created_at DESC LIMIT 2000"""
async def backup_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
return [_job_view(job_id, payload) for job_id, payload in sorted(jobs.items())]
async def backup_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
payload = jobs.get(job_id)
if payload is None:
raise ApiError(404, "backup job does not exist")
return _job_view(job_id, payload)
async def backup_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
job_id = str(payload.get("id") or f"backup-{secrets.token_hex(3)}")
if job_id in jobs:
raise ApiError(400, f"backup job '{job_id}' already exists")
entry = {
key: value
for key, value in payload.items()
if key not in {"delete", "digest", "node", "target"}
}
entry["id"] = job_id
entry.setdefault("enabled", 1)
entry.setdefault("storage", payload.get("storage") or "local")
jobs[job_id] = entry
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_update(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
payload = values(inputs)
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
current = jobs.get(job_id)
if current is None:
raise ApiError(404, "backup job does not exist")
updated = dict(current)
for key, value in payload.items():
if key in {"id", "delete", "digest"}:
continue
updated[key] = value
updated["id"] = job_id
jobs[job_id] = updated
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_delete(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
if job_id not in jobs:
raise ApiError(404, "backup job does not exist")
del jobs[job_id]
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_info(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("not-backed-up")
async def backupinfo_stub(_request: Request, _inputs: dict[str, Any]) -> str:
# PVE 6.x `/cluster/backupinfo` returns a stub string; 7+ uses `/backup-info` index.
return "Please use the 'not-backed-up' API"
async def backup_not_backed_up(
request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
covered: set[str] = set()
for payload in jobs.values():
if int(payload.get("all") or 0):
covered.add("*")
for item in str(payload.get("vmid") or "").split(","):
item = item.strip()
if item:
covered.add(item)
rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.kind, r.state FROM resources r
WHERE r.kind IN ('qemu', 'lxc')
ORDER BY r.external_id::integer"""
)
result: list[dict[str, Any]] = []
if "*" in covered:
return result
for row in rows:
metadata = state(row["metadata"])
vmid = str(row["external_id"])
if vmid in covered:
continue
guest_state = state(row["state"])
result.append(
{
"id": str(row["id"]),
"volid": str(row["volume_id"]),
"size": int(row["size_bytes"]),
"vmid": int(row["vmid"]) if row["vmid"] is not None else None,
"node": str(row["node"]) if row["node"] is not None else None,
"storage": str(row["storage_id"]),
"starttime": int(row["created_at"].timestamp()),
"mode": metadata.get("mode", "snapshot"),
"type": metadata.get("type", "vzdump"),
"vmid": int(vmid),
"type": "qemu" if row["kind"] == "qemu" else "lxc",
"name": str(guest_state.get("name") or f"{row['kind']}-{vmid}"),
}
)
return result
async def backup_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
backup_id = str(values(inputs)["id"])
row = await database(request).pool.fetchrow(
"""SELECT b.id, b.volume_id, b.size_bytes, b.metadata, b.created_at,
r.external_id AS vmid, n.name AS node, s.storage_id
FROM backups b
LEFT JOIN resources r ON r.id = b.resource_id
LEFT JOIN nodes n ON n.id = r.node_id
JOIN storages s ON s.resource_id = b.storage_resource_id
WHERE b.id::text = $1 OR b.volume_id = $1""",
backup_id,
)
if row is None:
raise ApiError(404, "backup does not exist")
metadata = state(row["metadata"])
return {
"id": str(row["id"]),
"volid": str(row["volume_id"]),
"size": int(row["size_bytes"]),
"vmid": int(row["vmid"]) if row["vmid"] is not None else None,
"node": str(row["node"]) if row["node"] is not None else None,
"storage": str(row["storage_id"]),
"starttime": int(row["created_at"].timestamp()),
"notes": metadata.get("notes-template"),
**metadata,
}
async def backup_included_volumes(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
job = jobs.get(job_id)
if job is None:
raise ApiError(404, "backup job does not exist")
vmids = [item.strip() for item in str(job.get("vmid") or "").split(",") if item.strip()]
guest_select = """SELECT r.external_id, r.kind, r.state,
v.config AS qemu_config, c.config AS lxc_config
FROM resources r
LEFT JOIN virtual_machines v ON v.resource_id = r.id
LEFT JOIN containers c ON c.resource_id = r.id"""
if int(job.get("all") or 0):
rows = await database(request).pool.fetch(
f"""{guest_select}
WHERE r.kind IN ('qemu', 'lxc')
ORDER BY r.external_id::integer"""
)
else:
rows = await database(request).pool.fetch(
f"""{guest_select}
WHERE r.kind IN ('qemu', 'lxc') AND r.external_id = ANY($1::text[])
ORDER BY r.external_id::integer""",
vmids,
)
children: list[dict[str, Any]] = []
for row in rows:
guest_state = state(row["state"])
config_raw = row["qemu_config"] if row["kind"] == "qemu" else row["lxc_config"]
config = state(config_raw) if config_raw is not None else {}
volumes: list[dict[str, Any]] = []
for key, value in sorted(config.items()):
if not (
key.startswith("scsi")
or key.startswith("virtio")
or key.startswith("sata")
or key.startswith("ide")
or key in {"rootfs", "mp0", "mp1", "mp2", "mp3"}
):
continue
volumes.append(
{
"id": key,
"name": str(value).split(",", 1)[0],
"included": True,
"reason": "included by default",
}
)
children.append(
{
"id": int(row["external_id"]),
"name": str(guest_state.get("name") or f"{row['kind']}-{row['external_id']}"),
"type": "qemu" if row["kind"] == "qemu" else "lxc",
"children": volumes,
}
)
return {"children": children}
async def backup_update(request: Request, inputs: dict[str, Any]) -> None:
backup_id = str(values(inputs)["id"])
row = await database(request).pool.fetchrow(
"SELECT id, metadata FROM backups WHERE id::text = $1",
backup_id,
)
if row is None:
raise ApiError(404, "backup does not exist")
metadata = state(row["metadata"])
payload = values(inputs)
if "notes" in payload:
metadata["notes-template"] = payload["notes"]
await database(request).pool.execute(
"UPDATE backups SET metadata=$2::jsonb WHERE id=$1",
row["id"],
json.dumps(metadata, sort_keys=True),
)
async def vzdump_defaults(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
from app.handlers.nodes import load_node_ops
async def backup_delete(request: Request, inputs: dict[str, Any]) -> None:
backup_id = str(values(inputs)["id"])
status = await database(request).pool.execute(
"DELETE FROM backups WHERE id::text = $1",
backup_id,
)
if status != "DELETE 1":
raise ApiError(404, "backup does not exist")
async def backup_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload.get("node") or payload.get("target") or "pve01")
node = str(values(inputs)["node"])
await require_node(request, node)
vmid = payload.get("vmid")
return await _schedule_vzdump(
request,
node=node,
vmids=[str(vmid)] if vmid is not None else None,
payload=payload,
)
async def backup_info(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"""SELECT r.external_id AS vmid, n.name AS node, max(b.created_at) AS last_backup
FROM resources r
JOIN nodes n ON n.id = r.node_id
LEFT JOIN backups b ON b.resource_id = r.id
WHERE r.kind = 'qemu'
GROUP BY r.external_id, n.name
ORDER BY r.external_id::integer
LIMIT 5000"""
)
return [
{
"vmid": int(row["vmid"]),
"node": str(row["node"]),
"lastbackup": int(row["last_backup"].timestamp()) if row["last_backup"] else 0,
"protected": 0,
}
for row in rows
]
async def backup_not_backed_up(_request: Request, _inputs: dict[str, Any]) -> list[int]:
rows = await database(_request).pool.fetch(
"""SELECT r.external_id::integer AS vmid
FROM resources r
LEFT JOIN backups b ON b.resource_id = r.id
WHERE r.kind = 'qemu' AND b.id IS NULL
ORDER BY r.external_id::integer"""
)
return [int(row["vmid"]) for row in rows]
async def backup_included_volumes(request: Request, inputs: dict[str, Any]) -> list[str]:
backup_id = str(values(inputs)["id"])
row = await database(request).pool.fetchrow(
"""SELECT b.volume_id, r.external_id AS vmid
FROM backups b LEFT JOIN resources r ON r.id = b.resource_id
WHERE b.id::text = $1""",
backup_id,
)
if row is None:
raise ApiError(404, "backup does not exist")
vmid = row["vmid"]
return [f"qemu/{vmid}"] if vmid is not None else [str(row["volume_id"])]
async def vzdump_defaults(_request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(_request, str(values(inputs)["node"]))
return {
"all": 0,
"bwlimit": 0,
"compress": "zstd",
"dumpdir": "backup",
"mode": "snapshot",
"remove": 0,
"storage": "nfs-backup",
"mailto": "",
"notes-template": "{{guestname}}",
}
ops = await load_node_ops(request, node)
vzdump = ops.get("vzdump")
defaults = vzdump.get("defaults") if isinstance(vzdump, dict) else None
return dict(defaults) if isinstance(defaults, dict) else {}
async def vzdump_extractconfig(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import load_node_ops
payload = values(inputs)
node = str(payload["node"])
volid = str(payload.get("volume") or payload.get("volid") or "")
if not volid:
raise ApiError(400, "volume parameter required")
return f"# simulated vzdump config extracted from {volid}\name: demo\nmemory: 2048\n"
await require_node(request, node)
ops = await load_node_ops(request, node)
vzdump = ops.get("vzdump")
configs = vzdump.get("extractconfig") if isinstance(vzdump, dict) else None
if not isinstance(configs, dict):
return ""
return str(configs.get(volid) or "")
async def vzdump_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
@@ -188,6 +242,7 @@ def register_backup_handlers(registry: HandlerRegistry) -> None:
registry.register("/cluster/backup", "GET", backup_list)
registry.register("/cluster/backup", "POST", backup_create)
registry.register("/cluster/backup-info", "GET", backup_info)
registry.register("/cluster/backupinfo", "GET", backupinfo_stub)
registry.register("/cluster/backup-info/not-backed-up", "GET", backup_not_backed_up)
registry.register("/cluster/backup/{id}", "GET", backup_get)
registry.register("/cluster/backup/{id}", "PUT", backup_update)
@@ -224,11 +279,11 @@ async def _schedule_vzdump(
payload={
"node": node,
"vmids": vmids,
"storage": str(payload.get("storage") or "nfs-backup"),
"storage": str(payload.get("storage") or "local"),
"mode": str(payload.get("mode") or "snapshot"),
"compress": str(payload.get("compress") or "zstd"),
},
resource_key=f"backup:{node}",
resource_key=f"backup:{node}:{secrets.token_hex(4)}",
idempotency_key=request.headers.get("Idempotency-Key"),
)
except ConflictError as error:
+182 -111
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
import json
import secrets
from typing import Any, cast
from typing import Any
from fastapi import Request
@@ -21,37 +21,6 @@ from app.handlers.common import (
)
from app.simulation.seed import CLUSTER_ID
DEFAULT_CLUSTER_CEPH = {
"initialized": True,
"config": {
"network": "10.10.10.0/24",
"cluster-network": "10.10.10.0/24",
"size": 3,
"min_size": 2,
"pg_bits": 7,
},
"cfg_db": [
{"section": "global", "name": "auth_client_required", "value": "cephx"},
{"section": "global", "name": "fsid", "value": "pve-simulator-fsid"},
],
"cfg_raw": "[global]\nfsid = pve-simulator-fsid\nauth_client_required = cephx\n",
"cfg_values": {},
"pools": {
"rbd": {
"pool": "rbd",
"size": 3,
"min_size": 2,
"pg_num": 128,
"application": "rbd",
"crush_rule": "replicated_rule",
}
},
"fs": {},
"rules": [{"name": "replicated_rule", "id": 0}],
"crush": "device 0 osd.0 class hdd\n",
"running": True,
}
async def _load_cluster_ceph(request: Request) -> dict[str, Any]:
row = await database(request).pool.fetchrow(
@@ -60,14 +29,9 @@ async def _load_cluster_ceph(request: Request) -> dict[str, Any]:
)
metadata = state(row["metadata"]) if row is not None else {}
ceph = metadata.get("ceph")
if not isinstance(ceph, dict) or not ceph:
return dict(DEFAULT_CLUSTER_CEPH)
merged = dict(DEFAULT_CLUSTER_CEPH)
merged.update(ceph)
for key in ("config", "pools", "fs", "cfg_values"):
if not isinstance(merged.get(key), dict):
merged[key] = dict(cast(dict[str, Any], DEFAULT_CLUSTER_CEPH[key]))
return merged
if not isinstance(ceph, dict):
return {}
return dict(ceph)
async def _save_cluster_ceph(request: Request, ceph: dict[str, Any]) -> None:
@@ -82,34 +46,27 @@ async def _save_cluster_ceph(request: Request, ceph: dict[str, Any]) -> None:
async def _load_node_ceph(request: Request, node: str) -> dict[str, Any]:
metadata = await node_metadata(request, node)
ops = metadata.setdefault("ops", {})
ceph = ops.setdefault(
"ceph",
{
"mds": {},
"mgr": {},
"mon": {f"{node}": {"name": node, "addr": f"{node}.local:6789", "rank": 0}},
"log": [{"t": 1_700_000_000, "n": 0, "line": "ceph simulator ready"}],
},
)
ops = metadata.get("ops")
if not isinstance(ops, dict):
return {"mds": {}, "mgr": {}, "mon": {}, "log": []}
ceph = ops.get("ceph")
if not isinstance(ceph, dict):
ceph = {
"mds": {},
"mgr": {},
"mon": {},
"log": [],
}
ops["ceph"] = ceph
ceph.setdefault("mds", {})
ceph.setdefault("mgr", {})
ceph.setdefault("mon", {})
ceph.setdefault("log", [])
return ceph
return {"mds": {}, "mgr": {}, "mon": {}, "log": []}
return {
"mds": ceph.get("mds") if isinstance(ceph.get("mds"), dict) else {},
"mgr": ceph.get("mgr") if isinstance(ceph.get("mgr"), dict) else {},
"mon": ceph.get("mon") if isinstance(ceph.get("mon"), dict) else {},
"log": list(ceph.get("log") or []) if isinstance(ceph.get("log"), list) else [],
**{key: value for key, value in ceph.items() if key not in {"mds", "mgr", "mon", "log"}},
}
async def _save_node_ceph(request: Request, node: str, ceph: dict[str, Any]) -> None:
metadata = await node_metadata(request, node)
ops = metadata.setdefault("ops", {})
ops = metadata.get("ops")
if not isinstance(ops, dict):
ops = {}
metadata["ops"] = ops
ops["ceph"] = ceph
await save_node_metadata(request, node, metadata)
@@ -175,10 +132,12 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
await require_node(request, str(payload["node"]))
ceph = await _load_cluster_ceph(request)
keys = [item.strip() for item in str(payload.get("config-keys") or "").split(",") if item]
stored = ceph.setdefault("cfg_values", {})
result = {key: stored.get(key, "") for key in keys} if keys else dict(stored)
await _save_cluster_ceph(request, ceph)
return result
stored = ceph.get("cfg_values")
if not isinstance(stored, dict):
stored = {}
if keys:
return {key: stored.get(key, "") for key in keys}
return dict(stored)
async def crush(request: Request, inputs: dict[str, Any]) -> str:
await require_node(request, str(values(inputs)["node"]))
@@ -202,12 +161,13 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
async def cmd_safety(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
return {
"safe": 1,
"action": payload.get("action"),
"service": payload.get("service"),
"id": payload.get("id"),
}
ceph = await _load_cluster_ceph(request)
safety = ceph.get("cmd_safety")
result = dict(safety) if isinstance(safety, dict) else {}
for key in ("action", "service", "id"):
if key in payload:
result[key] = payload[key]
return result
async def init(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
@@ -282,11 +242,45 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
await _save_cluster_ceph(request, ceph)
return _upid(node, "cephrestart")
def _wire_ceph_pool(name: str, item: dict[str, Any], *, index: int) -> dict[str, Any]:
pool_id = item.get("pool_id", item.get("pool"))
try:
pool_num = int(pool_id) if pool_id is not None else index
except (TypeError, ValueError):
pool_num = index
crush_raw = item.get("crush_rule", 0)
try:
crush_id = int(crush_raw)
crush_name = str(item.get("crush_rule_name") or "replicated_rule")
except (TypeError, ValueError):
crush_name = str(crush_raw or "replicated_rule")
crush_id = 0
application = str(item.get("application") or "rbd")
return {
"pool": pool_num,
"pool_name": str(item.get("pool_name") or name),
"size": int(item.get("size") or 3),
"min_size": int(item.get("min_size") or 2),
"pg_num": int(item.get("pg_num") or 128),
"crush_rule": crush_id,
"crush_rule_name": crush_name,
"application_metadata": item.get("application_metadata")
if isinstance(item.get("application_metadata"), dict)
else {application: {}},
"bytes_used": int(item.get("bytes_used") or 0),
"percent_used": float(item.get("percent_used") or 0.0),
"healthy": bool(item.get("healthy", True)),
}
async def pool_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
ceph = await _load_cluster_ceph(request)
pools = ceph.get("pools") or {}
return [dict(item) for _, item in sorted(pools.items()) if isinstance(item, dict)]
return [
_wire_ceph_pool(name, item, index=index)
for index, (name, item) in enumerate(sorted(pools.items()))
if isinstance(item, dict)
]
async def pool_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
@@ -318,7 +312,10 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
pool = (ceph.get("pools") or {}).get(name)
if not isinstance(pool, dict):
raise ApiError(404, "pool does not exist")
return [dict(pool)]
# Reuse list wire shape for detail GETs.
names = sorted((ceph.get("pools") or {}).keys())
index = names.index(name) if name in names else 0
return [_wire_ceph_pool(name, pool, index=index)]
async def pool_update(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
@@ -368,13 +365,7 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
pool = (ceph.get("pools") or {}).get(name)
if not isinstance(pool, dict):
raise ApiError(404, "pool does not exist")
return {
**pool,
"pg_num": pool.get("pg_num", 128),
"bytes_used": 0,
"percent_used": 0.0,
"healthy": True,
}
return dict(pool)
async def fs_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
@@ -528,30 +519,63 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
await _save_node_ceph(request, node, ceph)
return _upid(node, "cephdestroymon")
async def osd_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
async def osd_list(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
await require_node(request, node)
rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.state
"""SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='ceph-osd'
ORDER BY r.external_id""",
node,
WHERE r.kind='ceph-osd'
ORDER BY n.name, r.external_id"""
)
result: list[dict[str, Any]] = []
hosts: dict[str, dict[str, Any]] = {}
host_id = -2
for row in rows:
host_name = str(row["node"])
host = hosts.get(host_name)
if host is None:
host = {
"id": host_id,
"name": host_name,
"type": "host",
"type_id": 1,
"children": [],
}
hosts[host_name] = host
host_id -= 1
payload = state(row["state"])
osd_id = payload.get("osd_id", row["external_id"])
result.append(
osd_raw = payload.get("osd_id", row["external_id"])
try:
osd_id = int(osd_raw)
except (TypeError, ValueError):
digits = "".join(ch for ch in str(osd_raw) if ch.isdigit())
osd_id = int(digits) if digits else 0
host["children"].append(
{
"osd": int(osd_id) if str(osd_id).isdigit() else osd_id,
"id": osd_id,
"name": f"osd.{osd_id}",
"type": "osd",
"type_id": 0,
"status": payload.get("status", "up"),
"in": 1 if payload.get("in", True) else 0,
"weight": payload.get("weight", 1.0),
"weight": float(payload.get("weight", 1.0)),
"device_class": payload.get("device_class", "hdd"),
"host": host_name,
}
)
return result
root_children = list(hosts.values())
# Prefer the requested node first in the tree for local dumps.
root_children.sort(key=lambda item: (0 if item["name"] == node else 1, item["name"]))
return {
"flags": "",
"root": {
"id": -1,
"name": "default",
"type": "root",
"type_id": 10,
"children": root_children,
},
}
async def osd_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
@@ -562,9 +586,8 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
MAX(NULLIF(regexp_replace(external_id, '\\D', '', 'g'), '')::int),
-1
) + 1
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='ceph-osd'""",
node,
FROM resources
WHERE kind='ceph-osd'""",
)
osd_id = int(next_id or 0)
external_id = f"osd.{osd_id}"
@@ -583,8 +606,15 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
"used_bytes": 0,
}
await database(request).pool.execute(
"""INSERT INTO resources(id, node_id, kind, external_id, state)
VALUES(gen_random_uuid(), $1, 'ceph-osd', $2, $3::jsonb)""",
"""INSERT INTO resources(id, node_id, cluster_id, kind, external_id, state)
VALUES(
gen_random_uuid(),
$1,
(SELECT cluster_id FROM nodes WHERE id=$1),
'ceph-osd',
$2,
$3::jsonb
)""",
node_id,
external_id,
json.dumps(osd_state, sort_keys=True),
@@ -682,13 +712,16 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node)
row = await _osd_row(request, node, osdid)
current = state(row["state"])
metadata = current.get("metadata")
if isinstance(metadata, dict):
return dict(metadata)
return {
"osd": {
"id": int(osdid) if osdid.isdigit() else osdid,
"uuid": current.get("uuid") or f"osd-uuid-{osdid}",
"device_class": current.get("device_class", "hdd"),
"uuid": current.get("uuid", ""),
"device_class": current.get("device_class", ""),
},
"devices": [{"dev": current.get("dev") or f"/dev/sd{osdid}"}],
"devices": [{"dev": current.get("dev", "")}] if current.get("dev") else [],
}
async def cluster_ceph_status(_request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
@@ -698,20 +731,58 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
)
total = int(row["capacity_bytes"] or 0) if row is not None else 0
used = int(row["used_bytes"] or 0) if row is not None else 0
osd_count = await database(_request).pool.fetchval(
"SELECT count(*)::int FROM resources WHERE kind='ceph-osd'"
osd_rows = await database(_request).pool.fetch(
"SELECT state FROM resources WHERE kind='ceph-osd'"
)
num_osds = len(osd_rows)
num_up = 0
num_in = 0
for osd_row in osd_rows:
osd_state = state(osd_row["state"])
if str(osd_state.get("status") or "") == "up":
num_up += 1
if osd_state.get("in") in (True, 1, "1"):
num_in += 1
ceph = await _load_cluster_ceph(_request)
version = ceph.get("version")
version_str = (
str(version.get("str"))
if isinstance(version, dict) and version.get("str") is not None
else str(version or "")
)
health = ceph.get("health")
if not isinstance(health, dict):
health = {"status": "HEALTH_OK" if ceph.get("running") else "HEALTH_WARN"}
node_count = int(await database(_request).pool.fetchval("SELECT COUNT(*) FROM nodes") or 0)
return {
"version": "17.2.7",
"health": {"status": "HEALTH_OK" if ceph.get("running", True) else "HEALTH_WARN"},
"osdmap": {
"num_osds": osd_count,
"num_up_osds": osd_count - 1,
"num_in_osds": osd_count - 1,
"version": {
"version": version_str,
"release": "reef",
"epoch": 0,
},
"pgmap": {"bytes_used": used, "bytes_total": total},
"fsmap": {"filesystems": list((ceph.get("fs") or {}).keys())},
"health": health,
"osdmap": {
"epoch": 1,
"num_osds": num_osds,
"num_up_osds": num_up,
"num_in_osds": num_in,
"num_remapped_pgs": 0,
},
"pgmap": {
"bytes_used": used,
"bytes_total": total,
"bytes_avail": max(total - used, 0),
"data_bytes": used,
"num_pgs": max(num_osds * 32, 32),
},
"fsmap": {
"epoch": 1,
"by_rank": [],
"filesystems": list((ceph.get("fs") or {}).keys()),
},
"monmap": {"num_mons": max(node_count, 1)},
"mgrmap": {"available": True, "num_standbys": 0},
"servicemap": {"epoch": 1, "services": {}},
}
base = "/nodes/{node}/ceph"
+74 -20
View File
@@ -24,7 +24,43 @@ def _replication_jobs(metadata: dict[str, Any]) -> list[dict[str, Any]]:
jobs = metadata.get("replication", [])
if not isinstance(jobs, list):
return []
return [dict(item) for item in jobs if isinstance(item, dict)]
result: list[dict[str, Any]] = []
for item in jobs:
if not isinstance(item, dict):
continue
guest_raw = item.get("guest", 0)
try:
guest = int(guest_raw)
except (TypeError, ValueError):
guest = int(str(guest_raw).split(":")[-1] or 0)
jobnum_raw = item.get("jobnum")
if jobnum_raw is None:
id_text = str(item.get("id") or "0")
tail = id_text.rsplit("-", 1)[-1]
jobnum = int(tail) if tail.isdigit() else 0
else:
jobnum = int(jobnum_raw)
disable = item.get("disable")
if disable is None:
disable = 0 if int(item.get("enabled", 1) or 1) else 1
entry = {
"id": str(item.get("id") or f"{guest}-{jobnum}"),
"guest": guest,
"jobnum": jobnum,
"target": str(item.get("target") or ""),
"type": str(item.get("type") or "local"),
"disable": bool(int(disable)),
}
if item.get("schedule") is not None:
entry["schedule"] = str(item["schedule"])
if item.get("rate") is not None:
entry["rate"] = int(item["rate"])
if item.get("comment") is not None:
entry["comment"] = str(item["comment"])
if item.get("source") is not None:
entry["source"] = str(item["source"])
result.append(entry)
return result
def register_cluster_handlers(registry: HandlerRegistry) -> None:
@@ -46,24 +82,48 @@ def register_cluster_handlers(registry: HandlerRegistry) -> None:
"tasks",
)
async def cluster_status(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
async def cluster_status(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
from app.handlers.nodes import load_node_ops
rows = await database(request).pool.fetch(
"""SELECT id, name, status FROM nodes ORDER BY name"""
)
result: list[dict[str, Any]] = []
metadata = await cluster_metadata(request)
quorate = bool(int(metadata.get("quorate", 1) or 1))
cluster_name = metadata.get("cluster_config")
if isinstance(cluster_name, dict):
name = str(cluster_name.get("clustername") or "pve-simulator")
version = int(cluster_name.get("version") or 2)
else:
name = "pve-simulator"
version = 2
result: list[dict[str, Any]] = [
{
"type": "cluster",
"id": "cluster",
"name": name,
"nodes": len(rows),
"quorate": quorate,
"version": version,
}
]
for index, row in enumerate(rows):
online = str(row["status"]) == "online"
ops = await load_node_ops(request, str(row["name"]))
cluster_node = ops.get("cluster_status")
entry = dict(cluster_node) if isinstance(cluster_node, dict) else {}
local_raw = entry.get("local", index == 0)
node_name = str(row["name"])
result.append(
{
"id": str(row["id"]),
"name": str(row["name"]),
"nodeid": index,
"online": 1 if online else 0,
"local": 1 if index == 0 else 0,
"ip": f"10.32.{index // 254 + 1}.{index % 254 + 10}",
"level": "c",
"type": "node",
"quorate": 1,
"id": f"node/{node_name}",
"name": node_name,
"nodeid": int(entry.get("nodeid", index + 1)),
"online": online,
"local": bool(int(local_raw)) if not isinstance(local_raw, bool) else local_raw,
"ip": str(entry.get("ip") or ops.get("ip") or ""),
"level": str(entry.get("level", "")),
}
)
return result
@@ -95,14 +155,8 @@ def register_cluster_handlers(registry: HandlerRegistry) -> None:
metadata = state(row["metadata"]) if row is not None else {}
options = metadata.get("options", {})
if not isinstance(options, dict):
options = {}
return {
"keyboard": options.get("keyboard", "en-us"),
"email_from": options.get("email_from", "root@localhost"),
"http_proxy": options.get("http_proxy", ""),
"description": options.get("description", "Proxmox API emulator cluster"),
**options,
}
return {}
return dict(options)
async def cluster_options_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
current = await cluster_options_get(request, inputs)
+283 -78
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import hashlib
import secrets
from typing import Any
@@ -12,50 +13,145 @@ from app.api.registry import HandlerRegistry
from app.handlers.common import (
cluster_metadata,
database,
require_value,
save_cluster_metadata,
subdirs,
values,
)
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
def _config(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault(
"cluster_config",
{
"clustername": "pve-simulator",
"votes": 1,
"links": {},
"join_info": {},
"totem": {"version": 2, "secauth": "on", "cluster_name": "pve-simulator"},
"qdevice": {"status": "disabled"},
"apiversion": 1,
},
)
if not isinstance(current, dict):
current = {
"clustername": "pve-simulator",
"votes": 1,
"links": {},
"join_info": {},
"totem": {"version": 2, "secauth": "on", "cluster_name": "pve-simulator"},
"qdevice": {"status": "disabled"},
"apiversion": 1,
}
metadata["cluster_config"] = current
return current
current = metadata.get("cluster_config")
return current if isinstance(current, dict) else {}
def _principal(request: Request) -> str:
return str(getattr(request.state, "principal", None) or "root@pam")
def _fingerprint_sha256() -> str:
return ":".join(f"{byte:02X}" for byte in secrets.token_bytes(32))
def _stable_digest(material: str) -> str:
return hashlib.sha256(material.encode("utf-8")).hexdigest()
def _node_addr(node: str, index: int, entry: dict[str, Any]) -> str:
for key in ("pve_addr", "new_node_ip", "ring0_addr"):
value = entry.get(key)
if isinstance(value, str) and value and not value.endswith(".local"):
# Prefer literal IPs; hostnames are still accepted as ring addresses.
if key != "ring0_addr" or all(part.isdigit() for part in value.split(".")):
return value
return f"10.0.0.{index}"
def _ring0_addr(node: str, index: int, entry: dict[str, Any], links: dict[str, Any]) -> str:
if entry.get("ring0_addr"):
return str(entry["ring0_addr"])
link0 = links.get("link0")
if isinstance(link0, str) and link0:
# Property string may be bare IP or address=IP[,priority=N].
if link0.startswith("address="):
return link0.removeprefix("address=").split(",", 1)[0]
return link0.split(",", 1)[0]
if entry.get("new_node_ip"):
return str(entry["new_node_ip"])
return f"10.0.0.{index}"
def _render_corosync_conf(config: dict[str, Any], node_names: list[str]) -> str:
clustername = str(config.get("clustername") or "proxmox")
totem = dict(config.get("totem") or {})
added_raw = config.get("added_nodes")
added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
links = dict(config.get("links") or {})
lines = [
"totem {",
" version: 2",
f" cluster_name: {totem.get('cluster_name', clustername)}",
f" secauth: {totem.get('secauth', 'on')}",
"}",
"nodelist {",
]
for index, name in enumerate(node_names, start=1):
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
nodeid = int(entry.get("nodeid") or index)
votes = int(entry.get("quorum_votes") or entry.get("votes") or config.get("votes") or 1)
ring0 = _ring0_addr(name, index, entry, links)
lines.extend(
[
" node {",
f" name: {name}",
f" nodeid: {nodeid}",
f" quorum_votes: {votes}",
f" ring0_addr: {ring0}",
" }",
]
)
lines.append("}")
lines.extend(["quorum {", " provider: corosync_votequorum", "}"])
return "\n".join(lines) + "\n"
def _ensure_corosync_materials(
config: dict[str, Any], node_names: list[str]
) -> tuple[str, str, str]:
authkey = config.get("corosync_authkey")
if not isinstance(authkey, str) or not authkey:
authkey = secrets.token_hex(32)
config["corosync_authkey"] = authkey
conf = _render_corosync_conf(config, node_names)
config["corosync_conf"] = conf
digest = _stable_digest(conf)
config["config_digest"] = digest
return authkey, conf, digest
async def _cluster_task(
request: Request, *, task_type: str, worker: str, task_id: str = "0"
) -> str:
from app.db.primitives import ConflictError
pool = database(request).pool
node = await pool.fetchval("SELECT name FROM nodes ORDER BY name LIMIT 1") or "localhost"
upid = str(Upid.allocate(str(node), worker, task_id, _principal(request)))
try:
task = await TaskRepository(pool).create(
upid=upid,
task_type=task_type,
payload={"cluster": True, "worker": worker},
resource_key=f"cluster:{task_type}:{secrets.token_hex(4)}",
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("apiversion", "join", "nodes", "qdevice", "totem")
# PVE uses {name}; contract child link is href="{name}".
return [{"name": name} for name in ("nodes", "totem", "join", "qdevice", "apiversion")]
async def create(request: Request, inputs: dict[str, Any]) -> None:
async def create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
clustername = str(require_value(payload, "clustername"))
metadata = await cluster_metadata(request)
config = _config(metadata)
if payload.get("clustername"):
config["clustername"] = str(payload["clustername"])
config.setdefault("totem", {})["cluster_name"] = str(payload["clustername"])
config = dict(_config(metadata))
# Match PVE: refuse when corosync materials already exist (seeded or prior create).
if config.get("corosync_conf") or config.get("config_digest"):
raise ApiError(400, "cluster config already exists")
config["clustername"] = clustername
totem = dict(config.get("totem") or {})
totem["version"] = int(totem.get("version") or 2)
totem["secauth"] = totem.get("secauth") or "on"
totem["cluster_name"] = clustername
# PVE defaults token-coefficient to 125 when omitted (majors that expose it).
totem["token_coefficient"] = payload.get("token-coefficient", 125)
config["totem"] = totem
if "votes" in payload:
config["votes"] = payload["votes"]
if "nodeid" in payload:
@@ -64,12 +160,43 @@ def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
if links:
config["links"] = links
config["token"] = secrets.token_hex(16)
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows]
if not node_names:
node_names = ["localhost"]
creator = node_names[0]
added = dict(config.get("added_nodes") or {})
creator_entry = dict(added.get(creator) or {})
creator_entry.update(
{
"node": creator,
"nodeid": payload.get("nodeid") or creator_entry.get("nodeid") or 1,
"quorum_votes": payload.get("votes") or creator_entry.get("quorum_votes") or 1,
"ring0_addr": _ring0_addr(
creator, 1, creator_entry, dict(config.get("links") or {})
),
"pve_addr": creator_entry.get("pve_addr") or _node_addr(creator, 1, creator_entry),
"pve_fp": creator_entry.get("pve_fp") or _fingerprint_sha256(),
}
)
added[creator] = creator_entry
config["added_nodes"] = added
_ensure_corosync_materials(config, node_names)
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata)
await database(request).pool.execute(
"""UPDATE clusters
SET name=$1, updated_at=now()
WHERE id=(SELECT id FROM clusters LIMIT 1)""",
str(config["clustername"]),
clustername,
)
return await _cluster_task(
request,
task_type="cluster-create",
worker="clustercreate",
task_id=clustername,
)
async def apiversion(_request: Request, _inputs: dict[str, Any]) -> int:
@@ -79,84 +206,143 @@ def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
async def join_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
config = _config(metadata)
node = values(inputs).get("node")
preferred_raw = values(inputs).get("node")
# Contract documents the omitted-node default as this phrase; PVE uses local nodename.
if preferred_raw in {None, "", "current connected node"}:
preferred = None
else:
preferred = str(preferred_raw)
rows = await database(request).pool.fetch("SELECT name, status FROM nodes ORDER BY name")
nodelist = [
{"name": str(row["name"]), "online": 1 if row["status"] == "online" else 0}
for row in rows
]
added_raw = config.get("added_nodes")
added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
links = dict(config.get("links") or {})
nodelist: list[dict[str, Any]] = []
for index, row in enumerate(rows, start=1):
name = str(row["name"])
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
if "pve_fp" not in entry:
entry["pve_fp"] = _fingerprint_sha256()
added[name] = entry
if "pve_addr" not in entry:
entry["pve_addr"] = _node_addr(name, index, entry)
added[name] = entry
nodelist.append(
{
"name": name,
"nodeid": int(entry.get("nodeid") or index),
"quorum_votes": int(
entry.get("quorum_votes") or entry.get("votes") or config.get("votes") or 1
),
"ring0_addr": _ring0_addr(name, index, entry, links),
"pve_addr": str(entry["pve_addr"]),
"pve_fp": str(entry["pve_fp"]),
}
)
if added != dict(added_raw or {}):
updated = dict(config)
updated["added_nodes"] = added
if not updated.get("config_digest"):
_ensure_corosync_materials(updated, [item["name"] for item in nodelist])
metadata["cluster_config"] = updated
await save_cluster_metadata(request, metadata)
config = updated
preferred_node = preferred or (nodelist[0]["name"] if nodelist else None)
digest = config.get("config_digest")
if not isinstance(digest, str) or not digest:
updated = dict(config)
_, _, digest = _ensure_corosync_materials(updated, [item["name"] for item in nodelist])
metadata["cluster_config"] = updated
await save_cluster_metadata(request, metadata)
config = updated
return {
"clustername": config.get("clustername"),
"config_digest": secrets.token_hex(8),
"nodelist": nodelist,
"preferred_node": node or (nodelist[0]["name"] if nodelist else None),
"totem": config.get("totem", {}),
"links": config.get("links", {}),
"preferred_node": preferred_node,
"totem": dict(config.get("totem") or {}),
"config_digest": digest,
}
async def join_post(request: Request, inputs: dict[str, Any]) -> None:
async def join_post(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
hostname = str(require_value(payload, "hostname"))
require_value(payload, "fingerprint")
require_value(payload, "password")
metadata = await cluster_metadata(request)
config = _config(metadata)
hostname = str(payload.get("hostname") or payload.get("node") or "")
if not hostname:
raise ApiError(400, "parameter verification failed - 'hostname' missing")
joins = config.setdefault("join_info", {})
config = dict(_config(metadata))
joins = dict(config.get("join_info") or {})
joins[hostname] = {
"hostname": hostname,
"fingerprint": payload.get("fingerprint"),
"nodeid": payload.get("nodeid"),
"votes": payload.get("votes", 1),
"force": payload.get("force"),
"password_set": True,
}
# password accepted but not stored in clear form
if payload.get("password"):
joins[hostname]["password_set"] = True
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)",
hostname,
)
if not exists:
await database(request).pool.execute(
"""INSERT INTO nodes(id, name, status, metadata)
VALUES(gen_random_uuid(), $1, 'online', '{}'::jsonb)""",
hostname,
)
links = {key: value for key, value in payload.items() if key.startswith("link")}
if links:
joins[hostname]["links"] = links
config["join_info"] = joins
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows]
_ensure_corosync_materials(config, node_names or ["localhost"])
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata)
return await _cluster_task(
request,
task_type="cluster-join",
worker="clusterjoin",
)
async def nodes_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(request).pool.fetch("SELECT name, status FROM nodes ORDER BY name")
metadata = await cluster_metadata(request)
config = _config(metadata)
added_raw = config.get("added_nodes")
added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
links = dict(config.get("links") or {})
result = []
for index, row in enumerate(rows, start=1):
name = str(row["name"])
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
result.append(
{
"node": str(row["name"]),
"nodeid": index,
"ring0_addr": f"{row['name']}.local",
"quorum_votes": config.get("votes", 1),
"node": name,
"nodeid": entry.get("nodeid", index),
"ring0_addr": _ring0_addr(name, index, entry, links),
"quorum_votes": entry.get("quorum_votes", config.get("votes", 0)),
}
)
return result
async def nodes_add(request: Request, inputs: dict[str, Any]) -> None:
async def nodes_add(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
node = str(payload["node"])
node = str(require_value(payload, "node"))
metadata = await cluster_metadata(request)
config = _config(metadata)
added = config.setdefault("added_nodes", {})
added[node] = {
config = dict(_config(metadata))
added = dict(config.get("added_nodes") or {})
if node in added and not payload.get("force"):
raise ApiError(400, f"can't add existing node '{node}'")
index = len(added) + 1
links = {key: value for key, value in payload.items() if key.startswith("link")}
entry = {
"node": node,
"nodeid": payload.get("nodeid"),
"nodeid": payload.get("nodeid") or index,
"new_node_ip": payload.get("new_node_ip"),
"votes": payload.get("votes", 1),
"apiversion": payload.get("apiversion"),
"force": payload.get("force"),
"ring0_addr": payload.get("ring0_addr")
or payload.get("new_node_ip")
or _ring0_addr(node, index, {}, links or dict(config.get("links") or {})),
"quorum_votes": payload.get("quorum_votes", payload.get("votes", 1)),
"pve_addr": payload.get("new_node_ip") or _node_addr(node, index, {}),
"pve_fp": _fingerprint_sha256(),
}
links = {key: value for key, value in payload.items() if key.startswith("link")}
if links:
added[node]["links"] = links
entry["links"] = links
added[node] = entry
config["added_nodes"] = added
exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)",
node,
@@ -167,16 +353,33 @@ def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
VALUES(gen_random_uuid(), $1, 'online', '{}'::jsonb)""",
node,
)
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows]
if node not in node_names:
node_names.append(node)
authkey, conf, _digest = _ensure_corosync_materials(config, node_names)
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata)
return {
"corosync_authkey": authkey,
"corosync_conf": conf,
"warnings": [],
}
async def nodes_delete(request: Request, inputs: dict[str, Any]) -> None:
node = str(values(inputs)["node"])
metadata = await cluster_metadata(request)
config = _config(metadata)
added = config.setdefault("added_nodes", {})
config = dict(_config(metadata))
added = dict(config.get("added_nodes") or {})
added.pop(node, None)
joins = config.setdefault("join_info", {})
joins = dict(config.get("join_info") or {})
joins.pop(node, None)
config["added_nodes"] = added
config["join_info"] = joins
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows if str(row["name"]) != node]
_ensure_corosync_materials(config, node_names or ["localhost"])
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata)
# Keep node row; mark offline to avoid cascading guest deletes.
await database(request).pool.execute(
@@ -186,11 +389,13 @@ def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
async def qdevice(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
return dict(_config(metadata).get("qdevice") or {"status": "disabled"})
qdevice = _config(metadata).get("qdevice")
return dict(qdevice) if isinstance(qdevice, dict) else {}
async def totem(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
return dict(_config(metadata).get("totem") or {})
totem = _config(metadata).get("totem")
return dict(totem) if isinstance(totem, dict) else {}
registry.register("/cluster/config", "GET", index)
registry.register("/cluster/config", "POST", create)
+198 -111
View File
@@ -2,7 +2,6 @@
from __future__ import annotations
import copy
import secrets
import time
from typing import Any
@@ -16,62 +15,28 @@ from app.handlers.common import (
database,
require_node,
save_cluster_metadata,
state,
subdirs,
values,
)
from app.simulation.ceph_defaults import CEPH_FLAG_DESCRIPTIONS, default_ceph_flags
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
DEFAULT_CEPH_FLAGS: dict[str, int] = {
"nobackfill": 0,
"nodeep-scrub": 0,
"nodown": 0,
"noin": 0,
"noout": 0,
"norebalance": 0,
"norecover": 0,
"noscrub": 0,
"notieragent": 0,
"pause": 0,
}
DEFAULT_CPU_FLAGS: list[dict[str, Any]] = [
{"name": "aes", "introduces": "Westmere"},
{"name": "avx", "introduces": "SandyBridge"},
{"name": "avx2", "introduces": "Haswell"},
]
def _jobs(metadata: dict[str, Any]) -> dict[str, Any]:
jobs = metadata.setdefault("jobs", {})
if not isinstance(jobs, dict):
jobs = {}
metadata["jobs"] = jobs
sync = jobs.setdefault("realm_sync", {})
if not isinstance(sync, dict):
sync = {}
jobs["realm_sync"] = sync
return jobs
jobs = metadata.get("jobs")
return jobs if isinstance(jobs, dict) else {}
def _metrics(metadata: dict[str, Any]) -> dict[str, Any]:
metrics = metadata.setdefault("metrics", {})
if not isinstance(metrics, dict):
metrics = {}
metadata["metrics"] = metrics
servers = metrics.setdefault("servers", {})
if not isinstance(servers, dict):
servers = {}
metrics["servers"] = servers
return metrics
metrics = metadata.get("metrics")
return metrics if isinstance(metrics, dict) else {}
def _cpu_models(metadata: dict[str, Any]) -> dict[str, Any]:
models = metadata.get("qemu_cpu_models")
if not isinstance(models, dict):
models = {}
metadata["qemu_cpu_models"] = models
return models
return models if isinstance(models, dict) else {}
def _ha_rules_store(metadata: dict[str, Any]) -> list[dict[str, Any]]:
@@ -84,12 +49,7 @@ def _ha_rules_store(metadata: dict[str, Any]) -> list[dict[str, Any]]:
]
if isinstance(rules, list):
return [dict(item) for item in rules if isinstance(item, dict)]
defaults = [
{"rule": "node-fencing", "type": "node", "action": "restart"},
{"rule": "service-ha", "type": "resource", "action": "failover"},
]
metadata["ha_rules"] = defaults
return list(defaults)
return []
def _save_ha_rules(metadata: dict[str, Any], rules: list[dict[str, Any]]) -> None:
@@ -97,26 +57,14 @@ def _save_ha_rules(metadata: dict[str, Any], rules: list[dict[str, Any]]) -> Non
def _replication_jobs(metadata: dict[str, Any]) -> list[dict[str, Any]]:
jobs = metadata.get("replication", [])
if not isinstance(jobs, list):
return []
return [dict(item) for item in jobs if isinstance(item, dict)]
from app.handlers.cluster import _replication_jobs as normalize
return normalize(metadata)
def _ceph(metadata: dict[str, Any]) -> dict[str, Any]:
ceph = metadata.get("ceph")
if not isinstance(ceph, dict):
ceph = {}
flags = ceph.get("flags")
if not isinstance(flags, dict):
flags = copy.deepcopy(DEFAULT_CEPH_FLAGS)
else:
merged = copy.deepcopy(DEFAULT_CEPH_FLAGS)
merged.update({str(key): int(value) for key, value in flags.items()})
flags = merged
ceph["flags"] = flags
metadata["ceph"] = ceph
return ceph
return ceph if isinstance(ceph, dict) else {}
async def _cluster_task(request: Request, *, task_type: str, worker: str) -> str:
@@ -174,8 +122,8 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
job_id = str(payload["id"])
metadata = await cluster_metadata(request)
jobs = _jobs(metadata)
sync = jobs.setdefault("realm_sync", {})
jobs = dict(_jobs(metadata))
sync = dict(jobs.get("realm_sync") or {})
if job_id in sync:
raise ApiError(409, "realm-sync job already exists")
entry = {
@@ -185,6 +133,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
entry.setdefault("enabled", 1)
entry.setdefault("realm", str(payload.get("realm") or "pam"))
sync[job_id] = entry
jobs["realm_sync"] = sync
metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata)
return {"id": job_id, **entry}
@@ -193,8 +142,8 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
job_id = str(payload["id"])
metadata = await cluster_metadata(request)
jobs = _jobs(metadata)
sync = jobs.setdefault("realm_sync", {})
jobs = dict(_jobs(metadata))
sync = dict(jobs.get("realm_sync") or {})
if job_id not in sync:
raise ApiError(404, "realm-sync job does not exist")
updated = {
@@ -206,6 +155,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
},
}
sync[job_id] = updated
jobs["realm_sync"] = sync
metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata)
return {"id": job_id, **updated}
@@ -213,23 +163,26 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
async def realm_sync_delete(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _jobs(metadata)
sync = jobs.setdefault("realm_sync", {})
jobs = dict(_jobs(metadata))
sync = dict(jobs.get("realm_sync") or {})
if job_id not in sync:
raise ApiError(404, "realm-sync job does not exist")
del sync[job_id]
jobs["realm_sync"] = sync
metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def schedule_analyze(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
schedule = str(values(inputs).get("schedule") or "*/15")
metadata = await cluster_metadata(request)
jobs = _jobs(metadata)
jobs = dict(_jobs(metadata))
results = jobs.get("schedule_analyze_results")
if not isinstance(results, list):
results = []
jobs["last_schedule_analyze"] = {"schedule": schedule, "at": int(time.time())}
metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata)
now = int(time.time())
return [{"timestamp": now + offset * 900, "utc": True} for offset in range(4)]
return [dict(item) for item in results if isinstance(item, dict)]
async def metrics_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("export", "server")
@@ -237,20 +190,54 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
async def metrics_export(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
metrics = _metrics(metadata)
return {
"data": metrics.get("export_data")
or '# HELP pve_up Node is up\npve_up{node="pve01"} 1\n',
"timestamp": int(time.time()),
}
now = int(time.time())
seeded = metrics.get("export_series")
if isinstance(seeded, list) and seeded:
seeded_series = [dict(item) for item in seeded if isinstance(item, dict)]
return {"data": seeded_series, "timestamp": now}
# Build a PVE-shaped metric array from live inventory (scales with seed).
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
series: list[dict[str, Any]] = []
for row in rows:
node = str(row["name"])
series.append(
{
"id": f"node/{node}",
"metric": "up",
"timestamp": now,
"type": "gauge",
"value": 1,
}
)
guests = await database(request).pool.fetch(
"""SELECT r.kind, r.external_id, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind IN ('qemu', 'lxc')
ORDER BY r.external_id::integer"""
)
for guest in guests:
series.append(
{
"id": f"{guest['kind']}/{guest['external_id']}",
"metric": "up",
"timestamp": now,
"type": "gauge",
"value": 1,
}
)
return {"data": series, "timestamp": now}
async def metrics_server_list(
request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
metrics = _metrics(metadata)
servers = metrics.get("servers")
if not isinstance(servers, dict):
return []
return [
{"id": server_id, **dict(payload)}
for server_id, payload in sorted(metrics.get("servers", {}).items())
for server_id, payload in sorted(servers.items())
if isinstance(payload, dict)
]
@@ -267,8 +254,8 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
server_id = str(payload["id"])
metadata = await cluster_metadata(request)
metrics = _metrics(metadata)
servers = metrics.setdefault("servers", {})
metrics = dict(_metrics(metadata))
servers = dict(metrics.get("servers") or {})
if server_id in servers:
raise ApiError(409, "metrics server already exists")
entry = {
@@ -279,6 +266,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
entry.setdefault("port", 8086)
entry.setdefault("enable", 1)
servers[server_id] = entry
metrics["servers"] = servers
metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata)
return {"id": server_id, **entry}
@@ -287,8 +275,8 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
server_id = str(payload["id"])
metadata = await cluster_metadata(request)
metrics = _metrics(metadata)
servers = metrics.setdefault("servers", {})
metrics = dict(_metrics(metadata))
servers = dict(metrics.get("servers") or {})
if server_id not in servers:
raise ApiError(404, "metrics server does not exist")
updated = {
@@ -300,6 +288,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
},
}
servers[server_id] = updated
metrics["servers"] = servers
metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata)
return {"id": server_id, **updated}
@@ -307,19 +296,24 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
async def metrics_server_delete(request: Request, inputs: dict[str, Any]) -> None:
server_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
metrics = _metrics(metadata)
servers = metrics.setdefault("servers", {})
metrics = dict(_metrics(metadata))
servers = dict(metrics.get("servers") or {})
if server_id not in servers:
raise ApiError(404, "metrics server does not exist")
del servers[server_id]
metrics["servers"] = servers
metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata)
async def qemu_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("cpu-flags", "custom-cpu-models")
async def qemu_cpu_flags(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(DEFAULT_CPU_FLAGS)
async def qemu_cpu_flags(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
flags = metadata.get("qemu_cpu_flags")
if isinstance(flags, list):
return [dict(item) for item in flags if isinstance(item, dict)]
return []
async def cpu_models_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
@@ -437,41 +431,69 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
) -> list[dict[str, str]]:
return subdirs("flags", "metadata", "status")
async def ceph_flags_get(request: Request, _inputs: dict[str, Any]) -> dict[str, int]:
async def ceph_flags_get(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
ceph = _ceph(metadata)
await save_cluster_metadata(request, metadata)
return {str(key): int(value) for key, value in ceph["flags"].items()}
stored = ceph.get("flags")
flags = default_ceph_flags()
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
return [
{
"name": name,
"value": bool(flags[name]),
"description": description,
}
for name, description in sorted(CEPH_FLAG_DESCRIPTIONS.items())
]
async def ceph_flags_put(request: Request, inputs: dict[str, Any]) -> dict[str, int]:
async def ceph_flags_put(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
metadata = await cluster_metadata(request)
ceph = _ceph(metadata)
flags = dict(ceph["flags"])
ceph = dict(_ceph(metadata))
flags = default_ceph_flags()
stored = ceph.get("flags")
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
for key, value in payload.items():
if key in {"delete", "digest"}:
continue
if key not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(400, f"unknown ceph flag '{key}'")
flags[str(key)] = int(value)
ceph["flags"] = flags
metadata["ceph"] = ceph
await save_cluster_metadata(request, metadata)
return {str(key): int(value) for key, value in flags.items()}
return await _cluster_task(request, task_type="ceph-flags", worker="cephflags")
async def ceph_flag_get(request: Request, inputs: dict[str, Any]) -> dict[str, int]:
async def ceph_flag_get(request: Request, inputs: dict[str, Any]) -> bool:
flag = str(values(inputs)["flag"])
if flag not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(404, "ceph flag does not exist")
metadata = await cluster_metadata(request)
ceph = _ceph(metadata)
flags = ceph["flags"]
if flag not in flags:
raise ApiError(404, "ceph flag does not exist")
return {flag: int(flags[flag])}
flags = ceph.get("flags")
if isinstance(flags, dict) and flag in flags:
return bool(int(flags[flag]))
return False
async def ceph_flag_put(request: Request, inputs: dict[str, Any]) -> dict[str, int]:
async def ceph_flag_put(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
flag = str(payload["flag"])
if flag not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(404, "ceph flag does not exist")
metadata = await cluster_metadata(request)
ceph = _ceph(metadata)
flags = dict(ceph["flags"])
ceph = dict(_ceph(metadata))
flags = default_ceph_flags()
stored = ceph.get("flags")
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
if "value" in payload:
flags[flag] = int(payload["value"])
elif flag in payload:
@@ -481,19 +503,84 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
ceph["flags"] = flags
metadata["ceph"] = ceph
await save_cluster_metadata(request, metadata)
return {flag: int(flags[flag])}
async def ceph_metadata(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
ceph = _ceph(metadata)
await save_cluster_metadata(request, metadata)
config_raw = ceph.get("config")
config: dict[str, Any] = dict(config_raw) if isinstance(config_raw, dict) else {}
version = ceph.get("version")
nodes = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in nodes]
version_str = (
str(version.get("str"))
if isinstance(version, dict) and version.get("str")
else "18.2.2"
)
version_short = ".".join(version_str.split(".")[:2]) if version_str else "18.2"
osd_rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind='ceph-osd' ORDER BY r.external_id"""
)
osd_map: dict[str, Any] = {}
for row in osd_rows:
payload = state(row["state"])
external = str(row["external_id"])
osd_id = str(payload.get("osd_id", external.removeprefix("osd.")))
hostname = str(row["node"])
osd_map[osd_id] = {
"id": int(osd_id) if osd_id.isdigit() else osd_id,
"hostname": hostname,
"name": f"osd.{osd_id}",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
"device_class": payload.get("device_class", "hdd"),
"dev": payload.get("dev", ""),
}
if not osd_map:
fallback_host = node_names[0] if node_names else "pve01"
osd_map["0"] = {
"id": 0,
"hostname": fallback_host,
"name": "osd.0",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
}
def _service_entry(name: str, index: int) -> dict[str, Any]:
return {
"name": name,
"hostname": name,
"addr": f"10.0.0.{index}",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
}
return {
"version": ceph.get("version") or {"str": "18.2.2", "parts": [18, 2, 2]},
"fsid": ceph.get("config", {}).get("fsid")
if isinstance(ceph.get("config"), dict)
else "pve-simulator-fsid",
"initialized": int(bool(ceph.get("initialized", True))),
"flags": ceph.get("flags", {}),
"mds": {},
"mgr": {
f"mgr@{node}": _service_entry(node, index)
for index, node in enumerate(node_names[:1] or ["pve01"], start=1)
},
"mon": {
f"mon@{node}": _service_entry(node, index)
for index, node in enumerate(node_names[:3] or ["pve01"], start=1)
},
"node": {
node: {
"name": node,
"hostname": node,
"ceph_version": version_str,
"ceph_version_short": version_short,
}
for node in (node_names or ["pve01"])
},
"osd": list(osd_map.values()),
"fsid": str(config.get("fsid") or ""),
}
async def ha_rule_create(request: Request, inputs: dict[str, Any]) -> None:
@@ -588,7 +675,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
log = job.get("log")
if isinstance(log, list):
return [dict(item) for item in log if isinstance(item, dict)]
return [{"t": int(time.time()), "n": 0, "msg": f"replication idle for {job.get('id')}"}]
return []
async def node_replication_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job = await node_replication_get(request, inputs)
+29 -6
View File
@@ -76,7 +76,11 @@ async def require_node(request: Request, node: str) -> None:
node,
)
if not exists:
raise ApiError(404, "node does not exist")
raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
async def cluster_metadata(request: Request) -> dict[str, Any]:
@@ -105,7 +109,11 @@ async def node_metadata(request: Request, node: str) -> dict[str, Any]:
node,
)
if row is None:
raise ApiError(404, "node does not exist")
raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
return state(row["metadata"])
@@ -116,7 +124,11 @@ async def save_node_metadata(request: Request, node: str, metadata: dict[str, An
json.dumps(metadata, sort_keys=True),
)
if status != "UPDATE 1":
raise ApiError(404, "node does not exist")
raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
def storage_payload(row: Any) -> dict[str, Any]:
@@ -129,13 +141,24 @@ def storage_payload(row: Any) -> dict[str, Any]:
total = int(row["capacity_bytes"] or 0)
used = int(row["used_bytes"] or 0)
avail = max(total - used, 0)
storage_type = str(row["storage_type"])
formats = config.get("formats")
if not isinstance(formats, str) or not formats:
formats = {
"dir": "raw,qcow2,vmdk",
"lvmthin": "raw",
"rbd": "raw",
"zfspool": "raw,subvol",
}.get(storage_type, "raw")
payload: dict[str, Any] = {
"storage": str(row["storage_id"]),
"type": str(row["storage_type"]),
"type": storage_type,
"shared": int(bool(row["shared"])),
"content": content_str,
"active": 1,
"enabled": 1,
"active": int(bool(config.get("active", True))),
"enabled": int(bool(config.get("enabled", True))),
"formats": formats,
"select_existing": int(bool(config.get("select_existing", False))),
"total": total,
"used": used,
"avail": avail,
+331 -23
View File
@@ -19,7 +19,7 @@ from app.handlers.ceph import register_ceph_handlers
from app.handlers.cluster import register_cluster_handlers
from app.handlers.cluster_config import register_cluster_config_handlers
from app.handlers.cluster_extra import register_cluster_extra_handlers
from app.handlers.common import require_node, subdirs
from app.handlers.common import cluster_metadata, require_node, subdirs
from app.handlers.firewall import register_firewall_handlers
from app.handlers.ha import register_ha_handlers
from app.handlers.legacy_aliases import register_legacy_aliases
@@ -60,52 +60,360 @@ def build_core_handlers(settings: Settings) -> HandlerRegistry:
raise ApiError(401, "authentication failure")
key = settings.ticket_signing_key.get_secret_value().encode()
ticket = issue_ticket(username, key)
cluster_name = await _database(request).pool.fetchval(
"SELECT name FROM clusters ORDER BY created_at LIMIT 1"
)
return {
"username": username,
"ticket": ticket,
"CSRFPreventionToken": csrf_token(ticket, key),
"clustername": str(cluster_name or "pve-simulator"),
"cap": {"vms": {"VM.Audit": 1, "VM.PowerMgmt": 1}},
}
async def nodes(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
from app.handlers.nodes import load_node_ops
rows = await _database(request).pool.fetch(
"SELECT name AS node, status FROM nodes ORDER BY name"
)
return [{"node": str(row["node"]), "status": str(row["status"])} for row in rows]
result: list[dict[str, Any]] = []
for row in rows:
name = str(row["node"])
ops = await load_node_ops(request, name)
status_payload = ops.get("status")
status_dict = dict(status_payload) if isinstance(status_payload, dict) else {}
fingerprint = status_dict.get("ssl_fingerprint") or status_dict.get("fingerprint")
if fingerprint in (None, "", 0) or isinstance(fingerprint, dict | list):
from app.simulation.seed import _seed_fingerprint
fingerprint = _seed_fingerprint(name)
def _as_float(value: object, default: float) -> float:
if isinstance(value, bool) or value is None or isinstance(value, dict | list):
return default
try:
return float(str(value))
except (TypeError, ValueError):
return default
def _as_int(value: object, default: int) -> int:
if isinstance(value, bool) or value is None or isinstance(value, dict | list):
return default
try:
return int(float(str(value)))
except (TypeError, ValueError):
return default
item: dict[str, Any] = {
"node": name,
"status": str(row["status"]),
"type": "node",
"ssl_fingerprint": str(fingerprint),
"cpu": _as_float(status_dict.get("cpu"), 0.0),
"maxcpu": _as_int(status_dict.get("maxcpu"), 4),
"mem": _as_int(status_dict.get("mem"), _as_int(status_dict.get("memory"), 0)),
"maxmem": _as_int(status_dict.get("maxmem"), 8 * 1024**3),
"uptime": _as_int(status_dict.get("uptime"), 0),
"level": str(status_dict.get("level") or ""),
}
result.append(item)
return result
async def node_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
from app.handlers.nodes import load_node_ops
node = str(cast(dict[str, Any], inputs["values"])["node"])
row = await _database(request).pool.fetchrow(
"SELECT name, status FROM nodes WHERE name=$1", node
)
if row is None:
raise ApiError(404, "node does not exist")
ops = await load_node_ops(request, node)
status = ops.get("status")
payload = dict(status) if isinstance(status, dict) else {}
mem_used = int(payload.get("mem") or 0)
mem_total = int(payload.get("maxmem") or 8 * 1024**3)
memory = payload.get("memory")
if isinstance(memory, dict):
mem_used = int(memory.get("used") or mem_used)
mem_total = int(memory.get("total") or mem_total)
rootfs = payload.get("rootfs")
if not isinstance(rootfs, dict):
rootfs = {
"used": int(payload.get("rootfs_used") or mem_used // 8),
"total": int(payload.get("rootfs_total") or 100 * 1024**3),
"avail": int(payload.get("rootfs_avail") or 80 * 1024**3),
"free": int(payload.get("rootfs_free") or 80 * 1024**3),
}
cpuinfo = payload.get("cpuinfo")
if not isinstance(cpuinfo, dict):
cpuinfo = {
"cpus": int(payload.get("maxcpu") or 4),
"cores": int(payload.get("maxcpu") or 4),
"sockets": 1,
"model": str(payload.get("cpu_model") or "QEMU Virtual CPU"),
"flags": "",
}
boot_info = payload.get("boot-info")
if not isinstance(boot_info, dict):
boot_info = {"mode": "efi", "secureboot": 0}
current_kernel = payload.get("current-kernel")
if not isinstance(current_kernel, dict):
kversion = str(payload.get("kversion") or "6.8.12-1-pve")
current_kernel = {
"sysname": "Linux",
"release": kversion,
"version": f"#1 SMP PREEMPT_DYNAMIC {kversion}",
"machine": "x86_64",
}
return {
"status": str(row["status"]),
"node": str(row["name"]),
"uptime": 0,
"cpu": 0.0,
"memory": {"used": 0, "total": 0},
"uptime": int(payload.get("uptime") or 0),
"wait": float(payload.get("wait") or 0.0),
"idle": float(payload.get("idle") or 0.95),
"cpu": float(payload.get("cpu") or 0.0),
"loadavg": list(payload.get("loadavg") or ["0.00", "0.00", "0.00"]),
"memory": {
"used": mem_used,
"total": mem_total,
"free": max(mem_total - mem_used, 0),
"available": max(mem_total - mem_used, 0),
},
"rootfs": rootfs,
"swap": payload.get("swap")
if isinstance(payload.get("swap"), dict)
else {"used": 0, "total": 0, "free": 0},
"cpuinfo": cpuinfo,
"kversion": str(payload.get("kversion") or current_kernel.get("release") or ""),
"pveversion": str(payload.get("pveversion") or "pve-manager/9.2.3"),
"current-kernel": current_kernel,
"boot-info": boot_info,
}
async def resources(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await _database(request).pool.fetch(
"""SELECT r.kind AS type, r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
ORDER BY r.kind, r.external_id"""
)
result: list[dict[str, Any]] = []
for row in rows:
raw_state = row["state"]
state = json.loads(raw_state) if isinstance(raw_state, str) else dict(raw_state)
result.append(
{
"type": str(row["type"]),
"id": f"{row['type']}/{row['external_id']}",
"node": str(row["node"]),
**state,
}
async def resources(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
"""Cluster-wide inventory in Proxmox ``/cluster/resources`` shape.
Do not dump guest ``state`` / config blobs: QEMU ``cpu`` is a model
string (e.g. ``qemu64``), while this endpoint's ``cpu`` is utilization
(float). Bridged clients (bpg / pulumi-proxmoxve) decode strictly.
"""
def _as_dict(raw: object) -> dict[str, Any]:
if isinstance(raw, str):
loaded = json.loads(raw)
return dict(loaded) if isinstance(loaded, dict) else {}
return dict(raw) if isinstance(raw, dict) else {}
def _num(value: object, default: float | int) -> float | int:
if isinstance(value, bool) or value is None or isinstance(value, dict | list):
return default
try:
if isinstance(default, float):
return float(str(value))
return int(float(str(value)))
except (TypeError, ValueError):
return default
def _memory_bytes(state: dict[str, Any]) -> int:
raw = state.get("maxmem", state.get("memory"))
if raw in (None, ""):
return 0
value = _num(raw, 0)
# QEMU config stores memory in MiB; cluster resources use bytes.
if isinstance(raw, str) or (isinstance(value, int) and 0 < value < 10_000_000):
return int(value) * 1024 * 1024
return int(value)
def _maxcpu(state: dict[str, Any]) -> int:
cores = int(_num(state.get("cores", state.get("cpus", 1)), 1))
sockets = int(_num(state.get("sockets", 1), 1))
return max(cores * sockets, 1)
def _cpu_util(state: dict[str, Any], *, running: bool) -> float:
if not running:
return 0.0
samples = state.get("rrddata")
if isinstance(samples, list) and samples:
last = samples[-1]
if isinstance(last, dict):
return float(_num(last.get("cpu"), 0.0))
return (
float(_num(state.get("cpu"), 0.0)) if not isinstance(state.get("cpu"), str) else 0.0
)
type_filter = cast(dict[str, Any], inputs.get("values") or {}).get("type")
result: list[dict[str, Any]] = []
if type_filter in (None, "node"):
node_rows = await _database(request).pool.fetch(
"SELECT name AS node, status FROM nodes ORDER BY name"
)
from app.handlers.nodes import load_node_ops
for row in node_rows:
name = str(row["node"])
ops = await load_node_ops(request, name)
status_payload = ops.get("status")
status_dict = dict(status_payload) if isinstance(status_payload, dict) else {}
mem_raw = status_dict.get("mem", status_dict.get("memory"))
if isinstance(mem_raw, dict):
mem_val = int(_num(mem_raw.get("used"), 0))
maxmem_val = int(
_num(status_dict.get("maxmem", mem_raw.get("total")), 8 * 1024**3)
)
else:
mem_val = int(_num(mem_raw, 0))
maxmem_val = int(_num(status_dict.get("maxmem"), 8 * 1024**3))
result.append(
{
"type": "node",
"id": f"node/{name}",
"node": name,
"status": str(row["status"]),
"cpu": float(_num(status_dict.get("cpu"), 0.0)),
"maxcpu": int(_num(status_dict.get("maxcpu"), 4)),
"mem": mem_val,
"maxmem": maxmem_val,
"uptime": int(_num(status_dict.get("uptime"), 0)),
"level": str(status_dict.get("level") or ""),
}
)
kind_filter: tuple[str, ...] | None
if type_filter == "vm":
kind_filter = ("qemu", "lxc")
elif type_filter == "storage":
kind_filter = ("storage",)
elif type_filter in (None,):
kind_filter = ("qemu", "lxc", "storage", "pool")
elif type_filter in {"qemu", "lxc", "storage", "pool", "sdn"}:
kind_filter = (str(type_filter),)
else:
kind_filter = ()
ha_by_guest: dict[str, str] = {}
if kind_filter and ("qemu" in kind_filter or "lxc" in kind_filter):
ha_rows = await _database(request).pool.fetch(
"SELECT external_id, state FROM resources WHERE kind='ha'"
)
for ha_row in ha_rows:
sid = str(ha_row["external_id"])
ha_state = _as_dict(ha_row["state"])
ha_by_guest[sid] = str(ha_state.get("state") or "started")
pool_by_vmid: dict[str, str] = {}
if kind_filter and ("qemu" in kind_filter or "lxc" in kind_filter or "pool" in kind_filter):
membership = await _database(request).pool.fetch(
"""SELECT p.pool_id AS pool, r.external_id AS vmid
FROM pool_members pm
JOIN pools p ON p.id=pm.pool_id
JOIN resources r ON r.id=pm.resource_id"""
)
for row in membership:
pool_by_vmid[str(row["vmid"])] = str(row["pool"])
node_names = [
str(item["node"])
for item in await _database(request).pool.fetch(
"SELECT name AS node FROM nodes ORDER BY name"
)
]
if kind_filter:
rows = await _database(request).pool.fetch(
"""SELECT r.kind AS type, r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind = ANY($1::text[])
ORDER BY r.kind, r.external_id""",
list(kind_filter),
)
for row in rows:
kind = str(row["type"])
external_id = str(row["external_id"])
node = str(row["node"])
state = _as_dict(row["state"])
if kind in {"qemu", "lxc"}:
status = str(state.get("status") or "stopped")
running = status in {"running", "paused"}
vmid = int(external_id)
sid = f"{'vm' if kind == 'qemu' else 'ct'}:{external_id}"
item: dict[str, Any] = {
"type": kind,
"id": f"{kind}/{external_id}",
"node": node,
"vmid": vmid,
"name": str(state.get("name") or f"{kind}-{external_id}"),
"status": status,
"template": bool(state.get("template") in (True, "1", 1)),
"cpu": _cpu_util(state, running=running),
"maxcpu": _maxcpu(state),
"mem": int(_num(state.get("mem"), 0)) if running else 0,
"maxmem": _memory_bytes(state),
"disk": int(_num(state.get("disk"), 0)),
"maxdisk": int(_num(state.get("maxdisk"), 0)),
"uptime": int(_num(state.get("uptime"), 0)) if running else 0,
}
if sid in ha_by_guest:
item["hastate"] = ha_by_guest[sid]
if external_id in pool_by_vmid:
item["pool"] = pool_by_vmid[external_id]
result.append(item)
elif kind == "storage":
content = state.get("content")
if isinstance(content, list):
content_text = ",".join(str(part) for part in content)
else:
content_text = str(content or "")
shared = bool(_num(state.get("shared"), 0))
# Schema allows one storages row per storage_id; expand to every
# node like a real PVE dump (local + shared storages).
targets = node_names if (shared or len(node_names) > 1) else [node]
if not shared and external_id not in {"local", "local-lvm", "ceph"}:
targets = [node]
for target in targets:
result.append(
{
"type": "storage",
"id": f"storage/{target}/{external_id}",
"node": target,
"storage": external_id,
"status": str(state.get("status") or "available"),
"content": content_text,
"disk": int(_num(state.get("disk"), 0)),
"maxdisk": int(_num(state.get("maxdisk"), 1 * 1024**3)),
"shared": shared,
"plugintype": str(
state.get("plugintype") or state.get("type") or "dir"
),
}
)
elif kind == "pool":
result.append(
{
"type": "pool",
"id": f"pool/{external_id}",
"pool": external_id,
"comment": str(state.get("comment") or ""),
}
)
if type_filter in (None, "sdn"):
metadata = await cluster_metadata(request)
sdn = metadata.get("sdn")
zones = sdn.get("zones") if isinstance(sdn, dict) else None
if isinstance(zones, dict):
for zone_name, zone in sorted(zones.items()):
if not isinstance(zone, dict):
continue
result.append(
{
"type": "sdn",
"id": f"sdn/{zone_name}",
"sdn": zone_name,
"status": str(zone.get("status") or "ok"),
}
)
return result
async def node_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
+73 -41
View File
@@ -13,20 +13,6 @@ from app.api.registry import HandlerRegistry
from app.handlers.common import database, require_node, state, subdirs, values
from app.simulation.seed import CLUSTER_ID
DEFAULT_OPTIONS = {
"enable": 1,
"policy_in": "DROP",
"policy_out": "ACCEPT",
"log_level_in": "nolog",
"log_level_out": "nolog",
}
DEFAULT_MACROS = [
{"macro": "SSH", "descr": "Secure Shell"},
{"macro": "HTTPS", "descr": "Secure web server"},
{"macro": "HTTP", "descr": "Web server"},
]
ScopeFn = Callable[[dict[str, Any]], str]
@@ -50,19 +36,36 @@ async def _save_firewall(request: Request, firewall: dict[str, Any]) -> None:
)
def _scope_data(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
def _empty_scope() -> dict[str, Any]:
return {
"options": {},
"rules": [],
"aliases": {},
"ipset": {},
"groups": {},
"log": [],
}
def _get_scope(firewall: dict[str, Any], scope: str) -> dict[str, Any] | None:
scopes = firewall.get("scopes")
if not isinstance(scopes, dict):
return None
section = scopes.get(scope)
return cast(dict[str, Any], section) if isinstance(section, dict) else None
def _ensure_scope(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
"""Create an empty durable scope on mutation; never injects catalog defaults."""
scopes = firewall.setdefault("scopes", {})
if scope not in scopes or not isinstance(scopes[scope], dict):
scopes[scope] = {
"options": dict(DEFAULT_OPTIONS),
"rules": [],
"aliases": {},
"ipset": {},
"groups": {},
"log": [],
}
section = scopes[scope]
section.setdefault("options", dict(DEFAULT_OPTIONS))
if not isinstance(scopes, dict):
scopes = {}
firewall["scopes"] = scopes
section = scopes.get(scope)
if not isinstance(section, dict):
section = _empty_scope()
scopes[scope] = section
section.setdefault("options", {})
section.setdefault("rules", [])
section.setdefault("aliases", {})
section.setdefault("ipset", {})
@@ -71,6 +74,11 @@ def _scope_data(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
return cast(dict[str, Any], section)
def _scope_data(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
"""Mutation helper — ensure scope exists without template defaults."""
return _ensure_scope(firewall, scope)
def register_firewall_handlers(registry: HandlerRegistry) -> None:
def register_scope(
base: str,
@@ -98,25 +106,31 @@ def register_firewall_handlers(registry: HandlerRegistry) -> None:
async def options_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
return dict(_scope_data(firewall, scope_fn(payload)).get("options", DEFAULT_OPTIONS))
section = _get_scope(firewall, scope_fn(payload))
if section is None:
return {}
options = section.get("options", {})
return dict(options) if isinstance(options, dict) else {}
async def options_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
async def options_put(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _scope_data(firewall, scope_fn(payload))
current = dict(section.get("options", DEFAULT_OPTIONS))
section = _ensure_scope(firewall, scope_fn(payload))
current = dict(section.get("options") or {})
for key, value in payload.items():
if key in {"node", "vmid", "delete", "digest"}:
continue
current[key] = value
section["options"] = current
await _save_firewall(request, firewall)
return current
async def rules_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
rules = _scope_data(firewall, scope_fn(payload)).get("rules", [])
section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
rules = section.get("rules", [])
return list(rules) if isinstance(rules, list) else []
async def rules_create(request: Request, inputs: dict[str, Any]) -> None:
@@ -169,7 +183,8 @@ def register_firewall_handlers(registry: HandlerRegistry) -> None:
async def aliases_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
aliases = _scope_data(firewall, scope_fn(payload)).get("aliases", {})
section = _get_scope(firewall, scope_fn(payload))
aliases = section.get("aliases", {}) if section else {}
if not isinstance(aliases, dict):
return []
return [
@@ -236,7 +251,8 @@ def register_firewall_handlers(registry: HandlerRegistry) -> None:
async def ipset_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
ipsets = _scope_data(firewall, scope_fn(payload)).get("ipset", {})
section = _get_scope(firewall, scope_fn(payload))
ipsets = section.get("ipset", {}) if section else {}
if not isinstance(ipsets, dict):
return []
return [
@@ -352,27 +368,43 @@ def register_firewall_handlers(registry: HandlerRegistry) -> None:
async def refs_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
section = _scope_data(firewall, scope_fn(payload))
section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
refs: list[dict[str, Any]] = []
for name in section.get("aliases", {}):
for name in section.get("aliases", {}) or {}:
refs.append({"type": "alias", "name": name})
for name in section.get("ipset", {}):
for name in section.get("ipset", {}) or {}:
refs.append({"type": "ipset", "name": name})
return refs
async def log_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
log = _scope_data(firewall, scope_fn(payload)).get("log", [])
section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
log = section.get("log", [])
return list(log) if isinstance(log, list) else []
async def macros_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(DEFAULT_MACROS)
async def macros_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
from app.handlers.common import cluster_metadata
metadata = await cluster_metadata(request)
macros = metadata.get("firewall_macros")
if isinstance(macros, list):
return [dict(item) for item in macros if isinstance(item, dict)]
firewall = await _load_firewall(request)
nested = firewall.get("macros")
if isinstance(nested, list):
return [dict(item) for item in nested if isinstance(item, dict)]
return []
async def groups_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs)
firewall = await _load_firewall(request)
groups = _scope_data(firewall, scope_fn(payload)).get("groups", {})
section = _get_scope(firewall, scope_fn(payload))
groups = section.get("groups", {}) if section else {}
if not isinstance(groups, dict):
return []
return [
+102 -62
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import hashlib
import json
from typing import Any
@@ -27,6 +28,13 @@ def _ha_groups(metadata: dict[str, Any]) -> dict[str, dict[str, Any]]:
return {str(key): dict(value) for key, value in groups.items() if isinstance(value, dict)}
def _ha_digest(payload: dict[str, Any]) -> str:
material = {key: value for key, value in payload.items() if key != "digest"}
return hashlib.sha1( # noqa: S324 - Proxmox HA digests use SHA-1
json.dumps(material, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
def register_ha_handlers(registry: HandlerRegistry) -> None:
async def ha_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("groups", "resources", "rules", "status")
@@ -41,17 +49,17 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
for row in rows:
payload = state(row["state"])
sid = str(row["external_id"])
result.append(
{
"sid": sid,
"type": "vm" if sid.startswith("vm:") else "ct",
"state": payload.get("state", "started"),
"group": payload.get("group"),
"node": str(row["node"]),
"max_relocate": payload.get("max_relocate", 1),
"max_restart": payload.get("max_restart", 1),
}
)
entry = {
"sid": sid,
"type": "vm" if sid.startswith("vm:") else "ct",
"state": payload.get("state", "started"),
"group": payload.get("group"),
"node": str(row["node"]),
"max_relocate": payload.get("max_relocate", 1),
"max_restart": payload.get("max_restart", 1),
}
entry["digest"] = _ha_digest(entry)
result.append(entry)
return result
async def ha_resource_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -142,7 +150,7 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
async def ha_groups(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
configured = _ha_groups(metadata)
result = [
return [
{
"group": group_id,
"nodes": str(payload.get("nodes", "")),
@@ -153,26 +161,6 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
}
for group_id, payload in sorted(configured.items())
]
if result:
return result
rows = await database(request).pool.fetch(
"""SELECT DISTINCT state->>'group' AS group_id
FROM resources WHERE kind='ha' AND state ? 'group'
ORDER BY 1"""
)
node_names = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
nodes = ",".join(str(row["name"]) for row in node_names) or "pve01"
return [
{
"group": str(row["group_id"]),
"nodes": nodes,
"nofailback": 0,
"restricted": 0,
"type": "group",
}
for row in rows
if row["group_id"]
]
async def ha_group_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
group = str(values(inputs)["group"])
@@ -228,50 +216,97 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
async def ha_status(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("current", "manager_status")
async def ha_status_current(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
row = await database(request).pool.fetchrow(
"""SELECT count(*) FILTER (WHERE state->>'state' = 'started') AS started,
count(*) AS total
FROM resources WHERE kind='ha'"""
)
async def ha_status_current(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
master = await database(request).pool.fetchval(
"SELECT name FROM nodes WHERE status='online' ORDER BY name LIMIT 1"
)
metadata = await cluster_metadata(request)
ha = metadata.get("ha", {}) if isinstance(metadata.get("ha"), dict) else {}
return {
"quorate": 1,
"mode": "active" if ha.get("armed", True) else "disabled",
"master_node": str(master or "pve01"),
"ha_started": int(row["started"] or 0),
"ha_total": int(row["total"] or 0),
"armed": 1 if ha.get("armed", True) else 0,
}
ha_raw = metadata.get("ha")
ha: dict[str, Any] = dict(ha_raw) if isinstance(ha_raw, dict) else {}
armed = bool(ha.get("armed")) if "armed" in ha else False
status_raw = ha.get("status_current")
status: dict[str, Any] = dict(status_raw) if isinstance(status_raw, dict) else {}
quorate = bool(int(status.get("quorate", metadata.get("quorate", 1)) or 1))
master_node = str(status.get("master_node") or master or "")
rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind='ha' ORDER BY r.external_id"""
)
nodes = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
result: list[dict[str, Any]] = [
{
"id": "quorum",
"type": "quorum",
"node": master_node,
"status": "OK" if quorate else "offline",
"quorate": quorate,
},
{
"id": "master",
"type": "master",
"node": master_node,
"status": "active" if armed else "disabled",
},
{
"id": "fencing",
"type": "fencing",
"node": master_node,
"status": "armed" if armed else "disarmed",
"armed-state": "armed" if armed else "disarmed",
},
]
for node_row in nodes:
node_name = str(node_row["name"])
result.append(
{
"id": f"lrm:{node_name}",
"type": "lrm",
"node": node_name,
"status": "active" if quorate else "offline",
}
)
for row in rows:
payload = state(row["state"])
sid = str(row["external_id"])
crm = str(payload.get("state") or "started")
result.append(
{
"id": f"service:{sid}",
"type": "service",
"sid": sid,
"node": str(row["node"]),
"crm_state": crm,
"state": crm,
"status": crm,
"request_state": str(payload.get("request_state") or crm),
"max_relocate": int(payload.get("max_relocate") or 1),
"max_restart": int(payload.get("max_restart") or 1),
}
)
return result
async def ha_manager_status(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request)
ha = metadata.get("ha", {}) if isinstance(metadata.get("ha"), dict) else {}
armed = bool(ha.get("armed", True))
ha_raw = metadata.get("ha")
ha: dict[str, Any] = dict(ha_raw) if isinstance(ha_raw, dict) else {}
manager_raw = ha.get("manager_status")
manager: dict[str, Any] = dict(manager_raw) if isinstance(manager_raw, dict) else {}
armed = bool(ha.get("armed")) if "armed" in ha else False
return {
"manager_status": "active" if armed else "disabled",
"quorum": "OK",
"manager_status": manager.get("manager_status", "active" if armed else "disabled"),
"quorum": manager.get("quorum", ""),
"armed": 1 if armed else 0,
}
async def ha_rules(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
rules = metadata.get("ha_rules")
if isinstance(rules, list) and rules:
if isinstance(rules, list):
return [dict(item) for item in rules if isinstance(item, dict)]
defaults = [
{"rule": "node-fencing", "type": "node", "action": "restart"},
{"rule": "service-ha", "type": "resource", "action": "failover"},
]
metadata["ha_rules"] = defaults
await save_cluster_metadata(request, metadata)
return list(defaults)
return []
async def ha_relocate(request: Request, inputs: dict[str, Any]) -> None:
async def ha_relocate(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
sid = str(payload["sid"])
target = str(payload.get("node") or payload.get("target") or "")
@@ -311,9 +346,14 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
ha_row["id"],
json.dumps(current, sort_keys=True),
)
return {
"sid": sid,
"requested-node": target,
"comigrated-resources": [],
}
async def ha_migrate(request: Request, inputs: dict[str, Any]) -> None:
await ha_relocate(request, inputs)
async def ha_migrate(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await ha_relocate(request, inputs)
async def ha_arm(request: Request, _inputs: dict[str, Any]) -> None:
metadata = await cluster_metadata(request)
+1 -1
View File
@@ -19,7 +19,7 @@ def register_legacy_aliases(registry: HandlerRegistry) -> None:
alias("/access/tfa", "POST", "/access/tfa/{userid}", "POST")
alias("/access/tfa", "PUT", "/access/tfa/{userid}/{id}", "PUT")
alias("/cluster/backupinfo", "GET", "/cluster/backup-info", "GET")
# /cluster/backupinfo is registered as a PVE 6 stub string in backup.py
alias(
"/cluster/backupinfo/not_backed_up",
"GET",
+118 -10
View File
@@ -38,27 +38,120 @@ def _state(value: object) -> dict[str, Any]:
return dict(cast(Mapping[str, Any], value))
# Runtime-only keys stored in ``resources.state``; never echo on GET /config.
_LXC_CONFIG_INTERNAL = frozenset(
{
"agent",
"cloudinit_dump",
"config",
"consoles",
"interfaces",
"migrate_preconditions",
"name",
"rrd",
"rrddata",
"status",
"volume_moves",
}
)
def _public_lxc_config(
config: dict[str, Any], state: dict[str, Any], *, vmid: int
) -> dict[str, Any]:
import hashlib
merged = {**state, **config}
payload: dict[str, Any] = {"vmid": vmid}
for key, value in merged.items():
if key in _LXC_CONFIG_INTERNAL:
continue
if isinstance(value, dict | list):
continue
payload[key] = value
digest_src = {key: value for key, value in payload.items() if key != "digest"}
payload["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(digest_src, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
return payload
def _lxc_index_item(vmid: int, vm_state: dict[str, Any], config: dict[str, Any]) -> dict[str, Any]:
status = str(vm_state.get("status", "stopped"))
running = status in {"running", "paused"}
memory_mb = int(config.get("memory", config.get("mem", 512)))
maxmem = memory_mb * 2**20
return {
"vmid": vmid,
"name": str(config.get("hostname", config.get("name", f"ct-{vmid}"))),
"status": status,
"cpus": int(config.get("cores", config.get("cpus", 1))),
"cpu": float(vm_state.get("cpu", 0.05 if running else 0.0)),
"maxmem": maxmem,
"mem": int(vm_state.get("mem", maxmem // 2 if running else 0)),
"maxdisk": int(vm_state.get("maxdisk", config.get("maxdisk", 8 * 1024**3))),
"disk": int(vm_state.get("disk", 0)),
"diskread": int(vm_state.get("diskread", 0)),
"diskwrite": int(vm_state.get("diskwrite", 0)),
"netin": int(vm_state.get("netin", 0)),
"netout": int(vm_state.get("netout", 0)),
"maxswap": int(config.get("swap", 512)) * 2**20,
"uptime": int(vm_state.get("uptime", 0)) if running else 0,
"lock": str(vm_state.get("lock", "")),
"tags": str(config.get("tags", "")),
"template": int(bool(vm_state.get("template", False))),
"pressurecpusome": float(vm_state.get("pressurecpusome", 0.0)),
"pressurecpufull": float(vm_state.get("pressurecpufull", 0.0)),
"pressureiosome": float(vm_state.get("pressureiosome", 0.0)),
"pressureiofull": float(vm_state.get("pressureiofull", 0.0)),
"pressurememorysome": float(vm_state.get("pressurememorysome", 0.0)),
"pressurememoryfull": float(vm_state.get("pressurememoryfull", 0.0)),
}
def register_lxc_handlers(registry: HandlerRegistry) -> None:
async def lxc_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(_values(inputs)["node"])
rows = await _database(request).pool.fetch(
"""SELECT r.external_id::integer AS vmid, r.state
FROM resources r JOIN nodes n ON n.id=r.node_id
"""SELECT r.external_id::integer AS vmid, r.state, c.config
FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN containers c ON c.resource_id=r.id
WHERE n.name=$1 AND r.kind='lxc' ORDER BY r.external_id::integer""",
node,
)
return [{"vmid": int(row["vmid"]), **_state(row["state"])} for row in rows]
return [
_lxc_index_item(int(row["vmid"]), _state(row["state"]), _state(row["config"]))
for row in rows
]
async def lxc_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"])
row = await _lxc_resource(request, node, vmid)
return {"vmid": int(vmid), **_state(row["config"]), **_state(row["state"])}
return _public_lxc_config(_state(row["config"]), _state(row["state"]), vmid=int(vmid))
async def lxc_current(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await lxc_config(request, inputs)
payload = _values(inputs)
node, vmid = str(payload["node"]), int(payload["vmid"])
row = await _lxc_resource(request, node, str(vmid))
vm_state = _state(row["state"])
config = _state(row["config"])
item = _lxc_index_item(vmid, vm_state, config)
item["ha"] = {"managed": int(vm_state.get("ha_managed", 0))}
return item
async def lxc_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await lxc_config(request, inputs)
async def lxc_status(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs)
await _lxc_resource(request, str(payload["node"]), str(payload["vmid"]))
return subdirs(
"current",
"reboot",
"resume",
"shutdown",
"start",
"stop",
"suspend",
)
async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
@@ -323,7 +416,7 @@ def register_lxc_handlers(registry: HandlerRegistry) -> None:
async def migrate_preconditions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
target = values.get("target")
if target in {None, ""}:
raise ApiError(400, "parameter 'target' is required")
@@ -333,7 +426,13 @@ def register_lxc_handlers(registry: HandlerRegistry) -> None:
)
if not exists:
raise ApiError(404, "target node does not exist")
return {"local_disks": [], "local_resources": [], "running": False}
state = _state(resource["state"])
pre = state.get("migrate_preconditions")
payload = dict(pre) if isinstance(pre, dict) else {}
payload["running"] = str(state.get("status") or "") == "running"
payload.setdefault("local_disks", [])
payload.setdefault("local_resources", [])
return payload
async def migrate(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
@@ -415,7 +514,7 @@ def register_lxc_handlers(registry: HandlerRegistry) -> None:
}
}
async def lxc_resize(request: Request, inputs: dict[str, Any]) -> None:
async def lxc_resize(request: Request, inputs: dict[str, Any]) -> str:
payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"])
disk = str(payload.get("disk") or "rootfs")
@@ -440,6 +539,13 @@ def register_lxc_handlers(registry: HandlerRegistry) -> None:
resource["id"],
json.dumps({disk: config[disk]}, sort_keys=True),
)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-resize",
payload={"resource_id": str(resource["id"]), "disk": disk, "size": size},
)
async def lxc_template(request: Request, inputs: dict[str, Any]) -> None:
payload = _values(inputs)
@@ -531,6 +637,8 @@ async def _create_task(
"lxc-clone": "pctclone",
"lxc-migrate": "pctmigrate",
"lxc-remote-migrate": "pctremote",
"lxc-resize": "pctresize",
"lxc-move-volume": "pctmove",
}[task_type]
upid = str(Upid.allocate(node, worker_type, vmid, str(request.state.principal)))
try:
+21 -20
View File
@@ -39,16 +39,15 @@ def register_lxc_extra_handlers(registry: HandlerRegistry) -> None:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
ifaces = state.setdefault(
"interfaces",
[{"name": "eth0", "hwaddr": "02:00:00:00:00:11", "inet": "192.0.2.20/24"}],
)
await _save_state(request, resource["id"], state)
return list(ifaces) if isinstance(ifaces, list) else []
ifaces = state.get("interfaces")
return [dict(item) for item in ifaces] if isinstance(ifaces, list) else []
async def move_volume(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.lxc import _create_task
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
node, vmid = str(values["node"]), str(values["vmid"])
resource = await _lxc_resource(request, node, vmid)
volume = str(values.get("volume") or values.get("disk") or "rootfs")
storage = str(values.get("storage") or "local-lvm")
config = _state(resource["config"])
@@ -64,29 +63,31 @@ def register_lxc_extra_handlers(registry: HandlerRegistry) -> None:
moves = state["volume_moves"] = []
moves.append({"volume": volume, "storage": storage})
await _save_state(request, resource["id"], state)
return f"UPID:{values['node']}:lxc-move-volume:{values['vmid']}"
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-move-volume",
payload={
"resource_id": str(resource["id"]),
"volume": volume,
"storage": storage,
},
)
async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
rrd_state = state.setdefault("rrd", {"filename": f"pve-ct-{values['vmid']}.rrd"})
await _save_state(request, resource["id"], state)
return dict(rrd_state)
rrd_state = state.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
series = state.setdefault(
"rrddata",
[
{"time": 1_700_000_000, "cpu": 0.02, "mem": 64 * 1024 * 1024},
{"time": 1_700_000_060, "cpu": 0.03, "mem": 66 * 1024 * 1024},
],
)
await _save_state(request, resource["id"], state)
return list(series)
series = state.get("rrddata")
return [dict(item) for item in series] if isinstance(series, list) else []
async def _console(request: Request, inputs: dict[str, Any], kind: str) -> dict[str, Any]:
values = _values(inputs)
+16 -9
View File
@@ -12,13 +12,14 @@ from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs
def _mappings(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault("mapping", {"dir": {}, "pci": {}, "usb": {}})
current = metadata.get("mapping")
if not isinstance(current, dict):
current = {"dir": {}, "pci": {}, "usb": {}}
metadata["mapping"] = current
for kind in ("dir", "pci", "usb"):
current.setdefault(kind, {})
return current
return {"dir": {}, "pci": {}, "usb": {}}
return {
"dir": dict(current["dir"]) if isinstance(current.get("dir"), dict) else {},
"pci": dict(current["pci"]) if isinstance(current.get("pci"), dict) else {},
"usb": dict(current["usb"]) if isinstance(current.get("usb"), dict) else {},
}
def register_mapping_handlers(registry: HandlerRegistry) -> None:
@@ -42,12 +43,14 @@ def register_mapping_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
item_id = str(payload["id"])
metadata = await cluster_metadata(request)
store = _mappings(metadata)[kind]
mapping = _mappings(metadata)
store = mapping[kind]
if item_id in store:
raise ApiError(400, f"{kind} mapping '{item_id}' already exists")
store[item_id] = {
key: value for key, value in payload.items() if key not in {"delete", "digest"}
}
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata)
async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -62,7 +65,8 @@ def register_mapping_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
item_id = str(payload["id"])
metadata = await cluster_metadata(request)
store = _mappings(metadata)[kind]
mapping = _mappings(metadata)
store = mapping[kind]
if item_id not in store:
raise ApiError(404, f"{kind} mapping does not exist")
current = dict(store[item_id])
@@ -76,15 +80,18 @@ def register_mapping_handlers(registry: HandlerRegistry) -> None:
current[key] = value
current["id"] = item_id
store[item_id] = current
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata)
async def delete(request: Request, inputs: dict[str, Any]) -> None:
item_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
store = _mappings(metadata)[kind]
mapping = _mappings(metadata)
store = mapping[kind]
if item_id not in store:
raise ApiError(404, f"{kind} mapping does not exist")
del store[item_id]
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata)
registry.register(base, "GET", list_items)
+130 -126
View File
@@ -2,7 +2,6 @@
from __future__ import annotations
import copy
from typing import Any, cast
from fastapi import Request
@@ -20,92 +19,13 @@ from app.handlers.common import (
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
DEFAULT_NODE_OPS: dict[str, Any] = {
"network": [
{
"iface": "vmbr0",
"type": "bridge",
"active": 1,
"method": "static",
"address": "10.0.0.10/24",
},
{
"iface": "vmbr1",
"type": "bridge",
"active": 1,
"method": "static",
"address": "10.10.0.10/24",
},
{"iface": "eno1", "type": "eth", "active": 1, "method": "manual"},
],
"disks": {
"list": [
{
"devpath": "/dev/sda",
"size": 1_000_000_000_000,
"model": "SIM-DISK-01",
"serial": "SIM0001",
"gpt": 1,
},
{
"devpath": "/dev/sdb",
"size": 2_000_000_000_000,
"model": "SIM-SSD-01",
"serial": "SIM0002",
"gpt": 0,
},
],
"directory": [],
"lvm": [],
"lvmthin": [],
"zfs": [],
"smart": {},
},
"services": {
"pveproxy": {"state": "running", "enabled": 1},
"pvedaemon": {"state": "running", "enabled": 1},
"pvestatd": {"state": "running", "enabled": 1},
"corosync": {"state": "running", "enabled": 1},
},
"apt": {
"packages": [
{
"Package": "pve-manager",
"Version": "9.2.3",
"OldVersion": "9.2.2",
"Status": "upgradable",
},
{"Package": "libpve-common-perl", "Version": "9.0.3", "Status": "installed"},
],
"repositories": [
{
"path": "/etc/apt/sources.list.d/pve-enterprise.list",
"enabled": 1,
"types": "deb",
"uri": "http://download.proxmox.com/debian/pve",
"suites": "bookworm",
"components": "pve-no-subscription",
}
],
"update": {"status": "stopped", "exitstatus": "OK"},
"changelogs": {},
},
}
def default_node_ops() -> dict[str, Any]:
return copy.deepcopy(DEFAULT_NODE_OPS)
async def load_node_ops(request: Request, node: str) -> dict[str, Any]:
metadata = await node_metadata(request, node)
ops = metadata.get("ops")
if isinstance(ops, dict) and ops:
if isinstance(ops, dict):
return ops
ops = default_node_ops()
metadata["ops"] = ops
await save_node_metadata(request, node, metadata)
return ops
return {}
async def save_node_ops(request: Request, node: str, ops: dict[str, Any]) -> None:
@@ -123,31 +43,76 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
packages = ops.get("apt", {}).get("packages", [])
return list(packages) if isinstance(packages, list) else []
if not isinstance(packages, list):
return []
result: list[dict[str, Any]] = []
for item in packages:
if not isinstance(item, dict):
continue
entry = dict(item)
status = str(entry.pop("Status", entry.get("CurrentState", "Installed")))
if status.lower() == "upgradable":
entry["CurrentState"] = "Installed"
entry.setdefault("OldVersion", entry.get("Version", ""))
else:
entry["CurrentState"] = "Installed" if status.lower() == "installed" else status
entry.setdefault("Package", entry.get("Package") or "unknown")
entry.setdefault("Version", entry.get("Version") or "")
entry.setdefault("Arch", "amd64")
entry.setdefault("Origin", "Proxmox")
result.append(entry)
return result
async def apt_repositories(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
async def apt_repositories(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
repositories = ops.get("apt", {}).get("repositories", [])
return list(repositories) if isinstance(repositories, list) else []
repositories = ops.get("apt", {}).get("repositories")
if isinstance(repositories, dict) and "files" in repositories:
return {
"digest": str(repositories.get("digest") or "seedapt"),
"files": list(repositories.get("files") or []),
"errors": list(repositories.get("errors") or []),
"infos": list(repositories.get("infos") or []),
"standard-repos": list(repositories.get("standard-repos") or []),
}
legacy = list(repositories) if isinstance(repositories, list) else []
return {
"digest": "seedapt",
"files": [
{
"path": "/etc/apt/sources.list.d/pve-enterprise.list",
"file-type": "list",
"digest": "seedapt-file0",
"repositories": legacy,
}
],
"errors": [],
"infos": [],
"standard-repos": [
{
"handle": "no-subscription",
"name": "Proxmox VE No-Subscription Repository",
"status": 1,
}
],
}
async def apt_changelog(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
name = str(values(inputs).get("name") or "pve-manager")
ops = await load_node_ops(request, node)
changelogs = ops.setdefault("apt", {}).setdefault("changelogs", {})
if name not in changelogs:
changelogs[name] = f"simulated changelog for {name}\n\n * emulator build\n"
await save_node_ops(request, node, ops)
return str(changelogs[name])
apt = ops.get("apt")
changelogs = apt.get("changelogs") if isinstance(apt, dict) else None
if not isinstance(changelogs, dict):
return ""
return str(changelogs.get(name) or "")
async def apt_update_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
update = ops.get("apt", {}).get("update", {"status": "stopped", "exitstatus": "OK"})
if isinstance(update, dict):
return dict(update)
return {"status": "stopped", "exitstatus": "OK"}
apt = ops.get("apt")
update = apt.get("update") if isinstance(apt, dict) else None
return dict(update) if isinstance(update, dict) else {}
async def apt_update_start(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
@@ -170,7 +135,7 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
return item
raise ApiError(404, "interface does not exist")
async def network_mutate(request: Request, inputs: dict[str, Any]) -> None:
async def network_mutate(request: Request, inputs: dict[str, Any]) -> str | None:
node = str(values(inputs)["node"])
payload = values(inputs)
ops = await load_node_ops(request, node)
@@ -221,8 +186,14 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
raise ApiError(404, "interface does not exist")
ops["network"] = updated
else:
# PUT /nodes/{node}/network without iface → reload (returns UPID string).
ops["network_applied"] = True
await save_node_ops(request, node, ops)
return await _node_task(
request, node=node, task_type="network-reload", worker="srvreload"
)
await save_node_ops(request, node, ops)
return None
async def disks_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"]))
@@ -230,11 +201,9 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
async def _disks(request: Request, node: str) -> dict[str, Any]:
ops = await load_node_ops(request, node)
disks = ops.setdefault("disks", default_node_ops()["disks"])
disks = ops.get("disks")
if not isinstance(disks, dict):
disks = default_node_ops()["disks"]
ops["disks"] = disks
await save_node_ops(request, node, ops)
return {}
return cast(dict[str, Any], disks)
async def disks_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
@@ -247,17 +216,9 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or "/dev/sda")
disks = await _disks(request, node)
smart = disks.setdefault("smart", {})
if disk not in smart:
smart[disk] = {
"health": "PASSED",
"type": "scsi",
"model": "SIM-DISK",
"serial": disk.rsplit("/", 1)[-1],
}
ops = await load_node_ops(request, node)
ops["disks"] = disks
await save_node_ops(request, node, ops)
smart = disks.get("smart")
if not isinstance(smart, dict) or disk not in smart:
return {}
return dict(smart[disk])
async def disks_collection(
@@ -280,13 +241,16 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
async def disks_zfs(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
return await disks_collection(request, inputs, "zfs")
async def disks_initgpt(request: Request, inputs: dict[str, Any]) -> None:
async def disks_initgpt(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or values(inputs).get("device") or "")
if not disk:
raise ApiError(400, "parameter verification failed - 'disk' missing")
ops = await load_node_ops(request, node)
disks = ops.setdefault("disks", default_node_ops()["disks"])
disks = ops.get("disks")
if not isinstance(disks, dict):
disks = {}
ops["disks"] = disks
items = list(disks.get("list") or [])
found = False
for item in items:
@@ -307,14 +271,17 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
disks["list"] = items
ops["disks"] = disks
await save_node_ops(request, node, ops)
return await _node_task(request, node=node, task_type="disk-initgpt", worker="diskinit")
async def disks_wipedisk(request: Request, inputs: dict[str, Any]) -> None:
async def disks_wipedisk(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or values(inputs).get("device") or "")
if not disk:
raise ApiError(400, "parameter verification failed - 'disk' missing")
ops = await load_node_ops(request, node)
disks = ops.setdefault("disks", default_node_ops()["disks"])
disks = ops.get("disks")
if not isinstance(disks, dict):
raise ApiError(404, "disk does not exist")
items = list(disks.get("list") or [])
for item in items:
if item.get("devpath") == disk:
@@ -324,18 +291,35 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
else:
raise ApiError(404, "disk does not exist")
disks["list"] = items
smart = disks.setdefault("smart", {})
smart.pop(disk, None)
smart = disks.get("smart")
if isinstance(smart, dict):
smart.pop(disk, None)
ops["disks"] = disks
await save_node_ops(request, node, ops)
return await _node_task(request, node=node, task_type="disk-wipe", worker="diskwipe")
async def services_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
async def services_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
ops = await load_node_ops(request, node)
services = ops.get("services") or {}
return [{"subdir": name} for name in sorted(services)]
result: list[dict[str, Any]] = []
for name in sorted(services):
payload = dict(services[name]) if isinstance(services[name], dict) else {}
state_name = str(payload.get("state", "stopped"))
enabled = bool(int(payload.get("enabled", 0) or 0))
result.append(
{
"service": name,
"name": f"{name}.service",
"desc": str(payload.get("desc") or name),
"state": state_name,
"active-state": "active" if state_name == "running" else "inactive",
"unit-state": "enabled" if enabled else "disabled",
}
)
return result
async def service_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
async def service_get(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
node = str(values(inputs)["node"])
service = str(values(inputs)["service"])
ops = await load_node_ops(request, node)
@@ -343,12 +327,25 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
if service not in services:
services[service] = {"state": "stopped", "enabled": 0}
await save_node_ops(request, node, ops)
payload = dict(services[service])
payload["service"] = service
return payload
return subdirs("state", "start", "stop", "restart", "reload")
async def service_state(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await service_get(request, inputs)
node = str(values(inputs)["node"])
service = str(values(inputs)["service"])
ops = await load_node_ops(request, node)
services = ops.setdefault("services", {})
if service not in services:
services[service] = {"state": "stopped", "enabled": 0}
await save_node_ops(request, node, ops)
payload = dict(services[service]) if isinstance(services[service], dict) else {}
state_name = str(payload.get("state", "stopped"))
enabled = bool(int(payload.get("enabled", 0) or 0))
return {
"service": service,
"state": state_name,
"active-state": "active" if state_name == "running" else "inactive",
"unit-state": "enabled" if enabled else "disabled",
}
async def service_action(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
@@ -371,7 +368,12 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
services[service] = current
ops["services"] = services
await save_node_ops(request, node, ops)
return "OK"
return await _node_task(
request,
node=node,
task_type=f"service-{action}",
worker=f"srv{action}",
)
registry.register("/nodes/{node}/apt", "GET", apt_index)
registry.register("/nodes/{node}/apt/versions", "GET", apt_versions)
@@ -404,6 +406,8 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
async def _node_task(request: Request, *, node: str, task_type: str, worker: str) -> str:
import secrets
from app.api.errors import ApiError
from app.db.primitives import ConflictError
@@ -414,7 +418,7 @@ async def _node_task(request: Request, *, node: str, task_type: str, worker: str
upid=upid,
task_type=task_type,
payload={"node": node},
resource_key=f"node:{node}",
resource_key=f"node:{node}:{task_type}:{secrets.token_hex(4)}",
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
+310 -266
View File
@@ -2,7 +2,6 @@
from __future__ import annotations
import copy
import json
import secrets
import time
@@ -13,128 +12,49 @@ from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import database, require_node, subdirs, values
from app.handlers.nodes import default_node_ops, load_node_ops, save_node_ops
from app.handlers.nodes import load_node_ops, save_node_ops
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
DEFAULT_HARDWARE: dict[str, Any] = {
"pci": [
{
"id": "0000:00:1f.2",
"vendor_name": "Intel Corporation",
"device_name": "SATA Controller",
"iommugroup": 0,
},
{
"id": "0000:01:00.0",
"vendor_name": "NVIDIA Corporation",
"device_name": "GP102 [GeForce GTX 1080 Ti]",
"iommugroup": 1,
"mdev": 1,
},
],
"usb": [
{"busnum": 1, "devnum": 1, "level": 0, "port": "1", "prodid": "0002", "vendid": "1d6b"},
{"busnum": 2, "devnum": 2, "level": 1, "port": "2", "prodid": "5591", "vendid": "0781"},
],
"mdev": {
"0000:01:00.0": [
{"type": "nvidia-11", "available": 4, "description": "GRID profile"},
]
},
}
DEFAULT_SCAN: dict[str, list[dict[str, Any]]] = {
"cifs": [{"server": "files.local", "share": "backups"}],
"iscsi": [{"portal": "10.0.0.50:3260", "target": "iqn.2024-01.local:storage"}],
"lvm": [{"vg": "pve", "size": 500_000_000_000, "free": 100_000_000_000}],
"lvmthin": [{"lv": "data", "vg": "pve", "lv_size": 400_000_000_000}],
"nfs": [{"server": "nfs.local", "path": "/export/pve", "options": "vers=4"}],
"pbs": [{"server": "pbs.local", "datastore": "store1"}],
"zfs": [{"pool": "rpool", "name": "rpool/data", "size": 800_000_000_000}],
}
DEFAULT_SUBSCRIPTION: dict[str, Any] = {
"status": "notfound",
"message": "There is no subscription key",
"serverid": "SIMULATOR",
"sockets": 1,
"productname": "Proxmox VE",
"url": "https://www.proxmox.com/en/proxmox-virtual-environment/pricing",
}
DEFAULT_CONFIG: dict[str, Any] = {
"description": "Simulator node",
"startall-onboot-delay": 0,
"wakeonlan": "",
}
DEFAULT_DNS: dict[str, Any] = {
"search": "local",
"dns1": "1.1.1.1",
"dns2": "8.8.8.8",
"dns3": "",
}
DEFAULT_TIME: dict[str, Any] = {
"timezone": "UTC",
"time": 0,
"localtime": 0,
}
def _certificates(ops: dict[str, Any]) -> dict[str, Any]:
certs = ops.setdefault(
"certificates",
{
"custom": None,
"acme": {"account": "default", "domains": [], "certificate": None},
"info": [],
},
)
certs = ops.get("certificates")
if not isinstance(certs, dict):
certs = {"custom": None, "acme": {}, "info": []}
ops["certificates"] = certs
certs.setdefault("acme", {"account": "default", "domains": [], "certificate": None})
certs.setdefault("info", [])
return {"custom": None, "acme": {}, "info": []}
return certs
def _hardware(ops: dict[str, Any]) -> dict[str, Any]:
hardware = ops.get("hardware")
if not isinstance(hardware, dict) or not hardware:
hardware = copy.deepcopy(DEFAULT_HARDWARE)
ops["hardware"] = hardware
hardware.setdefault("pci", copy.deepcopy(DEFAULT_HARDWARE["pci"]))
hardware.setdefault("usb", copy.deepcopy(DEFAULT_HARDWARE["usb"]))
hardware.setdefault("mdev", copy.deepcopy(DEFAULT_HARDWARE["mdev"]))
return hardware
if not isinstance(hardware, dict):
return {"pci": [], "usb": [], "mdev": {}}
return {
"pci": list(hardware.get("pci") or []) if isinstance(hardware.get("pci"), list) else [],
"usb": list(hardware.get("usb") or []) if isinstance(hardware.get("usb"), list) else [],
"mdev": dict(hardware.get("mdev") or {}) if isinstance(hardware.get("mdev"), dict) else {},
}
def _scan_cache(ops: dict[str, Any]) -> dict[str, Any]:
scan = ops.get("scan")
if not isinstance(scan, dict) or not scan:
scan = copy.deepcopy(DEFAULT_SCAN)
ops["scan"] = scan
for key, value in DEFAULT_SCAN.items():
scan.setdefault(key, copy.deepcopy(value))
if not isinstance(scan, dict):
return {}
return scan
def _subscription(ops: dict[str, Any]) -> dict[str, Any]:
subscription = ops.get("subscription")
if not isinstance(subscription, dict) or not subscription:
subscription = copy.deepcopy(DEFAULT_SUBSCRIPTION)
ops["subscription"] = subscription
if not isinstance(subscription, dict):
return {}
return subscription
def _disk_items(ops: dict[str, Any], kind: str) -> list[dict[str, Any]]:
disks = ops.setdefault("disks", default_node_ops()["disks"])
disks = ops.get("disks")
if not isinstance(disks, dict):
disks = default_node_ops()["disks"]
disks = {}
ops["disks"] = disks
items = disks.setdefault(kind, [])
items = disks.get(kind)
if not isinstance(items, list):
items = []
disks[kind] = items
@@ -188,7 +108,9 @@ async def _migrate_guests(request: Request, node: str, target: str) -> None:
def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
async def disks_create(request: Request, inputs: dict[str, Any], kind: str) -> dict[str, Any]:
async def disks_create(request: Request, inputs: dict[str, Any], kind: str) -> str:
from app.handlers.nodes import _node_task
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
@@ -208,11 +130,19 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
}
entry["name"] = name
items.append(entry)
ops.setdefault("disks", default_node_ops()["disks"])[kind] = items
disks = ops.get("disks")
if not isinstance(disks, dict):
disks = {}
ops["disks"] = disks
disks[kind] = items
await save_node_ops(request, node, ops)
return entry
return await _node_task(
request, node=node, task_type=f"disk-{kind}-create", worker=f"disk{kind}"
)
async def disks_delete(request: Request, inputs: dict[str, Any], kind: str) -> str:
from app.handlers.nodes import _node_task
async def disks_delete(request: Request, inputs: dict[str, Any], kind: str) -> None:
payload = values(inputs)
node = str(payload["node"])
name = str(payload["name"])
@@ -224,6 +154,9 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
raise ApiError(404, f"{kind} does not exist")
ops.setdefault("disks", {})[kind] = remaining
await save_node_ops(request, node, ops)
return await _node_task(
request, node=node, task_type=f"disk-{kind}-delete", worker=f"diskdel{kind}"
)
async def disks_zfs_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
@@ -270,11 +203,11 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
certs["acme"] = acme
ops["certificates"] = certs
await save_node_ops(request, node, ops)
if method == "DELETE":
return None
return await _node_task(request, node=node, task_type="acme", worker="acme")
async def certificates_custom(request: Request, inputs: dict[str, Any]) -> None:
async def certificates_custom(
request: Request, inputs: dict[str, Any]
) -> dict[str, Any] | None:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
@@ -282,35 +215,40 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
certs = _certificates(ops)
if request.method.upper() == "DELETE":
certs["custom"] = None
else:
certificates = str(payload.get("certificates") or payload.get("cert") or "")
if not certificates:
raise ApiError(400, "parameter verification failed - 'certificates' missing")
key = str(payload.get("key") or payload.get("private-key") or "")
certs["custom"] = {
"certificates": certificates,
"key": key,
"restart": int(payload.get("restart") or 0),
"filename": str(payload.get("filename") or "pveproxy-ssl.pem"),
}
info = list(certs.get("info") or [])
info = [item for item in info if item.get("filename") != certs["custom"]["filename"]]
info.append(
{
"filename": certs["custom"]["filename"],
"fingerprint": secrets.token_hex(20),
"issuer": "CN=Simulator",
"subject": "CN=pve.local",
"notbefore": int(time.time()) - 86_400,
"notafter": int(time.time()) + 365 * 86_400,
"san": ["DNS:pve.local"],
"public-key-type": "rsa",
"public-key-bits": 2048,
}
)
certs["info"] = info
ops["certificates"] = certs
await save_node_ops(request, node, ops)
return None
certificates = str(payload.get("certificates") or payload.get("cert") or "")
if not certificates:
raise ApiError(400, "parameter verification failed - 'certificates' missing")
key = str(payload.get("key") or payload.get("private-key") or "")
filename = str(payload.get("filename") or "pveproxy-ssl.pem")
fingerprint = ":".join(secrets.token_hex(1) for _ in range(32)) # SHA-256 fingerprint style
summary = {
"filename": filename,
"fingerprint": fingerprint,
"issuer": "CN=Simulator",
"subject": "CN=pve.local",
"notbefore": int(time.time()) - 86_400,
"notafter": int(time.time()) + 365 * 86_400,
"san": ["DNS:pve.local"],
"public-key-type": "rsa",
"public-key-bits": 2048,
"pem": certificates,
}
certs["custom"] = {
"certificates": certificates,
"key": key,
"restart": int(payload.get("restart") or 0),
"filename": filename,
}
info = list(certs.get("info") or [])
info = [item for item in info if item.get("filename") != filename]
info.append({key: value for key, value in summary.items() if key != "pem"})
certs["info"] = info
ops["certificates"] = certs
await save_node_ops(request, node, ops)
return summary
async def certificates_info(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
@@ -333,7 +271,39 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
certs["info"] = info
ops["certificates"] = certs
await save_node_ops(request, node, ops)
return [dict(item) for item in info]
if not info:
from app.simulation.seed import _seed_fingerprint
info = [
{
"filename": "pve-ssl.pem",
"fingerprint": _seed_fingerprint(node),
"issuer": f"CN={node}.proxmox.local",
"subject": f"CN={node}.proxmox.local",
"notbefore": 1_700_000_000,
"notafter": 2_000_000_000,
"pem": (
"-----BEGIN CERTIFICATE-----\n"
"SIMULATOR-SEED-CERT\n"
"-----END CERTIFICATE-----\n"
),
"san": [f"{node}.proxmox.local"],
"public-key-type": "rsa",
"public-key-bits": 4096,
}
]
certs["info"] = info
ops["certificates"] = certs
await save_node_ops(request, node, ops)
result: list[dict[str, Any]] = []
for item in info:
entry = dict(item)
entry.setdefault("pem", "-----BEGIN CERTIFICATE-----\nSIM\n-----END CERTIFICATE-----\n")
entry.setdefault("san", [str(entry.get("subject", "CN=pve.local")).removeprefix("CN=")])
entry.setdefault("public-key-type", "rsa")
entry.setdefault("public-key-bits", 4096)
result.append(entry)
return result
async def scan_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"]))
@@ -360,35 +330,40 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
return subdirs("cpu", "cpu-flags", "machines", "migration")
async def capabilities_cpu(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
return [
{"name": "host", "vendor": "QEMU", "custom": 0},
{"name": "x86-64-v2-AES", "vendor": "QEMU", "custom": 0},
{"name": "kvm64", "vendor": "QEMU", "custom": 0},
]
node = str(values(inputs)["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
caps = ops.get("capabilities")
items = caps.get("cpu") if isinstance(caps, dict) else None
return [dict(item) for item in items] if isinstance(items, list) else []
async def capabilities_cpu_flags(
request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
return [
{"name": "aes", "introduces": "Westmere"},
{"name": "avx", "introduces": "SandyBridge"},
{"name": "avx2", "introduces": "Haswell"},
]
node = str(values(inputs)["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
caps = ops.get("capabilities")
items = caps.get("cpu_flags") if isinstance(caps, dict) else None
return [dict(item) for item in items] if isinstance(items, list) else []
async def capabilities_machines(
request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
return [
{"id": "pc-i440fx-9.0", "type": "i440fx", "version": "9.0"},
{"id": "pc-q35-9.0", "type": "q35", "version": "9.0"},
]
node = str(values(inputs)["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
caps = ops.get("capabilities")
items = caps.get("machines") if isinstance(caps, dict) else None
return [dict(item) for item in items] if isinstance(items, list) else []
async def capabilities_migration(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(request, str(values(inputs)["node"]))
return {"network": "", "type": "secure", "enabled": 1}
node = str(values(inputs)["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
caps = ops.get("capabilities")
migration = caps.get("migration") if isinstance(caps, dict) else None
return dict(migration) if isinstance(migration, dict) else {}
async def hardware_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"]))
@@ -399,8 +374,6 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node)
ops = await load_node_ops(request, node)
hardware = _hardware(ops)
ops["hardware"] = hardware
await save_node_ops(request, node, ops)
return [dict(item) for item in hardware.get("pci", [])]
async def hardware_pci_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -430,8 +403,6 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node)
ops = await load_node_ops(request, node)
hardware = _hardware(ops)
ops["hardware"] = hardware
await save_node_ops(request, node, ops)
return [dict(item) for item in hardware.get("usb", [])]
async def subscription_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -452,7 +423,10 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
current = _subscription(ops)
method = request.method.upper()
if method == "DELETE":
ops["subscription"] = copy.deepcopy(DEFAULT_SUBSCRIPTION)
ops["subscription"] = {
"status": "notfound",
"message": "There is no subscription key",
}
await save_node_ops(request, node, ops)
return None
if method == "POST":
@@ -481,17 +455,8 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
ops = await load_node_ops(request, node)
items = ops.get("aplinfo")
if not isinstance(items, list):
items = [
{
"package": "alpine-3-standard",
"section": "system",
"type": "lxc",
"version": "3.20",
}
]
ops["aplinfo"] = items
await save_node_ops(request, node, ops)
return [dict(item) for item in items]
return []
return [dict(item) for item in items if isinstance(item, dict)]
async def aplinfo_download(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
@@ -514,45 +479,140 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
node = str(payload["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
apt = ops.setdefault("apt", copy.deepcopy(default_node_ops()["apt"]))
repositories = list(apt.get("repositories") or [])
apt = ops.get("apt")
if not isinstance(apt, dict):
apt = {}
ops["apt"] = apt
envelope = apt.get("repositories")
if isinstance(envelope, dict) and "files" in envelope:
files = [
dict(item) for item in list(envelope.get("files") or []) if isinstance(item, dict)
]
standard = [
dict(item)
for item in list(envelope.get("standard-repos") or [])
if isinstance(item, dict)
]
else:
legacy = list(envelope) if isinstance(envelope, list) else []
files = [
{
"path": "/etc/apt/sources.list.d/pve-enterprise.list",
"file-type": "list",
"digest": "seedapt-file0",
"repositories": legacy,
}
]
standard = []
method = request.method.upper()
if method == "POST":
entry = {
handle = payload.get("handle")
if handle is not None:
updated_standard: list[dict[str, Any]] = []
found = False
for item in standard:
if item.get("handle") == handle:
found = True
merged = dict(item)
if "enabled" in payload:
merged["status"] = int(bool(payload.get("enabled")))
updated_standard.append(merged)
else:
updated_standard.append(item)
if not found:
updated_standard.append(
{
"handle": str(handle),
"name": str(payload.get("name") or handle),
"status": int(bool(payload.get("enabled", 1))),
}
)
standard = updated_standard
elif method == "POST":
repo_entry = {
key: value
for key, value in payload.items()
if key not in {"node", "delete", "digest"}
if key not in {"node", "delete", "digest", "path", "handle", "index", "file-type"}
}
entry.setdefault("path", f"/etc/apt/sources.list.d/sim-{secrets.token_hex(2)}.list")
entry.setdefault("enabled", 1)
repositories.append(entry)
else:
path = payload.get("path")
handle = payload.get("handle")
index = payload.get("index")
updated: list[dict[str, Any]] = []
for idx, item in enumerate(repositories):
match = False
if path is not None and item.get("path") == path:
match = True
if handle is not None and item.get("handle") == handle:
match = True
if index is not None and idx == int(index):
match = True
if match or (path is None and handle is None and index is None and idx == 0):
merged = {
**item,
**{
key: value
for key, value in payload.items()
if key not in {"node", "delete", "digest", "path", "handle", "index"}
},
path = str(
payload.get("path") or f"/etc/apt/sources.list.d/sim-{secrets.token_hex(2)}.list"
)
matched = False
for file_entry in files:
if file_entry.get("path") == path:
repos = list(file_entry.get("repositories") or [])
repos.append(repo_entry)
file_entry["repositories"] = repos
matched = True
break
if not matched:
files.append(
{
"path": path,
"file-type": str(payload.get("file-type") or "list"),
"digest": secrets.token_hex(4),
"repositories": [repo_entry],
}
updated.append(merged)
else:
updated.append(item)
repositories = updated
apt["repositories"] = repositories
)
else:
target_path = payload.get("path")
index = payload.get("index")
for file_idx, file_entry in enumerate(files):
if target_path is not None and file_entry.get("path") != target_path:
continue
repos = [
dict(item)
for item in list(file_entry.get("repositories") or [])
if isinstance(item, dict)
]
if index is not None:
idx = int(index)
if 0 <= idx < len(repos):
repos[idx] = {
**repos[idx],
**{
key: value
for key, value in payload.items()
if key
not in {
"node",
"delete",
"digest",
"path",
"handle",
"index",
"file-type",
}
},
}
elif target_path is not None or file_idx == 0:
if repos:
repos[0] = {
**repos[0],
**{
key: value
for key, value in payload.items()
if key
not in {
"node",
"delete",
"digest",
"path",
"handle",
"index",
"file-type",
}
},
}
file_entry["repositories"] = repos
if target_path is not None:
break
apt["repositories"] = {
"digest": secrets.token_hex(4),
"errors": [],
"infos": [],
"standard-repos": standard,
"files": files,
}
ops["apt"] = apt
await save_node_ops(request, node, ops)
@@ -561,18 +621,14 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node)
ops = await load_node_ops(request, node)
config = ops.get("config")
if not isinstance(config, dict):
config = copy.deepcopy(DEFAULT_CONFIG)
ops["config"] = config
await save_node_ops(request, node, ops)
return dict(config)
return dict(config) if isinstance(config, dict) else {}
async def node_config_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
config = dict(ops.get("config") or DEFAULT_CONFIG)
config = dict(ops.get("config") or {})
config.update(
{
key: value
@@ -589,18 +645,14 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node)
ops = await load_node_ops(request, node)
dns = ops.get("dns")
if not isinstance(dns, dict):
dns = copy.deepcopy(DEFAULT_DNS)
ops["dns"] = dns
await save_node_ops(request, node, ops)
return dict(dns)
return dict(dns) if isinstance(dns, dict) else {}
async def dns_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
node = str(payload["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
dns = dict(ops.get("dns") or DEFAULT_DNS)
dns = dict(ops.get("dns") or {})
dns.update(
{
key: value
@@ -616,13 +668,10 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
node = str(values(inputs)["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
current = dict(ops.get("time") or DEFAULT_TIME)
current = dict(ops.get("time") or {})
now = int(time.time())
current["time"] = now
current["localtime"] = now
current.setdefault("timezone", "UTC")
ops["time"] = current
await save_node_ops(request, node, ops)
return current
async def time_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -630,7 +679,7 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
node = str(payload["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
current = dict(ops.get("time") or DEFAULT_TIME)
current = dict(ops.get("time") or {})
if "timezone" in payload:
current["timezone"] = str(payload["timezone"])
now = int(time.time())
@@ -668,12 +717,7 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
ops = await load_node_ops(request, node)
hosts = ops.get("hosts")
if not isinstance(hosts, dict):
hosts = {
"data": f"127.0.0.1 localhost\n10.0.0.10 {node}\n",
"digest": secrets.token_hex(8),
}
ops["hosts"] = hosts
await save_node_ops(request, node, ops)
return {"data": "", "digest": ""}
return {"data": str(hosts.get("data", "")), "digest": str(hosts.get("digest", ""))}
async def hosts_post(request: Request, inputs: dict[str, Any]) -> None:
@@ -692,46 +736,50 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node)
start = int(values(inputs).get("startcursor") or values(inputs).get("start") or 0)
limit = int(values(inputs).get("limit") or 50)
lines = [
f"{index}: {node} systemd[1]: Started simulated service {index}."
for index in range(start, start + limit)
]
return lines
ops = await load_node_ops(request, node)
lines = ops.get("journal")
if not isinstance(lines, list):
return []
sliced = lines[start : start + limit]
return [str(item) for item in sliced]
async def syslog(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
limit = int(values(inputs).get("limit") or 50)
return [
{"n": index, "t": f"{node} kernel: simulated syslog line {index}"}
for index in range(limit)
]
ops = await load_node_ops(request, node)
lines = ops.get("syslog")
if not isinstance(lines, list):
return []
return [dict(item) for item in lines[:limit] if isinstance(item, dict)]
async def netstat(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
return [
{"in": 1_000_000, "out": 900_000, "vnet": "vmbr0", "hwaddr": "bc:24:11:00:00:01"},
{"in": 500_000, "out": 450_000, "vnet": "vmbr1", "hwaddr": "bc:24:11:00:00:02"},
]
ops = await load_node_ops(request, node)
items = ops.get("netstat")
return [dict(item) for item in items] if isinstance(items, list) else []
async def report(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
await require_node(request, node)
return f"==== Proxmox node report for {node} ====\nuptime: simulated\n"
ops = await load_node_ops(request, node)
report_text = ops.get("report")
return str(report_text) if report_text is not None else ""
async def rrd(_request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(_request, str(values(inputs)["node"]))
return {"filename": "/var/lib/rrdcached/db/pve-node.rrd"}
async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
rrd_state = ops.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def rrddata(_request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(_request, str(values(inputs)["node"]))
now = int(time.time())
return [
{"time": now - 120, "cpu": 0.05, "memused": 1_000_000_000},
{"time": now - 60, "cpu": 0.07, "memused": 1_100_000_000},
{"time": now, "cpu": 0.04, "memused": 1_050_000_000},
]
async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"])
await require_node(request, node)
ops = await load_node_ops(request, node)
series = ops.get("rrddata")
return [dict(item) for item in series] if isinstance(series, list) else []
async def startall(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"])
@@ -809,7 +857,7 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node)
ops = await load_node_ops(request, node)
if not isinstance(ops.get("network"), list):
ops["network"] = copy.deepcopy(default_node_ops()["network"])
ops["network"] = []
ops["network_applied"] = False
await save_node_ops(request, node, ops)
@@ -818,28 +866,24 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node)
repo = str(values(inputs).get("repo") or "library/alpine")
ops = await load_node_ops(request, node)
cache = ops.setdefault("oci_tags", {})
if repo not in cache:
cache[repo] = [{"tag": "latest"}, {"tag": "3.20"}]
ops["oci_tags"] = cache
await save_node_ops(request, node, ops)
return [dict(item) for item in cache[repo]]
cache = ops.get("oci_tags")
if not isinstance(cache, dict):
return []
items = cache.get(repo)
if not isinstance(items, list):
return []
return [dict(item) for item in items if isinstance(item, dict)]
async def query_url_metadata(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
await require_node(request, node)
url = str(values(inputs).get("url") or "")
ops = await load_node_ops(request, node)
cache = ops.setdefault("url_metadata", {})
if url not in cache:
cache[url] = {
"filename": url.rsplit("/", 1)[-1] or "download.bin",
"mimetype": "application/octet-stream",
"size": 1024,
}
ops["url_metadata"] = cache
await save_node_ops(request, node, ops)
return dict(cache[url])
cache = ops.get("url_metadata")
if not isinstance(cache, dict):
return {}
payload = cache.get(url)
return dict(payload) if isinstance(payload, dict) else {}
async def vncwebsocket(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"])
+57 -50
View File
@@ -13,53 +13,40 @@ from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs
_SECRET_KEYS = frozenset({"token", "password", "secret"})
DEFAULT_MATCHER_FIELDS = [
{"name": "type", "type": "string"},
{"name": "hostname", "type": "string"},
{"name": "job-id", "type": "string"},
{"name": "severity", "type": "string"},
]
DEFAULT_MATCHER_FIELD_VALUES = [
{"field": "type", "value": "fencing"},
{"field": "type", "value": "package-updates"},
{"field": "type", "value": "replication"},
{"field": "type", "value": "system-mail"},
]
def _public(endpoint: dict[str, Any]) -> dict[str, Any]:
return {key: value for key, value in endpoint.items() if key not in _SECRET_KEYS}
def _notifications(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault(
"notifications",
{
"endpoints": {
"gotify": {},
"sendmail": {},
"smtp": {},
"webhook": {},
},
"matchers": {},
"tests": [],
},
)
current = metadata.get("notifications")
if not isinstance(current, dict):
current = {
return {
"endpoints": {"gotify": {}, "sendmail": {}, "smtp": {}, "webhook": {}},
"matchers": {},
"tests": [],
}
metadata["notifications"] = current
current.setdefault(
"endpoints",
{"gotify": {}, "sendmail": {}, "smtp": {}, "webhook": {}},
endpoints = current.get("endpoints")
if not isinstance(endpoints, dict):
endpoints = {}
result = {
key: value
for key, value in current.items()
if key not in {"endpoints", "matchers", "tests"}
}
result["endpoints"] = {
"gotify": dict(endpoints["gotify"]) if isinstance(endpoints.get("gotify"), dict) else {},
"sendmail": (
dict(endpoints["sendmail"]) if isinstance(endpoints.get("sendmail"), dict) else {}
),
"smtp": dict(endpoints["smtp"]) if isinstance(endpoints.get("smtp"), dict) else {},
"webhook": dict(endpoints["webhook"]) if isinstance(endpoints.get("webhook"), dict) else {},
}
result["matchers"] = (
dict(current["matchers"]) if isinstance(current.get("matchers"), dict) else {}
)
current.setdefault("matchers", {})
current.setdefault("tests", [])
return current
result["tests"] = list(current["tests"]) if isinstance(current.get("tests"), list) else []
return result
def register_notifications_handlers(registry: HandlerRegistry) -> None:
@@ -80,26 +67,28 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
async def list_endpoints(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {})
store = _notifications(metadata)["endpoints"].get(kind) or {}
return [_public({"name": name, **item}) for name, item in sorted(store.items())]
async def create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {})
notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name in store:
raise ApiError(400, f"{kind} endpoint '{name}' already exists")
entry = {key: payload[key] for key in create_keys if key in payload}
entry["name"] = name
entry.setdefault("disable", 0)
store[name] = entry
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {})
store = _notifications(metadata)["endpoints"].get(kind) or {}
if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist")
return _public({"name": name, **store[name]})
@@ -108,7 +97,8 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {})
notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist")
current = dict(store[name])
@@ -123,15 +113,18 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
current[key] = value
current["name"] = name
store[name] = current
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def delete(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {})
notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist")
del store[name]
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
registry.register(base, "GET", list_endpoints)
@@ -149,12 +142,14 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["matchers"]
notifications = _notifications(metadata)
store = notifications["matchers"]
if name in store:
raise ApiError(400, f"matcher '{name}' already exists")
store[name] = {
key: value for key, value in payload.items() if key not in {"delete", "digest"}
}
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def matchers_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -169,7 +164,8 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
name = str(payload["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["matchers"]
notifications = _notifications(metadata)
store = notifications["matchers"]
if name not in store:
raise ApiError(404, "matcher does not exist")
current = dict(store[name])
@@ -183,24 +179,35 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
current[key] = value
current["name"] = name
store[name] = current
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def matchers_delete(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["name"])
metadata = await cluster_metadata(request)
store = _notifications(metadata)["matchers"]
notifications = _notifications(metadata)
store = notifications["matchers"]
if name not in store:
raise ApiError(404, "matcher does not exist")
del store[name]
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
async def matcher_fields(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(DEFAULT_MATCHER_FIELDS)
async def matcher_fields(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
fields = _notifications(metadata).get("matcher_fields")
if isinstance(fields, list):
return [dict(item) for item in fields if isinstance(item, dict)]
return []
async def matcher_field_values(
_request: Request, _inputs: dict[str, Any]
request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]:
return list(DEFAULT_MATCHER_FIELD_VALUES)
metadata = await cluster_metadata(request)
values_list = _notifications(metadata).get("matcher_field_values")
if isinstance(values_list, list):
return [dict(item) for item in values_list if isinstance(item, dict)]
return []
async def targets(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
@@ -231,10 +238,10 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
break
if not found:
raise ApiError(404, "notification target does not exist")
tests = notifications.setdefault("tests", [])
if not isinstance(tests, list):
tests = notifications["tests"] = []
tests = list(notifications.get("tests") or [])
tests.append({"name": name, "tested_at": int(time.time()), "ok": True})
notifications["tests"] = tests
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata)
registry.register("/cluster/notifications", "GET", index)
+37 -11
View File
@@ -10,18 +10,35 @@ from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import database, state, values
from app.handlers.common import database, require_value, state, values
from app.simulation.seed import CLUSTER_ID, stable_id
async def _pool_members(request: Request, pool_id: uuid.UUID) -> list[str]:
async def _pool_members(request: Request, pool_id: uuid.UUID) -> list[dict[str, Any]]:
rows = await database(request).pool.fetch(
"""SELECT r.external_id FROM pool_members pm
"""SELECT r.external_id, r.kind, r.state, n.name AS node
FROM pool_members pm
JOIN resources r ON r.id = pm.resource_id
WHERE pm.pool_id=$1 ORDER BY r.external_id::integer""",
JOIN nodes n ON n.id = r.node_id
WHERE pm.pool_id=$1
ORDER BY r.external_id::integer""",
pool_id,
)
return [str(row["external_id"]) for row in rows]
members: list[dict[str, Any]] = []
for row in rows:
guest_state = state(row["state"])
kind = str(row["kind"])
vmid = int(row["external_id"])
members.append(
{
"id": f"{kind}/{vmid}",
"node": str(row["node"]),
"type": kind,
"vmid": vmid,
"name": str(guest_state.get("name") or f"{kind}-{vmid}"),
}
)
return members
def register_pool_handlers(registry: HandlerRegistry) -> None:
@@ -39,16 +56,25 @@ def register_pool_handlers(registry: HandlerRegistry) -> None:
metadata = state(row["metadata"])
members = await _pool_members(request, row["id"])
if not members and isinstance(metadata.get("members"), list):
members = [str(item) for item in metadata["members"]]
item: dict[str, Any] = {
# Fallback for metadata-only membership before pool_members rows exist.
for item in metadata["members"]:
members.append(
{
"id": f"qemu/{item}",
"node": "",
"type": "qemu",
"vmid": int(item) if str(item).isdigit() else 0,
}
)
pool_item: dict[str, Any] = {
"poolid": str(row["pool_id"]),
"members": members,
}
if row["comment"] is not None:
item["comment"] = str(row["comment"])
pool_item["comment"] = str(row["comment"])
elif metadata.get("comment"):
item["comment"] = str(metadata["comment"])
result.append(item)
pool_item["comment"] = str(metadata["comment"])
result.append(pool_item)
return result
async def pool_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -117,7 +143,7 @@ def register_pool_handlers(registry: HandlerRegistry) -> None:
)
async def pool_delete(request: Request, inputs: dict[str, Any]) -> None:
poolid = str(values(inputs)["poolid"])
poolid = str(require_value(values(inputs), "poolid"))
status = await database(request).pool.execute(
"DELETE FROM pools WHERE pool_id=$1",
poolid,
+225 -76
View File
@@ -33,16 +33,124 @@ def _state(value: object) -> dict[str, Any]:
return dict(cast(Mapping[str, Any], value))
# Runtime-only keys stored in ``resources.state``; never echo on GET /config.
_QEMU_CONFIG_INTERNAL = frozenset(
{
"agent",
"cloudinit_dump",
"config",
"interfaces",
"migrate_preconditions",
"rrd",
"rrddata",
"sendkey",
"status",
}
)
def _agent_config_string(value: object) -> str:
"""Proxmox QEMU config ``agent`` is a string (e.g. ``1`` / ``0``), not an object."""
if isinstance(value, dict):
return "1" if value.get("enabled", True) else "0"
if isinstance(value, bool):
return "1" if value else "0"
if value in (None, ""):
return "0"
text = str(value).strip()
return text or "0"
def _public_qemu_config(config: dict[str, Any], state: dict[str, Any]) -> dict[str, Any]:
"""Wire-format QEMU config for clients (bpg / pulumi-proxmoxve)."""
import hashlib
merged = {**state, **config}
payload: dict[str, Any] = {}
for key, value in merged.items():
if key in _QEMU_CONFIG_INTERNAL:
continue
if isinstance(value, dict | list):
continue
payload[key] = value
agent = config.get("agent", state.get("agent", "0"))
payload["agent"] = _agent_config_string(agent)
digest_src = {key: value for key, value in payload.items() if key != "digest"}
payload["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(digest_src, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
return payload
def _cpu_utilization(vm_state: Mapping[str, Any], *, running: bool, idle: float = 0.1) -> float:
"""Status ``cpu`` is utilization; config may store model strings like ``qemu64``."""
raw = vm_state.get("cpu")
if isinstance(raw, (int, float)):
return float(raw)
if isinstance(raw, str):
try:
return float(raw)
except ValueError:
pass
return idle if running else 0.0
def _qemu_index_item(vmid: int, vm_state: dict[str, Any], config: dict[str, Any]) -> dict[str, Any]:
status = str(vm_state.get("status", "stopped"))
running = status in {"running", "paused"}
memory_mb = int(config.get("memory", config.get("mem", 2048)))
maxmem = memory_mb * 2**20
mem = int(vm_state.get("mem", maxmem // 2 if running else 0))
return {
"vmid": vmid,
"name": str(config.get("name", f"vm-{vmid}")),
"status": status,
"qmpstatus": status if running else "stopped",
"cpus": int(config.get("cores", config.get("cpus", 1))),
"cpu": _cpu_utilization(vm_state, running=running),
"maxmem": maxmem,
"mem": mem,
"memhost": int(vm_state.get("memhost", mem)),
"maxdisk": int(vm_state.get("maxdisk", config.get("maxdisk", 32 * 1024**3))),
"disk": int(vm_state.get("disk", 0)),
"diskread": int(vm_state.get("diskread", 0)),
"diskwrite": int(vm_state.get("diskwrite", 0)),
"netin": int(vm_state.get("netin", 0)),
"netout": int(vm_state.get("netout", 0)),
"uptime": int(vm_state.get("uptime", 0)) if running else 0,
"pid": int(vm_state.get("pid", 12_345 if running else 0)),
"lock": str(vm_state.get("lock", "")),
"tags": str(config.get("tags", "")),
"template": int(bool(vm_state.get("template", False))),
"running-qemu": str(vm_state.get("running-qemu", "9.2.0")) if running else "",
"running-machine": str(vm_state.get("running-machine", "pc-i440fx-9.2")) if running else "",
"pressurecpusome": float(vm_state.get("pressurecpusome", 0.0)),
"pressurecpufull": float(vm_state.get("pressurecpufull", 0.0)),
"pressureiosome": float(vm_state.get("pressureiosome", 0.0)),
"pressureiofull": float(vm_state.get("pressureiofull", 0.0)),
"pressurememorysome": float(vm_state.get("pressurememorysome", 0.0)),
"pressurememoryfull": float(vm_state.get("pressurememoryfull", 0.0)),
}
def register_qemu_handlers(registry: HandlerRegistry) -> None:
async def qemu_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(_values(inputs)["node"])
rows = await _database(request).pool.fetch(
"""SELECT r.external_id::integer AS vmid, r.state
FROM resources r JOIN nodes n ON n.id=r.node_id
"""SELECT r.external_id::integer AS vmid, r.state, v.config
FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN virtual_machines v ON v.resource_id=r.id
WHERE n.name=$1 AND r.kind='qemu' ORDER BY r.external_id::integer""",
node,
)
return [{"vmid": int(row["vmid"]), **_state(row["state"])} for row in rows]
return [
_qemu_index_item(int(row["vmid"]), _state(row["state"]), _state(row["config"]))
for row in rows
]
async def qemu_status_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs)
@@ -82,28 +190,28 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
else 0,
)
)
return {
"vmid": vmid,
"name": str(config.get("name", f"vm-{vmid}")),
"status": status,
"qmpstatus": status if running else "stopped",
"lock": str(vm_state.get("lock", "")),
"pid": int(vm_state.get("pid", 12_345 if running else 0)),
"cpus": int(config.get("cores", config.get("cpus", 1))),
"maxmem": maxmem,
"mem": mem_used,
"balloon": int(vm_state.get("balloon", 0)),
"ballooninfo": {
"actual": mem_used,
"max_mem": maxmem,
"mem_swapped_in": 0,
"mem_swapped_out": 0,
},
"uptime": uptime,
"template": int(bool(vm_state.get("template", False))),
"ha": {"managed": int(vm_state.get("ha_managed", 0))},
"agent": 1 if running and str(config.get("agent", "0")).startswith("1") else 0,
}
agent_enabled = str(config.get("agent", "0")).startswith("1")
item = _qemu_index_item(vmid, vm_state, config)
item.update(
{
"uptime": uptime,
"mem": mem_used,
"balloon": int(vm_state.get("balloon", 0)),
"ballooninfo": {
"actual": mem_used,
"max_mem": maxmem,
"mem_swapped_in": 0,
"mem_swapped_out": 0,
},
"template": bool(vm_state.get("template", False)),
"ha": {"managed": int(vm_state.get("ha_managed", 0))},
"agent": bool(running and agent_enabled),
"clipboard": str(vm_state.get("clipboard", "")),
"spice": bool(vm_state.get("spice", False)),
"serial": bool(vm_state.get("serial", False)),
}
)
return item
async def qemu_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"])
@@ -117,7 +225,10 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
)
if row is None:
raise ApiError(404, "virtual machine does not exist")
return {"vmid": int(vmid), **_state(row["config"]), **_state(row["state"])}
return {
"vmid": int(vmid),
**_public_qemu_config(_state(row["config"]), _state(row["state"])),
}
async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
@@ -291,24 +402,30 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
rows = await _database(request).pool.fetch(
"""SELECT name, parent_name, description, created_at FROM snapshots
"""SELECT name, parent_name, description, created_at, state FROM snapshots
WHERE resource_id=$1 ORDER BY created_at, name""",
resource["id"],
)
return [
{
"name": row["name"],
"parent": row["parent_name"],
"description": row["description"] or "",
"snaptime": int(row["created_at"].timestamp()),
}
for row in rows
]
result: list[dict[str, Any]] = []
for row in rows:
raw_state = row["state"] if "state" in row else None
snap_state = _state(raw_state) if raw_state is not None else {}
result.append(
{
"name": row["name"],
"parent": row["parent_name"],
"description": row["description"] or "",
"snaptime": int(row["created_at"].timestamp()),
"vmstate": int(bool(snap_state.get("vmstate"))),
}
)
return result
async def snapshot_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
row = await _snapshot(request, values)
state = _state(row["state"])
raw_state = row["state"] if "state" in row else None
state = _state(raw_state) if raw_state is not None else {}
return {
"name": row["name"],
"parent": row["parent_name"],
@@ -406,7 +523,7 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
async def migrate_preconditions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
target = values.get("target")
if target in {None, ""}:
raise ApiError(400, "parameter 'target' is required")
@@ -416,7 +533,13 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
)
if not exists:
raise ApiError(404, "target node does not exist")
return {"local_disks": [], "local_resources": [], "running": False}
state = _state(resource["state"])
pre = state.get("migrate_preconditions")
payload = dict(pre) if isinstance(pre, dict) else {}
payload["running"] = str(state.get("status") or "") == "running"
payload.setdefault("local_disks", [])
payload.setdefault("local_resources", [])
return payload
async def migrate(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
@@ -474,7 +597,7 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
},
)
async def resize(request: Request, inputs: dict[str, Any]) -> None:
async def resize(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
node, vmid, disk = str(values["node"]), str(values["vmid"]), str(values["disk"])
resource = await _qemu_resource(request, node, vmid)
@@ -507,6 +630,13 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
str(config[disk]).split(":", 1)[0],
size,
)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-resize",
payload={"resource_id": str(resource["id"]), "disk": disk, "size": size},
)
async def move_disk(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
@@ -544,39 +674,16 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
command: str, request: Request, inputs: dict[str, Any]
) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
config = _state(resource["config"])
vmid = str(_values(inputs)["vmid"])
results: dict[str, Any] = {
"info": {
"version": "9.2.0-simulator",
"supported_commands": [
{"name": name, "enabled": True, "success-response": True}
for name in ("guest-ping", "guest-info", "guest-get-osinfo")
],
},
"get-osinfo": {
"name": str(config.get("ostype", "linux")),
"pretty-name": "Proxmox Simulator Guest",
"version": "1.0",
"machine": "x86_64",
},
"get-host-name": {"host-name": str(config.get("name", f"vm-{vmid}"))},
"network-get-interfaces": [
{
"name": "eth0",
"hardware-address": "02:00:00:00:00:01",
"ip-addresses": [
{"ip-address": "192.0.2.10", "ip-address-type": "ipv4", "prefix": 24}
],
}
],
"ping": {},
}
if command == "get-time":
seconds = int(
await _database(request).pool.fetchval("SELECT extract(epoch from now())::bigint")
)
return {"result": {"seconds": seconds, "nanoseconds": 0}}
state = _state(resource["state"])
agent = state.get("agent")
results = agent.get("results") if isinstance(agent, dict) else None
if not isinstance(results, dict) or command not in results:
raise ApiError(404, f"agent command '{command}' has no stored result")
return {"result": results[command]}
async def agent_info(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -586,6 +693,12 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
return await agent_result("get-osinfo", request, inputs)
async def agent_hostname(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
config = _state(resource["config"])
name = state.get("name") or config.get("name")
if name:
return {"result": {"host-name": str(name)}}
return await agent_result("get-host-name", request, inputs)
async def agent_network(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -598,12 +711,32 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
return await agent_result("ping", request, inputs)
async def task_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
tasks = await TaskRepository(_database(request).pool).list_for_node(
str(_values(inputs)["node"])
)
return [
{"upid": task.upid, "status": task.status, "type": task.task_type} for task in tasks
]
from app.tasks.upid import Upid
node = str(_values(inputs)["node"])
tasks = await TaskRepository(_database(request).pool).list_for_node(node)
result: list[dict[str, Any]] = []
for task in tasks:
try:
parsed = Upid.parse(task.upid)
except ValueError:
continue
finished = task.status in {"success", "error", "cancelled"}
entry: dict[str, Any] = {
"upid": task.upid,
"id": parsed.task_id,
"node": parsed.node,
"pid": parsed.pid,
"pstart": parsed.process_start,
"starttime": parsed.start_time,
"type": parsed.task_type,
"user": parsed.user,
}
if finished:
entry["status"] = "OK" if task.status == "success" else str(task.status).upper()
entry["endtime"] = parsed.start_time + max(task.progress, 1)
result.append(entry)
return result
async def task_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
task = await TaskRepository(_database(request).pool).get_by_upid(
@@ -644,7 +777,7 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
}
}
async def qemu_template(request: Request, inputs: dict[str, Any]) -> None:
async def qemu_template(request: Request, inputs: dict[str, Any]) -> str:
payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"])
resource = await _qemu_resource(request, node, vmid)
@@ -661,6 +794,13 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
resource["id"],
json.dumps(state, sort_keys=True),
)
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-template",
payload={"resource_id": str(resource["id"])},
)
async def qemu_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs)
@@ -778,6 +918,8 @@ async def _create_task(
"qemu-migrate": "qmigrate",
"qemu-remote-migrate": "qmremote",
"qemu-move-disk": "qmmove",
"qemu-resize": "qmresize",
"qemu-template": "qmtemplate",
}[task_type]
upid = str(Upid.allocate(node, worker_type, vmid, str(request.state.principal)))
try:
@@ -825,7 +967,14 @@ async def _agent_resource(request: Request, values: dict[str, Any]) -> Any:
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
config = _state(resource["config"])
state = _state(resource["state"])
if str(config.get("agent", "0")).split(",", 1)[0].lower() not in {"1", "true", "yes"}:
agent = state.get("agent")
if not isinstance(agent, dict):
agent = config.get("agent")
if isinstance(agent, dict):
enabled = bool(agent.get("enabled", True))
else:
enabled = str(config.get("agent", "0")).split(",", 1)[0].lower() in {"1", "true", "yes"}
if not enabled:
raise ApiError(409, "QEMU guest agent is not enabled")
if state.get("status") != "running":
raise ApiError(409, "QEMU guest agent is not running")
+23 -49
View File
@@ -79,20 +79,10 @@ def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
async def _agent_blob(command: str, request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
agent = state.setdefault("agent", {})
blobs = agent.setdefault("results", {})
defaults: dict[str, Any] = {
"get-users": [{"user": "root", "login-time": 0}],
"get-fsinfo": [{"name": "/", "type": "ext4", "total-bytes": 32 * 1024**3}],
"get-memory-block-info": {"size": 1024**3},
"get-memory-blocks": [{"start": 0, "size": 1024**3}],
"get-timezone": {"zone": "UTC", "offset": 0},
"get-vcpus": [{"online": True, "can-offline": False}],
"fsfreeze-status": "thawed",
}
if command not in blobs:
blobs[command] = defaults.get(command, {})
await _save_guest_state(request, resource["id"], state)
agent = state.get("agent")
blobs = agent.get("results") if isinstance(agent, dict) else None
if not isinstance(blobs, dict) or command not in blobs:
raise ApiError(404, f"agent command '{command}' has no stored result")
return {"result": blobs[command]}
async def agent_users(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -146,10 +136,10 @@ def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
resource = await _agent_resource(request, payload)
path = str(payload.get("file") or payload.get("path") or "/etc/hostname")
state = _state(resource["state"])
files = state.setdefault("agent", {}).setdefault("files", {})
if path not in files:
files[path] = f"simulated:{path}\n"
await _save_guest_state(request, resource["id"], state)
agent = state.get("agent")
files = agent.get("files") if isinstance(agent, dict) else None
if not isinstance(files, dict) or path not in files:
raise ApiError(404, "file does not exist")
content = str(files[path])
return {"result": {"content": content, "truncated": True}}
@@ -188,9 +178,15 @@ def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
async def agent_fstrim(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
state.setdefault("agent", {})["last_fstrim"] = True
agent = dict(state.get("agent") or {})
agent["last_fstrim"] = True
state["agent"] = agent
await _save_guest_state(request, resource["id"], state)
return {"result": {"paths": [{"path": "/", "trimmed": 0}]}}
results = agent.get("results") if isinstance(agent.get("results"), dict) else {}
fstrim = results.get("fstrim") if isinstance(results, dict) else None
if isinstance(fstrim, dict):
return {"result": fstrim}
return {"result": {}}
async def agent_set_password(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs)
@@ -258,45 +254,23 @@ def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
async def cloudinit_dump(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
config = _state(resource["config"])
return (
f"#cloud-config\nhostname: {config.get('name', values['vmid'])}\n"
f"manage_etc_hosts: true\n"
)
state = _state(resource["state"])
dump = state.get("cloudinit_dump")
return str(dump) if dump is not None else ""
async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
rrd_state = state.setdefault("rrd", {"filename": f"pve-vm-{values['vmid']}.rrd"})
await _save_guest_state(request, resource["id"], state)
return dict(rrd_state)
rrd_state = state.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"])
series = state.setdefault(
"rrddata",
[
{
"time": 1_700_000_000,
"cpu": 0.05,
"mem": 256 * 1024 * 1024,
"netin": 0,
"netout": 0,
},
{
"time": 1_700_000_060,
"cpu": 0.08,
"mem": 260 * 1024 * 1024,
"netin": 100,
"netout": 80,
},
],
)
await _save_guest_state(request, resource["id"], state)
return list(series)
series = state.get("rrddata")
return [dict(item) for item in series] if isinstance(series, list) else []
async def monitor(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs)
+180 -122
View File
@@ -21,65 +21,77 @@ _SECRET_KEYS = frozenset({"key", "token", "fingerprint"})
def _sdn(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault(
"sdn",
{
"zones": {},
"vnets": {},
"controllers": {},
"dns": {},
"ipams": {},
"fabrics": {},
"fabric_nodes": {},
"prefix_lists": {},
"route_maps": {},
"lock": None,
"pending": False,
"running_version": 1,
},
)
if not isinstance(current, dict):
current = {
"zones": {},
"vnets": {},
"controllers": {},
"dns": {},
"ipams": {},
"fabrics": {},
"fabric_nodes": {},
"prefix_lists": {},
"route_maps": {},
"lock": None,
"pending": False,
"running_version": 1,
}
metadata["sdn"] = current
for key in (
"zones",
"vnets",
"controllers",
current = metadata.get("sdn")
return current if isinstance(current, dict) else {}
_VNET_LIST_KEYS = frozenset(
{
"vnet",
"type",
"zone",
"alias",
"tag",
"vlanaware",
"digest",
"isolate-ports",
"pending",
"state",
}
)
_ZONE_LIST_KEYS = frozenset(
{
"zone",
"type",
"bridge",
"mtu",
"nodes",
"ipam",
"dns",
"ipams",
"fabrics",
"fabric_nodes",
"prefix_lists",
"route_maps",
):
current.setdefault(key, {})
return current
"reversedns",
"vlan-protocol",
"digest",
"pending",
"state",
"comment",
}
)
def _public(item: dict[str, Any]) -> dict[str, Any]:
return {key: value for key, value in item.items() if key not in _SECRET_KEYS}
def _vnet_list_item(name: str, item: dict[str, Any]) -> dict[str, Any]:
payload = {key: value for key, value in item.items() if key in _VNET_LIST_KEYS}
payload["vnet"] = str(payload.get("vnet") or name)
payload.setdefault("type", "vnet")
return _public(payload)
def _zone_list_item(name: str, item: dict[str, Any]) -> dict[str, Any]:
payload = {key: value for key, value in item.items() if key in _ZONE_LIST_KEYS}
payload["zone"] = str(payload.get("zone") or name)
return _public(payload)
def _store_list(store: dict[str, Any], *, id_key: str) -> list[dict[str, Any]]:
if id_key == "vnet":
return [_vnet_list_item(name, item) for name, item in sorted(store.items())]
if id_key == "zone":
return [_zone_list_item(name, item) for name, item in sorted(store.items())]
return [_public({id_key: name, **item}) for name, item in sorted(store.items())]
async def _load(request: Request) -> tuple[dict[str, Any], dict[str, Any]]:
async def _load(
request: Request, *, for_write: bool = False
) -> tuple[dict[str, Any], dict[str, Any]]:
metadata = await cluster_metadata(request)
return metadata, _sdn(metadata)
sdn = _sdn(metadata)
if for_write and metadata.get("sdn") is not sdn:
metadata["sdn"] = sdn
return metadata, sdn
def register_sdn_handlers(registry: HandlerRegistry) -> None:
@@ -98,9 +110,11 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
"zones",
)
async def apply(request: Request, inputs: dict[str, Any]) -> None:
async def apply(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.cluster_extra import _cluster_task
payload = values(inputs)
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
lock = sdn.get("lock")
token = payload.get("lock-token")
if lock and token and lock.get("token") != token:
@@ -110,19 +124,20 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
if payload.get("release-lock"):
sdn["lock"] = None
await save_cluster_metadata(request, metadata)
return await _cluster_task(request, task_type="sdn-apply", worker="sdnapply")
async def lock_create(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
metadata, sdn = await _load(request)
async def lock_create(request: Request, inputs: dict[str, Any]) -> str:
metadata, sdn = await _load(request, for_write=True)
if sdn.get("lock") and not values(inputs).get("allow-pending"):
raise ApiError(400, "SDN is already locked")
token = secrets.token_hex(8)
sdn["lock"] = {"token": token}
await save_cluster_metadata(request, metadata)
return {"digest": token, "token": token}
return token
async def lock_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
lock = sdn.get("lock")
if lock is None:
return None
@@ -133,14 +148,14 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def rollback(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
sdn["pending"] = False
if payload.get("release-lock"):
sdn["lock"] = None
await save_cluster_metadata(request, metadata)
async def dry_run(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
return [
{"type": "zone", "name": name, "action": "noop"}
for name in sorted(sdn.get("zones") or {})
@@ -154,7 +169,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
create_required: str | None = None,
) -> None:
async def list_items(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
items = _store_list(sdn.get(store_key) or {}, id_key=id_param)
type_filter = values(inputs).get("type")
if type_filter:
@@ -164,7 +179,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
item_id = str(payload[create_required or id_param])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault(store_key, {})
if item_id in store:
raise ApiError(400, f"{store_key} '{item_id}' already exists")
@@ -179,7 +194,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
item_id = str(values(inputs)[id_param])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = (sdn.get(store_key) or {}).get(item_id)
if not isinstance(item, dict):
raise ApiError(404, f"{store_key} entry does not exist")
@@ -188,7 +203,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
item_id = str(payload[id_param])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault(store_key, {})
if item_id not in store:
raise ApiError(404, f"{store_key} entry does not exist")
@@ -208,7 +223,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def delete(request: Request, inputs: dict[str, Any]) -> None:
item_id = str(values(inputs)[id_param])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault(store_key, {})
if item_id not in store:
raise ApiError(404, f"{store_key} entry does not exist")
@@ -230,20 +245,20 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def ipam_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
ipam = str(values(inputs)["ipam"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
if ipam not in (sdn.get("ipams") or {}):
raise ApiError(404, "ipam does not exist")
return {"status": "ok", "ipam": ipam}
# vnets + nested
async def vnets_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
return _store_list(sdn.get("vnets") or {}, id_key="vnet")
async def vnets_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("vnets", {})
if vnet in store:
raise ApiError(400, f"vnet '{vnet}' already exists")
@@ -259,7 +274,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def vnet_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
vnet = str(values(inputs)["vnet"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("vnets") or {}).get(vnet)
if not isinstance(item, dict):
raise ApiError(404, "vnet does not exist")
@@ -268,7 +283,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def vnet_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("vnets", {})
if vnet not in store:
raise ApiError(404, "vnet does not exist")
@@ -288,7 +303,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def vnet_delete(request: Request, inputs: dict[str, Any]) -> None:
vnet = str(values(inputs)["vnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("vnets", {})
if vnet not in store:
raise ApiError(404, "vnet does not exist")
@@ -307,7 +322,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def subnets_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
vnet = str(values(inputs)["vnet"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
return _store_list(item.get("subnets") or {}, id_key="subnet")
@@ -315,7 +330,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
subnet = str(payload["subnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
subnets = item.setdefault("subnets", {})
if subnet in subnets:
@@ -331,7 +346,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
subnet = str(payload["subnet"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
data = (item.get("subnets") or {}).get(subnet)
if not isinstance(data, dict):
@@ -342,7 +357,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
subnet = str(payload["subnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
subnets = item.setdefault("subnets", {})
if subnet not in subnets:
@@ -365,7 +380,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
subnet = str(payload["subnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
subnets = item.setdefault("subnets", {})
if subnet not in subnets:
@@ -377,7 +392,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def ips_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
ips = item.setdefault("ips", [])
if not isinstance(ips, list):
@@ -398,7 +413,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def ips_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
ips = item.setdefault("ips", [])
if not isinstance(ips, list):
@@ -416,14 +431,14 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fw_options_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
vnet = str(values(inputs)["vnet"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
return dict(item.get("firewall", {}).get("options") or {"enable": 0})
async def fw_options_put(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
options = dict(item.setdefault("firewall", {}).setdefault("options", {"enable": 0}))
for key, value in payload.items():
@@ -435,7 +450,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fw_rules_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
vnet = str(values(inputs)["vnet"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
rules = item.get("firewall", {}).get("rules") or []
return list(rules) if isinstance(rules, list) else []
@@ -443,7 +458,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fw_rules_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
rules = item.setdefault("firewall", {}).setdefault("rules", [])
if not isinstance(rules, list):
@@ -464,7 +479,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
pos = int(payload["pos"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
rules = item.setdefault("firewall", {}).setdefault("rules", [])
if not isinstance(rules, list) or pos < 0 or pos >= len(rules):
@@ -479,7 +494,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
vnet = str(payload["vnet"])
pos = int(payload["pos"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet)
rules = item.setdefault("firewall", {}).setdefault("rules", [])
if not isinstance(rules, list) or pos < 0 or pos >= len(rules):
@@ -492,7 +507,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
return subdirs("all", "fabric", "node")
async def fabrics_all(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
return _store_list(sdn.get("fabrics") or {}, id_key="id")
async def fabric_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
@@ -501,7 +516,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fabric_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
fabric_id = str(payload["id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabrics", {})
if fabric_id in store:
raise ApiError(400, f"fabric '{fabric_id}' already exists")
@@ -514,7 +529,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fabric_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
fabric_id = str(values(inputs)["id"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("fabrics") or {}).get(fabric_id)
if not isinstance(item, dict):
raise ApiError(404, "fabric does not exist")
@@ -523,7 +538,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fabric_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
fabric_id = str(payload["id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabrics", {})
if fabric_id not in store:
raise ApiError(404, "fabric does not exist")
@@ -543,7 +558,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fabric_delete(request: Request, inputs: dict[str, Any]) -> None:
fabric_id = str(values(inputs)["id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabrics", {})
if fabric_id not in store:
raise ApiError(404, "fabric does not exist")
@@ -554,7 +569,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
await save_cluster_metadata(request, metadata)
async def fabric_nodes_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
fabric_id = values(inputs).get("fabric_id")
nodes = sdn.get("fabric_nodes") or {}
result: list[dict[str, Any]] = []
@@ -571,7 +586,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
fabric_id = str(payload["fabric_id"])
node_id = str(payload["node_id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
if fabric_id not in (sdn.get("fabrics") or {}):
raise ApiError(404, "fabric does not exist")
store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {})
@@ -588,7 +603,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
fabric_id = str(payload["fabric_id"])
node_id = str(payload["node_id"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = ((sdn.get("fabric_nodes") or {}).get(fabric_id) or {}).get(node_id)
if not isinstance(item, dict):
raise ApiError(404, "fabric node does not exist")
@@ -598,7 +613,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
fabric_id = str(payload["fabric_id"])
node_id = str(payload["node_id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {})
if node_id not in store:
raise ApiError(404, "fabric node does not exist")
@@ -620,7 +635,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
fabric_id = str(payload["fabric_id"])
node_id = str(payload["node_id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {})
if node_id not in store:
raise ApiError(404, "fabric node does not exist")
@@ -630,13 +645,13 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
# prefix lists
async def prefix_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
return _store_list(sdn.get("prefix_lists") or {}, id_key="id")
async def prefix_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
list_id = str(payload["id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("prefix_lists", {})
if list_id in store:
raise ApiError(400, f"prefix-list '{list_id}' already exists")
@@ -654,7 +669,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def prefix_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
list_id = str(values(inputs)["id"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist")
@@ -663,7 +678,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def prefix_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
list_id = str(payload["id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("prefix_lists", {})
if list_id not in store:
raise ApiError(404, "prefix-list does not exist")
@@ -676,7 +691,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def prefix_delete(request: Request, inputs: dict[str, Any]) -> None:
list_id = str(values(inputs)["id"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("prefix_lists", {})
if list_id not in store:
raise ApiError(404, "prefix-list does not exist")
@@ -686,7 +701,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def prefix_entries(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
list_id = str(values(inputs)["id"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist")
@@ -699,7 +714,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
list_id = str(payload["id"])
seq = str(payload.get("seq") or secrets.randbelow(10000))
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist")
@@ -720,7 +735,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
list_id = str(payload["id"])
seq = str(payload["url_seq"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist")
@@ -733,7 +748,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
list_id = str(payload["id"])
seq = str(payload["url_seq"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist")
@@ -752,7 +767,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
list_id = str(payload["id"])
seq = str(payload["url_seq"])
metadata, sdn = await _load(request)
metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist")
@@ -767,16 +782,44 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def route_maps_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("entries")
def _route_map_entries(item: object) -> dict[str, Any]:
if not isinstance(item, dict):
return {}
nested = item.get("entries")
if isinstance(nested, dict):
return {
str(key): value
for key, value in nested.items()
if isinstance(value, dict) and str(key).isdigit()
}
return {
str(key): value
for key, value in item.items()
if isinstance(value, dict) and str(key).isdigit()
}
def _route_map_entries_mutable(sdn: dict[str, Any], map_id: str) -> dict[str, Any]:
bucket = sdn.setdefault("route_maps", {}).setdefault(map_id, {})
if not isinstance(bucket, dict):
bucket = {}
sdn["route_maps"][map_id] = bucket
if "entries" in bucket or "name" in bucket:
entries = bucket.setdefault("entries", {})
if not isinstance(entries, dict):
entries = {}
bucket["entries"] = entries
return entries
return bucket
async def route_entries_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
maps = sdn.get("route_maps") or {}
route_map_id = values(inputs).get("route-map-id")
result: list[dict[str, Any]] = []
for map_id, entries in sorted(maps.items()):
for map_id, raw_entries in sorted(maps.items()):
if route_map_id and map_id != route_map_id:
continue
if not isinstance(entries, dict):
continue
entries = _route_map_entries(raw_entries)
for order, entry in sorted(entries.items(), key=lambda pair: int(pair[0])):
result.append({"route-map-id": map_id, "order": int(order), **entry})
return result
@@ -785,8 +828,8 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
map_id = str(payload["route-map-id"])
order = str(payload.get("order") or 10)
metadata, sdn = await _load(request)
entries = sdn.setdefault("route_maps", {}).setdefault(map_id, {})
metadata, sdn = await _load(request, for_write=True)
entries = _route_map_entries_mutable(sdn, map_id)
if order in entries:
raise ApiError(400, f"route-map entry '{order}' already exists")
entries[order] = {
@@ -809,8 +852,8 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
map_id = str(payload["route-map-id"])
order = str(payload["order"])
_metadata, sdn = await _load(request)
entry = ((sdn.get("route_maps") or {}).get(map_id) or {}).get(order)
_metadata, sdn = await _load(request, for_write=True)
entry = _route_map_entries((sdn.get("route_maps") or {}).get(map_id)).get(order)
if not isinstance(entry, dict):
raise ApiError(404, "route-map entry does not exist")
return {"route-map-id": map_id, "order": int(order), **entry}
@@ -819,8 +862,8 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
map_id = str(payload["route-map-id"])
order = str(payload["order"])
metadata, sdn = await _load(request)
entries = sdn.setdefault("route_maps", {}).setdefault(map_id, {})
metadata, sdn = await _load(request, for_write=True)
entries = _route_map_entries_mutable(sdn, map_id)
if order not in entries:
raise ApiError(404, "route-map entry does not exist")
current = dict(entries[order])
@@ -837,8 +880,8 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
map_id = str(payload["route-map-id"])
order = str(payload["order"])
metadata, sdn = await _load(request)
entries = sdn.setdefault("route_maps", {}).setdefault(map_id, {})
metadata, sdn = await _load(request, for_write=True)
entries = _route_map_entries_mutable(sdn, map_id)
if order not in entries:
raise ApiError(404, "route-map entry does not exist")
del entries[order]
@@ -852,13 +895,13 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def node_zones(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
return _store_list(sdn.get("zones") or {}, id_key="zone")
async def node_zone(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(request, str(values(inputs)["node"]))
zone = str(values(inputs)["zone"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("zones") or {}).get(zone)
if not isinstance(item, dict):
raise ApiError(404, "zone does not exist")
@@ -866,13 +909,16 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def node_zone_bridges(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
zone = await node_zone(request, inputs)
bridge = zone.get("bridge") or f"vmbr-{zone.get('zone')}"
return [{"iface": bridge, "active": 1}]
bridges = zone.get("bridges")
if isinstance(bridges, list):
return [dict(item) for item in bridges if isinstance(item, dict)]
bridge = zone.get("bridge")
return [{"iface": bridge, "active": 1}] if bridge else []
async def node_zone_content(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
zone = str(values(inputs)["zone"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
return [
{"vnet": name, **item}
for name, item in sorted((sdn.get("vnets") or {}).items())
@@ -881,7 +927,11 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def node_zone_ip_vrf(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
zone = await node_zone(request, inputs)
return {"zone": zone.get("zone"), "vrf": f"vrf-{zone.get('zone')}", "table": 100}
return {
"zone": zone.get("zone"),
"vrf": zone.get("vrf") or "",
"table": zone.get("table") if zone.get("table") is not None else 0,
}
async def node_vnet(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(request, str(values(inputs)["node"]))
@@ -889,7 +939,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def node_vnet_mac_vrf(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
vnet = await node_vnet(request, inputs)
return {"vnet": vnet.get("vnet"), "mac-vrf": f"macvrf-{vnet.get('vnet')}"}
return {"vnet": vnet.get("vnet"), "mac-vrf": vnet.get("mac-vrf") or ""}
async def node_fabric(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(request, str(values(inputs)["node"]))
@@ -901,7 +951,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
fabric = str(values(inputs)["fabric"])
_metadata, sdn = await _load(request)
_metadata, sdn = await _load(request, for_write=True)
nodes = (sdn.get("fabric_nodes") or {}).get(fabric) or {}
result = []
for node_id, item in nodes.items():
@@ -919,15 +969,23 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
fabric = str(values(inputs)["fabric"])
_metadata, sdn = await _load(request)
nodes = (sdn.get("fabric_nodes") or {}).get(fabric) or {}
return [{"node": node_id, "state": "up"} for node_id in sorted(nodes)]
return [
{
"node": node_id,
"state": item.get("state") if isinstance(item, dict) else "",
}
for node_id, item in sorted(nodes.items())
]
async def node_fabric_routes(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"]))
fabric = str(values(inputs)["fabric"])
_metadata, sdn = await _load(request)
item = (sdn.get("fabrics") or {}).get(fabric) or {}
prefix = item.get("ip_prefix") or "10.0.0.0/24"
return [{"dst": prefix, "protocol": item.get("protocol") or "ospf"}]
routes = item.get("routes") if isinstance(item, dict) else None
if isinstance(routes, list):
return [dict(route) for route in routes if isinstance(route, dict)]
return []
# registrations
registry.register("/cluster/sdn", "GET", index)
+110 -27
View File
@@ -9,7 +9,15 @@ from fastapi import Request
from app.api.errors import ApiError
from app.api.registry import HandlerRegistry
from app.handlers.common import database, require_node, state, storage_payload, subdirs, values
from app.handlers.common import (
database,
require_node,
require_value,
state,
storage_payload,
subdirs,
values,
)
from app.simulation.seed import CLUSTER_ID, stable_id
@@ -140,6 +148,7 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
JOIN resources r ON r.id = s.resource_id
JOIN nodes n ON n.id = r.node_id
WHERE n.name=$1 OR s.shared = true
OR s.storage_id IN ('local', 'local-lvm')
ORDER BY s.storage_id""",
node,
)
@@ -286,7 +295,55 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
if status != "DELETE 1":
raise ApiError(404, "volume does not exist")
async def node_storage_upload(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
async def node_storage_allocate(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import _node_task
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
payload = values(inputs)
filename = str(payload.get("filename") or f"vm-{payload.get('vmid', '0')}-disk-0")
content_type = str(payload.get("content") or "images")
size_raw = str(payload.get("size") or "1G")
# Accept either integer bytes or Proxmox size strings like 32G.
try:
size = int(size_raw)
except ValueError:
from app.handlers.common import parse_size_bytes
try:
size = parse_size_bytes(size_raw)
except ValueError as error:
raise ApiError(400, f"invalid size '{size_raw}'") from error
format_name = str(payload.get("format") or "raw")
volume_id = f"{storage_id}:{content_type}/{filename}"
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
await database(request).pool.execute(
"""INSERT INTO storage_contents(
id, storage_resource_id, volume_id, content_type, size_bytes, metadata
) VALUES(gen_random_uuid(), $1, $2, $3, $4, $5::jsonb)
ON CONFLICT (storage_resource_id, volume_id) DO UPDATE
SET size_bytes=EXCLUDED.size_bytes,
content_type=EXCLUDED.content_type,
metadata=EXCLUDED.metadata""",
resource_id,
volume_id,
content_type,
size,
json.dumps(
{"filename": filename, "format": format_name, "source": "allocate"},
sort_keys=True,
),
)
return await _node_task(request, node=node, task_type="storage-allocate", worker="imgcopy")
async def node_storage_upload(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import _node_task
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
@@ -318,11 +375,11 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
size,
json.dumps({"filename": filename, "source": "upload"}, sort_keys=True),
)
return {"uploadid": volume_id, "filename": filename, "size": size, "volid": volume_id}
return await _node_task(request, node=node, task_type="storage-upload", worker="download")
async def node_storage_prunebackups(
request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]] | None:
) -> list[dict[str, Any]] | str:
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
@@ -332,6 +389,8 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
storage_id,
)
if request.method == "DELETE":
from app.handlers.nodes import _node_task
keep = int(values(inputs).get("keep-last") or values(inputs).get("keep_last") or 1)
await database(request).pool.execute(
"""DELETE FROM backups
@@ -344,7 +403,9 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
resource_id,
keep,
)
return None
return await _node_task(
request, node=node, task_type="prune-backups", worker="prunebackups"
)
rows = await database(request).pool.fetch(
"""SELECT volume_id, size_bytes, created_at FROM backups
WHERE storage_resource_id=$1 ORDER BY created_at DESC""",
@@ -485,18 +546,29 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
async def file_restore_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
await _storage_row(request, None, str(payload["storage"]))
row = await _storage_row(request, None, str(payload["storage"]))
filepath = str(payload.get("filepath") or "/")
return [{"filepath": filepath.rstrip("/") + "/etc", "type": "d", "text": "etc"}]
config = state(row["config"])
restore = config.get("file_restore")
if not isinstance(restore, dict):
return []
items = restore.get(filepath) or restore.get(filepath.rstrip("/") or "/")
return [dict(item) for item in items] if isinstance(items, list) else []
async def file_restore_download(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
await _storage_row(request, None, str(payload["storage"]))
row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
downloads = config.get("file_restore_downloads")
filepath = str(payload.get("filepath") or "/")
if isinstance(downloads, dict) and filepath in downloads:
entry = downloads[filepath]
return dict(entry) if isinstance(entry, dict) else {"filepath": filepath}
return {
"download-url": f"/api2/json/nodes/{payload['node']}/storage/"
f"{payload['storage']}/file-restore/download",
"filepath": payload.get("filepath") or "/",
"filepath": filepath,
"volume": payload.get("volume"),
}
@@ -504,40 +576,51 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs)
await require_node(request, str(payload["node"]))
row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
identity = config.get("identity")
if isinstance(identity, dict):
return {
"storage": str(row["storage_id"]),
"type": str(identity.get("type") or row["storage_type"]),
"fingerprint": str(identity.get("fingerprint") or ""),
}
return {
"storage": str(row["storage_id"]),
"type": str(row["storage_type"]),
"fingerprint": f"sim-{row['storage_id']}",
"fingerprint": "",
}
async def import_metadata(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
storage_id = str(payload["storage"])
volume = str(payload["volume"])
await _storage_row(request, None, storage_id)
return {
"type": "qemu",
"source": volume,
"disks": {"scsi0": f"{storage_id}:0/vm-import.raw"},
"net0": "virtio,bridge=vmbr0",
}
volume = str(require_value(payload, "volume"))
row = await _storage_row(request, None, storage_id)
config = state(row["config"])
by_volume = config.get("import_metadata_by_volume")
if isinstance(by_volume, dict) and volume in by_volume:
meta = by_volume[volume]
return dict(meta) if isinstance(meta, dict) else {}
meta = config.get("import_metadata")
if isinstance(meta, dict):
return {"source": volume, **meta}
return {}
async def storage_rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
storage_id = str(payload["storage"])
await _storage_row(request, None, storage_id)
return {"filename": f"pve-storage-{storage_id}.rrd"}
row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
rrd_state = config.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def storage_rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = values(inputs)
await require_node(request, str(payload["node"]))
await _storage_row(request, None, str(payload["storage"]))
return [
{"time": 1_700_000_000, "used": 10, "total": 100},
{"time": 1_700_000_060, "used": 12, "total": 100},
]
row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
series = config.get("rrddata")
return [dict(item) for item in series] if isinstance(series, list) else []
registry.register("/storage", "GET", storage_ids)
registry.register("/storage", "POST", storage_create)
@@ -548,7 +631,7 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
registry.register("/nodes/{node}/storage/{storage}", "GET", node_storage_index)
registry.register("/nodes/{node}/storage/{storage}/status", "GET", node_storage_status)
registry.register("/nodes/{node}/storage/{storage}/content", "GET", node_storage_content)
registry.register("/nodes/{node}/storage/{storage}/content", "POST", node_storage_upload)
registry.register("/nodes/{node}/storage/{storage}/content", "POST", node_storage_allocate)
registry.register(
"/nodes/{node}/storage/{storage}/content/{volume}", "GET", node_storage_content_get
)
+28 -1
View File
@@ -25,6 +25,7 @@ from app.tasks.lxc import lxc_handler
from app.tasks.qemu import qemu_handler
from app.tasks.repository import TaskRepository
from app.tasks.worker import TaskWorker
from app.version import get_app_version
from app.web.routes import router as web_router
@@ -68,12 +69,15 @@ def create_app(
"qemu-stop": qemu,
"qemu-suspend": qemu,
"qemu-update": qemu,
"qemu-resize": qemu,
"qemu-template": qemu,
"lxc-clone": lxc,
"lxc-create": lxc,
"lxc-delete": lxc,
"lxc-migrate": lxc,
"lxc-reboot": lxc,
"lxc-resume": lxc,
"lxc-resize": lxc,
"lxc-shutdown": lxc,
"lxc-snapshot-create": lxc,
"lxc-snapshot-delete": lxc,
@@ -83,6 +87,29 @@ def create_app(
"lxc-suspend": lxc,
"vzdump": backup,
"aptupdate": backup,
"network-reload": backup,
"disk-initgpt": backup,
"disk-wipe": backup,
"disk-directory-create": backup,
"disk-directory-delete": backup,
"disk-lvm-create": backup,
"disk-lvm-delete": backup,
"disk-lvmthin-create": backup,
"disk-lvmthin-delete": backup,
"disk-zfs-create": backup,
"disk-zfs-delete": backup,
"service-start": backup,
"service-stop": backup,
"service-restart": backup,
"service-reload": backup,
"storage-allocate": backup,
"storage-upload": backup,
"prune-backups": backup,
"acme": backup,
"sdn-apply": backup,
"ceph-flags": backup,
"cluster-create": backup,
"cluster-join": backup,
},
concurrency=resolved.task_worker_concurrency,
lease_seconds=resolved.task_lease_seconds,
@@ -91,7 +118,7 @@ def create_app(
resolved_workers = (task_worker,)
app = FastAPI(
title=resolved.app_name,
version="0.1.0",
version=get_app_version(),
openapi_tags=openapi_tag_metadata(),
lifespan=create_lifespan(resolved, database_factory, resolved_workers or ()),
)
+25
View File
@@ -0,0 +1,25 @@
"""Canonical Ceph cluster flag names (PVE /cluster/ceph/flags)."""
from __future__ import annotations
# Keep in sync with Proxmox `ceph_flags` API (majors 6-9).
CEPH_FLAG_DESCRIPTIONS: dict[str, str] = {
"nobackfill": "Pause backfill operations",
"nodeep-scrub": "Disable deep scrubbing",
"nodown": "Do not mark OSDs down",
"noin": "Do not mark OSDs in",
"noout": "Do not mark OSDs out",
"norebalance": "Pause rebalance",
"norecover": "Pause recovery",
"noscrub": "Disable scrubbing",
"notieragent": "Pause tiering agent",
"noup": "Do not mark OSDs up",
"pause": "Pause I/O",
}
def default_ceph_flags(*, enabled: frozenset[str] | None = None) -> dict[str, int]:
"""Return the full flag map; unset flags are ``0`` (PVE GET → ``false``)."""
active = enabled or frozenset()
return {name: (1 if name in active else 0) for name in CEPH_FLAG_DESCRIPTIONS}
+2020 -93
View File
File diff suppressed because it is too large Load Diff
+7 -7
View File
@@ -27,7 +27,7 @@ from app.contracts.model import Method, Snapshot
from app.db.migrations import migrate
from app.db.pool import AsyncpgDatabase
from app.main import create_app
from app.simulation.seed import apply_seed, small_profile
from app.simulation.seed import apply_seed, lab_profile
from app.web.contract_catalog import get_major_releases
_BUNDLED_9 = Path(
@@ -119,7 +119,7 @@ async def prepare_db(url: str) -> None:
connection = await asyncpg.connect(url)
try:
await migrate(connection)
await apply_seed(connection, small_profile())
await apply_seed(connection, lab_profile())
finally:
await connection.close()
@@ -159,16 +159,16 @@ async def probe_major(
verb = method.verb.upper()
url = f"/api2/json{render_path(path_template)}"
headers = {"CSRFPreventionToken": csrf} if verb != "GET" else {}
body = body_for(method, path_template) if verb in {"PUT", "POST"} else None
payload = body_for(method, path_template)
try:
if verb == "GET":
response = await client.get(url, headers=headers)
response = await client.get(url, headers=headers, params=payload or None)
elif verb == "PUT":
response = await client.put(url, data=body or {}, headers=headers)
response = await client.put(url, data=payload or {}, headers=headers)
elif verb == "POST":
response = await client.post(url, data=body or {}, headers=headers)
response = await client.post(url, data=payload or {}, headers=headers)
elif verb == "DELETE":
response = await client.delete(url, headers=headers)
response = await client.request("DELETE", url, data=payload or {}, headers=headers)
else:
continue
except Exception as exc:
+5
View File
@@ -43,6 +43,11 @@ def backup_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
await repository.append_log(task.id, "apt update finished")
return {"status": "OK"}
# Node/cluster ops whose durable side effects were applied by the API handler.
if task.task_type != "vzdump":
await repository.append_log(task.id, f"{task.task_type} completed")
return {"status": "OK"}
node = str(task.payload["node"])
vmids = [str(item) for item in task.payload.get("vmids", [])]
storage_id = str(task.payload.get("storage") or "nfs-backup")
+3
View File
@@ -29,6 +29,9 @@ def lxc_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
)
if operation == "migrate" or operation == "remote-migrate":
return await _migrate(repository, task, resource_id, clock)
if operation == "resize":
await repository.append_log(task.id, "container resize completed")
return {"status": "OK"}
async with repository.pool.acquire() as connection:
row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id)
if row is None:
+7 -1
View File
@@ -33,6 +33,10 @@ def qemu_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
return await _migrate(repository, task, resource_id, clock)
if operation == "move-disk":
return await _move_disk(repository, task, resource_id)
if operation in {"resize", "template"}:
# Side effects already persisted by the API handler; complete the worker.
await repository.append_log(task.id, f"VM {operation} completed")
return {"status": "OK"}
async with repository.pool.acquire() as connection:
row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id)
if row is None:
@@ -62,10 +66,12 @@ def qemu_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
async def _create(repository: TaskRepository, task: Task) -> dict[str, Any]:
from app.simulation.seed import enrich_guest_state
node, vmid = str(task.payload["node"]), int(task.payload["vmid"])
config = dict(task.payload.get("config", {}))
resource_id = uuid.uuid4()
state = {"status": "stopped", **config}
state = enrich_guest_state({"status": "stopped", **config}, kind="qemu", vmid=str(vmid))
async with repository.pool.acquire() as connection:
async with connection.transaction():
node_row = await connection.fetchrow(
+130
View File
@@ -0,0 +1,130 @@
"""SemVer source of truth: VERSION file (synced to pyproject + Helm)."""
from __future__ import annotations
import re
from functools import lru_cache
from pathlib import Path
_VERSION_FILE = Path(__file__).resolve().parents[1] / "VERSION"
_PROJECT_ROOT = _VERSION_FILE.parent
_PYPROJECT_FILE = _PROJECT_ROOT / "pyproject.toml"
_CHART_FILE = _PROJECT_ROOT / "helm" / "proxmox-api-simulator" / "Chart.yaml"
_VALUES_FILE = _PROJECT_ROOT / "helm" / "proxmox-api-simulator" / "values.yaml"
_DEFAULT_VERSION = "0.1.0"
_VERSION_RE = re.compile(r"^(\d+)\.(\d+)\.(\d+)$")
def _read_version_file(path: Path) -> str | None:
try:
text = path.read_text(encoding="utf-8").strip()
except OSError:
return None
if _VERSION_RE.match(text):
return text
return None
@lru_cache
def get_app_version() -> str:
for path in (
Path("/app/VERSION"),
_VERSION_FILE,
Path.cwd() / "VERSION",
):
found = _read_version_file(path)
if found:
return found
try:
from importlib.metadata import version
meta = version("proxmox-api-simulator")
if _VERSION_RE.match(meta):
return meta
except Exception:
pass
return _DEFAULT_VERSION
def get_app_version_label() -> str:
clear_version_cache()
return f"v{get_app_version()}"
def parse_version(value: str) -> tuple[int, int, int]:
match = _VERSION_RE.match(value.strip())
if not match:
raise ValueError(f"Invalid semantic version: {value!r}")
return int(match.group(1)), int(match.group(2)), int(match.group(3))
def format_version(major: int, minor: int, patch: int) -> str:
return f"{major}.{minor}.{patch}"
def clear_version_cache() -> None:
get_app_version.cache_clear()
def write_project_version(version: str) -> None:
parse_version(version)
_VERSION_FILE.write_text(f"{version}\n", encoding="utf-8")
pyproject = _PYPROJECT_FILE.read_text(encoding="utf-8")
pyproject, count = re.subn(
r'^version = ".*"$',
f'version = "{version}"',
pyproject,
count=1,
flags=re.MULTILINE,
)
if count != 1:
raise RuntimeError("Failed to update pyproject.toml version")
_PYPROJECT_FILE.write_text(pyproject, encoding="utf-8")
chart = _CHART_FILE.read_text(encoding="utf-8")
chart, chart_count = re.subn(
r"^version: .*$",
f"version: {version}",
chart,
count=1,
flags=re.MULTILINE,
)
chart, app_count = re.subn(
r'^appVersion: ".*"$',
f'appVersion: "{version}"',
chart,
count=1,
flags=re.MULTILINE,
)
if chart_count != 1 or app_count != 1:
raise RuntimeError("Failed to update Chart.yaml version")
_CHART_FILE.write_text(chart, encoding="utf-8")
values = _VALUES_FILE.read_text(encoding="utf-8")
values, values_count = re.subn(
r'^(\s*tag:\s*).*$',
rf'\1"{version}"',
values,
count=1,
flags=re.MULTILINE,
)
if values_count != 1:
raise RuntimeError("Failed to update helm values.yaml image.tag")
_VALUES_FILE.write_text(values, encoding="utf-8")
clear_version_cache()
def bump_patch_version() -> str:
major, minor, patch = parse_version(get_app_version())
version = format_version(major, minor, patch + 1)
write_project_version(version)
return version
def bump_minor_version() -> str:
major, minor, patch = parse_version(get_app_version())
version = format_version(major, minor + 1, 0)
write_project_version(version)
return version
+140 -12
View File
@@ -11,7 +11,7 @@ from pathlib import Path
from app.api.openapi import contract_openapi_tag
from app.config import Settings
from app.contracts.examples import path_param_example, schema_example
from app.contracts.examples import path_param_example, schema_example, wire_param_name
from app.contracts.importer import RemoteSourceImporter
from app.contracts.model import Method, Parameter, Snapshot
from app.contracts.normalize import normalize_snapshot
@@ -102,8 +102,10 @@ def list_majors(
*,
runtime_version: str | None,
settings: Settings | None = None,
app_version: str | None = None,
) -> dict[str, object]:
return {
"app_version": app_version,
"runtime_version": runtime_version,
"majors": [
{
@@ -226,16 +228,25 @@ def _path_param_names(path: str) -> tuple[str, ...]:
return tuple(match.group(1) for match in _PATH_PARAM.finditer(path))
def _parameter_payload(parameter: Parameter) -> dict[str, object]:
def _parameter_payload(parameter: Parameter, *, wire_name: str | None = None) -> dict[str, object]:
schema = parameter.definition
return {
"name": parameter.name,
name = wire_name or parameter.name
typetext = schema.extra.get("typetext")
payload: dict[str, object] = {
"name": name,
"type": schema.type,
"description": schema.description,
"optional": bool(schema.optional),
"enum": list(schema.enum),
"example": schema_example(schema, name=parameter.name),
}
if wire_name and wire_name != parameter.name:
payload["template"] = parameter.name
if isinstance(schema.format, str):
payload["format"] = schema.format
if isinstance(typetext, str) and typetext:
payload["typetext"] = typetext
return payload
def method_payload(
@@ -274,15 +285,20 @@ def method_payload(
"example": path_param_example(name) or name,
}
)
body_fields = [
contract_body_fields = [
_parameter_payload(parameter)
for parameter in method.parameters
if parameter.name not in path_params and "[n]" not in parameter.name
]
# Contract uses foo[n]; the wire form is foo0..fooN — seed the console with foo0.
indexed_fields = [
_parameter_payload(parameter) for parameter in method.parameters if "[n]" in parameter.name
_parameter_payload(parameter, wire_name=wire_param_name(parameter.name))
for parameter in method.parameters
if "[n]" in parameter.name
]
body_example = _body_example(method, path_params)
# Flatten nested scalars from body_example (and nested field examples) into PARAMS.
body_fields = _body_fields_with_nested(contract_body_fields, body_example)
resolved_path = _resolve_path(path, path_fields)
implemented = (
(path, method.verb.upper()) in implemented_methods
@@ -316,21 +332,133 @@ def _resolve_path(path: str, path_fields: list[dict[str, object]]) -> str:
def _body_example(method: Method, path_params: tuple[str, ...]) -> dict[str, object]:
"""Minimal request body: required parameters only, Proxmox wire names/values."""
body: dict[str, object] = {}
for parameter in method.parameters:
if parameter.name in path_params:
continue
if "[n]" in parameter.name:
concrete = parameter.name.replace("[n]", "0")
if not parameter.definition.optional:
body[concrete] = schema_example(parameter.definition, name=concrete)
continue
if parameter.definition.optional:
continue
body[parameter.name] = schema_example(parameter.definition, name=parameter.name)
wire = wire_param_name(parameter.name)
body[wire] = schema_example(parameter.definition, name=parameter.name)
return body
def _leaf_type(value: object) -> str:
if isinstance(value, bool):
return "boolean"
if isinstance(value, int) and not isinstance(value, bool):
return "integer"
if isinstance(value, float):
return "number"
return "string"
def _body_fields_from_example(body_example: dict[str, object] | None) -> list[dict[str, object]]:
"""PARAM inputs derived from body_example, including nested scalar paths."""
if not isinstance(body_example, dict) or not body_example:
return []
inner: object = body_example
# Soft unwrap a single root object envelope (Engine-style), never a scalar/list.
if len(body_example) == 1:
only = next(iter(body_example.values()))
if isinstance(only, dict):
inner = only
if not isinstance(inner, dict):
return []
fields: list[dict[str, object]] = []
def _walk(prefix: str, value: object) -> None:
if isinstance(value, dict):
for key, child in value.items():
path = f"{prefix}.{key}" if prefix else str(key)
_walk(path, child)
return
if isinstance(value, list):
for index, child in enumerate(value):
path = f"{prefix}.{index}" if prefix else str(index)
_walk(path, child)
return
fields.append(
{
"name": prefix,
"type": _leaf_type(value),
"description": prefix,
"optional": True,
"enum": [],
"example": value,
}
)
_walk("", inner)
return fields
def _body_fields_with_nested(
contract_fields: list[dict[str, object]],
body_example: dict[str, object],
) -> list[dict[str, object]]:
"""Merge contract PARAMS with nested scalar paths from body_example."""
example_fields = _body_fields_from_example(body_example)
nested_tops = {
str(item["name"]).split(".", 1)[0] for item in example_fields if "." in str(item["name"])
}
contract_by_name = {str(item["name"]): item for item in contract_fields}
merged: list[dict[str, object]] = []
seen: set[str] = set()
for item in example_fields:
name = str(item["name"])
top = name.split(".", 1)[0]
parent = contract_by_name.get(name) or contract_by_name.get(top)
row = dict(item)
if parent is not None:
row["optional"] = bool(parent.get("optional"))
description = parent.get("description")
if isinstance(description, str) and description:
row["description"] = description
typetext = parent.get("typetext")
if isinstance(typetext, str) and typetext:
row["typetext"] = typetext
fmt = parent.get("format")
if isinstance(fmt, str) and fmt:
row["format"] = fmt
else:
# body_example is required-only — treat uncovered leaves as required.
row["optional"] = False
merged.append(row)
seen.add(name)
for field in contract_fields:
name = str(field["name"])
if name in nested_tops or name in seen:
continue
example = field.get("example")
if isinstance(example, dict | list):
nested = _body_fields_from_example({name: example})
if nested:
for item in nested:
nested_name = str(item["name"])
if nested_name in seen:
continue
row = dict(item)
row["optional"] = bool(field.get("optional"))
description = field.get("description")
if isinstance(description, str) and description:
row["description"] = description
merged.append(row)
seen.add(nested_name)
continue
# Empty object/array examples stay as a top-level PARAM.
merged.append(field)
seen.add(name)
return merged
@lru_cache(maxsize=1)
def default_store_root() -> Path:
return _DEFAULT_STORE
+686 -87
View File
File diff suppressed because it is too large Load Diff
+26 -3
View File
@@ -16,6 +16,7 @@ from app.contracts.source import SourceError
from app.db.pool import AsyncpgDatabase
from app.dependencies import get_database
from app.simulation.seed import apply_seed, build_profile, simulation_state_summary
from app.version import get_app_version
from app.web.assets import console_html
from app.web.compatibility_catalog import compatibility_payload
from app.web.contract_catalog import catalog_payload, list_majors, load_snapshot, method_payload
@@ -37,7 +38,13 @@ async def console() -> HTMLResponse:
async def ui_versions(request: Request) -> JSONResponse:
settings = _settings(request)
runtime_version = _runtime_version(request)
return JSONResponse(list_majors(runtime_version=runtime_version, settings=settings))
return JSONResponse(
list_majors(
runtime_version=runtime_version,
settings=settings,
app_version=get_app_version(),
)
)
@router.get("/ui/api/catalog", include_in_schema=False)
@@ -161,7 +168,23 @@ async def ui_demo_state(request: Request) -> JSONResponse:
@router.post("/ui/api/demo/load", include_in_schema=False)
async def ui_demo_load(request: Request) -> JSONResponse:
pool = _database_pool(request)
profile = build_profile("demo-cluster")
requested = "small"
if request.query_params.get("profile"):
requested = str(request.query_params.get("profile"))
else:
try:
body = await request.json()
except Exception:
body = None
if isinstance(body, dict) and body.get("profile"):
requested = str(body["profile"])
allowed = {"small", "large", "big"}
if requested not in allowed:
raise HTTPException(
status_code=400,
detail=f"unknown cluster profile '{requested}' (expected: small, large, big)",
)
profile = build_profile(requested)
async with pool.acquire() as connection:
await apply_seed(connection, profile)
summary = await simulation_state_summary(connection)
@@ -179,7 +202,7 @@ async def ui_demo_unload(request: Request) -> JSONResponse:
summary = await simulation_state_summary(connection)
except Exception as error:
raise HTTPException(
status_code=500, detail=f"failed to remove demo data: {error}"
status_code=500, detail=f"failed to reset to minimal cluster: {error}"
) from error
return JSONResponse({"ok": True, "profile": profile.name, "summary": summary})
+51 -2
View File
@@ -1,12 +1,28 @@
# Quick start with the published Docker Hub runtime image.
#
# WARNING: Laboratory / CI only.
# Default TICKET_SIGNING_KEY and PostgreSQL password are intentional lab
# defaults. Do NOT expose host :8006 to untrusted networks without replacing
# secrets and adding your own controls. See SECURITY.md and docs/security.md.
#
# Requires this repository checkout (compose file).
#
# docker compose -f docker-compose.release.yml up -d
# docker compose -f docker-compose.release.yml run --rm --entrypoint python simulator -m app.simulation.seed_cli
# docker compose -f docker-compose.release.yml run --rm --entrypoint python \
# simulator -m app.simulation.seed_cli
# curl -sS http://localhost:8006/health/ready
#
# Override the image tag:
# IMAGE_TAG=0.1.0 docker compose -f docker-compose.release.yml up -d
#
# Change TICKET_SIGNING_KEY before exposing the stack beyond a local lab.
# Rotate lab secrets before any shared or networked demo:
# TICKET_SIGNING_KEY=$(openssl rand -hex 32) \
# POSTGRES_PASSWORD=$(openssl rand -hex 16) \
# docker compose -f docker-compose.release.yml up -d
#
# Optional HTTPS for proxmoxer-style clients (separate port):
# docker compose -f docker-compose.release.yml --profile tls up -d
# curl -sk https://localhost:8443/health/ready
name: proxmox-api-simulator-release
@@ -18,6 +34,7 @@ x-app-env: &app-env
CONTRACT_SNAPSHOT: /app/contracts/pve-9.2.3.json
COMPATIBILITY_EVIDENCE: /app/evidence/pve-9.2.3.json
LOG_LEVEL: ${LOG_LEVEL:-INFO}
# Lab default — replace for any shared or networked use.
TICKET_SIGNING_KEY: ${TICKET_SIGNING_KEY:-development-only-signing-key-change-me}
TASK_WORKER_CONCURRENCY: ${TASK_WORKER_CONCURRENCY:-2}
SIMULATION_TIME_SCALE: ${SIMULATION_TIME_SCALE:-10}
@@ -86,3 +103,35 @@ services:
start_period: 20s
ports:
- "${SIMULATOR_PORT:-8006}:8006"
tls-gateway:
profiles: [tls]
image: nginx:1.28.0-alpine
restart: unless-stopped
networks: [simulator]
depends_on:
simulator:
condition: service_healthy
ports:
- "${TLS_GATEWAY_PORT:-8443}:8443"
volumes:
- ./docker/tls/gateway.conf:/etc/nginx/conf.d/default.conf:ro
- ./docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
- ./docker/tls/server.key:/etc/nginx/tls/server.key:ro
healthcheck:
test:
[
"CMD-SHELL",
"wget -qO- --no-check-certificate https://127.0.0.1:8443/health/live || exit 1",
]
interval: 10s
timeout: 3s
retries: 5
start_period: 5s
read_only: true
tmpfs:
- /var/cache/nginx
- /tmp
- /var/run
security_opt:
- no-new-privileges:true
+8 -4
View File
@@ -12,8 +12,9 @@ x-dev-env: &dev-env
TEST_DATABASE_URL: postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator
CONTRACT_SNAPSHOT: /workspace/contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json
COMPATIBILITY_EVIDENCE: /workspace/evidence/pve-9.2.3.json
PROXMOXER_HOST: tls-gateway
PROXMOXER_PORT: "8443"
# proxmoxer is HTTPS-only — set when using --profile tls (see Makefile test-compatibility)
PROXMOXER_HOST: ${PROXMOXER_HOST:-}
PROXMOXER_PORT: ${PROXMOXER_PORT:-8443}
LOG_LEVEL: INFO
TICKET_SIGNING_KEY: development-only-signing-key-change-me
@@ -108,9 +109,12 @@ services:
retries: 8
start_period: 20s
ports:
- "8006:8006"
- "${SIMULATOR_PORT:-8006}:8006"
# Optional HTTPS front for clients that cannot speak plain HTTP (proxmoxer).
# Default lab URL is http://localhost:8006/ — K8s TLS is Ingress-only.
tls-gateway:
profiles: [tls]
image: nginx:1.28.0-alpine
restart: unless-stopped
networks: [simulator]
@@ -118,7 +122,7 @@ services:
simulator:
condition: service_healthy
ports:
- "8007:8443"
- "${TLS_GATEWAY_PORT:-8443}:8443"
volumes:
- ./docker/tls/gateway.conf:/etc/nginx/conf.d/default.conf:ro
- ./docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
+23 -2
View File
@@ -1,5 +1,26 @@
# Development HTTPS gateway for proxmoxer and other TLS clients.
# Upstream hostnames are resolved at request time via Docker embedded DNS.
# Optional / internal TLS terminator.
#
# - Host lab default is plain HTTP :8006 on the simulator (no TLS).
# - Kubernetes HTTPS terminates at Ingress (cert-manager).
# - This gateway is for HTTPS-only clients (proxmoxer, pulumi-proxmoxve).
# - Also listen on :80: some bridged providers rewrite https://host:8443 → http://host
# on later requests (delete); without :80 those calls fail.
server {
listen 80;
server_name _;
resolver 127.0.0.11 valid=10s ipv6=off;
location / {
set $upstream simulator:8006;
proxy_pass http://$upstream;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-ID $request_id;
}
}
server {
listen 8443 ssl;
+39
View File
@@ -0,0 +1,39 @@
**Language / Язык:** [English](README.md) | [Русский](ru/README.md)
# Documentation
Guides for the Proxmox VE API simulator. Switch language with the header on each
page. Russian mirrors live under [`ru/`](ru/README.md).
![Web UI main](images/web-ui-main.png)
| Guide | Description |
|---|---|
| [Getting started](getting-started.md) | First successful lab session |
| [Configuration](configuration.md) | Environment variables and Compose |
| [Authentication](authentication.md) | Tickets, CSRF, API tokens, ACLs |
| [API versions](api-versions.md) | Contracts 6–9 and hot-swap |
| [Clients & examples](clients.md) | Python, Go, Java, Perl, Ansible, Terraform, Pulumi |
| [Seed profiles](seed-profiles.md) | Deterministic cluster fixtures |
| [API surface](api-surface.md) | Routing, handlers, fallbacks |
| [Domains](domains/README.md) | QEMU, LXC, storage, HA, SDN, … |
| [Web UI](web-ui.md) | Interactive console and catalogs |
| [Operations](operations.md) | Migrate, reseed, upgrade, Hub publish |
| [CI/CD (Jenkins)](ci-cd.md) | VERSION, Harbor/Hub push, deploy to proxmox.devops.org.ru |
| [Docker Hub overview](docker-hub-overview.md) | Paste-ready Hub repository description |
| [Kubernetes / Helm](kubernetes.md) | Hub image + Ingress + Let's Encrypt |
| [Security](security.md) | Lab threat model and credentials |
| [Observability](observability.md) | Health endpoints and logging |
| [Troubleshooting](troubleshooting.md) | Common failure modes |
| [FAQ](faq.md) | Short answers |
| [Architecture](architecture.md) | Component boundaries |
| [Compatibility](compatibility.md) | Evidence model and release matrix |
| [Testing](testing.md) | Suites layout, Make targets, latest results |
Runnable cookbooks: [`examples/`](../examples/README.md).
Integration suites: [`pulumi-tests/`](../pulumi-tests/README.md).
Test map and latest pass notes: [Testing](testing.md).
## See also
- [Web UI](web-ui.md) — interactive console, catalogs, DATA panel, and screenshots
+4 -1
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](api-surface.md) | [Русский](ru/api-surface.md)
# API surface
## Request path
@@ -33,7 +35,8 @@ not part of the product contract. See the workspace durable-simulator rule.
- Interactive FastAPI docs: `/docs`
- Web UI method inspector: `/` → catalog → method
- UI APIs: `/ui/api/catalog`, `/ui/api/method`, `/ui/api/compatibility`
- UI APIs: `/ui/api/versions`, `/ui/api/catalog`, `/ui/api/method`,
`/ui/api/compatibility`, `/ui/api/contract/apply`, `/ui/api/demo/*`
## Compatibility endpoints
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](api-versions.md) | [Русский](ru/api-versions.md)
# API versions (PVE 6–9)
The simulator ships authoritative imported contracts for four Proxmox VE majors.
+3 -1
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](architecture.md) | [Русский](ru/architecture.md)
# Architecture
## Goals
@@ -30,7 +32,7 @@ flowchart LR
Obs["Logs / Prometheus / OpenTelemetry"]
Client -->|"/api2/json"| API
Admin -->|"CLI and /_simulator"| API
Admin -->|"CLI, Make/Helm, Web UI /ui/api"| API
Docs -->|"explicit import only"| Importer
Importer --> Contract
Contract --> DB
+5 -2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](authentication.md) | [Русский](ru/authentication.md)
# Authentication
The simulator implements Proxmox-compatible ticket and API-token authentication
@@ -54,8 +56,9 @@ beyond its owner.
## Seeded development principals
Loaded by every standard seed profile (unless replaced by UI demo unload →
`minimal`):
Seeded for **every** profile — including `minimal` and after Web UI demo unload.
Unload shrinks guests/nodes/storages; lab principals and tokens are still
inserted by `apply_seed`:
| Principal | Password | Token | Notes |
|---|---|---|---|
+78
View File
@@ -0,0 +1,78 @@
**Language / Язык:** [English](ci-cd.md) | [Русский](ru/ci-cd.md)
# CI/CD (Jenkins + Gitea)
SemVer lives in [`VERSION`](../VERSION). Bump locally with `make bump-patch` /
`make bump-minor` (also syncs `pyproject.toml` and Helm chart/values). Jenkins
does **not** bump — it builds whatever is in the commit.
Public lab URL after deploy: **https://proxmox.devops.org.ru/**
## Pipeline
Single [`Jenkinsfile`](../Jenkinsfile) on `main` / `master`:
1. **Build & push** (agent `docker` / DinD) — multi-arch `linux/amd64,linux/arm64`
to Harbor and Docker Hub with tags **`:{VERSION}`** and **`:latest`** only.
2. **Deploy** — `kubectl set image` on `simulators/simulators-proxmox` (main
container `simulator` + initContainer `migrate`), then rollout status
(`--timeout=1200s`).
First cluster install (Ingress, Postgres, secrets) is **not** this job — use
`make addon-simulators` in DevOpsTools/K3S (`simulators_proxmox_host:
proxmox.devops.org.ru`).
### Images
| Registry | Repository |
|---|---|
| Harbor | `hub.antropoff.ru/devops-tools/proxmox-api-simulator` |
| Docker Hub | `inecs/proxmox-api-simulator` |
### Credentials (Jenkins Global)
| ID | Type | Use |
|---|---|---|
| `harbor-devops-tools-push-pull-access` | Username/password | Harbor robot push |
| `docker-hub` | Username/password | Docker Hub `inecs` |
| `k3s-kubeconfig` | Secret file | Kubeconfig for deploy |
| `ssh-gitea-key` | SSH private key | Gitea SCM checkout |
## Gitea setup
1. Ensure the repo is on Gitea (mirror or primary) with branch `main`.
2. Deploy key / Jenkins credential `ssh-gitea-key` can clone the repo
(`git@…:…/proxmox_api_simulator.git` or your path).
3. Optional: webhook from Gitea → Jenkins Multibranch (or rely on Jenkins SCM
polling / Organization Folder).
## Jenkins setup
1. **Credentials** — create the four IDs above if missing (same as Wrapped).
2. **Agent** — Kubernetes cloud pod template with label `docker` and a
`docker` container (DinD / buildx), same as Wrapped.
3. **Job** — Multibranch Pipeline (or Pipeline from SCM):
- Script Path: `Jenkinsfile`
- Branch discover: `main` (and `master` if needed)
- SCM: Gitea SSH URL + credential `ssh-gitea-key`
4. **Harbor project** — `devops-tools` must allow the robot to push
`proxmox-api-simulator` (create the repository on first push or pre-create).
5. **Docker Hub** — `inecs/proxmox-api-simulator` exists / push rights for
credential `docker-hub`.
6. **Cluster** — release `simulators` in namespace `simulators` already
installed (`make addon-simulators`). Deployment name must be
`simulators-proxmox`.
7. Run once on `main` after bumping `VERSION` and pushing. Confirm:
- Harbor/Hub tags `:0.x.y` and `:latest`
- `kubectl -n simulators get deploy,ing`
- Help → About shows badge `v0.x.y` on https://proxmox.devops.org.ru/
## Local release (without Jenkins)
```bash
make bump-patch # or bump-minor
make version-commit
git push origin HEAD
# or manual Hub-only:
make release # DOCKER_IMAGE=inecs/proxmox-api-simulator
```
+18 -27
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](clients.md) | [Русский](ru/clients.md)
# Clients
Use the simulator from common automation stacks. Each cookbook aims for the
@@ -13,39 +15,28 @@ same laboratory flow where the tool allows it:
## Connection matrix
| Stack | Transport | Notes | Docs | Code |
Real Proxmox VE clients talk to **HTTPS `:8006`**. This lab’s Compose stack
publishes plain **HTTP `:8006`** (same port number). HTTPS belongs on
**Kubernetes Ingress** (cert-manager). Clients that cannot speak HTTP
(proxmoxer) use the optional profile: `docker compose --profile tls` →
`https://localhost:8443/` (see [Ports and TLS](configuration.md#ports-and-tls)).
| Stack | Compose transport | Notes | Docs | Code |
|---|---|---|---|---|
| Python (proxmoxer) | HTTPS `:8007` | Unmodified library; `verify_ssl=False` for local cert | [guide](examples/python-proxmoxer.md) | [`examples/python`](../examples/python) |
| Python (proxmoxer) | HTTPS `:8443` (`--profile tls`) | HTTPS-only library; `verify_ssl=False` for lab cert | [guide](examples/python-proxmoxer.md) | [`examples/python`](../examples/python) |
| Python (requests) | HTTP `:8006` | Raw `/api2/json` | [guide](examples/python-requests.md) | [`examples/python`](../examples/python) |
| Go | HTTP `:8006` | stdlib `net/http` | [guide](examples/go.md) | [`examples/go`](../examples/go) |
| Java | HTTP `:8006` | Java 11+ `HttpClient` | [guide](examples/java.md) | [`examples/java`](../examples/java) |
| Perl | HTTP `:8006` | `HTTP::Tiny` + JSON | [guide](examples/perl.md) | [`examples/perl`](../examples/perl) |
| Ansible | HTTP `:8006` | `uri` module cookbook | [guide](examples/ansible.md) | [`examples/ansible`](../examples/ansible) |
| Terraform | HTTPS `:8007` | Provider + insecure TLS for local gateway | [guide](examples/terraform.md) | [`examples/terraform`](../examples/terraform) |
| Pulumi | HTTPS `:8007` | Python program against the API | [guide](examples/pulumi.md) | [`examples/pulumi`](../examples/pulumi) |
| Terraform | HTTP `:8006` (or TLS `:8443`) | Prefer HTTP; use `insecure` only with `--profile tls` | [guide](examples/terraform.md) | [`examples/terraform`](../examples/terraform) |
| Pulumi | HTTP `:8006` | `pulumi-proxmoxve` or HTTP cookbooks | [guide](examples/pulumi.md) | [`examples/pulumi`](../examples/pulumi) |
Shared prerequisites: [examples overview](examples/overview.md).
On Kubernetes with Ingress + cert-manager, point every client at
`https://<your-host>/` instead.
## Credentials (seed)
## More
| Use | Value |
|---|---|
| User | `root@pam` |
| Password | `secret` |
| Token | `root@pam!automation=automation-secret` |
| Default node (`small`) | `pve01` |
## API major
Pin the major before long runs:
- Cold start: `CONTRACT_SNAPSHOT`
- Runtime: Web UI apply or `POST /ui/api/contract/apply?major=N`
Confirm with `GET /api2/json/version`. Coverage is **100%** for declared methods
on majors 6–9.
## Troubleshooting clients
See [troubleshooting-clients](examples/troubleshooting-clients.md) and the
global [Troubleshooting](troubleshooting.md) guide.
- Cookbooks index: [examples/overview.md](examples/overview.md)
- Troubleshooting: [examples/troubleshooting-clients.md](examples/troubleshooting-clients.md)
- Pulumi full suite: [`pulumi-tests/`](../pulumi-tests/README.md)
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](compatibility-0.1.0.md) | [Русский](ru/compatibility-0.1.0.md)
# Compatibility report — 0.1.0
This report records evidence for simulator release 0.1.0 against the bundled
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](compatibility.md) | [Русский](ru/compatibility.md)
# Compatibility
This document explains how the simulator claims compatibility with Proxmox VE
+32 -3
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](configuration.md) | [Русский](ru/configuration.md)
# Configuration
Application settings are loaded from the environment (see `.env.example`).
@@ -53,14 +55,41 @@ any accidental gap surfaces as HTTP 501.
| `SEED_LARGE_NODES` | Node count for `large` |
| `SEED_LARGE_RESOURCES` | Guest count for `large` (default 10 000) |
| `TEST_DATABASE_URL` | Integration-test DSN |
| `PROXMOXER_HOST` / `PROXMOXER_PORT` | Compatibility test client target (`tls-gateway` / `8443` in Compose) |
| `PROXMOXER_HOST` / `PROXMOXER_PORT` | Compatibility test client target with `--profile tls` (`tls-gateway` / `8443`) |
## Ports and TLS
### Real Proxmox VE (reference)
On a physical / production PVE node the management API listens on **HTTPS
`:8006`** (`/api2/json/...`). Related management ports (not separate REST APIs):
| Port | Protocol | Role |
|---|---|---|
| `8006` | TCP, HTTPS | Web UI + REST API |
| `3128` | TCP | SPICE proxy (graphical console) |
| `5900–5999` | TCP (WebSocket) | VNC web console |
| `22` | TCP | SSH / cluster actions |
| `5405–5412` | UDP | Corosync cluster traffic |
Port **`8007`** is **not** the PVE API — it is the usual Proxmox Backup Server
(PBS) management port. Do not point PVE clients at `:8007` on real hardware.
### Simulator lab endpoints
| Endpoint | Use |
|---|---|
| `http://localhost:8006` | Direct HTTP (curl, browsers, most examples) |
| `https://localhost:8007` | TLS gateway for TLS-assuming clients (proxmoxer, etc.) |
| `http://localhost:8006` | Primary client URL — simulator (curl, browsers, requests, Terraform, …) |
| `https://localhost:8443` | Optional — `docker compose --profile tls` for proxmoxer-style HTTPS-only clients |
Compose publishes plain **HTTP on host `:8006`** (same port number as real PVE,
which uses HTTPS). The simulator process speaks HTTP on `:8006` inside the Docker
network as well. For Kubernetes, TLS terminates at Ingress (cert-manager). Host
**`:8007` is not used** for the lab API (on real hardware that port is typically
PBS, not PVE).
Optional Compose TLS: `docker compose --profile tls` starts an nginx gateway on
host `:8443` (requires `docker/tls/`). It proxies to `simulator:8006`.
The checked-in certificate under `docker/tls/` is disposable development
material. Never reuse it outside local labs. See [Security](security.md).
+40
View File
@@ -0,0 +1,40 @@
# Docker Hub overview (paste into Hub)
Copy the block below into the **Full description** of
[`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator)
so Hub wording matches GitHub (stateful simulator — not a thin mock).
---
**proxmox-api-simulator** — stateful asynchronous Proxmox VE API simulator for
labs and CI. PostgreSQL-backed mutations, durable UPIDs, official API contracts
for PVE 6–9, and the same `/api2/json` surface clients already speak.
**Laboratory / CI only.** Default credentials and signing keys are intentional
lab defaults. Do **not** expose this image to the public Internet without
replacing secrets and adding your own network controls.
### Quick start
```bash
# from a git checkout (needs docker-compose.release.yml + docker/tls/)
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
```
- HTTP API + Web UI: `http://localhost:8006/`
- Optional HTTPS for proxmoxer: `docker compose --profile tls` → `https://localhost:8443/`
- Seeded admin: `root@pam` / `secret`
- Source & docs: https://github.com/sergeyantropoff/proxmox-api-simulator
- Helm chart: `helm/proxmox-api-simulator` in the same repository
### Tags
| Tag | Meaning |
|---|---|
| `0.1.0`, `…` | Immutable release from `pyproject.toml` / `make release` |
| `latest` | Most recent `make release` (when `PUSH_LATEST=1`) |
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](README.md) | [Русский](../ru/domains/README.md)
# Domain guides
These pages summarize durable semantics by area. For exhaustive method lists,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](access.md) | [Русский](../ru/domains/access.md)
# Access
Durable identity and authorization: users, groups, roles, ACL entries, realms,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](ceph.md) | [Русский](../ru/domains/ceph.md)
# Ceph
Ceph-related API paths persist simulated cluster, pool, OSD, and monitor state.
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](cluster-extras.md) | [Русский](../ru/domains/cluster-extras.md)
# Cluster extras
Additional cluster-scoped domains with durable handlers:
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](core-cluster.md) | [Русский](../ru/domains/core-cluster.md)
# Core & cluster
## Version
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](firewall.md) | [Русский](../ru/domains/firewall.md)
# Firewall
Cluster, node, and guest firewall configuration — rules, aliases, IP sets,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](ha.md) | [Русский](../ru/domains/ha.md)
# HA
High-availability groups, resources, status, and rules persist in cluster
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](lxc.md) | [Русский](../ru/domains/lxc.md)
# LXC
Container APIs mirror the QEMU lifecycle patterns where the contract declares
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](pools.md) | [Русский](../ru/domains/pools.md)
# Pools
Pool CRUD and resource membership are fully covered and durable. The `medium`
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](qemu.md) | [Русский](../ru/domains/qemu.md)
# QEMU
Full contract surface for QEMU guests on the active major, including:
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](sdn.md) | [Русский](../ru/domains/sdn.md)
# SDN
Software-defined networking handlers cover declared zones, VNets, subnets,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](storage-backup.md) | [Русский](../ru/domains/storage-backup.md)
# Storage & backup
## Storage
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](tasks.md) | [Русский](../ru/domains/tasks.md)
# Tasks
Long-running operations return a Proxmox-style **UPID**. Task rows, events,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](ansible.md) | [Русский](../ru/examples/ansible.md)
# Ansible
Playbook uses the `uri` module against HTTP `:8006` with token auth, then
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](go.md) | [Русский](../ru/examples/go.md)
# Go
Uses the Go standard library against `http://localhost:8006` with API-token
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](java.md) | [Русский](../ru/examples/java.md)
# Java
Java 11+ `HttpClient` cookbook using API-token auth against `:8006`.
+3 -1
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](overview.md) | [Русский](../ru/examples/overview.md)
# Client examples overview
## Bring-up checklist
@@ -20,7 +22,7 @@ curl -s -X POST 'http://localhost:8006/ui/api/contract/apply?major=8'
| URL | When |
|---|---|
| `http://localhost:8006` | curl, Go, Java, Perl, Ansible, requests |
| `https://localhost:8007` | proxmoxer, many Terraform/Pulumi TLS clients |
| `http://localhost:8006` | proxmoxer, many Terraform/Pulumi TLS clients |
## Auth quick reference
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](perl.md) | [Русский](../ru/examples/perl.md)
# Perl
`HTTP::Tiny` + JSON cookbook with API-token auth.
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](pulumi.md) | [Русский](../ru/examples/pulumi.md)
# Pulumi
Python Pulumi program that drives the simulator over HTTPS using token auth via
+3 -1
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](python-proxmoxer.md) | [Русский](../ru/examples/python-proxmoxer.md)
# Python — proxmoxer
Canonical library path against the HTTPS gateway.
@@ -11,7 +13,7 @@ python examples/python/proxmoxer_cookbook.py
```
Environment overrides: `PVE_HOST` (default `localhost`), `PVE_PORT` (default
`8007`), `PVE_USER`, `PVE_PASSWORD`, or token via `PVE_TOKEN_NAME` /
`8006`), `PVE_USER`, `PVE_PASSWORD`, or token via `PVE_TOKEN_NAME` /
`PVE_TOKEN_VALUE`.
## Notes
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](python-requests.md) | [Русский](../ru/examples/python-requests.md)
# Python — requests
Raw HTTP against `:8006` without proxmoxer.
+3 -1
View File
@@ -1,7 +1,9 @@
**Language / Язык:** [English](terraform.md) | [Русский](../ru/examples/terraform.md)
# Terraform
Example uses a Proxmox provider pointed at the local HTTPS gateway
(`https://localhost:8007`) with `insecure = true` for the development
(`http://localhost:8006`) with `insecure = true` for the development
certificate.
```bash
+3 -1
View File
@@ -1,8 +1,10 @@
**Language / Язык:** [English](troubleshooting-clients.md) | [Русский](../ru/examples/troubleshooting-clients.md)
# Troubleshooting clients
| Symptom | Fix |
|---|---|
| TLS certificate errors | Use `:8007` with verify disabled **only** locally, or use HTTP `:8006` |
| TLS certificate errors | Compose default is `http://localhost:8006` (no TLS). For proxmoxer use `docker compose --profile tls` and `https://localhost:8443` with verify disabled **only** locally (`curl -sk`, `verify_ssl=False`, `insecure=true`). On Kubernetes use your Ingress hostname with cert-manager TLS. |
| CSRF failure | Send `CSRFPreventionToken` with ticket mutations; prefer token auth in scripts |
| Node not found | `small` seed uses `pve01` |
| 403 on power | You may be using `auditor@pve` / readonly token — switch to root or operator |
+19 -2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](faq.md) | [Русский](ru/faq.md)
# FAQ
## Is this a real Proxmox hypervisor?
@@ -14,7 +16,10 @@ See [API versions](api-versions.md) and [Compatibility](compatibility.md).
## Can I use this in CI for Terraform / Ansible / custom clients?
Yes. That is a primary use case. Pin the API major, seed a profile, and point
clients at HTTP `:8006` or HTTPS `:8007`. See [Clients](clients.md).
clients at **HTTP `:8006`** (Compose) or your Ingress **HTTPS** hostname on
Kubernetes. See [Clients](clients.md). For the
Pulumi surface suite see [`pulumi-tests/`](../pulumi-tests/README.md).
Full test map and latest recorded results: [Testing](testing.md).
## Why do some OpenID / LDAP / ACME / Ceph calls “succeed” without remotes?
@@ -40,4 +45,16 @@ Hub image. Ingress + cert-manager Let's Encrypt is supported — see
## Which node name does the small seed use?
`pve01`.
`pve01`. Profiles `medium` and `ha-demo` use **`pve1` / `pve2` / `pve3`**.
## What ports does real Proxmox VE use vs this simulator?
Real PVE serves the Web UI and REST API on **HTTPS `:8006`** only. Related
management ports include SPICE `:3128`, VNC `:5900–5999`, SSH `:22`, and
Corosync UDP `:5405–5412`. Port `:8007` on real hardware is typically
**Proxmox Backup Server**, not PVE.
This lab publishes plain **HTTP `:8006`** in Compose (same port number as real
PVE). HTTPS belongs on Kubernetes Ingress. Optional proxmoxer TLS:
`docker compose --profile tls` on `:8443`. Host `:8007` is **not** used. Details:
[Ports and TLS](configuration.md#ports-and-tls).
+24 -14
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](getting-started.md) | [Русский](ru/getting-started.md)
# Getting started
Bring up a local laboratory cluster, authenticate, and exercise a first
@@ -17,15 +19,18 @@ Python toolchain for day-to-day use.
|---|---|
| [Published image](#1a-published-image-docker-hub) | Fastest lab using `inecs/proxmox-api-simulator` |
| [Helm / Kubernetes](kubernetes.md) | Cluster install with Ingress + Let's Encrypt |
| [Development checkout](#1b-development-checkout) | Contribute / bind-mount source / HTTPS gateway on `:8007` |
| [Development checkout](#1b-development-checkout) | Contribute / bind-mount source / HTTP API on `:8006` |
## 1a. Published image (Docker Hub)
Uses [`docker-compose.release.yml`](../docker-compose.release.yml) — PostgreSQL +
runtime simulator from Hub. No source build required.
> Laboratory / CI only — rotate `TICKET_SIGNING_KEY` and the DB password before
> any shared or networked demo. See [SECURITY.md](../SECURITY.md).
```bash
# from this repository, or download docker-compose.release.yml alone
# from this repository (compose file + docker/tls/)
docker compose -f docker-compose.release.yml pull
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
@@ -47,7 +52,7 @@ make release-seed PROFILE=small
| Host port | Service |
|---|---|
| `8006` | HTTP API + Web UI |
| `8006` | HTTP API + Web UI (same port as real PVE; real PVE uses HTTPS) |
| `5432` | PostgreSQL (localhost only) |
Migrations run automatically via the `migrate` one-shot service.
@@ -65,17 +70,22 @@ Services:
| Host port | Service |
|---|---|
| `8006` | HTTP API + Web UI |
| `8007` | HTTPS nginx gateway → simulator |
| `8006` | HTTP API + Web UI (same port as real PVE; real PVE uses HTTPS) |
| `5432` | PostgreSQL (localhost only) |
On real Proxmox VE the REST API is **only** `https://<host>:8006/api2/json/...`.
The lab publishes plain **HTTP** on host `:8006`; see
[Ports and TLS](configuration.md#ports-and-tls). Optional HTTPS for proxmoxer:
`docker compose --profile tls` → `https://localhost:8443/`. Host `:8007` is
**not** used (on hardware it is typically PBS, not PVE API).
Migrations apply automatically before the simulator becomes ready.
## 2. Wait until ready
```bash
curl http://localhost:8006/health/live
curl http://localhost:8006/health/ready
curl -sS http://localhost:8006/health/live
curl -sS http://localhost:8006/health/ready
```
`/health/ready` returns HTTP 503 until PostgreSQL is reachable **and** the
@@ -94,7 +104,7 @@ local storages, and the standard development principals. See
## 4. Check the API version
```bash
curl -s http://localhost:8006/api2/json/version | jq .
curl -sS http://localhost:8006/api2/json/version | jq .
```
The cold-start contract defaults to the bundled PVE **9.2.3** snapshot in Docker
@@ -104,7 +114,7 @@ Compose. Switch majors 6–9 from the Web UI or
## 5. Authenticate
```bash
curl -s -X POST \
curl -sS -X POST \
-d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket | jq .
```
@@ -120,10 +130,10 @@ Details: [Authentication](authentication.md).
```bash
# replace TICKET / CSRF from the previous response
curl -s -H "Cookie: PVEAuthCookie=$TICKET" \
curl -sS -H "Cookie: PVEAuthCookie=$TICKET" \
http://localhost:8006/api2/json/nodes/pve01/qemu | jq .
curl -s -X POST \
curl -sS -X POST \
-H "Cookie: PVEAuthCookie=$TICKET" \
-H "CSRFPreventionToken: $CSRF" \
http://localhost:8006/api2/json/nodes/pve01/qemu/100/status/start | jq .
@@ -132,7 +142,7 @@ curl -s -X POST \
Async operations return a UPID string. Poll until the task finishes:
```bash
curl -s -H "Cookie: PVEAuthCookie=$TICKET" \
curl -sS -H "Cookie: PVEAuthCookie=$TICKET" \
"http://localhost:8006/api2/json/nodes/pve01/tasks/${UPID}/status" | jq .
```
@@ -140,8 +150,8 @@ curl -s -H "Cookie: PVEAuthCookie=$TICKET" \
Visit [http://localhost:8006/](http://localhost:8006/) for the interactive
console, contract catalog (PVE 6–9), compatibility view, runtime contract apply,
and demo-cluster controls. See [Web UI](web-ui.md) for light/dark theme
screenshots and the full feature list.
and demo-cluster controls. See [Web UI](web-ui.md) for screenshots and the
full feature list.
## 8. Try a client library
Binary file not shown.

After

Width:  |  Height:  |  Size: 140 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 53 KiB

After

Width:  |  Height:  |  Size: 175 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 135 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 222 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 171 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 86 KiB

Some files were not shown because too many files have changed in this diff Show More