Compare commits

..

9 Commits

Author SHA1 Message Date
Sergey Antropoff 98ba5004ad Fix Deploy stage for Alpine ash: drop bash arrays.
Jenkins docker agents run sh without array syntax; set simulator/migrate images with POSIX conditionals.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-28 02:12:16 +03:00
Sergey Antropoff c64cd4ca8b Raise deploy rollout timeout to 1200s and set images in one step.
Avoid double ReplicaSet churn and ImagePullBackOff races during Hub pulls.
2026-07-28 00:56:46 +03:00
Sergey Antropoff f81d6186ab Add SemVer VERSION, Jenkins CI/CD, and release 0.1.1.
Ship Harbor/Hub push with latest+version tags, deploy to proxmox.devops.org.ru, and show the package version badge in Help → About.
2026-07-28 00:24:05 +03:00
Sergey Antropoff f981ca1bf1 Return clear node-missing JSON and reset Web UI auth on 401.
Keep Ingress from rewriting API 404/405 into branded HTML; document curl/auth and a prompt for sibling sims.
2026-07-22 06:52:45 +03:00
Sergey Antropoff baa1f58ad0 Fix CI/pulumi blockers and document test suites with latest PASS results.
Stabilize node status, cluster-wide Ceph OSD ids, and QEMU cpu utilization so
status/list no longer 500 on model strings; align pulumi defaults to pve1;
add bilingual testing docs with the 2026-07-18 ci-all + pulumi-tests pass.
2026-07-18 09:38:28 +03:00
Sergey Antropoff 6f42d4c318 Simplify bilingual README and docs index around the Web UI.
Drop CI/Docker/License badges, keep a single main-window screenshot, and
link to the Web UI guide at the bottom of the English and Russian pages.
2026-07-18 09:01:44 +03:00
Sergey Antropoff 176c8fd71c Ignore generated pulumi-tests report artifacts.
Keep HTML/JSON/JUnit output from make pulumi-tests out of the repo.
2026-07-18 08:51:52 +03:00
Sergey Antropoff 0773f721ea Align sized cluster seeds and GET dumps with PVE wire shapes; restyle DATA panel.
- Scale small/large/big seeds (3×50 / 10×1000 / 20×2000) with proportional
  backups, snapshots, HA, replication, Ceph capacity, and OSD totals
  (10 / 100 / 500) plus matching node disks and crush/pg metadata
- Enrich handler responses for apt, certificates, qemu/lxc status, storage,
  SDN, metrics export, and related cluster/node dumps
- Flatten nested body_example fields into PARAMS and sync the request body
  via dotted paths (oVirt-style)
- Restyle DATA controls as size cards with full-width Reset to minimal /
  Refresh stats; unload reloads the minimal cluster
2026-07-18 08:46:11 +03:00
Sergey Antropoff 48df10b17e Prepare 0.1.0 for lab release: durable handlers, HTTP Compose, CI, and pulumi-tests.
- Harden DB-backed handlers and seed profiles; align client wire shapes for
  cluster resources, QEMU config, and node SSL fields
- Serve plain HTTP on Compose :8006; keep TLS optional (--profile tls) and
  terminate HTTPS at Kubernetes Ingress
- Add pulumi-tests (full contract surface majors 6–9 + BPG lifecycle) and
  make pulumi-tests
- Ship bilingual docs, CHANGELOG, SECURITY, CONTRIBUTING, and GitHub Actions
  (make ci + Compose/Helm validation)
2026-07-18 04:18:05 +03:00
225 changed files with 13188 additions and 1861 deletions
+3 -1
View File
@@ -23,5 +23,7 @@ SIMULATION_SEED=42
SIMULATION_TIME_SCALE=10 SIMULATION_TIME_SCALE=10
SIMULATOR_ADMIN_ENABLED=false SIMULATOR_ADMIN_ENABLED=false
SIMULATOR_ADMIN_TOKEN=replace-with-a-long-random-secret SIMULATOR_ADMIN_TOKEN=replace-with-a-long-random-secret
PROXMOXER_HOST=tls-gateway PROXMOXER_HOST=
PROXMOXER_PORT=8443 PROXMOXER_PORT=8443
# For proxmoxer (HTTPS-only): docker compose --profile tls up -d
# then PROXMOXER_HOST=tls-gateway PROXMOXER_PORT=8443
+60
View File
@@ -0,0 +1,60 @@
name: CI
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Compose / Helm validate
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Validate Compose files
run: |
docker compose -f docker-compose.yml config --quiet
docker compose -f docker-compose.release.yml config --quiet
- uses: azure/setup-helm@v4
with:
version: v3.16.4
- name: Lint and render Helm chart
run: |
make helm-lint
make helm-template >/dev/null
quality:
name: make ci
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- name: Prepare lab env
run: test -f .env || cp .env.example .env
- name: Build runtime and development images
run: make install
- name: Ruff + mypy + offline pytest + API surface probe
run: make ci
- name: Upload coverage XML
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-xml
path: coverage.xml
if-no-files-found: ignore
+4 -1
View File
@@ -67,9 +67,12 @@ hypothesis/
*.sublime-project *.sublime-project
*.sublime-workspace *.sublime-workspace
# Docker / local runtime # Docker / local runtime (`make up-local` writes port remaps here)
*.log *.log
docker-compose.override.yml docker-compose.override.yml
# Lab probe output (regenerated by scripts/probe_api_surface.py) # Lab probe output (regenerated by scripts/probe_api_surface.py)
evidence/_api_surface_probe.json evidence/_api_surface_probe.json
# Pulumi suite reports (HTML/JSON/JUnit from make pulumi-tests)
pulumi-tests/reports/*
+30
View File
@@ -0,0 +1,30 @@
# Changelog
All notable changes to this project are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.1.0] - 2026-07-17
### Added
- Stateful Proxmox VE API simulator backed by PostgreSQL with durable UPID task
workers.
- Imported official API contracts for PVE majors **6.4-15**, **7.4-16**,
**8.4.5**, and **9.2.3** with full handler registration for declared methods.
- Interactive Web UI (catalog, runtime contract Apply, task monitor).
- Development Compose stack with plain HTTP on host `:8006` (real PVE port; real
PVE uses HTTPS). Optional TLS gateway via `--profile tls` on `:8443`.
- Published runtime image workflow (`make release`) for
[`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator).
- `docker-compose.release.yml` + Helm chart with optional Ingress / cert-manager.
- Bilingual documentation (English + Russian) and client cookbooks.
- GitHub Actions CI (`make ci`, Compose/Helm validation).
### Security
- Explicit lab-only threat model: default secrets and open `/ui` / `/admin`
helpers are for local/CI use. See [SECURITY.md](SECURITY.md).
[0.1.0]: https://github.com/sergeyantropoff/proxmox-api-simulator/releases/tag/v0.1.0
+69
View File
@@ -0,0 +1,69 @@
**Language / Язык:** [English](CONTRIBUTING.md) | [Русский](CONTRIBUTING.ru.md)
# Contributing
Thanks for helping improve the Proxmox VE API laboratory simulator.
## Prerequisites
- Docker + Docker Compose
- `make`
- No local Python toolchain required for day-to-day work (tools run in Compose)
## Local loop
```bash
cp -n .env.example .env
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
```
Primary HTTP endpoint (same port as real PVE): `http://localhost:8006/`
Optional HTTPS for proxmoxer: `docker compose --profile tls` →
`https://localhost:8443/` (self-signed — use `curl -sk` or accept the browser
warning).
## Quality gates (must pass before a PR)
```bash
make ci # ruff format/check + mypy + offline pytest + surface probe
make ci-all # also remaining integration + proxmoxer compatibility
make helm-lint # chart lint (+ ingress example values)
make pulumi-tests # Pulumi surface (majors 6–9) + pulumi-proxmoxve lifecycle
```
GitHub Actions runs `make ci` plus Compose/Helm validation on every push and PR
to `main`. Run `make ci-all`, `make helm-lint`, and `make pulumi-tests` locally
before larger API or client-facing changes.
Suite layout, Make targets, and the latest recorded local results:
[docs/testing.md](docs/testing.md).
## Project rules worth remembering
1. Mutations must **persist to PostgreSQL** (tables and/or jsonb metadata).
2. Do **not** add user-facing “not supported in the simulator” style errors —
see `.cursor/rules/durable-simulator.mdc`.
3. Prefer matching Proxmox request/response shapes from the contract snapshot.
4. Keep EN and RU docs in sync when you change operator-facing behaviour.
## Docs
- Index: [docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md)
- Security / lab threat model: [SECURITY.md](SECURITY.md)
## Releases
Maintainers publish the runtime image with:
```bash
docker login
make release # pushes inecs/proxmox-api-simulator:<version> (+ :latest)
```
After a public release, paste the overview from
[docs/docker-hub-overview.md](docs/docker-hub-overview.md) into the Docker Hub
repository description if it drifted.
+66
View File
@@ -0,0 +1,66 @@
**Language / Язык:** [English](CONTRIBUTING.md) | [Русский](CONTRIBUTING.ru.md)
# Участие в разработке
Спасибо за помощь с лабораторным симулятором Proxmox VE API.
## Требования
- Docker + Docker Compose
- `make`
- Локальный Python для повседневной работы не нужен (инструменты в Compose)
## Локальный цикл
```bash
cp -n .env.example .env
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
```
Основной HTTPS-эндпоинт (как у реального PVE): `http://localhost:8006/`
(self-signed — `curl -sk` или принять предупреждение в браузере).
## Quality gates (обязательны перед PR)
```bash
make ci # ruff format/check + mypy + offline pytest + surface probe
make ci-all # ещё integration + proxmoxer compatibility
make helm-lint # lint chart (+ ingress example values)
make pulumi-tests # Pulumi surface (majors 6–9) + pulumi-proxmoxve lifecycle
```
GitHub Actions на каждый push/PR в `main` запускает `make ci` и проверку
Compose/Helm. Перед крупными API/клиентскими изменениями локально прогоняйте
`make ci-all`, `make helm-lint` и `make pulumi-tests`.
Карта наборов, Make-цели и последние записанные локальные результаты:
[docs/ru/testing.md](docs/ru/testing.md).
## Важные правила проекта
1. Мутации должны **persist в PostgreSQL** (таблицы и/или jsonb metadata).
2. Не добавляйте user-facing сообщения вроде «not supported in the simulator» —
см. `.cursor/rules/durable-simulator.mdc`.
3. Сохраняйте формы запросов/ответов Proxmox из снимка контракта.
4. При изменении операторского поведения синхронизируйте EN и RU docs.
## Документация
- Индекс: [docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md)
- Безопасность / модель угроз лаборатории: [SECURITY.md](SECURITY.md)
## Релизы
Maintainers публикуют runtime-образ так:
```bash
docker login
make release # пушит inecs/proxmox-api-simulator:<version> (+ :latest)
```
После публичного релиза при необходимости вставьте overview из
[docs/docker-hub-overview.md](docs/docker-hub-overview.md) в описание репозитория
Docker Hub.
+4 -1
View File
@@ -16,7 +16,9 @@ FROM python:3.13-slim AS runtime
ARG APP_VERSION=0.1.0 ARG APP_VERSION=0.1.0
LABEL org.opencontainers.image.title="proxmox-api-simulator" \ LABEL org.opencontainers.image.title="proxmox-api-simulator" \
org.opencontainers.image.version="$APP_VERSION" \ org.opencontainers.image.version="$APP_VERSION" \
org.opencontainers.image.source="https://github.com/example/proxmox-api-simulator" org.opencontainers.image.description="Stateful Proxmox VE API simulator for labs and CI" \
org.opencontainers.image.source="https://github.com/sergeyantropoff/proxmox-api-simulator" \
org.opencontainers.image.licenses="Apache-2.0"
ENV PATH="/opt/venv/bin:$PATH" \ ENV PATH="/opt/venv/bin:$PATH" \
PYTHONDONTWRITEBYTECODE=1 \ PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \ PYTHONUNBUFFERED=1 \
@@ -25,6 +27,7 @@ ENV PATH="/opt/venv/bin:$PATH" \
RUN groupadd --system --gid 10001 simulator \ RUN groupadd --system --gid 10001 simulator \
&& useradd --system --uid 10001 --gid simulator --home-dir /app --no-create-home simulator && useradd --system --uid 10001 --gid simulator --home-dir /app --no-create-home simulator
COPY --from=builder /opt/venv /opt/venv COPY --from=builder /opt/venv /opt/venv
COPY VERSION /app/VERSION
COPY contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json /app/contracts/pve-9.2.3.json COPY contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json /app/contracts/pve-9.2.3.json
COPY evidence/ /app/evidence/ COPY evidence/ /app/evidence/
WORKDIR /app WORKDIR /app
Vendored
+205
View File
@@ -0,0 +1,205 @@
// Proxmox API Simulator — CI/CD: multi-arch push (Harbor + Docker Hub) + deploy
//
// Один файл: Build (DinD) → Deploy (kubectl set image в релиз simulators).
// Первый install lab — через DevOpsTools/K3S: make addon-simulators
// (Ingress https://proxmox.devops.org.ru уже из addon values).
//
// Версия образа = VERSION из коммита. Локальный bump: make bump-patch / make push.
// Теги: :{VERSION} и :latest (без sha).
//
// Credentials (Global):
// harbor-devops-tools-push-pull-access — Harbor devops-tools (robot)
// docker-hub — Docker Hub (inecs)
// k3s-kubeconfig — kubeconfig к K3S
// ssh-gitea-key — SCM checkout Gitea (в job)
pipeline {
agent none
options {
buildDiscarder(logRotator(numToKeepStr: '20'))
disableConcurrentBuilds()
timeout(time: 60, unit: 'MINUTES')
timestamps()
}
environment {
HARBOR_REGISTRY = 'hub.antropoff.ru'
HARBOR_IMAGE = 'hub.antropoff.ru/devops-tools/proxmox-api-simulator'
DOCKERHUB_IMAGE = 'inecs/proxmox-api-simulator'
BUILDX_BUILDER = "jenkins-proxmox-api-simulator-${env.BUILD_NUMBER}"
HELM_NAMESPACE = 'simulators'
DEPLOYMENT_NAME = 'simulators-proxmox'
INGRESS_HOST = 'proxmox.devops.org.ru'
TZ = 'Europe/Moscow'
}
stages {
stage('Build & push') {
when {
anyOf {
branch 'main'
branch 'master'
}
}
agent { label 'docker' }
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Version') {
steps {
script {
env.IMAGE_VERSION = readFile('VERSION').trim()
if (!env.IMAGE_VERSION) {
error('VERSION file is empty')
}
echo "IMAGE_VERSION from VERSION → ${env.IMAGE_VERSION}"
}
}
}
stage('Buildx push') {
steps {
withCredentials([
usernamePassword(
credentialsId: 'harbor-devops-tools-push-pull-access',
usernameVariable: 'HARBOR_USER',
passwordVariable: 'HARBOR_PASS'
),
usernamePassword(
credentialsId: 'docker-hub',
usernameVariable: 'DOCKERHUB_USER',
passwordVariable: 'DOCKERHUB_PASS'
)
]) {
container('docker') {
sh '''
set -eux
test -n "${IMAGE_VERSION}"
echo "Building tags: ${IMAGE_VERSION}, latest (amd64+arm64)"
echo "$HARBOR_PASS" | docker login "$HARBOR_REGISTRY" -u "$HARBOR_USER" --password-stdin
echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USER" --password-stdin
docker buildx rm "$BUILDX_BUILDER" 2>/dev/null || true
docker buildx create --name "$BUILDX_BUILDER" --driver docker-container --use
docker buildx inspect --bootstrap >/dev/null
docker buildx build \
--platform linux/amd64,linux/arm64 \
--target runtime \
--build-arg "APP_VERSION=${IMAGE_VERSION}" \
--provenance=false --sbom=false --push \
-t "${HARBOR_IMAGE}:${IMAGE_VERSION}" \
-t "${HARBOR_IMAGE}:latest" \
-t "${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" \
-t "${DOCKERHUB_IMAGE}:latest" \
-f Dockerfile .
echo "--- Harbor ---"
docker buildx imagetools inspect "${HARBOR_IMAGE}:${IMAGE_VERSION}" | sed -n '1,40p'
echo "--- Docker Hub ---"
docker buildx imagetools inspect "${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" | sed -n '1,40p'
docker buildx rm "$BUILDX_BUILDER" || true
'''
}
}
}
}
stage('Deploy') {
steps {
withCredentials([
file(credentialsId: 'k3s-kubeconfig', variable: 'KUBECONFIG')
]) {
container('docker') {
sh '''
set -eux
if ! command -v kubectl >/dev/null 2>&1; then
apk add --no-cache curl >/dev/null
ARCH="$(uname -m)"
case "$ARCH" in
x86_64) ARCH=amd64 ;;
aarch64|arm64) ARCH=arm64 ;;
esac
KVER="$(curl -fsSL https://dl.k8s.io/release/stable.txt)"
curl -fsSLo /usr/local/bin/kubectl \
"https://dl.k8s.io/release/${KVER}/bin/linux/${ARCH}/kubectl"
chmod +x /usr/local/bin/kubectl
fi
kubectl version --client || true
if ! kubectl -n "$HELM_NAMESPACE" get deploy "$DEPLOYMENT_NAME" >/dev/null 2>&1; then
echo "Deployment ${HELM_NAMESPACE}/${DEPLOYMENT_NAME} not found." >&2
echo "Сначала: make addon-simulators (Ingress ${INGRESS_HOST})." >&2
exit 1
fi
echo "Rolling ${DOCKERHUB_IMAGE}:${IMAGE_VERSION} → ${HELM_NAMESPACE}/${DEPLOYMENT_NAME}"
echo "Public URL: https://${INGRESS_HOST}/"
# POSIX sh (Alpine ash): no bash arrays. One set-image when migrate exists.
INITS="$(kubectl -n "$HELM_NAMESPACE" get deploy "$DEPLOYMENT_NAME" \
-o jsonpath='{.spec.template.spec.initContainers[*].name}' || true)"
if printf '%s\n' $INITS | grep -qx migrate; then
kubectl -n "$HELM_NAMESPACE" set image \
"deployment/${DEPLOYMENT_NAME}" \
"simulator=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}" \
"migrate=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}"
else
kubectl -n "$HELM_NAMESPACE" set image \
"deployment/${DEPLOYMENT_NAME}" \
"simulator=${DOCKERHUB_IMAGE}:${IMAGE_VERSION}"
fi
kubectl -n "$HELM_NAMESPACE" rollout status \
"deployment/${DEPLOYMENT_NAME}" --timeout=1200s
kubectl -n "$HELM_NAMESPACE" get deploy,po,ing -o wide
'''
}
}
}
}
}
post {
always {
script {
try {
container('docker') {
sh 'docker buildx rm "$BUILDX_BUILDER" 2>/dev/null || true'
}
} catch (Ignored) {
// pod may already be gone
}
}
}
success {
echo "✓ ${DOCKERHUB_IMAGE}:{${IMAGE_VERSION},latest} → https://${INGRESS_HOST}/"
}
failure {
echo "✗ Build/push/deploy failed — см. лог"
}
}
}
}
post {
success {
echo "✓ Version: ${IMAGE_VERSION}"
echo "✓ Harbor: ${HARBOR_IMAGE}:{${IMAGE_VERSION},latest}"
echo "✓ Hub: ${DOCKERHUB_IMAGE}:{${IMAGE_VERSION},latest}"
echo "✓ Live: https://${INGRESS_HOST}/"
}
failure {
echo "✗ Pipeline failed"
}
}
}
+82 -12
View File
@@ -4,19 +4,22 @@ SERVICE_SIM := simulator
PYTEST_OFFLINE := -m "not integration and not compatibility" PYTEST_OFFLINE := -m "not integration and not compatibility"
# Docker Hub release image (runtime target only — not the local bind-mount "dev" image). # Docker Hub release image (runtime target only — not the local bind-mount "dev" image).
# SemVer source of truth: VERSION (synced to pyproject + Helm via bump targets).
DOCKERHUB_USER ?= inecs DOCKERHUB_USER ?= inecs
IMAGE_NAME ?= proxmox-api-simulator IMAGE_NAME ?= proxmox-api-simulator
VERSION ?= $(shell sed -n 's/^version = "\(.*\)"/\1/p' pyproject.toml) VERSION ?= $(shell cat VERSION 2>/dev/null || echo 0.1.0)
DOCKER_IMAGE ?= $(DOCKERHUB_USER)/$(IMAGE_NAME) DOCKER_IMAGE ?= $(DOCKERHUB_USER)/$(IMAGE_NAME)
PUSH_LATEST ?= 1 PUSH_LATEST ?= 1
COMPOSE_RELEASE ?= $(COMPOSE) -f docker-compose.release.yml COMPOSE_RELEASE ?= $(COMPOSE) -f docker-compose.release.yml
HELM_CHART ?= ./helm/proxmox-api-simulator HELM_CHART ?= ./helm/proxmox-api-simulator
.PHONY: help install format lint typecheck test test-unit test-integration test-contract test-compatibility test-surface evidence coverage run dev up down restart logs docker-build docker-up docker-down docker-logs docker-restart db-up db-down db-migrate db-reset api-import api-diff seed clean ci ci-all shell release release-build release-up release-down release-seed helm-deps helm-template .PHONY: help install format lint typecheck test test-unit test-integration test-contract test-compatibility test-surface evidence coverage run dev up up-local down down-local restart logs docker-build docker-up docker-down docker-logs docker-restart db-up db-down db-migrate db-reset api-import api-diff seed clean ci ci-all shell release release-build release-up release-down release-seed helm-deps helm-template helm-lint pulumi-tests push bump-patch bump-minor version-commit
help: ## Show available commands help: ## Show available commands
@awk 'BEGIN {FS = ":.*## "}; /^[a-zA-Z0-9_-]+:.*## / {printf "%-22s %s\n", $$1, $$2}' $(MAKEFILE_LIST) @awk 'BEGIN {FS = ":.*## "}; /^[a-zA-Z0-9_-]+:.*## / {printf "%-22s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
@echo
@echo "Version: $$(cat VERSION 2>/dev/null || echo 0.1.0) (make bump-patch | bump-minor)"
install: ## Build runtime and development images install: ## Build runtime and development images
@test -f .env || cp .env.example .env @test -f .env || cp .env.example .env
@@ -45,11 +48,16 @@ test-integration: ## Run tests that require PostgreSQL
test-contract: ## Run offline API contract tests test-contract: ## Run offline API contract tests
$(COMPOSE) run --rm --no-deps $(SERVICE_DEV) pytest -m contract $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) pytest -m contract
test-compatibility: ## Run proxmoxer smoke flow against the Compose stack test-compatibility: ## Run proxmoxer smoke flow (needs --profile tls; proxmoxer is HTTPS-only)
@test -f .env || cp .env.example .env @test -f .env || cp .env.example .env
$(COMPOSE) up -d --build --wait $(COMPOSE) --profile tls up -d --build --wait
$(COMPOSE) run --rm --entrypoint python $(SERVICE_SIM) -m app.simulation.seed_cli # Medium profile provides pve1/pve2/pve3 required by the proxmoxer migration smoke.
$(COMPOSE) run --rm $(SERVICE_DEV) pytest -m compatibility $(COMPOSE) run --rm -e SEED_PROFILE=medium --entrypoint python $(SERVICE_SIM) \
-m app.simulation.seed_cli
$(COMPOSE) run --rm \
-e PROXMOXER_HOST=tls-gateway \
-e PROXMOXER_PORT=8443 \
$(SERVICE_DEV) pytest -m compatibility
test-surface: ## Probe every declared method on majors 6-9 (0x501 / 0xexception) test-surface: ## Probe every declared method on majors 6-9 (0x501 / 0xexception)
@test -f .env || cp .env.example .env @test -f .env || cp .env.example .env
@@ -66,13 +74,21 @@ run: ## Run the application in the foreground
@test -f .env || cp .env.example .env @test -f .env || cp .env.example .env
$(COMPOSE) up --build $(COMPOSE) up --build
up: ## Start PostgreSQL, simulator, and TLS gateway up: ## Start PostgreSQL and simulator (plain HTTP :8006)
@test -f .env || cp .env.example .env @test -f .env || cp .env.example .env
$(COMPOSE) up -d --build --wait $(COMPOSE) up -d --build --wait
up-local: ## Start locally; remap busy ports via gitignored docker-compose.override.yml
@COMPOSE="$(COMPOSE)" ./scripts/up-local.sh
down: ## Stop local services down: ## Stop local services
$(COMPOSE) down $(COMPOSE) down
down-local: ## Stop local stack and remove gitignored docker-compose.override.yml
@$(COMPOSE) down
@rm -f docker-compose.override.yml
@echo "Stopped local stack and removed docker-compose.override.yml"
restart: ## Rebuild and restart the stack restart: ## Rebuild and restart the stack
@test -f .env || cp .env.example .env @test -f .env || cp .env.example .env
$(COMPOSE) up -d --build --force-recreate --wait $(COMPOSE) up -d --build --force-recreate --wait
@@ -90,8 +106,8 @@ docker-build: ## Build runtime and development images
docker-up: up ## Alias for up docker-up: up ## Alias for up
docker-restart: ## Rebuild and recreate simulator and TLS gateway only docker-restart: ## Rebuild and recreate the simulator
$(COMPOSE) up -d --build --force-recreate simulator tls-gateway $(COMPOSE) up -d --build --force-recreate simulator
docker-down: down ## Alias for down docker-down: down ## Alias for down
@@ -122,7 +138,11 @@ api-diff: ## Compare API snapshots
seed: ## Seed simulation data seed: ## Seed simulation data
@test -f .env || cp .env.example .env @test -f .env || cp .env.example .env
SEED_PROFILE="$${PROFILE:-small}" $(COMPOSE) run --rm --entrypoint python $(SERVICE_SIM) -m app.simulation.seed_cli $(COMPOSE) run --rm \
-e SEED_PROFILE="$${PROFILE:-small}" \
-e SEED_LARGE_NODES="$${SEED_LARGE_NODES:-10}" \
-e SEED_LARGE_RESOURCES="$${SEED_LARGE_RESOURCES:-1000}" \
--entrypoint python $(SERVICE_SIM) -m app.simulation.seed_cli
shell: ## Open an interactive shell in the development container shell: ## Open an interactive shell in the development container
$(COMPOSE) run --rm --no-deps $(SERVICE_DEV) bash $(COMPOSE) run --rm --no-deps $(SERVICE_DEV) bash
@@ -170,14 +190,64 @@ release-down: ## Stop the published Hub stack
$(COMPOSE_RELEASE) down $(COMPOSE_RELEASE) down
release-seed: ## Seed the published Hub stack (PROFILE=small by default) release-seed: ## Seed the published Hub stack (PROFILE=small by default)
SEED_PROFILE="$${PROFILE:-small}" IMAGE_TAG="$${IMAGE_TAG:-$(VERSION)}" DOCKER_IMAGE="$(DOCKER_IMAGE)" \ IMAGE_TAG="$${IMAGE_TAG:-$(VERSION)}" DOCKER_IMAGE="$(DOCKER_IMAGE)" \
$(COMPOSE_RELEASE) run --rm --entrypoint python simulator -m app.simulation.seed_cli $(COMPOSE_RELEASE) run --rm \
-e SEED_PROFILE="$${PROFILE:-small}" \
-e SEED_LARGE_NODES="$${SEED_LARGE_NODES:-10}" \
-e SEED_LARGE_RESOURCES="$${SEED_LARGE_RESOURCES:-1000}" \
--entrypoint python simulator -m app.simulation.seed_cli
helm-deps: ## No-op placeholder (chart has no OCI dependencies) helm-deps: ## No-op placeholder (chart has no OCI dependencies)
@echo "Chart $(HELM_CHART) vendors PostgreSQL templates; no helm dependency update required." @echo "Chart $(HELM_CHART) vendors PostgreSQL templates; no helm dependency update required."
helm-lint: ## Lint the Helm chart (requires helm)
helm lint $(HELM_CHART)
helm lint $(HELM_CHART) -f $(HELM_CHART)/values-ingress-example.yaml \
--set certManager.email=docs@example.com \
--set secret.ticketSigningKey=docs-only-signing-key
helm-template: ## Render Helm manifests locally (requires helm) helm-template: ## Render Helm manifests locally (requires helm)
helm template pve-sim $(HELM_CHART) \ helm template pve-sim $(HELM_CHART) \
-f $(HELM_CHART)/values-ingress-example.yaml \ -f $(HELM_CHART)/values-ingress-example.yaml \
--set certManager.email=docs@example.com \ --set certManager.email=docs@example.com \
--set secret.ticketSigningKey=docs-only-signing-key --set secret.ticketSigningKey=docs-only-signing-key
pulumi-tests: ## Full Pulumi suite (surface majors 6–9 + lifecycle, HTML report)
$(MAKE) -C pulumi-tests up
$(MAKE) -C pulumi-tests test
bump-patch: ## VERSION +0.0.1 (pyproject + Helm)
@PYTHONPATH=. python3 scripts/bump_version.py patch
@echo "VERSION → $$(cat VERSION)"
bump-minor: ## VERSION +0.1.0 (pyproject + Helm)
@PYTHONPATH=. python3 scripts/bump_version.py minor
@echo "VERSION → $$(cat VERSION)"
version-commit: ## Stage version files and commit "Bump version to …"
@git add VERSION pyproject.toml helm/proxmox-api-simulator/Chart.yaml helm/proxmox-api-simulator/values.yaml
@if git diff --cached --quiet; then \
echo "Nothing to commit for version files."; \
else \
git commit -m "Bump version to $$(cat VERSION)."; \
fi
push: ## bump-patch + commit version + push origin (GitHub + antropoff.ru)
@$(MAKE) bump-patch
@$(MAKE) version-commit
@set -e; \
git add .; \
echo "=== staged ==="; \
git status --short; \
echo; \
if git diff --cached --quiet; then \
echo "Nothing else to commit — pushing current branch."; \
else \
echo "Enter commit message for remaining changes, then Ctrl-D:"; \
msg=$$(cat </dev/tty); \
if [ -z "$$msg" ]; then echo "Empty commit message, aborting." >&2; exit 1; fi; \
git commit -m "$$msg"; \
fi; \
echo "Pushing to both remotes:"; \
git remote get-url --push --all origin | sed 's/^/ - /'; \
git push origin HEAD
+47 -15
View File
@@ -1,9 +1,16 @@
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
# proxmox-api-simulator # proxmox-api-simulator
Stateful asynchronous [Proxmox VE](https://www.proxmox.com/) API simulator for Stateful asynchronous [Proxmox VE](https://www.proxmox.com/) API simulator for
testing API clients and infrastructure tooling without a real hypervisor testing API clients and infrastructure tooling without a real hypervisor
cluster. cluster.
> **Laboratory / CI only.** Default credentials, signing keys, and open UI/admin
> helpers are intentional lab defaults. Do **not** expose this stack to the
> public Internet without replacing secrets and adding your own controls.
> See [SECURITY.md](SECURITY.md) and [Security](docs/security.md).
The simulator is backed by PostgreSQL, driven by imported official API The simulator is backed by PostgreSQL, driven by imported official API
contracts, and exposes the same `/api2/json` and `/api2/extjs` surfaces as contracts, and exposes the same `/api2/json` and `/api2/extjs` surfaces as
Proxmox VE. Semantic handlers persist mutations; long-running work returns Proxmox VE. Semantic handlers persist mutations; long-running work returns
@@ -37,13 +44,16 @@ Image: [`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-ap
### Docker Compose ### Docker Compose
Needs a checkout that includes `docker-compose.release.yml`. Do not publish host
`:8006` beyond a trusted lab without rotating `TICKET_SIGNING_KEY` / DB password.
```bash ```bash
docker compose -f docker-compose.release.yml up -d docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \ docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli simulator -m app.simulation.seed_cli
curl http://localhost:8006/health/ready curl -sS http://localhost:8006/health/ready
curl http://localhost:8006/api2/json/version curl -sS http://localhost:8006/api2/json/version
``` ```
Or: `make release-up && make release-seed PROFILE=small` Or: `make release-up && make release-seed PROFILE=small`
@@ -57,10 +67,14 @@ helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
--set certManager.email=you@example.com \ --set certManager.email=you@example.com \
--set ingress.hosts[0].host=pve-sim.example.com \ --set ingress.hosts[0].host=pve-sim.example.com \
--set ingress.tls[0].hosts[0]=pve-sim.example.com \ --set ingress.tls[0].hosts[0]=pve-sim.example.com \
--set secret.ticketSigningKey="$(openssl rand -hex 32)" --set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set postgresql.auth.password="$(openssl rand -hex 16)"
``` ```
Requires an Ingress controller and cert-manager. Details: Requires an Ingress controller and cert-manager. The chart Service speaks
**HTTP** `:8006`; TLS terminates at Ingress. Compose also serves plain HTTP on
`:8006`; optional HTTPS for proxmoxer-style clients is
`docker compose --profile tls` on host `:8443`. Details:
[Kubernetes / Helm](docs/kubernetes.md). [Kubernetes / Helm](docs/kubernetes.md).
- HTTP API and Web UI (Compose): [http://localhost:8006/](http://localhost:8006/) - HTTP API and Web UI (Compose): [http://localhost:8006/](http://localhost:8006/)
@@ -76,29 +90,34 @@ make install
make up make up
make seed PROFILE=small make seed PROFILE=small
curl http://localhost:8006/health/ready curl -sS http://localhost:8006/health/ready
curl http://localhost:8006/api2/json/version curl -sS http://localhost:8006/api2/json/version
curl -X POST -d 'username=root@pam&password=secret' \ curl -sS -X POST -d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket http://localhost:8006/api2/json/access/ticket
``` ```
- HTTP API and Web UI: [http://localhost:8006/](http://localhost:8006/) - HTTP API and Web UI: [http://localhost:8006/](http://localhost:8006/)
- HTTPS gateway (self-signed, development only): `https://localhost:8007` (real PVE uses **HTTPS** on `:8006`; this lab uses plain HTTP on the same
— checked-in `docker/tls/server.key` is a **lab-only** localhost cert; do not port. Optional TLS for proxmoxer: `docker compose --profile tls` →
reuse it outside local Compose. `https://localhost:8443/`)
- Port map and TLS notes: [Ports and TLS](docs/configuration.md#ports-and-tls).
- FastAPI schema docs: [http://localhost:8006/docs](http://localhost:8006/docs) - FastAPI schema docs: [http://localhost:8006/docs](http://localhost:8006/docs)
### Web UI ### Web UI
Interactive console with light/dark themes, endpoint catalog for PVE 6–9, and Interactive console with light/dark themes, endpoint catalog for PVE 6–9,
runtime contract hot-swap. More detail: [Web UI](docs/web-ui.md). runtime contract hot-swap, and a UPID task monitor.
![Web UI light theme](docs/images/web-ui-light.png) ![Web UI main](docs/images/web-ui-main.png)
![Web UI dark theme](docs/images/web-ui-dark.png) More screens and details: [Web UI](docs/web-ui.md).
## Documentation ## Documentation
Documentation is bilingual. Use the **Language / Язык** switcher at the top of
each page, or open the Russian root [README.ru.md](README.ru.md). Index:
[docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md).
| Guide | Description | | Guide | Description |
|---|---| |---|---|
| [Getting started](docs/getting-started.md) | First successful lab session | | [Getting started](docs/getting-started.md) | First successful lab session |
@@ -120,6 +139,8 @@ runtime contract hot-swap. More detail: [Web UI](docs/web-ui.md).
| [Compatibility](docs/compatibility.md) | Evidence model and release matrix | | [Compatibility](docs/compatibility.md) | Evidence model and release matrix |
Runnable cookbooks live under [`examples/`](examples/README.md). Runnable cookbooks live under [`examples/`](examples/README.md).
Pulumi integration suite (contract surface majors 6–9 + lifecycle, HTML report):
[`pulumi-tests/`](pulumi-tests/README.md) (`make pulumi-tests`).
## proxmoxer (HTTPS gateway) ## proxmoxer (HTTPS gateway)
@@ -128,7 +149,7 @@ from proxmoxer import ProxmoxAPI
proxmox = ProxmoxAPI( proxmox = ProxmoxAPI(
"localhost", "localhost",
port=8007, port=8006,
user="root@pam", user="root@pam",
password="secret", password="secret",
verify_ssl=False, # local self-signed development certificate only verify_ssl=False, # local self-signed development certificate only
@@ -169,9 +190,20 @@ make release-build # build/tag only, no push
make release-up && make release-seed # run the published stack locally make release-up && make release-seed # run the published stack locally
``` ```
## Contributing / security / changelog
- [CONTRIBUTING.md](CONTRIBUTING.md) · [CONTRIBUTING.ru.md](CONTRIBUTING.ru.md)
- [Testing](docs/testing.md) · [Тесты](docs/ru/testing.md) — suites, Make targets, latest results
- [SECURITY.md](SECURITY.md)
- [CHANGELOG.md](CHANGELOG.md)
## What this is not ## What this is not
- Not a hypervisor: no KVM/LXC execution on bare metal or nested hosts. - Not a hypervisor: no KVM/LXC execution on bare metal or nested hosts.
- Not a drop-in multi-tenant production Proxmox replacement. - Not a drop-in multi-tenant production Proxmox replacement.
- Remote IdP / LDAP / live Ceph / live ACME directories are simulated locally; - Remote IdP / LDAP / live Ceph / live ACME directories are simulated locally;
they do not call real external systems. they do not call real external systems.
## See also
- [Web UI](docs/web-ui.md) — interactive console, catalogs, DATA panel, and screenshots
+210
View File
@@ -0,0 +1,210 @@
**Language / Язык:** [English](README.md) | [Русский](README.ru.md)
# proxmox-api-simulator
Stateful-асинхронный симулятор API [Proxmox VE](https://www.proxmox.com/) для
тестирования API-клиентов и инфраструктурных инструментов без реального
гипервизорного кластера.
> **Только лаборатория / CI.** Учётные данные, signing keys и открытые UI/admin
> helpers по умолчанию — намеренные лабораторные значения. **Не** выставляйте
> стек в публичный Интернет без замены секретов и своих сетевых ограничений.
> См. [SECURITY.md](SECURITY.md) и [Безопасность](docs/ru/security.md).
Симулятор работает на PostgreSQL, опирается на импортированные официальные
контракты API и предоставляет те же поверхности `/api2/json` и `/api2/extjs`,
что и Proxmox VE. Семантические обработчики сохраняют мутации; длительные
операции возвращают устойчивые UPID, которые выполняют воркеры с арендой задач.
## Проверенное покрытие API
Реестр обработчиков и верифицированные ledger поверхности — **100%** для каждого
включённого major:
| Контракт | Declared | Implemented | Verified |
|---|---:|---:|---:|
| PVE 6.4-15 | 504 | 504 | 504 |
| PVE 7.4-16 | 540 | 540 | 540 |
| PVE 8.4.5 | 605 | 605 | 605 |
| PVE 9.2.3 | 675 | 675 | 675 |
Переключение активного контракта в runtime — из Web UI (**Apply as runtime**) или
`POST /ui/api/contract/apply?major=N` — каждый Apply загружает
`evidence/pve-{version}.json`, чтобы observed/verified следовали выбранному
major. После импорта нового контракта перегенерируйте ledger: `make evidence`.
Живые отчёты: `/admin/compatibility` (также `.md` / `.html`). См.
[Совместимость](docs/ru/compatibility.md) и [Версии API](docs/ru/api-versions.md).
> Это измеримое покрытие контракта и обработчиков лабораторного симулятора —
> не утверждение, что каждый краевой случай Proxmox или удалённая интеграция
> ведёт себя идентично продакшен-железу.
## Быстрый старт (опубликованный образ)
Образ: [`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator)
### Docker Compose
Нужен checkout с `docker-compose.release.yml`. Не публикуйте хост `:8006` за
пределы доверенной лаборатории без ротации `TICKET_SIGNING_KEY` / пароля БД.
```bash
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
```
Или: `make release-up && make release-seed PROFILE=small`
### Helm (Kubernetes + Ingress + Let's Encrypt)
```bash
helm upgrade --install pve-sim ./helm/proxmox-api-simulator \
-n proxmox-sim --create-namespace \
-f ./helm/proxmox-api-simulator/values-ingress-example.yaml \
--set certManager.email=you@example.com \
--set ingress.hosts[0].host=pve-sim.example.com \
--set ingress.tls[0].hosts[0]=pve-sim.example.com \
--set secret.ticketSigningKey="$(openssl rand -hex 32)" \
--set postgresql.auth.password="$(openssl rand -hex 16)"
```
Нужны Ingress-контроллер и cert-manager. Service чарта говорит по **HTTP**
`:8006`; TLS — на Ingress. Compose тоже отдаёт plain HTTP на `:8006`; опциональный
HTTPS для proxmoxer — `docker compose --profile tls` на хосте `:8443`.
Подробности: [Kubernetes / Helm](docs/ru/kubernetes.md).
- HTTP API и Web UI (Compose): [http://localhost:8006/](http://localhost:8006/)
- Схема FastAPI: [http://localhost:8006/docs](http://localhost:8006/docs)
- Админ по умолчанию после seed: `root@pam` / `secret`
## Быстрый старт (разработка из репозитория)
Сборка и запуск development-стека с bind-mount из этого репозитория:
```bash
make install
make up
make seed PROFILE=small
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
curl -sS -X POST -d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket
```
- HTTP API и Web UI: [http://localhost:8006/](http://localhost:8006/)
(реальный PVE — **HTTPS** на `:8006`; лаборатория — plain HTTP на том же
порту. Опциональный TLS для proxmoxer: `docker compose --profile tls` →
`https://localhost:8443/`)
- Карта портов и TLS: [Порты и TLS](docs/ru/configuration.md#порты-и-tls).
- Схема FastAPI: [http://localhost:8006/docs](http://localhost:8006/docs)
### Web UI
Интерактивная консоль со светлой/тёмной темой, каталогом эндпоинтов PVE 6–9,
горячей сменой runtime-контракта и монитором задач UPID.
![Web UI, главный экран](docs/images/web-ui-main.png)
Больше экранов и подробностей: [Web UI](docs/ru/web-ui.md).
## Документация
Документация двуязычная. Переключатель **Language / Язык** — в шапке каждой
страницы; английский корень — [README.md](README.md). Индекс:
[docs/README.md](docs/README.md) · [docs/ru/README.md](docs/ru/README.md).
| Руководство | Описание |
|---|---|
| [Начало работы](docs/ru/getting-started.md) | Первая успешная лабораторная сессия |
| [Конфигурация](docs/ru/configuration.md) | Переменные окружения и Compose |
| [Аутентификация](docs/ru/authentication.md) | Тикеты, CSRF, API-токены, ACL |
| [Версии API](docs/ru/api-versions.md) | Контракты 6–9 и hot-swap |
| [Клиенты и примеры](docs/ru/clients.md) | Python, Go, Java, Perl, Ansible, Terraform, Pulumi |
| [Профили seed](docs/ru/seed-profiles.md) | Детерминированные фикстуры кластера |
| [Поверхность API](docs/ru/api-surface.md) | Маршрутизация, обработчики, fallback |
| [Домены](docs/ru/domains/README.md) | QEMU, LXC, storage, HA, SDN, … |
| [Web UI](docs/ru/web-ui.md) | Интерактивная консоль и каталоги |
| [Эксплуатация](docs/ru/operations.md) | Миграции, reseed, обновления |
| [Kubernetes / Helm](docs/ru/kubernetes.md) | Образ Hub + Ingress + Let's Encrypt |
| [Безопасность](docs/ru/security.md) | Модель угроз лаборатории и учётные данные |
| [Наблюдаемость](docs/ru/observability.md) | Health-эндпоинты и логирование |
| [Устранение неполадок](docs/ru/troubleshooting.md) | Типичные сбои |
| [FAQ](docs/ru/faq.md) | Краткие ответы |
| [Архитектура](docs/ru/architecture.md) | Границы компонентов |
| [Совместимость](docs/ru/compatibility.md) | Модель evidence и матрица релиза |
Индекс гайдов: [`docs/ru/README.md`](docs/ru/README.md).
Запускаемые cookbook: [`examples/`](examples/README.ru.md).
Интеграционный набор Pulumi (surface majors 6–9 + lifecycle, HTML-отчёт):
[`pulumi-tests/`](pulumi-tests/README.ru.md) (`make pulumi-tests`).
## proxmoxer (HTTPS-шлюз)
```python
from proxmoxer import ProxmoxAPI
proxmox = ProxmoxAPI(
"localhost",
port=8006,
user="root@pam",
password="secret",
verify_ssl=False, # только локальный self-signed сертификат разработки
)
print(proxmox.version.get())
print(proxmox.nodes("pve01").qemu.get())
```
Пример API-токена: пользователь `root@pam`, `token_name="automation"`,
`token_value="automation-secret"`. Запросам с токеном CSRF не нужен; мутациям
по тикету — нужен.
## Частые цели Make
```bash
make up / make down / make logs / make dev
make test # unit + contract (включая verified surface)
make test-integration # с PostgreSQL
make test-surface # все глаголы × majors 6-9 (0x501 / 0xexception)
make test-compatibility # proxmoxer против Compose
make evidence # перегенерация evidence/pve-*.json
make seed PROFILE=small
make db-migrate
make shell
make ci # ruff + mypy + offline pytest + surface probe
make release # сборка + push runtime-образа в Docker Hub
make release-up # pull/start docker-compose.release.yml
make release-seed PROFILE=small
```
Релиз в Docker Hub (нужен `docker login` владельца Hub; см.
[Эксплуатация](docs/ru/operations.md)):
```bash
make release # inecs/proxmox-api-simulator:<версия pyproject> + :latest
make release VERSION=0.2.0 # переопределить тег
make release-build # только build/tag, без push
make release-up && make release-seed # запустить опубликованный стек локально
```
## Участие / безопасность / changelog
- [CONTRIBUTING.md](CONTRIBUTING.md) · [CONTRIBUTING.ru.md](CONTRIBUTING.ru.md)
- [Testing](docs/testing.md) · [Тесты](docs/ru/testing.md) — наборы, Make-цели, последние результаты
- [SECURITY.md](SECURITY.md)
- [CHANGELOG.md](CHANGELOG.md)
## Чем это не является
- Не гипервизор: нет выполнения KVM/LXC на железе или nested-хостах.
- Не drop-in замена мультиарендного продакшен-Proxmox.
- Удалённые IdP / LDAP / live Ceph / live ACME эндпоинты симулируются локально;
к реальным внешним системам они не обращаются.
## См. также
- [Web UI](docs/ru/web-ui.md) — интерактивная консоль, каталоги, панель DATA и скриншоты
+52
View File
@@ -0,0 +1,52 @@
**Language / Язык:** [English](SECURITY.md) | [Русский](docs/ru/security.md)
# Security policy
## Supported versions
| Version | Supported |
|---|---|
| `0.1.x` (latest) | Yes — security reports accepted |
| older / untagged | Best-effort only |
## Threat model (read this first)
This repository is a **local / CI laboratory simulator**, not a hardened
multi-tenant public Proxmox deployment.
Default Compose and Helm values intentionally ship convenient lab secrets,
seeded passwords, open Web UI helper routes (`/ui/api/*`), and compatibility
endpoints (`/admin/compatibility*`). Treat network reachability as the trust
boundary.
**Do not** expose host port `8006` (or a public Ingress) to untrusted networks
without replacing every default secret and adding controls you own.
Full lab notes: [docs/security.md](docs/security.md) ·
[docs/ru/security.md](docs/ru/security.md).
## Reporting a vulnerability
Please **do not** open a public GitHub issue for sensitive reports.
Email the maintainer privately (account that owns the GitHub repository /
Docker Hub `inecs` namespace), or use GitHub
[ privately reported vulnerabilities](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability)
if enabled on the repository.
Include:
- Affected version / image tag (`inecs/proxmox-api-simulator:…`)
- Reproduction steps against a **local** lab (not third-party instances)
- Impact assessment (auth bypass, secret leak, RCE, etc.)
You should receive an acknowledgement within a few business days.
## Lab secrets that must be rotated outside toy labs
| Secret | Where |
|---|---|
| `TICKET_SIGNING_KEY` | Compose / Helm |
| PostgreSQL password | Compose / Helm |
| Seeded `root@pam` / API tokens | After `seed` |
| `docker/tls/server.key` | Checked-in self-signed material — never reuse outside local Compose |
+1
View File
@@ -0,0 +1 @@
0.1.1
+66 -2
View File
@@ -17,7 +17,7 @@ from app.api.errors import ApiError, ContractValidationError
from app.api.openapi import contract_openapi_tags from app.api.openapi import contract_openapi_tags
from app.config import Settings from app.config import Settings
from app.contracts.examples import schema_example from app.contracts.examples import schema_example
from app.contracts.model import Method, Schema, Snapshot from app.contracts.model import Method, Parameter, Schema, Snapshot
from app.db.pool import AsyncpgDatabase from app.db.pool import AsyncpgDatabase
from app.security.acl import AclEntry, CapabilityRequirement, authorize, requirement_from_contract from app.security.acl import AclEntry, CapabilityRequirement, authorize, requirement_from_contract
from app.security.auth import parse_api_token, verify_csrf, verify_secret, verify_ticket from app.security.auth import parse_api_token, verify_csrf, verify_secret, verify_ticket
@@ -137,6 +137,64 @@ def register_legacy_handler_routes(
return seen return seen
_PATH_PARAM_RE = re.compile(r"\{([^{}]+)\}")
def _unbound_method(path: str, verb: str) -> Method:
"""Minimal contract method so handler-only paths can be mounted."""
parameters = tuple(
Parameter(name=name, definition=Schema(type="string", optional=True))
for name in _PATH_PARAM_RE.findall(path)
)
return Method(
verb=verb.upper(),
name=f"unbound-{verb.lower()}-{path}",
description="Handler-backed route not present in the active contract snapshot",
parameters=parameters,
returns=Schema(),
permissions=None,
checksum="0" * 64,
)
def register_unbound_handler_routes(
app: FastAPI,
handlers: HandlerRegistry,
fallback: FallbackMode = "error",
*,
existing: set[tuple[str, str, str]] | None = None,
) -> set[tuple[str, str, str]]:
"""Mount semantic handlers that no cached contract currently declares.
Keeps DB-backed handlers reachable (no FastAPI 404) when Proxmox schema
omits a path that the simulator still implements.
"""
seen = existing if existing is not None else set()
for path, verb in sorted(handlers.keys()):
for renderer in ("json", "extjs"):
route = f"/api2/{renderer}{path}"
key = (route, verb, renderer)
if key in seen:
continue
method = _unbound_method(path, verb)
seen.add(key)
app.add_api_route(
route,
_endpoint(path, method, renderer, handlers, fallback),
methods=[verb],
name=f"unbound:{renderer}:{verb}:{path}",
tags=cast(list[str | Enum], contract_openapi_tags(path, renderer)),
openapi_extra={
"x-proxmox-method-checksum": method.checksum,
"x-proxmox-implementation": "implemented",
"x-proxmox-unbound-handler": True,
},
)
return seen
def _endpoint( def _endpoint(
semantic_path: str, semantic_path: str,
method: Method, method: Method,
@@ -302,8 +360,14 @@ async def _parse_inputs(request: Request, method: Method) -> dict[str, Any]:
continue continue
if name not in supplied: if name not in supplied:
if definition.optional: if definition.optional:
# Proxmox schema ``default`` is often UI documentation text (e.g.
# bwlimit default = "restore limit from datacenter…"), not a value
# to materialize. Only inject defaults that coerce to the type.
if definition.default is not None: if definition.default is not None:
parsed[name] = definition.default try:
parsed[name] = _coerce(definition.default, definition)
except (TypeError, ValueError):
pass
continue continue
errors[name] = "property is missing and it is not optional" errors[name] = "property is missing and it is not optional"
continue continue
+177 -5
View File
@@ -1,7 +1,15 @@
"""Generate example values from Proxmox contract schemas.""" """Generate example values from Proxmox contract schemas.
Examples follow the official API-viewer dialect: named ``format`` tokens,
property-string ``format`` objects (``key=value,key2=value2`` / bare
``default_key``), and common path-parameter placeholders.
"""
from __future__ import annotations from __future__ import annotations
from collections.abc import Mapping
from typing import Any
from app.contracts.model import Schema from app.contracts.model import Schema
_PATH_PARAM_EXAMPLES: dict[str, object] = { _PATH_PARAM_EXAMPLES: dict[str, object] = {
@@ -22,6 +30,53 @@ _PATH_PARAM_EXAMPLES: dict[str, object] = {
"key": "cpu", "key": "cpu",
"digest": "00000000", "digest": "00000000",
"name": "example", "name": "example",
"clustername": "example",
}
_FORMAT_EXAMPLES: dict[str, object] = {
"pve-node": "pve01",
"pve-node-list": "pve01,pve02",
"pve-vmid": 100,
"pve-vmid-list": "100,101",
"pve-storage-id": "local",
"pve-storage-server": "192.168.0.10",
"pve-poolid": "testpool",
"pve-userid": "root@pam",
"pve-userid-list": "root@pam,admin@pve",
"pve-realm": "pam",
"pve-groupid": "admins",
"pve-groupid-list": "admins,ops",
"pve-roleid": "Administrator",
"pve-roleid-list": "Administrator,PVEAdmin",
"pve-tokenid": "automation",
"pve-tokenid-list": "automation",
"pve-priv-list": "VM.Allocate,VM.Config.Options",
"pve-iface": "net0",
"pve-iface-list": "net0,net1",
"pve-configid": "default",
"pve-configid-list": "default",
"pve-sdn-vnet-id": "vnet0",
"pve-replication-job-id": "100-0",
"pve-ha-resource-or-vm-id": "vm:100",
"pve-ha-group-node-list": "pve01:1,pve02:1",
"pve-day-of-week-list": "mon,tue,wed",
"pve-fw-addr-spec": "192.168.0.0/24",
"pve-fw-dport-spec": "22",
"pve-fw-sport-spec": "1024:65535",
"pve-fw-protocol-spec": "tcp",
"pve-fw-icmp-type-spec": "echo-request",
"address": "192.168.0.1",
"ip": "192.168.0.1",
"IPorCIDR": "192.168.0.0/24",
"IPorCIDRorAlias": "192.168.0.0/24",
"dns-name": "example.local",
"email-opt": "admin@example.local",
"email-list": "admin@example.local",
"email-or-username-list": "admin@example.local",
"ldap-simple-attr": "uid",
"ldap-simple-attr-list": "posixGroup,groupOfNames",
"string-alist": "/var/lib/vz",
"pve-tfa-secret": "ABCDEFGHIJKLMNOP",
} }
@@ -31,6 +86,14 @@ def path_param_example(name: str) -> object | None:
return _PATH_PARAM_EXAMPLES.get(name) return _PATH_PARAM_EXAMPLES.get(name)
def wire_param_name(name: str, *, index: int = 0) -> str:
"""Map contract ``foo[n]`` names to a concrete wire key (``foo0``)."""
if "[n]" in name:
return name.replace("[n]", str(index))
return name
def schema_example(schema: Schema, *, name: str | None = None) -> object: def schema_example(schema: Schema, *, name: str | None = None) -> object:
"""Build a representative example value for a contract schema.""" """Build a representative example value for a contract schema."""
@@ -47,6 +110,19 @@ def schema_example(schema: Schema, *, name: str | None = None) -> object:
hinted = path_param_example(indexed.rstrip("0123456789")) hinted = path_param_example(indexed.rstrip("0123456789"))
if hinted is not None: if hinted is not None:
return hinted return hinted
base = indexed.rstrip("0123456789")
if base in _PATH_PARAM_EXAMPLES:
return _PATH_PARAM_EXAMPLES[base]
if isinstance(schema.format, Mapping):
return property_string_example(schema.format)
if isinstance(schema.format, str):
formatted = _named_format_example(schema.format, name=name)
if formatted is not None:
return formatted
if schema.format == "email":
return "user@example.com"
if schema.format == "uri":
return "https://example.com"
if schema.type == "array": if schema.type == "array":
if schema.items is not None: if schema.items is not None:
return [schema_example(schema.items)] return [schema_example(schema.items)]
@@ -68,9 +144,105 @@ def schema_example(schema: Schema, *, name: str | None = None) -> object:
return float(schema.minimum) return float(schema.minimum)
return 1.0 return 1.0
if schema.type == "string" or schema.type is None: if schema.type == "string" or schema.type is None:
if schema.format == "email":
return "user@example.com"
if schema.format == "uri":
return "https://example.com"
return "example" return "example"
return None return None
def property_string_example(fmt: Mapping[str, Any]) -> str:
"""Render a Proxmox property-string example from a ``format`` object.
Mirrors the API viewer dialect: a ``default_key`` may appear as a bare
value; other required keys use ``key=value``; optional keys are omitted
from the minimal example (they appear in ``typetext`` as ``[,key=…]``).
"""
bare: str | None = None
keyed: list[str] = []
for key, raw in sorted(fmt.items()):
if not isinstance(raw, Mapping):
continue
value = _format_key_example(key, raw)
rendered = _stringify_property_value(value)
if raw.get("default_key"):
bare = rendered
continue
if _is_optional_flag(raw.get("optional")):
continue
keyed.append(f"{key}={rendered}")
if bare is not None and not keyed:
return bare
if bare is not None:
return ",".join([bare, *keyed])
if keyed:
return ",".join(keyed)
# All keys optional — include defaults / default_key style extras for a
# usable stub that still matches typetext shape.
fallback: list[str] = []
for key, raw in sorted(fmt.items()):
if not isinstance(raw, Mapping):
continue
value = _format_key_example(key, raw)
rendered = _stringify_property_value(value)
if raw.get("default_key"):
return rendered
fallback.append(f"{key}={rendered}")
return ",".join(fallback) if fallback else "example"
def _named_format_example(fmt: str, *, name: str | None) -> object | None:
if fmt in _FORMAT_EXAMPLES:
return _FORMAT_EXAMPLES[fmt]
if fmt.endswith("-list") and fmt[: -len("-list")] in _FORMAT_EXAMPLES:
base = _FORMAT_EXAMPLES[fmt[: -len("-list")]]
return str(base)
if name and name in _FORMAT_EXAMPLES:
return _FORMAT_EXAMPLES[name]
return None
def _format_key_example(key: str, raw: Mapping[str, Any]) -> object:
if "default" in raw and raw["default"] is not None:
return raw["default"]
enum = raw.get("enum")
if isinstance(enum, list | tuple) and enum:
return enum[0]
nested_format = raw.get("format")
if isinstance(nested_format, Mapping):
return property_string_example(nested_format)
if isinstance(nested_format, str):
named = _named_format_example(nested_format, name=key)
if named is not None:
return named
if nested_format == "email":
return "user@example.com"
if nested_format == "uri":
return "https://example.com"
hinted = path_param_example(key)
if hinted is not None:
return hinted
typ = raw.get("type")
if typ == "boolean":
return 1
if typ == "integer":
minimum = raw.get("minimum")
return int(minimum) if isinstance(minimum, int | float) else 1
if typ == "number":
minimum = raw.get("minimum")
return float(minimum) if isinstance(minimum, int | float) else 1.0
pattern = raw.get("pattern")
if isinstance(pattern, str) and "second|minute|hour|day" in pattern:
return "1/second"
return "example"
def _stringify_property_value(value: object) -> str:
if isinstance(value, bool):
return "1" if value else "0"
return str(value)
def _is_optional_flag(value: object) -> bool:
if value is True or value == 1 or value == "1":
return True
return False
+3 -1
View File
@@ -14,6 +14,7 @@ from app.api.registry import (
HandlerRegistry, HandlerRegistry,
register_contract_routes, register_contract_routes,
register_legacy_handler_routes, register_legacy_handler_routes,
register_unbound_handler_routes,
) )
from app.compatibility import ( from app.compatibility import (
CompatibilityDimension, CompatibilityDimension,
@@ -106,7 +107,7 @@ def apply_runtime_contract(
clear_contract_routes(app) clear_contract_routes(app)
registered = register_contract_routes(app, snapshot, handlers, fallback) registered = register_contract_routes(app, snapshot, handlers, fallback)
register_legacy_handler_routes( registered = register_legacy_handler_routes(
app, app,
handlers, handlers,
store_root, store_root,
@@ -114,6 +115,7 @@ def apply_runtime_contract(
primary_version=snapshot.source_version, primary_version=snapshot.source_version,
existing=registered, existing=registered,
) )
register_unbound_handler_routes(app, handlers, fallback, existing=registered)
report = build_compatibility_for_snapshot( report = build_compatibility_for_snapshot(
snapshot, snapshot,
handlers, handlers,
+85 -14
View File
@@ -125,7 +125,7 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
return [ return [
{ {
"userid": str(row["name"]), "userid": str(row["name"]),
"enable": 1 if row["enabled"] else 0, "enable": bool(row["enabled"]),
"realm-type": str(row["realm_kind"]), "realm-type": str(row["realm_kind"]),
} }
for row in rows for row in rows
@@ -168,10 +168,25 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
) )
if row is None: if row is None:
raise ApiError(404, "user does not exist") raise ApiError(404, "user does not exist")
groups = await database(request).pool.fetch(
"""SELECT g.group_id FROM identity_group_members m
JOIN identity_groups g ON g.id = m.group_id
JOIN principals p ON p.id = m.principal_id
WHERE p.name=$1 ORDER BY g.group_id""",
userid,
)
tokens = await database(request).pool.fetch(
"""SELECT t.token_id FROM api_tokens t
JOIN principals p ON p.id = t.principal_id
WHERE p.name=$1 ORDER BY t.token_id""",
userid,
)
return { return {
"userid": str(row["name"]), "userid": str(row["name"]),
"enable": 1 if row["enabled"] else 0, "enable": bool(row["enabled"]),
"realm-type": str(row["realm_kind"]), "realm-type": str(row["realm_kind"]),
"groups": [str(item["group_id"]) for item in groups],
"tokens": [{"tokenid": str(item["token_id"]), "privsep": 1} for item in tokens],
} }
async def user_update(request: Request, inputs: dict[str, Any]) -> None: async def user_update(request: Request, inputs: dict[str, Any]) -> None:
@@ -397,10 +412,18 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
path = str(payload["path"]) path = str(payload["path"])
roleid = str(payload["roles"]) roleid = str(payload["roles"])
propagate = bool(int(payload.get("propagate", 1))) propagate = bool(int(payload.get("propagate", 1)))
delete = bool(int(payload.get("delete", 0)))
users = [item.strip() for item in str(payload.get("users", "")).split(",") if item.strip()] users = [item.strip() for item in str(payload.get("users", "")).split(",") if item.strip()]
groups = [ groups = [
item.strip() for item in str(payload.get("groups", "")).split(",") if item.strip() item.strip() for item in str(payload.get("groups", "")).split(",") if item.strip()
] ]
tokens = [
item.strip() for item in str(payload.get("tokens", "")).split(",") if item.strip()
]
await database(request).pool.execute(
"""INSERT INTO roles(name) VALUES($1) ON CONFLICT DO NOTHING""",
roleid,
)
for userid in users: for userid in users:
principal_id = await database(request).pool.fetchval( principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1", "SELECT id FROM principals WHERE name=$1",
@@ -408,10 +431,15 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
) )
if principal_id is None: if principal_id is None:
raise ApiError(404, f"user {userid} does not exist") raise ApiError(404, f"user {userid} does not exist")
await database(request).pool.execute( if delete:
"""INSERT INTO roles(name) VALUES($1) ON CONFLICT DO NOTHING""", await database(request).pool.execute(
roleid, """DELETE FROM acl_entries
) WHERE principal_id=$1 AND role_name=$2 AND path=$3""",
principal_id,
roleid,
path,
)
continue
await database(request).pool.execute( await database(request).pool.execute(
"""INSERT INTO acl_entries(principal_id, role_name, path, propagate) """INSERT INTO acl_entries(principal_id, role_name, path, propagate)
VALUES($1, $2, $3, $4) VALUES($1, $2, $3, $4)
@@ -429,10 +457,15 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
) )
if group_id is None: if group_id is None:
raise ApiError(404, f"group {groupid} does not exist") raise ApiError(404, f"group {groupid} does not exist")
await database(request).pool.execute( if delete:
"""INSERT INTO roles(name) VALUES($1) ON CONFLICT DO NOTHING""", await database(request).pool.execute(
roleid, """DELETE FROM group_acl_entries
) WHERE group_id=$1 AND role_name=$2 AND path=$3""",
group_id,
roleid,
path,
)
continue
await database(request).pool.execute( await database(request).pool.execute(
"""INSERT INTO group_acl_entries(group_id, role_name, path, propagate) """INSERT INTO group_acl_entries(group_id, role_name, path, propagate)
VALUES($1, $2, $3, $4) VALUES($1, $2, $3, $4)
@@ -443,6 +476,46 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
path, path,
propagate, propagate,
) )
for token_ref in tokens:
# Wire form: userid!tokenid
userid, _, tokenid = token_ref.partition("!")
if not userid or not tokenid:
raise ApiError(400, f"invalid token ACL subject '{token_ref}'")
token_row = await database(request).pool.fetchrow(
"""SELECT t.id FROM api_tokens t
JOIN principals p ON p.id = t.principal_id
WHERE p.name=$1 AND t.token_id=$2""",
userid,
tokenid,
)
if token_row is None:
raise ApiError(404, f"token {token_ref} does not exist")
# Persist as principal ACL under the owning user with token scope in path metadata.
# Keep durable by tagging path; GET ACL still lists principal entries.
scoped_path = f"{path}#token:{tokenid}" if path != "/" else f"/#token:{tokenid}"
principal_id = await database(request).pool.fetchval(
"SELECT id FROM principals WHERE name=$1",
userid,
)
if delete:
await database(request).pool.execute(
"""DELETE FROM acl_entries
WHERE principal_id=$1 AND role_name=$2 AND path=$3""",
principal_id,
roleid,
scoped_path,
)
else:
await database(request).pool.execute(
"""INSERT INTO acl_entries(principal_id, role_name, path, propagate)
VALUES($1, $2, $3, $4)
ON CONFLICT (principal_id, role_name, path) DO UPDATE
SET propagate=EXCLUDED.propagate""",
principal_id,
roleid,
scoped_path,
propagate,
)
async def token_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def token_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
userid = str(values(inputs)["userid"]) userid = str(values(inputs)["userid"])
@@ -562,10 +635,8 @@ def register_access_handlers(registry: HandlerRegistry) -> None:
) )
if row is None: if row is None:
raise ApiError(404, "role does not exist") raise ApiError(404, "role does not exist")
return { # Contract returns privilege→boolean map (additionalProperties: 0 on named privs).
"roleid": str(row["name"]), return {str(priv): 1 for priv in row["privileges"]}
"privs": ",".join(str(item) for item in row["privileges"]),
}
async def role_create(request: Request, inputs: dict[str, Any]) -> None: async def role_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
+60 -51
View File
@@ -2,6 +2,8 @@
from __future__ import annotations from __future__ import annotations
import hashlib
import json
from typing import Any from typing import Any
from fastapi import Request from fastapi import Request
@@ -10,41 +12,34 @@ from app.api.errors import ApiError
from app.api.registry import HandlerRegistry from app.api.registry import HandlerRegistry
from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs, values from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs, values
_DEFAULT_DIRECTORIES = [
{
"name": "Let's Encrypt V2",
"url": "https://acme-v02.api.letsencrypt.org/directory",
},
{
"name": "Let's Encrypt V2 Staging",
"url": "https://acme-staging-v02.api.letsencrypt.org/directory",
},
]
_CHALLENGE_SCHEMA = [
{
"id": "dns",
"name": "DNS plugin",
"type": "dns",
"fields": [{"name": "api", "type": "string"}],
}
]
def _acme(metadata: dict[str, Any]) -> dict[str, Any]: def _acme(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault( current = metadata.get("acme")
"acme",
{"accounts": {}, "plugins": {}, "meta": {}},
)
if not isinstance(current, dict): if not isinstance(current, dict):
current = {"accounts": {}, "plugins": {}, "meta": {}} current = {}
metadata["acme"] = current metadata["acme"] = current
current.setdefault("accounts", {}) if not isinstance(current.get("accounts"), dict):
current.setdefault("plugins", {}) current["accounts"] = {}
current.setdefault("meta", {}) if not isinstance(current.get("plugins"), dict):
current["plugins"] = {}
if not isinstance(current.get("meta"), dict):
current["meta"] = {}
if not isinstance(current.get("directories"), list):
current["directories"] = []
if not isinstance(current.get("challenge_schema"), list):
current["challenge_schema"] = []
return current return current
def _default_directory(acme: dict[str, Any]) -> str:
directories = acme.get("directories")
if isinstance(directories, list):
for item in directories:
if isinstance(item, dict) and item.get("url"):
return str(item["url"])
return "https://acme-v02.api.letsencrypt.org/directory"
def register_acme_handlers(registry: HandlerRegistry) -> None: def register_acme_handlers(registry: HandlerRegistry) -> None:
async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]: async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs( return subdirs(
@@ -68,24 +63,26 @@ def register_acme_handlers(registry: HandlerRegistry) -> None:
for name, item in sorted(accounts.items()) for name, item in sorted(accounts.items())
] ]
async def account_create(request: Request, inputs: dict[str, Any]) -> None: async def account_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
name = str(payload.get("name") or "default") name = str(payload.get("name") or "default")
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
accounts = _acme(metadata)["accounts"] acme = _acme(metadata)
accounts = acme["accounts"]
if name in accounts: if name in accounts:
raise ApiError(400, f"ACME account '{name}' already exists") raise ApiError(400, f"ACME account '{name}' already exists")
directory = str(payload.get("directory") or _default_directory(acme))
accounts[name] = { accounts[name] = {
"name": name, "name": name,
"contact": payload.get("contact"), "contact": payload.get("contact"),
"directory": payload.get("directory") or _DEFAULT_DIRECTORIES[0]["url"], "directory": directory,
"tos_url": payload.get("tos_url"), "tos_url": payload.get("tos_url"),
"eab-kid": payload.get("eab-kid"), "eab-kid": payload.get("eab-kid"),
# eab-hmac-key stored but never returned
"eab-hmac-key": payload.get("eab-hmac-key"), "eab-hmac-key": payload.get("eab-hmac-key"),
"location": f"https://acme.example.local/acct/{name}", "location": f"https://acme.example.local/acct/{name}",
} }
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return name
async def account_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def account_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
name = str(values(inputs)["name"]) name = str(values(inputs)["name"])
@@ -94,8 +91,10 @@ def register_acme_handlers(registry: HandlerRegistry) -> None:
if not isinstance(account, dict): if not isinstance(account, dict):
raise ApiError(404, "ACME account does not exist") raise ApiError(404, "ACME account does not exist")
return { return {
"name": name, "account": {
"contact": account.get("contact"), "name": name,
"contact": account.get("contact"),
},
"directory": account.get("directory"), "directory": account.get("directory"),
"tos": account.get("tos_url"), "tos": account.get("tos_url"),
"location": account.get("location"), "location": account.get("location"),
@@ -129,7 +128,13 @@ def register_acme_handlers(registry: HandlerRegistry) -> None:
for plugin_id, item in sorted(plugins.items()): for plugin_id, item in sorted(plugins.items()):
if plugin_type and item.get("type") != plugin_type: if plugin_type and item.get("type") != plugin_type:
continue continue
result.append({"plugin": plugin_id, **{k: v for k, v in item.items() if k != "data"}}) entry = {"plugin": plugin_id, **{k: v for k, v in item.items() if k != "data"}}
if "digest" not in entry:
material = {k: v for k, v in entry.items() if k != "digest"}
entry["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(material, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
result.append(entry)
return result return result
async def plugins_create(request: Request, inputs: dict[str, Any]) -> None: async def plugins_create(request: Request, inputs: dict[str, Any]) -> None:
@@ -181,29 +186,33 @@ def register_acme_handlers(registry: HandlerRegistry) -> None:
del plugins[plugin_id] del plugins[plugin_id]
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def directories(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def directories(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(_DEFAULT_DIRECTORIES) metadata = await cluster_metadata(request)
directories = _acme(metadata).get("directories")
if isinstance(directories, list):
return [dict(item) for item in directories if isinstance(item, dict)]
return []
async def challenge_schema(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def challenge_schema(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(_CHALLENGE_SCHEMA) metadata = await cluster_metadata(request)
schema = _acme(metadata).get("challenge_schema")
if isinstance(schema, list):
return [dict(item) for item in schema if isinstance(item, dict)]
return []
async def meta(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def meta(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
directory = str(values(inputs).get("directory") or _DEFAULT_DIRECTORIES[0]["url"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
meta_store = _acme(metadata).setdefault("meta", {}) acme = _acme(metadata)
payload = meta_store.setdefault( directory = str(values(inputs).get("directory") or _default_directory(acme))
directory, meta_raw = acme.get("meta")
{ meta_store: dict[str, Any] = dict(meta_raw) if isinstance(meta_raw, dict) else {}
"termsOfService": f"{directory.rstrip('/')}/tos", payload = meta_store.get(directory)
"caaIdentities": ["letsencrypt.org"], if not isinstance(payload, dict):
}, return {}
)
await save_cluster_metadata(request, metadata)
return dict(payload) return dict(payload)
async def tos(request: Request, inputs: dict[str, Any]) -> str: async def tos(request: Request, inputs: dict[str, Any]) -> str:
directory = str(values(inputs).get("directory") or _DEFAULT_DIRECTORIES[0]["url"]) result = await meta(request, inputs)
result = await meta(request, {"values": {"directory": directory}, "provided": frozenset()})
return str(result.get("termsOfService") or "") return str(result.get("termsOfService") or "")
registry.register("/cluster/acme", "GET", index) registry.register("/cluster/acme", "GET", index)
+200 -145
View File
@@ -1,8 +1,8 @@
"""Cluster backup and vzdump handlers.""" """Cluster backup jobs and node vzdump handlers."""
from __future__ import annotations from __future__ import annotations
import json import secrets
from typing import Any from typing import Any
from fastapi import Request from fastapi import Request
@@ -10,170 +10,224 @@ from fastapi import Request
from app.api.errors import ApiError from app.api.errors import ApiError
from app.api.registry import HandlerRegistry from app.api.registry import HandlerRegistry
from app.db.primitives import ConflictError from app.db.primitives import ConflictError
from app.handlers.common import database, require_node, state, values from app.handlers.common import (
cluster_metadata,
database,
require_node,
save_cluster_metadata,
state,
subdirs,
values,
)
from app.tasks.repository import TaskRepository from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid from app.tasks.upid import Upid
def _backup_jobs(metadata: dict[str, Any]) -> dict[str, dict[str, Any]]:
jobs = metadata.get("backup_jobs")
if not isinstance(jobs, dict):
return {}
return {
str(job_id): dict(payload) for job_id, payload in jobs.items() if isinstance(payload, dict)
}
def _job_view(job_id: str, payload: dict[str, Any]) -> dict[str, Any]:
view = dict(payload)
view["id"] = job_id
enabled = view.get("enabled", 1)
view["enabled"] = bool(int(enabled)) if not isinstance(enabled, bool) else enabled
return view
def register_backup_handlers(registry: HandlerRegistry) -> None: def register_backup_handlers(registry: HandlerRegistry) -> None:
async def backup_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def backup_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch( metadata = await cluster_metadata(request)
"""SELECT b.id, b.volume_id, b.size_bytes, b.metadata, b.created_at, jobs = _backup_jobs(metadata)
r.external_id AS vmid, n.name AS node, s.storage_id return [_job_view(job_id, payload) for job_id, payload in sorted(jobs.items())]
FROM backups b
LEFT JOIN resources r ON r.id = b.resource_id async def backup_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
LEFT JOIN nodes n ON n.id = r.node_id job_id = str(values(inputs)["id"])
JOIN storages s ON s.resource_id = b.storage_resource_id metadata = await cluster_metadata(request)
ORDER BY b.created_at DESC LIMIT 2000""" jobs = _backup_jobs(metadata)
payload = jobs.get(job_id)
if payload is None:
raise ApiError(404, "backup job does not exist")
return _job_view(job_id, payload)
async def backup_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs)
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
job_id = str(payload.get("id") or f"backup-{secrets.token_hex(3)}")
if job_id in jobs:
raise ApiError(400, f"backup job '{job_id}' already exists")
entry = {
key: value
for key, value in payload.items()
if key not in {"delete", "digest", "node", "target"}
}
entry["id"] = job_id
entry.setdefault("enabled", 1)
entry.setdefault("storage", payload.get("storage") or "local")
jobs[job_id] = entry
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_update(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
payload = values(inputs)
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
current = jobs.get(job_id)
if current is None:
raise ApiError(404, "backup job does not exist")
updated = dict(current)
for key, value in payload.items():
if key in {"id", "delete", "digest"}:
continue
updated[key] = value
updated["id"] = job_id
jobs[job_id] = updated
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_delete(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
if job_id not in jobs:
raise ApiError(404, "backup job does not exist")
del jobs[job_id]
metadata["backup_jobs"] = jobs
await save_cluster_metadata(request, metadata)
async def backup_info(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("not-backed-up")
async def backupinfo_stub(_request: Request, _inputs: dict[str, Any]) -> str:
# PVE 6.x `/cluster/backupinfo` returns a stub string; 7+ uses `/backup-info` index.
return "Please use the 'not-backed-up' API"
async def backup_not_backed_up(
request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request)
jobs = _backup_jobs(metadata)
covered: set[str] = set()
for payload in jobs.values():
if int(payload.get("all") or 0):
covered.add("*")
for item in str(payload.get("vmid") or "").split(","):
item = item.strip()
if item:
covered.add(item)
rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.kind, r.state FROM resources r
WHERE r.kind IN ('qemu', 'lxc')
ORDER BY r.external_id::integer"""
) )
result: list[dict[str, Any]] = [] result: list[dict[str, Any]] = []
if "*" in covered:
return result
for row in rows: for row in rows:
metadata = state(row["metadata"]) vmid = str(row["external_id"])
if vmid in covered:
continue
guest_state = state(row["state"])
result.append( result.append(
{ {
"id": str(row["id"]), "vmid": int(vmid),
"volid": str(row["volume_id"]), "type": "qemu" if row["kind"] == "qemu" else "lxc",
"size": int(row["size_bytes"]), "name": str(guest_state.get("name") or f"{row['kind']}-{vmid}"),
"vmid": int(row["vmid"]) if row["vmid"] is not None else None,
"node": str(row["node"]) if row["node"] is not None else None,
"storage": str(row["storage_id"]),
"starttime": int(row["created_at"].timestamp()),
"mode": metadata.get("mode", "snapshot"),
"type": metadata.get("type", "vzdump"),
} }
) )
return result return result
async def backup_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def backup_included_volumes(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
backup_id = str(values(inputs)["id"]) job_id = str(values(inputs)["id"])
row = await database(request).pool.fetchrow( metadata = await cluster_metadata(request)
"""SELECT b.id, b.volume_id, b.size_bytes, b.metadata, b.created_at, jobs = _backup_jobs(metadata)
r.external_id AS vmid, n.name AS node, s.storage_id job = jobs.get(job_id)
FROM backups b if job is None:
LEFT JOIN resources r ON r.id = b.resource_id raise ApiError(404, "backup job does not exist")
LEFT JOIN nodes n ON n.id = r.node_id vmids = [item.strip() for item in str(job.get("vmid") or "").split(",") if item.strip()]
JOIN storages s ON s.resource_id = b.storage_resource_id guest_select = """SELECT r.external_id, r.kind, r.state,
WHERE b.id::text = $1 OR b.volume_id = $1""", v.config AS qemu_config, c.config AS lxc_config
backup_id, FROM resources r
) LEFT JOIN virtual_machines v ON v.resource_id = r.id
if row is None: LEFT JOIN containers c ON c.resource_id = r.id"""
raise ApiError(404, "backup does not exist") if int(job.get("all") or 0):
metadata = state(row["metadata"]) rows = await database(request).pool.fetch(
return { f"""{guest_select}
"id": str(row["id"]), WHERE r.kind IN ('qemu', 'lxc')
"volid": str(row["volume_id"]), ORDER BY r.external_id::integer"""
"size": int(row["size_bytes"]), )
"vmid": int(row["vmid"]) if row["vmid"] is not None else None, else:
"node": str(row["node"]) if row["node"] is not None else None, rows = await database(request).pool.fetch(
"storage": str(row["storage_id"]), f"""{guest_select}
"starttime": int(row["created_at"].timestamp()), WHERE r.kind IN ('qemu', 'lxc') AND r.external_id = ANY($1::text[])
"notes": metadata.get("notes-template"), ORDER BY r.external_id::integer""",
**metadata, vmids,
} )
children: list[dict[str, Any]] = []
for row in rows:
guest_state = state(row["state"])
config_raw = row["qemu_config"] if row["kind"] == "qemu" else row["lxc_config"]
config = state(config_raw) if config_raw is not None else {}
volumes: list[dict[str, Any]] = []
for key, value in sorted(config.items()):
if not (
key.startswith("scsi")
or key.startswith("virtio")
or key.startswith("sata")
or key.startswith("ide")
or key in {"rootfs", "mp0", "mp1", "mp2", "mp3"}
):
continue
volumes.append(
{
"id": key,
"name": str(value).split(",", 1)[0],
"included": True,
"reason": "included by default",
}
)
children.append(
{
"id": int(row["external_id"]),
"name": str(guest_state.get("name") or f"{row['kind']}-{row['external_id']}"),
"type": "qemu" if row["kind"] == "qemu" else "lxc",
"children": volumes,
}
)
return {"children": children}
async def backup_update(request: Request, inputs: dict[str, Any]) -> None: async def vzdump_defaults(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
backup_id = str(values(inputs)["id"]) from app.handlers.nodes import load_node_ops
row = await database(request).pool.fetchrow(
"SELECT id, metadata FROM backups WHERE id::text = $1",
backup_id,
)
if row is None:
raise ApiError(404, "backup does not exist")
metadata = state(row["metadata"])
payload = values(inputs)
if "notes" in payload:
metadata["notes-template"] = payload["notes"]
await database(request).pool.execute(
"UPDATE backups SET metadata=$2::jsonb WHERE id=$1",
row["id"],
json.dumps(metadata, sort_keys=True),
)
async def backup_delete(request: Request, inputs: dict[str, Any]) -> None: node = str(values(inputs)["node"])
backup_id = str(values(inputs)["id"])
status = await database(request).pool.execute(
"DELETE FROM backups WHERE id::text = $1",
backup_id,
)
if status != "DELETE 1":
raise ApiError(404, "backup does not exist")
async def backup_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs)
node = str(payload.get("node") or payload.get("target") or "pve01")
await require_node(request, node) await require_node(request, node)
vmid = payload.get("vmid") ops = await load_node_ops(request, node)
return await _schedule_vzdump( vzdump = ops.get("vzdump")
request, defaults = vzdump.get("defaults") if isinstance(vzdump, dict) else None
node=node, return dict(defaults) if isinstance(defaults, dict) else {}
vmids=[str(vmid)] if vmid is not None else None,
payload=payload,
)
async def backup_info(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch(
"""SELECT r.external_id AS vmid, n.name AS node, max(b.created_at) AS last_backup
FROM resources r
JOIN nodes n ON n.id = r.node_id
LEFT JOIN backups b ON b.resource_id = r.id
WHERE r.kind = 'qemu'
GROUP BY r.external_id, n.name
ORDER BY r.external_id::integer
LIMIT 5000"""
)
return [
{
"vmid": int(row["vmid"]),
"node": str(row["node"]),
"lastbackup": int(row["last_backup"].timestamp()) if row["last_backup"] else 0,
"protected": 0,
}
for row in rows
]
async def backup_not_backed_up(_request: Request, _inputs: dict[str, Any]) -> list[int]:
rows = await database(_request).pool.fetch(
"""SELECT r.external_id::integer AS vmid
FROM resources r
LEFT JOIN backups b ON b.resource_id = r.id
WHERE r.kind = 'qemu' AND b.id IS NULL
ORDER BY r.external_id::integer"""
)
return [int(row["vmid"]) for row in rows]
async def backup_included_volumes(request: Request, inputs: dict[str, Any]) -> list[str]:
backup_id = str(values(inputs)["id"])
row = await database(request).pool.fetchrow(
"""SELECT b.volume_id, r.external_id AS vmid
FROM backups b LEFT JOIN resources r ON r.id = b.resource_id
WHERE b.id::text = $1""",
backup_id,
)
if row is None:
raise ApiError(404, "backup does not exist")
vmid = row["vmid"]
return [f"qemu/{vmid}"] if vmid is not None else [str(row["volume_id"])]
async def vzdump_defaults(_request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(_request, str(values(inputs)["node"]))
return {
"all": 0,
"bwlimit": 0,
"compress": "zstd",
"dumpdir": "backup",
"mode": "snapshot",
"remove": 0,
"storage": "nfs-backup",
"mailto": "",
"notes-template": "{{guestname}}",
}
async def vzdump_extractconfig(request: Request, inputs: dict[str, Any]) -> str: async def vzdump_extractconfig(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import load_node_ops
payload = values(inputs) payload = values(inputs)
node = str(payload["node"])
volid = str(payload.get("volume") or payload.get("volid") or "") volid = str(payload.get("volume") or payload.get("volid") or "")
if not volid: if not volid:
raise ApiError(400, "volume parameter required") raise ApiError(400, "volume parameter required")
return f"# simulated vzdump config extracted from {volid}\name: demo\nmemory: 2048\n" await require_node(request, node)
ops = await load_node_ops(request, node)
vzdump = ops.get("vzdump")
configs = vzdump.get("extractconfig") if isinstance(vzdump, dict) else None
if not isinstance(configs, dict):
return ""
return str(configs.get(volid) or "")
async def vzdump_create(request: Request, inputs: dict[str, Any]) -> str: async def vzdump_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
@@ -188,6 +242,7 @@ def register_backup_handlers(registry: HandlerRegistry) -> None:
registry.register("/cluster/backup", "GET", backup_list) registry.register("/cluster/backup", "GET", backup_list)
registry.register("/cluster/backup", "POST", backup_create) registry.register("/cluster/backup", "POST", backup_create)
registry.register("/cluster/backup-info", "GET", backup_info) registry.register("/cluster/backup-info", "GET", backup_info)
registry.register("/cluster/backupinfo", "GET", backupinfo_stub)
registry.register("/cluster/backup-info/not-backed-up", "GET", backup_not_backed_up) registry.register("/cluster/backup-info/not-backed-up", "GET", backup_not_backed_up)
registry.register("/cluster/backup/{id}", "GET", backup_get) registry.register("/cluster/backup/{id}", "GET", backup_get)
registry.register("/cluster/backup/{id}", "PUT", backup_update) registry.register("/cluster/backup/{id}", "PUT", backup_update)
@@ -224,11 +279,11 @@ async def _schedule_vzdump(
payload={ payload={
"node": node, "node": node,
"vmids": vmids, "vmids": vmids,
"storage": str(payload.get("storage") or "nfs-backup"), "storage": str(payload.get("storage") or "local"),
"mode": str(payload.get("mode") or "snapshot"), "mode": str(payload.get("mode") or "snapshot"),
"compress": str(payload.get("compress") or "zstd"), "compress": str(payload.get("compress") or "zstd"),
}, },
resource_key=f"backup:{node}", resource_key=f"backup:{node}:{secrets.token_hex(4)}",
idempotency_key=request.headers.get("Idempotency-Key"), idempotency_key=request.headers.get("Idempotency-Key"),
) )
except ConflictError as error: except ConflictError as error:
+182 -111
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
import json import json
import secrets import secrets
from typing import Any, cast from typing import Any
from fastapi import Request from fastapi import Request
@@ -21,37 +21,6 @@ from app.handlers.common import (
) )
from app.simulation.seed import CLUSTER_ID from app.simulation.seed import CLUSTER_ID
DEFAULT_CLUSTER_CEPH = {
"initialized": True,
"config": {
"network": "10.10.10.0/24",
"cluster-network": "10.10.10.0/24",
"size": 3,
"min_size": 2,
"pg_bits": 7,
},
"cfg_db": [
{"section": "global", "name": "auth_client_required", "value": "cephx"},
{"section": "global", "name": "fsid", "value": "pve-simulator-fsid"},
],
"cfg_raw": "[global]\nfsid = pve-simulator-fsid\nauth_client_required = cephx\n",
"cfg_values": {},
"pools": {
"rbd": {
"pool": "rbd",
"size": 3,
"min_size": 2,
"pg_num": 128,
"application": "rbd",
"crush_rule": "replicated_rule",
}
},
"fs": {},
"rules": [{"name": "replicated_rule", "id": 0}],
"crush": "device 0 osd.0 class hdd\n",
"running": True,
}
async def _load_cluster_ceph(request: Request) -> dict[str, Any]: async def _load_cluster_ceph(request: Request) -> dict[str, Any]:
row = await database(request).pool.fetchrow( row = await database(request).pool.fetchrow(
@@ -60,14 +29,9 @@ async def _load_cluster_ceph(request: Request) -> dict[str, Any]:
) )
metadata = state(row["metadata"]) if row is not None else {} metadata = state(row["metadata"]) if row is not None else {}
ceph = metadata.get("ceph") ceph = metadata.get("ceph")
if not isinstance(ceph, dict) or not ceph: if not isinstance(ceph, dict):
return dict(DEFAULT_CLUSTER_CEPH) return {}
merged = dict(DEFAULT_CLUSTER_CEPH) return dict(ceph)
merged.update(ceph)
for key in ("config", "pools", "fs", "cfg_values"):
if not isinstance(merged.get(key), dict):
merged[key] = dict(cast(dict[str, Any], DEFAULT_CLUSTER_CEPH[key]))
return merged
async def _save_cluster_ceph(request: Request, ceph: dict[str, Any]) -> None: async def _save_cluster_ceph(request: Request, ceph: dict[str, Any]) -> None:
@@ -82,34 +46,27 @@ async def _save_cluster_ceph(request: Request, ceph: dict[str, Any]) -> None:
async def _load_node_ceph(request: Request, node: str) -> dict[str, Any]: async def _load_node_ceph(request: Request, node: str) -> dict[str, Any]:
metadata = await node_metadata(request, node) metadata = await node_metadata(request, node)
ops = metadata.setdefault("ops", {}) ops = metadata.get("ops")
ceph = ops.setdefault( if not isinstance(ops, dict):
"ceph", return {"mds": {}, "mgr": {}, "mon": {}, "log": []}
{ ceph = ops.get("ceph")
"mds": {},
"mgr": {},
"mon": {f"{node}": {"name": node, "addr": f"{node}.local:6789", "rank": 0}},
"log": [{"t": 1_700_000_000, "n": 0, "line": "ceph simulator ready"}],
},
)
if not isinstance(ceph, dict): if not isinstance(ceph, dict):
ceph = { return {"mds": {}, "mgr": {}, "mon": {}, "log": []}
"mds": {}, return {
"mgr": {}, "mds": ceph.get("mds") if isinstance(ceph.get("mds"), dict) else {},
"mon": {}, "mgr": ceph.get("mgr") if isinstance(ceph.get("mgr"), dict) else {},
"log": [], "mon": ceph.get("mon") if isinstance(ceph.get("mon"), dict) else {},
} "log": list(ceph.get("log") or []) if isinstance(ceph.get("log"), list) else [],
ops["ceph"] = ceph **{key: value for key, value in ceph.items() if key not in {"mds", "mgr", "mon", "log"}},
ceph.setdefault("mds", {}) }
ceph.setdefault("mgr", {})
ceph.setdefault("mon", {})
ceph.setdefault("log", [])
return ceph
async def _save_node_ceph(request: Request, node: str, ceph: dict[str, Any]) -> None: async def _save_node_ceph(request: Request, node: str, ceph: dict[str, Any]) -> None:
metadata = await node_metadata(request, node) metadata = await node_metadata(request, node)
ops = metadata.setdefault("ops", {}) ops = metadata.get("ops")
if not isinstance(ops, dict):
ops = {}
metadata["ops"] = ops
ops["ceph"] = ceph ops["ceph"] = ceph
await save_node_metadata(request, node, metadata) await save_node_metadata(request, node, metadata)
@@ -175,10 +132,12 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
await require_node(request, str(payload["node"])) await require_node(request, str(payload["node"]))
ceph = await _load_cluster_ceph(request) ceph = await _load_cluster_ceph(request)
keys = [item.strip() for item in str(payload.get("config-keys") or "").split(",") if item] keys = [item.strip() for item in str(payload.get("config-keys") or "").split(",") if item]
stored = ceph.setdefault("cfg_values", {}) stored = ceph.get("cfg_values")
result = {key: stored.get(key, "") for key in keys} if keys else dict(stored) if not isinstance(stored, dict):
await _save_cluster_ceph(request, ceph) stored = {}
return result if keys:
return {key: stored.get(key, "") for key in keys}
return dict(stored)
async def crush(request: Request, inputs: dict[str, Any]) -> str: async def crush(request: Request, inputs: dict[str, Any]) -> str:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
@@ -202,12 +161,13 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
async def cmd_safety(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def cmd_safety(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
await require_node(request, str(payload["node"])) await require_node(request, str(payload["node"]))
return { ceph = await _load_cluster_ceph(request)
"safe": 1, safety = ceph.get("cmd_safety")
"action": payload.get("action"), result = dict(safety) if isinstance(safety, dict) else {}
"service": payload.get("service"), for key in ("action", "service", "id"):
"id": payload.get("id"), if key in payload:
} result[key] = payload[key]
return result
async def init(request: Request, inputs: dict[str, Any]) -> None: async def init(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
@@ -282,11 +242,45 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
await _save_cluster_ceph(request, ceph) await _save_cluster_ceph(request, ceph)
return _upid(node, "cephrestart") return _upid(node, "cephrestart")
def _wire_ceph_pool(name: str, item: dict[str, Any], *, index: int) -> dict[str, Any]:
pool_id = item.get("pool_id", item.get("pool"))
try:
pool_num = int(pool_id) if pool_id is not None else index
except (TypeError, ValueError):
pool_num = index
crush_raw = item.get("crush_rule", 0)
try:
crush_id = int(crush_raw)
crush_name = str(item.get("crush_rule_name") or "replicated_rule")
except (TypeError, ValueError):
crush_name = str(crush_raw or "replicated_rule")
crush_id = 0
application = str(item.get("application") or "rbd")
return {
"pool": pool_num,
"pool_name": str(item.get("pool_name") or name),
"size": int(item.get("size") or 3),
"min_size": int(item.get("min_size") or 2),
"pg_num": int(item.get("pg_num") or 128),
"crush_rule": crush_id,
"crush_rule_name": crush_name,
"application_metadata": item.get("application_metadata")
if isinstance(item.get("application_metadata"), dict)
else {application: {}},
"bytes_used": int(item.get("bytes_used") or 0),
"percent_used": float(item.get("percent_used") or 0.0),
"healthy": bool(item.get("healthy", True)),
}
async def pool_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def pool_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
ceph = await _load_cluster_ceph(request) ceph = await _load_cluster_ceph(request)
pools = ceph.get("pools") or {} pools = ceph.get("pools") or {}
return [dict(item) for _, item in sorted(pools.items()) if isinstance(item, dict)] return [
_wire_ceph_pool(name, item, index=index)
for index, (name, item) in enumerate(sorted(pools.items()))
if isinstance(item, dict)
]
async def pool_create(request: Request, inputs: dict[str, Any]) -> str: async def pool_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
@@ -318,7 +312,10 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
pool = (ceph.get("pools") or {}).get(name) pool = (ceph.get("pools") or {}).get(name)
if not isinstance(pool, dict): if not isinstance(pool, dict):
raise ApiError(404, "pool does not exist") raise ApiError(404, "pool does not exist")
return [dict(pool)] # Reuse list wire shape for detail GETs.
names = sorted((ceph.get("pools") or {}).keys())
index = names.index(name) if name in names else 0
return [_wire_ceph_pool(name, pool, index=index)]
async def pool_update(request: Request, inputs: dict[str, Any]) -> str: async def pool_update(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
@@ -368,13 +365,7 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
pool = (ceph.get("pools") or {}).get(name) pool = (ceph.get("pools") or {}).get(name)
if not isinstance(pool, dict): if not isinstance(pool, dict):
raise ApiError(404, "pool does not exist") raise ApiError(404, "pool does not exist")
return { return dict(pool)
**pool,
"pg_num": pool.get("pg_num", 128),
"bytes_used": 0,
"percent_used": 0.0,
"healthy": True,
}
async def fs_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def fs_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
@@ -528,30 +519,63 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
await _save_node_ceph(request, node, ceph) await _save_node_ceph(request, node, ceph)
return _upid(node, "cephdestroymon") return _upid(node, "cephdestroymon")
async def osd_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def osd_list(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
await require_node(request, node) await require_node(request, node)
rows = await database(request).pool.fetch( rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.state """SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE n.name=$1 AND r.kind='ceph-osd' WHERE r.kind='ceph-osd'
ORDER BY r.external_id""", ORDER BY n.name, r.external_id"""
node,
) )
result: list[dict[str, Any]] = [] hosts: dict[str, dict[str, Any]] = {}
host_id = -2
for row in rows: for row in rows:
host_name = str(row["node"])
host = hosts.get(host_name)
if host is None:
host = {
"id": host_id,
"name": host_name,
"type": "host",
"type_id": 1,
"children": [],
}
hosts[host_name] = host
host_id -= 1
payload = state(row["state"]) payload = state(row["state"])
osd_id = payload.get("osd_id", row["external_id"]) osd_raw = payload.get("osd_id", row["external_id"])
result.append( try:
osd_id = int(osd_raw)
except (TypeError, ValueError):
digits = "".join(ch for ch in str(osd_raw) if ch.isdigit())
osd_id = int(digits) if digits else 0
host["children"].append(
{ {
"osd": int(osd_id) if str(osd_id).isdigit() else osd_id, "id": osd_id,
"name": f"osd.{osd_id}",
"type": "osd",
"type_id": 0,
"status": payload.get("status", "up"), "status": payload.get("status", "up"),
"in": 1 if payload.get("in", True) else 0, "in": 1 if payload.get("in", True) else 0,
"weight": payload.get("weight", 1.0), "weight": float(payload.get("weight", 1.0)),
"device_class": payload.get("device_class", "hdd"), "device_class": payload.get("device_class", "hdd"),
"host": host_name,
} }
) )
return result root_children = list(hosts.values())
# Prefer the requested node first in the tree for local dumps.
root_children.sort(key=lambda item: (0 if item["name"] == node else 1, item["name"]))
return {
"flags": "",
"root": {
"id": -1,
"name": "default",
"type": "root",
"type_id": 10,
"children": root_children,
},
}
async def osd_create(request: Request, inputs: dict[str, Any]) -> str: async def osd_create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
@@ -562,9 +586,8 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
MAX(NULLIF(regexp_replace(external_id, '\\D', '', 'g'), '')::int), MAX(NULLIF(regexp_replace(external_id, '\\D', '', 'g'), '')::int),
-1 -1
) + 1 ) + 1
FROM resources r JOIN nodes n ON n.id=r.node_id FROM resources
WHERE n.name=$1 AND r.kind='ceph-osd'""", WHERE kind='ceph-osd'""",
node,
) )
osd_id = int(next_id or 0) osd_id = int(next_id or 0)
external_id = f"osd.{osd_id}" external_id = f"osd.{osd_id}"
@@ -583,8 +606,15 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
"used_bytes": 0, "used_bytes": 0,
} }
await database(request).pool.execute( await database(request).pool.execute(
"""INSERT INTO resources(id, node_id, kind, external_id, state) """INSERT INTO resources(id, node_id, cluster_id, kind, external_id, state)
VALUES(gen_random_uuid(), $1, 'ceph-osd', $2, $3::jsonb)""", VALUES(
gen_random_uuid(),
$1,
(SELECT cluster_id FROM nodes WHERE id=$1),
'ceph-osd',
$2,
$3::jsonb
)""",
node_id, node_id,
external_id, external_id,
json.dumps(osd_state, sort_keys=True), json.dumps(osd_state, sort_keys=True),
@@ -682,13 +712,16 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node) await require_node(request, node)
row = await _osd_row(request, node, osdid) row = await _osd_row(request, node, osdid)
current = state(row["state"]) current = state(row["state"])
metadata = current.get("metadata")
if isinstance(metadata, dict):
return dict(metadata)
return { return {
"osd": { "osd": {
"id": int(osdid) if osdid.isdigit() else osdid, "id": int(osdid) if osdid.isdigit() else osdid,
"uuid": current.get("uuid") or f"osd-uuid-{osdid}", "uuid": current.get("uuid", ""),
"device_class": current.get("device_class", "hdd"), "device_class": current.get("device_class", ""),
}, },
"devices": [{"dev": current.get("dev") or f"/dev/sd{osdid}"}], "devices": [{"dev": current.get("dev", "")}] if current.get("dev") else [],
} }
async def cluster_ceph_status(_request: Request, _inputs: dict[str, Any]) -> dict[str, Any]: async def cluster_ceph_status(_request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
@@ -698,20 +731,58 @@ def register_ceph_handlers(registry: HandlerRegistry) -> None:
) )
total = int(row["capacity_bytes"] or 0) if row is not None else 0 total = int(row["capacity_bytes"] or 0) if row is not None else 0
used = int(row["used_bytes"] or 0) if row is not None else 0 used = int(row["used_bytes"] or 0) if row is not None else 0
osd_count = await database(_request).pool.fetchval( osd_rows = await database(_request).pool.fetch(
"SELECT count(*)::int FROM resources WHERE kind='ceph-osd'" "SELECT state FROM resources WHERE kind='ceph-osd'"
) )
num_osds = len(osd_rows)
num_up = 0
num_in = 0
for osd_row in osd_rows:
osd_state = state(osd_row["state"])
if str(osd_state.get("status") or "") == "up":
num_up += 1
if osd_state.get("in") in (True, 1, "1"):
num_in += 1
ceph = await _load_cluster_ceph(_request) ceph = await _load_cluster_ceph(_request)
version = ceph.get("version")
version_str = (
str(version.get("str"))
if isinstance(version, dict) and version.get("str") is not None
else str(version or "")
)
health = ceph.get("health")
if not isinstance(health, dict):
health = {"status": "HEALTH_OK" if ceph.get("running") else "HEALTH_WARN"}
node_count = int(await database(_request).pool.fetchval("SELECT COUNT(*) FROM nodes") or 0)
return { return {
"version": "17.2.7", "version": {
"health": {"status": "HEALTH_OK" if ceph.get("running", True) else "HEALTH_WARN"}, "version": version_str,
"osdmap": { "release": "reef",
"num_osds": osd_count, "epoch": 0,
"num_up_osds": osd_count - 1,
"num_in_osds": osd_count - 1,
}, },
"pgmap": {"bytes_used": used, "bytes_total": total}, "health": health,
"fsmap": {"filesystems": list((ceph.get("fs") or {}).keys())}, "osdmap": {
"epoch": 1,
"num_osds": num_osds,
"num_up_osds": num_up,
"num_in_osds": num_in,
"num_remapped_pgs": 0,
},
"pgmap": {
"bytes_used": used,
"bytes_total": total,
"bytes_avail": max(total - used, 0),
"data_bytes": used,
"num_pgs": max(num_osds * 32, 32),
},
"fsmap": {
"epoch": 1,
"by_rank": [],
"filesystems": list((ceph.get("fs") or {}).keys()),
},
"monmap": {"num_mons": max(node_count, 1)},
"mgrmap": {"available": True, "num_standbys": 0},
"servicemap": {"epoch": 1, "services": {}},
} }
base = "/nodes/{node}/ceph" base = "/nodes/{node}/ceph"
+74 -20
View File
@@ -24,7 +24,43 @@ def _replication_jobs(metadata: dict[str, Any]) -> list[dict[str, Any]]:
jobs = metadata.get("replication", []) jobs = metadata.get("replication", [])
if not isinstance(jobs, list): if not isinstance(jobs, list):
return [] return []
return [dict(item) for item in jobs if isinstance(item, dict)] result: list[dict[str, Any]] = []
for item in jobs:
if not isinstance(item, dict):
continue
guest_raw = item.get("guest", 0)
try:
guest = int(guest_raw)
except (TypeError, ValueError):
guest = int(str(guest_raw).split(":")[-1] or 0)
jobnum_raw = item.get("jobnum")
if jobnum_raw is None:
id_text = str(item.get("id") or "0")
tail = id_text.rsplit("-", 1)[-1]
jobnum = int(tail) if tail.isdigit() else 0
else:
jobnum = int(jobnum_raw)
disable = item.get("disable")
if disable is None:
disable = 0 if int(item.get("enabled", 1) or 1) else 1
entry = {
"id": str(item.get("id") or f"{guest}-{jobnum}"),
"guest": guest,
"jobnum": jobnum,
"target": str(item.get("target") or ""),
"type": str(item.get("type") or "local"),
"disable": bool(int(disable)),
}
if item.get("schedule") is not None:
entry["schedule"] = str(item["schedule"])
if item.get("rate") is not None:
entry["rate"] = int(item["rate"])
if item.get("comment") is not None:
entry["comment"] = str(item["comment"])
if item.get("source") is not None:
entry["source"] = str(item["source"])
result.append(entry)
return result
def register_cluster_handlers(registry: HandlerRegistry) -> None: def register_cluster_handlers(registry: HandlerRegistry) -> None:
@@ -46,24 +82,48 @@ def register_cluster_handlers(registry: HandlerRegistry) -> None:
"tasks", "tasks",
) )
async def cluster_status(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def cluster_status(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(_request).pool.fetch( from app.handlers.nodes import load_node_ops
rows = await database(request).pool.fetch(
"""SELECT id, name, status FROM nodes ORDER BY name""" """SELECT id, name, status FROM nodes ORDER BY name"""
) )
result: list[dict[str, Any]] = [] metadata = await cluster_metadata(request)
quorate = bool(int(metadata.get("quorate", 1) or 1))
cluster_name = metadata.get("cluster_config")
if isinstance(cluster_name, dict):
name = str(cluster_name.get("clustername") or "pve-simulator")
version = int(cluster_name.get("version") or 2)
else:
name = "pve-simulator"
version = 2
result: list[dict[str, Any]] = [
{
"type": "cluster",
"id": "cluster",
"name": name,
"nodes": len(rows),
"quorate": quorate,
"version": version,
}
]
for index, row in enumerate(rows): for index, row in enumerate(rows):
online = str(row["status"]) == "online" online = str(row["status"]) == "online"
ops = await load_node_ops(request, str(row["name"]))
cluster_node = ops.get("cluster_status")
entry = dict(cluster_node) if isinstance(cluster_node, dict) else {}
local_raw = entry.get("local", index == 0)
node_name = str(row["name"])
result.append( result.append(
{ {
"id": str(row["id"]),
"name": str(row["name"]),
"nodeid": index,
"online": 1 if online else 0,
"local": 1 if index == 0 else 0,
"ip": f"10.32.{index // 254 + 1}.{index % 254 + 10}",
"level": "c",
"type": "node", "type": "node",
"quorate": 1, "id": f"node/{node_name}",
"name": node_name,
"nodeid": int(entry.get("nodeid", index + 1)),
"online": online,
"local": bool(int(local_raw)) if not isinstance(local_raw, bool) else local_raw,
"ip": str(entry.get("ip") or ops.get("ip") or ""),
"level": str(entry.get("level", "")),
} }
) )
return result return result
@@ -95,14 +155,8 @@ def register_cluster_handlers(registry: HandlerRegistry) -> None:
metadata = state(row["metadata"]) if row is not None else {} metadata = state(row["metadata"]) if row is not None else {}
options = metadata.get("options", {}) options = metadata.get("options", {})
if not isinstance(options, dict): if not isinstance(options, dict):
options = {} return {}
return { return dict(options)
"keyboard": options.get("keyboard", "en-us"),
"email_from": options.get("email_from", "root@localhost"),
"http_proxy": options.get("http_proxy", ""),
"description": options.get("description", "Proxmox API emulator cluster"),
**options,
}
async def cluster_options_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def cluster_options_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
current = await cluster_options_get(request, inputs) current = await cluster_options_get(request, inputs)
+283 -78
View File
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
import hashlib
import secrets import secrets
from typing import Any from typing import Any
@@ -12,50 +13,145 @@ from app.api.registry import HandlerRegistry
from app.handlers.common import ( from app.handlers.common import (
cluster_metadata, cluster_metadata,
database, database,
require_value,
save_cluster_metadata, save_cluster_metadata,
subdirs,
values, values,
) )
from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid
def _config(metadata: dict[str, Any]) -> dict[str, Any]: def _config(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault( current = metadata.get("cluster_config")
"cluster_config", return current if isinstance(current, dict) else {}
{
"clustername": "pve-simulator",
"votes": 1, def _principal(request: Request) -> str:
"links": {}, return str(getattr(request.state, "principal", None) or "root@pam")
"join_info": {},
"totem": {"version": 2, "secauth": "on", "cluster_name": "pve-simulator"},
"qdevice": {"status": "disabled"}, def _fingerprint_sha256() -> str:
"apiversion": 1, return ":".join(f"{byte:02X}" for byte in secrets.token_bytes(32))
},
)
if not isinstance(current, dict): def _stable_digest(material: str) -> str:
current = { return hashlib.sha256(material.encode("utf-8")).hexdigest()
"clustername": "pve-simulator",
"votes": 1,
"links": {}, def _node_addr(node: str, index: int, entry: dict[str, Any]) -> str:
"join_info": {}, for key in ("pve_addr", "new_node_ip", "ring0_addr"):
"totem": {"version": 2, "secauth": "on", "cluster_name": "pve-simulator"}, value = entry.get(key)
"qdevice": {"status": "disabled"}, if isinstance(value, str) and value and not value.endswith(".local"):
"apiversion": 1, # Prefer literal IPs; hostnames are still accepted as ring addresses.
} if key != "ring0_addr" or all(part.isdigit() for part in value.split(".")):
metadata["cluster_config"] = current return value
return current return f"10.0.0.{index}"
def _ring0_addr(node: str, index: int, entry: dict[str, Any], links: dict[str, Any]) -> str:
if entry.get("ring0_addr"):
return str(entry["ring0_addr"])
link0 = links.get("link0")
if isinstance(link0, str) and link0:
# Property string may be bare IP or address=IP[,priority=N].
if link0.startswith("address="):
return link0.removeprefix("address=").split(",", 1)[0]
return link0.split(",", 1)[0]
if entry.get("new_node_ip"):
return str(entry["new_node_ip"])
return f"10.0.0.{index}"
def _render_corosync_conf(config: dict[str, Any], node_names: list[str]) -> str:
clustername = str(config.get("clustername") or "proxmox")
totem = dict(config.get("totem") or {})
added_raw = config.get("added_nodes")
added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
links = dict(config.get("links") or {})
lines = [
"totem {",
" version: 2",
f" cluster_name: {totem.get('cluster_name', clustername)}",
f" secauth: {totem.get('secauth', 'on')}",
"}",
"nodelist {",
]
for index, name in enumerate(node_names, start=1):
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
nodeid = int(entry.get("nodeid") or index)
votes = int(entry.get("quorum_votes") or entry.get("votes") or config.get("votes") or 1)
ring0 = _ring0_addr(name, index, entry, links)
lines.extend(
[
" node {",
f" name: {name}",
f" nodeid: {nodeid}",
f" quorum_votes: {votes}",
f" ring0_addr: {ring0}",
" }",
]
)
lines.append("}")
lines.extend(["quorum {", " provider: corosync_votequorum", "}"])
return "\n".join(lines) + "\n"
def _ensure_corosync_materials(
config: dict[str, Any], node_names: list[str]
) -> tuple[str, str, str]:
authkey = config.get("corosync_authkey")
if not isinstance(authkey, str) or not authkey:
authkey = secrets.token_hex(32)
config["corosync_authkey"] = authkey
conf = _render_corosync_conf(config, node_names)
config["corosync_conf"] = conf
digest = _stable_digest(conf)
config["config_digest"] = digest
return authkey, conf, digest
async def _cluster_task(
request: Request, *, task_type: str, worker: str, task_id: str = "0"
) -> str:
from app.db.primitives import ConflictError
pool = database(request).pool
node = await pool.fetchval("SELECT name FROM nodes ORDER BY name LIMIT 1") or "localhost"
upid = str(Upid.allocate(str(node), worker, task_id, _principal(request)))
try:
task = await TaskRepository(pool).create(
upid=upid,
task_type=task_type,
payload={"cluster": True, "worker": worker},
resource_key=f"cluster:{task_type}:{secrets.token_hex(4)}",
)
except ConflictError as error:
raise ApiError(409, str(error)) from error
return task.upid
def register_cluster_config_handlers(registry: HandlerRegistry) -> None: def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]: async def index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("apiversion", "join", "nodes", "qdevice", "totem") # PVE uses {name}; contract child link is href="{name}".
return [{"name": name} for name in ("nodes", "totem", "join", "qdevice", "apiversion")]
async def create(request: Request, inputs: dict[str, Any]) -> None: async def create(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
clustername = str(require_value(payload, "clustername"))
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
config = _config(metadata) config = dict(_config(metadata))
if payload.get("clustername"): # Match PVE: refuse when corosync materials already exist (seeded or prior create).
config["clustername"] = str(payload["clustername"]) if config.get("corosync_conf") or config.get("config_digest"):
config.setdefault("totem", {})["cluster_name"] = str(payload["clustername"]) raise ApiError(400, "cluster config already exists")
config["clustername"] = clustername
totem = dict(config.get("totem") or {})
totem["version"] = int(totem.get("version") or 2)
totem["secauth"] = totem.get("secauth") or "on"
totem["cluster_name"] = clustername
# PVE defaults token-coefficient to 125 when omitted (majors that expose it).
totem["token_coefficient"] = payload.get("token-coefficient", 125)
config["totem"] = totem
if "votes" in payload: if "votes" in payload:
config["votes"] = payload["votes"] config["votes"] = payload["votes"]
if "nodeid" in payload: if "nodeid" in payload:
@@ -64,12 +160,43 @@ def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
if links: if links:
config["links"] = links config["links"] = links
config["token"] = secrets.token_hex(16) config["token"] = secrets.token_hex(16)
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows]
if not node_names:
node_names = ["localhost"]
creator = node_names[0]
added = dict(config.get("added_nodes") or {})
creator_entry = dict(added.get(creator) or {})
creator_entry.update(
{
"node": creator,
"nodeid": payload.get("nodeid") or creator_entry.get("nodeid") or 1,
"quorum_votes": payload.get("votes") or creator_entry.get("quorum_votes") or 1,
"ring0_addr": _ring0_addr(
creator, 1, creator_entry, dict(config.get("links") or {})
),
"pve_addr": creator_entry.get("pve_addr") or _node_addr(creator, 1, creator_entry),
"pve_fp": creator_entry.get("pve_fp") or _fingerprint_sha256(),
}
)
added[creator] = creator_entry
config["added_nodes"] = added
_ensure_corosync_materials(config, node_names)
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
await database(request).pool.execute( await database(request).pool.execute(
"""UPDATE clusters """UPDATE clusters
SET name=$1, updated_at=now() SET name=$1, updated_at=now()
WHERE id=(SELECT id FROM clusters LIMIT 1)""", WHERE id=(SELECT id FROM clusters LIMIT 1)""",
str(config["clustername"]), clustername,
)
return await _cluster_task(
request,
task_type="cluster-create",
worker="clustercreate",
task_id=clustername,
) )
async def apiversion(_request: Request, _inputs: dict[str, Any]) -> int: async def apiversion(_request: Request, _inputs: dict[str, Any]) -> int:
@@ -79,84 +206,143 @@ def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
async def join_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def join_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
config = _config(metadata) config = _config(metadata)
node = values(inputs).get("node") preferred_raw = values(inputs).get("node")
# Contract documents the omitted-node default as this phrase; PVE uses local nodename.
if preferred_raw in {None, "", "current connected node"}:
preferred = None
else:
preferred = str(preferred_raw)
rows = await database(request).pool.fetch("SELECT name, status FROM nodes ORDER BY name") rows = await database(request).pool.fetch("SELECT name, status FROM nodes ORDER BY name")
nodelist = [ added_raw = config.get("added_nodes")
{"name": str(row["name"]), "online": 1 if row["status"] == "online" else 0} added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
for row in rows links = dict(config.get("links") or {})
] nodelist: list[dict[str, Any]] = []
for index, row in enumerate(rows, start=1):
name = str(row["name"])
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
if "pve_fp" not in entry:
entry["pve_fp"] = _fingerprint_sha256()
added[name] = entry
if "pve_addr" not in entry:
entry["pve_addr"] = _node_addr(name, index, entry)
added[name] = entry
nodelist.append(
{
"name": name,
"nodeid": int(entry.get("nodeid") or index),
"quorum_votes": int(
entry.get("quorum_votes") or entry.get("votes") or config.get("votes") or 1
),
"ring0_addr": _ring0_addr(name, index, entry, links),
"pve_addr": str(entry["pve_addr"]),
"pve_fp": str(entry["pve_fp"]),
}
)
if added != dict(added_raw or {}):
updated = dict(config)
updated["added_nodes"] = added
if not updated.get("config_digest"):
_ensure_corosync_materials(updated, [item["name"] for item in nodelist])
metadata["cluster_config"] = updated
await save_cluster_metadata(request, metadata)
config = updated
preferred_node = preferred or (nodelist[0]["name"] if nodelist else None)
digest = config.get("config_digest")
if not isinstance(digest, str) or not digest:
updated = dict(config)
_, _, digest = _ensure_corosync_materials(updated, [item["name"] for item in nodelist])
metadata["cluster_config"] = updated
await save_cluster_metadata(request, metadata)
config = updated
return { return {
"clustername": config.get("clustername"),
"config_digest": secrets.token_hex(8),
"nodelist": nodelist, "nodelist": nodelist,
"preferred_node": node or (nodelist[0]["name"] if nodelist else None), "preferred_node": preferred_node,
"totem": config.get("totem", {}), "totem": dict(config.get("totem") or {}),
"links": config.get("links", {}), "config_digest": digest,
} }
async def join_post(request: Request, inputs: dict[str, Any]) -> None: async def join_post(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
hostname = str(require_value(payload, "hostname"))
require_value(payload, "fingerprint")
require_value(payload, "password")
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
config = _config(metadata) config = dict(_config(metadata))
hostname = str(payload.get("hostname") or payload.get("node") or "") joins = dict(config.get("join_info") or {})
if not hostname:
raise ApiError(400, "parameter verification failed - 'hostname' missing")
joins = config.setdefault("join_info", {})
joins[hostname] = { joins[hostname] = {
"hostname": hostname, "hostname": hostname,
"fingerprint": payload.get("fingerprint"), "fingerprint": payload.get("fingerprint"),
"nodeid": payload.get("nodeid"), "nodeid": payload.get("nodeid"),
"votes": payload.get("votes", 1), "votes": payload.get("votes", 1),
"force": payload.get("force"), "force": payload.get("force"),
"password_set": True,
} }
# password accepted but not stored in clear form links = {key: value for key, value in payload.items() if key.startswith("link")}
if payload.get("password"): if links:
joins[hostname]["password_set"] = True joins[hostname]["links"] = links
exists = await database(request).pool.fetchval( config["join_info"] = joins
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
hostname, node_names = [str(row["name"]) for row in rows]
) _ensure_corosync_materials(config, node_names or ["localhost"])
if not exists: metadata["cluster_config"] = config
await database(request).pool.execute(
"""INSERT INTO nodes(id, name, status, metadata)
VALUES(gen_random_uuid(), $1, 'online', '{}'::jsonb)""",
hostname,
)
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return await _cluster_task(
request,
task_type="cluster-join",
worker="clusterjoin",
)
async def nodes_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def nodes_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await database(request).pool.fetch("SELECT name, status FROM nodes ORDER BY name") rows = await database(request).pool.fetch("SELECT name, status FROM nodes ORDER BY name")
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
config = _config(metadata) config = _config(metadata)
added_raw = config.get("added_nodes")
added: dict[str, Any] = dict(added_raw) if isinstance(added_raw, dict) else {}
links = dict(config.get("links") or {})
result = [] result = []
for index, row in enumerate(rows, start=1): for index, row in enumerate(rows, start=1):
name = str(row["name"])
entry_raw = added.get(name)
entry: dict[str, Any] = dict(entry_raw) if isinstance(entry_raw, dict) else {}
result.append( result.append(
{ {
"node": str(row["name"]), "node": name,
"nodeid": index, "nodeid": entry.get("nodeid", index),
"ring0_addr": f"{row['name']}.local", "ring0_addr": _ring0_addr(name, index, entry, links),
"quorum_votes": config.get("votes", 1), "quorum_votes": entry.get("quorum_votes", config.get("votes", 0)),
} }
) )
return result return result
async def nodes_add(request: Request, inputs: dict[str, Any]) -> None: async def nodes_add(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
node = str(payload["node"]) node = str(require_value(payload, "node"))
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
config = _config(metadata) config = dict(_config(metadata))
added = config.setdefault("added_nodes", {}) added = dict(config.get("added_nodes") or {})
added[node] = { if node in added and not payload.get("force"):
raise ApiError(400, f"can't add existing node '{node}'")
index = len(added) + 1
links = {key: value for key, value in payload.items() if key.startswith("link")}
entry = {
"node": node, "node": node,
"nodeid": payload.get("nodeid"), "nodeid": payload.get("nodeid") or index,
"new_node_ip": payload.get("new_node_ip"), "new_node_ip": payload.get("new_node_ip"),
"votes": payload.get("votes", 1), "votes": payload.get("votes", 1),
"apiversion": payload.get("apiversion"), "apiversion": payload.get("apiversion"),
"force": payload.get("force"), "force": payload.get("force"),
"ring0_addr": payload.get("ring0_addr")
or payload.get("new_node_ip")
or _ring0_addr(node, index, {}, links or dict(config.get("links") or {})),
"quorum_votes": payload.get("quorum_votes", payload.get("votes", 1)),
"pve_addr": payload.get("new_node_ip") or _node_addr(node, index, {}),
"pve_fp": _fingerprint_sha256(),
} }
links = {key: value for key, value in payload.items() if key.startswith("link")}
if links: if links:
added[node]["links"] = links entry["links"] = links
added[node] = entry
config["added_nodes"] = added
exists = await database(request).pool.fetchval( exists = await database(request).pool.fetchval(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)", "SELECT EXISTS(SELECT 1 FROM nodes WHERE name=$1)",
node, node,
@@ -167,16 +353,33 @@ def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
VALUES(gen_random_uuid(), $1, 'online', '{}'::jsonb)""", VALUES(gen_random_uuid(), $1, 'online', '{}'::jsonb)""",
node, node,
) )
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows]
if node not in node_names:
node_names.append(node)
authkey, conf, _digest = _ensure_corosync_materials(config, node_names)
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return {
"corosync_authkey": authkey,
"corosync_conf": conf,
"warnings": [],
}
async def nodes_delete(request: Request, inputs: dict[str, Any]) -> None: async def nodes_delete(request: Request, inputs: dict[str, Any]) -> None:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
config = _config(metadata) config = dict(_config(metadata))
added = config.setdefault("added_nodes", {}) added = dict(config.get("added_nodes") or {})
added.pop(node, None) added.pop(node, None)
joins = config.setdefault("join_info", {}) joins = dict(config.get("join_info") or {})
joins.pop(node, None) joins.pop(node, None)
config["added_nodes"] = added
config["join_info"] = joins
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in rows if str(row["name"]) != node]
_ensure_corosync_materials(config, node_names or ["localhost"])
metadata["cluster_config"] = config
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
# Keep node row; mark offline to avoid cascading guest deletes. # Keep node row; mark offline to avoid cascading guest deletes.
await database(request).pool.execute( await database(request).pool.execute(
@@ -186,11 +389,13 @@ def register_cluster_config_handlers(registry: HandlerRegistry) -> None:
async def qdevice(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]: async def qdevice(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
return dict(_config(metadata).get("qdevice") or {"status": "disabled"}) qdevice = _config(metadata).get("qdevice")
return dict(qdevice) if isinstance(qdevice, dict) else {}
async def totem(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]: async def totem(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
return dict(_config(metadata).get("totem") or {}) totem = _config(metadata).get("totem")
return dict(totem) if isinstance(totem, dict) else {}
registry.register("/cluster/config", "GET", index) registry.register("/cluster/config", "GET", index)
registry.register("/cluster/config", "POST", create) registry.register("/cluster/config", "POST", create)
+198 -111
View File
@@ -2,7 +2,6 @@
from __future__ import annotations from __future__ import annotations
import copy
import secrets import secrets
import time import time
from typing import Any from typing import Any
@@ -16,62 +15,28 @@ from app.handlers.common import (
database, database,
require_node, require_node,
save_cluster_metadata, save_cluster_metadata,
state,
subdirs, subdirs,
values, values,
) )
from app.simulation.ceph_defaults import CEPH_FLAG_DESCRIPTIONS, default_ceph_flags
from app.tasks.repository import TaskRepository from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid from app.tasks.upid import Upid
DEFAULT_CEPH_FLAGS: dict[str, int] = {
"nobackfill": 0,
"nodeep-scrub": 0,
"nodown": 0,
"noin": 0,
"noout": 0,
"norebalance": 0,
"norecover": 0,
"noscrub": 0,
"notieragent": 0,
"pause": 0,
}
DEFAULT_CPU_FLAGS: list[dict[str, Any]] = [
{"name": "aes", "introduces": "Westmere"},
{"name": "avx", "introduces": "SandyBridge"},
{"name": "avx2", "introduces": "Haswell"},
]
def _jobs(metadata: dict[str, Any]) -> dict[str, Any]: def _jobs(metadata: dict[str, Any]) -> dict[str, Any]:
jobs = metadata.setdefault("jobs", {}) jobs = metadata.get("jobs")
if not isinstance(jobs, dict): return jobs if isinstance(jobs, dict) else {}
jobs = {}
metadata["jobs"] = jobs
sync = jobs.setdefault("realm_sync", {})
if not isinstance(sync, dict):
sync = {}
jobs["realm_sync"] = sync
return jobs
def _metrics(metadata: dict[str, Any]) -> dict[str, Any]: def _metrics(metadata: dict[str, Any]) -> dict[str, Any]:
metrics = metadata.setdefault("metrics", {}) metrics = metadata.get("metrics")
if not isinstance(metrics, dict): return metrics if isinstance(metrics, dict) else {}
metrics = {}
metadata["metrics"] = metrics
servers = metrics.setdefault("servers", {})
if not isinstance(servers, dict):
servers = {}
metrics["servers"] = servers
return metrics
def _cpu_models(metadata: dict[str, Any]) -> dict[str, Any]: def _cpu_models(metadata: dict[str, Any]) -> dict[str, Any]:
models = metadata.get("qemu_cpu_models") models = metadata.get("qemu_cpu_models")
if not isinstance(models, dict): return models if isinstance(models, dict) else {}
models = {}
metadata["qemu_cpu_models"] = models
return models
def _ha_rules_store(metadata: dict[str, Any]) -> list[dict[str, Any]]: def _ha_rules_store(metadata: dict[str, Any]) -> list[dict[str, Any]]:
@@ -84,12 +49,7 @@ def _ha_rules_store(metadata: dict[str, Any]) -> list[dict[str, Any]]:
] ]
if isinstance(rules, list): if isinstance(rules, list):
return [dict(item) for item in rules if isinstance(item, dict)] return [dict(item) for item in rules if isinstance(item, dict)]
defaults = [ return []
{"rule": "node-fencing", "type": "node", "action": "restart"},
{"rule": "service-ha", "type": "resource", "action": "failover"},
]
metadata["ha_rules"] = defaults
return list(defaults)
def _save_ha_rules(metadata: dict[str, Any], rules: list[dict[str, Any]]) -> None: def _save_ha_rules(metadata: dict[str, Any], rules: list[dict[str, Any]]) -> None:
@@ -97,26 +57,14 @@ def _save_ha_rules(metadata: dict[str, Any], rules: list[dict[str, Any]]) -> Non
def _replication_jobs(metadata: dict[str, Any]) -> list[dict[str, Any]]: def _replication_jobs(metadata: dict[str, Any]) -> list[dict[str, Any]]:
jobs = metadata.get("replication", []) from app.handlers.cluster import _replication_jobs as normalize
if not isinstance(jobs, list):
return [] return normalize(metadata)
return [dict(item) for item in jobs if isinstance(item, dict)]
def _ceph(metadata: dict[str, Any]) -> dict[str, Any]: def _ceph(metadata: dict[str, Any]) -> dict[str, Any]:
ceph = metadata.get("ceph") ceph = metadata.get("ceph")
if not isinstance(ceph, dict): return ceph if isinstance(ceph, dict) else {}
ceph = {}
flags = ceph.get("flags")
if not isinstance(flags, dict):
flags = copy.deepcopy(DEFAULT_CEPH_FLAGS)
else:
merged = copy.deepcopy(DEFAULT_CEPH_FLAGS)
merged.update({str(key): int(value) for key, value in flags.items()})
flags = merged
ceph["flags"] = flags
metadata["ceph"] = ceph
return ceph
async def _cluster_task(request: Request, *, task_type: str, worker: str) -> str: async def _cluster_task(request: Request, *, task_type: str, worker: str) -> str:
@@ -174,8 +122,8 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
job_id = str(payload["id"]) job_id = str(payload["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
jobs = _jobs(metadata) jobs = dict(_jobs(metadata))
sync = jobs.setdefault("realm_sync", {}) sync = dict(jobs.get("realm_sync") or {})
if job_id in sync: if job_id in sync:
raise ApiError(409, "realm-sync job already exists") raise ApiError(409, "realm-sync job already exists")
entry = { entry = {
@@ -185,6 +133,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
entry.setdefault("enabled", 1) entry.setdefault("enabled", 1)
entry.setdefault("realm", str(payload.get("realm") or "pam")) entry.setdefault("realm", str(payload.get("realm") or "pam"))
sync[job_id] = entry sync[job_id] = entry
jobs["realm_sync"] = sync
metadata["jobs"] = jobs metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return {"id": job_id, **entry} return {"id": job_id, **entry}
@@ -193,8 +142,8 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
job_id = str(payload["id"]) job_id = str(payload["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
jobs = _jobs(metadata) jobs = dict(_jobs(metadata))
sync = jobs.setdefault("realm_sync", {}) sync = dict(jobs.get("realm_sync") or {})
if job_id not in sync: if job_id not in sync:
raise ApiError(404, "realm-sync job does not exist") raise ApiError(404, "realm-sync job does not exist")
updated = { updated = {
@@ -206,6 +155,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
}, },
} }
sync[job_id] = updated sync[job_id] = updated
jobs["realm_sync"] = sync
metadata["jobs"] = jobs metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return {"id": job_id, **updated} return {"id": job_id, **updated}
@@ -213,23 +163,26 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
async def realm_sync_delete(request: Request, inputs: dict[str, Any]) -> None: async def realm_sync_delete(request: Request, inputs: dict[str, Any]) -> None:
job_id = str(values(inputs)["id"]) job_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
jobs = _jobs(metadata) jobs = dict(_jobs(metadata))
sync = jobs.setdefault("realm_sync", {}) sync = dict(jobs.get("realm_sync") or {})
if job_id not in sync: if job_id not in sync:
raise ApiError(404, "realm-sync job does not exist") raise ApiError(404, "realm-sync job does not exist")
del sync[job_id] del sync[job_id]
jobs["realm_sync"] = sync
metadata["jobs"] = jobs metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def schedule_analyze(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def schedule_analyze(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
schedule = str(values(inputs).get("schedule") or "*/15") schedule = str(values(inputs).get("schedule") or "*/15")
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
jobs = _jobs(metadata) jobs = dict(_jobs(metadata))
results = jobs.get("schedule_analyze_results")
if not isinstance(results, list):
results = []
jobs["last_schedule_analyze"] = {"schedule": schedule, "at": int(time.time())} jobs["last_schedule_analyze"] = {"schedule": schedule, "at": int(time.time())}
metadata["jobs"] = jobs metadata["jobs"] = jobs
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
now = int(time.time()) return [dict(item) for item in results if isinstance(item, dict)]
return [{"timestamp": now + offset * 900, "utc": True} for offset in range(4)]
async def metrics_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]: async def metrics_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("export", "server") return subdirs("export", "server")
@@ -237,20 +190,54 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
async def metrics_export(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]: async def metrics_export(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
metrics = _metrics(metadata) metrics = _metrics(metadata)
return { now = int(time.time())
"data": metrics.get("export_data") seeded = metrics.get("export_series")
or '# HELP pve_up Node is up\npve_up{node="pve01"} 1\n', if isinstance(seeded, list) and seeded:
"timestamp": int(time.time()), seeded_series = [dict(item) for item in seeded if isinstance(item, dict)]
} return {"data": seeded_series, "timestamp": now}
# Build a PVE-shaped metric array from live inventory (scales with seed).
rows = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
series: list[dict[str, Any]] = []
for row in rows:
node = str(row["name"])
series.append(
{
"id": f"node/{node}",
"metric": "up",
"timestamp": now,
"type": "gauge",
"value": 1,
}
)
guests = await database(request).pool.fetch(
"""SELECT r.kind, r.external_id, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind IN ('qemu', 'lxc')
ORDER BY r.external_id::integer"""
)
for guest in guests:
series.append(
{
"id": f"{guest['kind']}/{guest['external_id']}",
"metric": "up",
"timestamp": now,
"type": "gauge",
"value": 1,
}
)
return {"data": series, "timestamp": now}
async def metrics_server_list( async def metrics_server_list(
request: Request, _inputs: dict[str, Any] request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
metrics = _metrics(metadata) metrics = _metrics(metadata)
servers = metrics.get("servers")
if not isinstance(servers, dict):
return []
return [ return [
{"id": server_id, **dict(payload)} {"id": server_id, **dict(payload)}
for server_id, payload in sorted(metrics.get("servers", {}).items()) for server_id, payload in sorted(servers.items())
if isinstance(payload, dict) if isinstance(payload, dict)
] ]
@@ -267,8 +254,8 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
server_id = str(payload["id"]) server_id = str(payload["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
metrics = _metrics(metadata) metrics = dict(_metrics(metadata))
servers = metrics.setdefault("servers", {}) servers = dict(metrics.get("servers") or {})
if server_id in servers: if server_id in servers:
raise ApiError(409, "metrics server already exists") raise ApiError(409, "metrics server already exists")
entry = { entry = {
@@ -279,6 +266,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
entry.setdefault("port", 8086) entry.setdefault("port", 8086)
entry.setdefault("enable", 1) entry.setdefault("enable", 1)
servers[server_id] = entry servers[server_id] = entry
metrics["servers"] = servers
metadata["metrics"] = metrics metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return {"id": server_id, **entry} return {"id": server_id, **entry}
@@ -287,8 +275,8 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
server_id = str(payload["id"]) server_id = str(payload["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
metrics = _metrics(metadata) metrics = dict(_metrics(metadata))
servers = metrics.setdefault("servers", {}) servers = dict(metrics.get("servers") or {})
if server_id not in servers: if server_id not in servers:
raise ApiError(404, "metrics server does not exist") raise ApiError(404, "metrics server does not exist")
updated = { updated = {
@@ -300,6 +288,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
}, },
} }
servers[server_id] = updated servers[server_id] = updated
metrics["servers"] = servers
metadata["metrics"] = metrics metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return {"id": server_id, **updated} return {"id": server_id, **updated}
@@ -307,19 +296,24 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
async def metrics_server_delete(request: Request, inputs: dict[str, Any]) -> None: async def metrics_server_delete(request: Request, inputs: dict[str, Any]) -> None:
server_id = str(values(inputs)["id"]) server_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
metrics = _metrics(metadata) metrics = dict(_metrics(metadata))
servers = metrics.setdefault("servers", {}) servers = dict(metrics.get("servers") or {})
if server_id not in servers: if server_id not in servers:
raise ApiError(404, "metrics server does not exist") raise ApiError(404, "metrics server does not exist")
del servers[server_id] del servers[server_id]
metrics["servers"] = servers
metadata["metrics"] = metrics metadata["metrics"] = metrics
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def qemu_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]: async def qemu_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("cpu-flags", "custom-cpu-models") return subdirs("cpu-flags", "custom-cpu-models")
async def qemu_cpu_flags(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def qemu_cpu_flags(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(DEFAULT_CPU_FLAGS) metadata = await cluster_metadata(request)
flags = metadata.get("qemu_cpu_flags")
if isinstance(flags, list):
return [dict(item) for item in flags if isinstance(item, dict)]
return []
async def cpu_models_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def cpu_models_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
@@ -437,41 +431,69 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
) -> list[dict[str, str]]: ) -> list[dict[str, str]]:
return subdirs("flags", "metadata", "status") return subdirs("flags", "metadata", "status")
async def ceph_flags_get(request: Request, _inputs: dict[str, Any]) -> dict[str, int]: async def ceph_flags_get(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
ceph = _ceph(metadata) ceph = _ceph(metadata)
await save_cluster_metadata(request, metadata) stored = ceph.get("flags")
return {str(key): int(value) for key, value in ceph["flags"].items()} flags = default_ceph_flags()
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
return [
{
"name": name,
"value": bool(flags[name]),
"description": description,
}
for name, description in sorted(CEPH_FLAG_DESCRIPTIONS.items())
]
async def ceph_flags_put(request: Request, inputs: dict[str, Any]) -> dict[str, int]: async def ceph_flags_put(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
ceph = _ceph(metadata) ceph = dict(_ceph(metadata))
flags = dict(ceph["flags"]) flags = default_ceph_flags()
stored = ceph.get("flags")
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
for key, value in payload.items(): for key, value in payload.items():
if key in {"delete", "digest"}: if key in {"delete", "digest"}:
continue continue
if key not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(400, f"unknown ceph flag '{key}'")
flags[str(key)] = int(value) flags[str(key)] = int(value)
ceph["flags"] = flags ceph["flags"] = flags
metadata["ceph"] = ceph metadata["ceph"] = ceph
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return {str(key): int(value) for key, value in flags.items()} return await _cluster_task(request, task_type="ceph-flags", worker="cephflags")
async def ceph_flag_get(request: Request, inputs: dict[str, Any]) -> dict[str, int]: async def ceph_flag_get(request: Request, inputs: dict[str, Any]) -> bool:
flag = str(values(inputs)["flag"]) flag = str(values(inputs)["flag"])
if flag not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(404, "ceph flag does not exist")
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
ceph = _ceph(metadata) ceph = _ceph(metadata)
flags = ceph["flags"] flags = ceph.get("flags")
if flag not in flags: if isinstance(flags, dict) and flag in flags:
raise ApiError(404, "ceph flag does not exist") return bool(int(flags[flag]))
return {flag: int(flags[flag])} return False
async def ceph_flag_put(request: Request, inputs: dict[str, Any]) -> dict[str, int]: async def ceph_flag_put(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
flag = str(payload["flag"]) flag = str(payload["flag"])
if flag not in CEPH_FLAG_DESCRIPTIONS:
raise ApiError(404, "ceph flag does not exist")
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
ceph = _ceph(metadata) ceph = dict(_ceph(metadata))
flags = dict(ceph["flags"]) flags = default_ceph_flags()
stored = ceph.get("flags")
if isinstance(stored, dict):
for name, value in stored.items():
if name in flags:
flags[name] = int(value)
if "value" in payload: if "value" in payload:
flags[flag] = int(payload["value"]) flags[flag] = int(payload["value"])
elif flag in payload: elif flag in payload:
@@ -481,19 +503,84 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
ceph["flags"] = flags ceph["flags"] = flags
metadata["ceph"] = ceph metadata["ceph"] = ceph
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return {flag: int(flags[flag])}
async def ceph_metadata(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]: async def ceph_metadata(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
ceph = _ceph(metadata) ceph = _ceph(metadata)
await save_cluster_metadata(request, metadata) config_raw = ceph.get("config")
config: dict[str, Any] = dict(config_raw) if isinstance(config_raw, dict) else {}
version = ceph.get("version")
nodes = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
node_names = [str(row["name"]) for row in nodes]
version_str = (
str(version.get("str"))
if isinstance(version, dict) and version.get("str")
else "18.2.2"
)
version_short = ".".join(version_str.split(".")[:2]) if version_str else "18.2"
osd_rows = await database(request).pool.fetch(
"""SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind='ceph-osd' ORDER BY r.external_id"""
)
osd_map: dict[str, Any] = {}
for row in osd_rows:
payload = state(row["state"])
external = str(row["external_id"])
osd_id = str(payload.get("osd_id", external.removeprefix("osd.")))
hostname = str(row["node"])
osd_map[osd_id] = {
"id": int(osd_id) if osd_id.isdigit() else osd_id,
"hostname": hostname,
"name": f"osd.{osd_id}",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
"device_class": payload.get("device_class", "hdd"),
"dev": payload.get("dev", ""),
}
if not osd_map:
fallback_host = node_names[0] if node_names else "pve01"
osd_map["0"] = {
"id": 0,
"hostname": fallback_host,
"name": "osd.0",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
}
def _service_entry(name: str, index: int) -> dict[str, Any]:
return {
"name": name,
"hostname": name,
"addr": f"10.0.0.{index}",
"ceph_version": version_str,
"ceph_version_short": version_short,
"ceph_release": "reef",
}
return { return {
"version": ceph.get("version") or {"str": "18.2.2", "parts": [18, 2, 2]}, "mds": {},
"fsid": ceph.get("config", {}).get("fsid") "mgr": {
if isinstance(ceph.get("config"), dict) f"mgr@{node}": _service_entry(node, index)
else "pve-simulator-fsid", for index, node in enumerate(node_names[:1] or ["pve01"], start=1)
"initialized": int(bool(ceph.get("initialized", True))), },
"flags": ceph.get("flags", {}), "mon": {
f"mon@{node}": _service_entry(node, index)
for index, node in enumerate(node_names[:3] or ["pve01"], start=1)
},
"node": {
node: {
"name": node,
"hostname": node,
"ceph_version": version_str,
"ceph_version_short": version_short,
}
for node in (node_names or ["pve01"])
},
"osd": list(osd_map.values()),
"fsid": str(config.get("fsid") or ""),
} }
async def ha_rule_create(request: Request, inputs: dict[str, Any]) -> None: async def ha_rule_create(request: Request, inputs: dict[str, Any]) -> None:
@@ -588,7 +675,7 @@ def register_cluster_extra_handlers(registry: HandlerRegistry) -> None:
log = job.get("log") log = job.get("log")
if isinstance(log, list): if isinstance(log, list):
return [dict(item) for item in log if isinstance(item, dict)] return [dict(item) for item in log if isinstance(item, dict)]
return [{"t": int(time.time()), "n": 0, "msg": f"replication idle for {job.get('id')}"}] return []
async def node_replication_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_replication_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
job = await node_replication_get(request, inputs) job = await node_replication_get(request, inputs)
+29 -6
View File
@@ -76,7 +76,11 @@ async def require_node(request: Request, node: str) -> None:
node, node,
) )
if not exists: if not exists:
raise ApiError(404, "node does not exist") raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
async def cluster_metadata(request: Request) -> dict[str, Any]: async def cluster_metadata(request: Request) -> dict[str, Any]:
@@ -105,7 +109,11 @@ async def node_metadata(request: Request, node: str) -> dict[str, Any]:
node, node,
) )
if row is None: if row is None:
raise ApiError(404, "node does not exist") raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
return state(row["metadata"]) return state(row["metadata"])
@@ -116,7 +124,11 @@ async def save_node_metadata(request: Request, node: str, metadata: dict[str, An
json.dumps(metadata, sort_keys=True), json.dumps(metadata, sort_keys=True),
) )
if status != "UPDATE 1": if status != "UPDATE 1":
raise ApiError(404, "node does not exist") raise ApiError(
404,
f"No such node ('{node}')",
errors={"node": f"No such node ('{node}')"},
)
def storage_payload(row: Any) -> dict[str, Any]: def storage_payload(row: Any) -> dict[str, Any]:
@@ -129,13 +141,24 @@ def storage_payload(row: Any) -> dict[str, Any]:
total = int(row["capacity_bytes"] or 0) total = int(row["capacity_bytes"] or 0)
used = int(row["used_bytes"] or 0) used = int(row["used_bytes"] or 0)
avail = max(total - used, 0) avail = max(total - used, 0)
storage_type = str(row["storage_type"])
formats = config.get("formats")
if not isinstance(formats, str) or not formats:
formats = {
"dir": "raw,qcow2,vmdk",
"lvmthin": "raw",
"rbd": "raw",
"zfspool": "raw,subvol",
}.get(storage_type, "raw")
payload: dict[str, Any] = { payload: dict[str, Any] = {
"storage": str(row["storage_id"]), "storage": str(row["storage_id"]),
"type": str(row["storage_type"]), "type": storage_type,
"shared": int(bool(row["shared"])), "shared": int(bool(row["shared"])),
"content": content_str, "content": content_str,
"active": 1, "active": int(bool(config.get("active", True))),
"enabled": 1, "enabled": int(bool(config.get("enabled", True))),
"formats": formats,
"select_existing": int(bool(config.get("select_existing", False))),
"total": total, "total": total,
"used": used, "used": used,
"avail": avail, "avail": avail,
+331 -23
View File
@@ -19,7 +19,7 @@ from app.handlers.ceph import register_ceph_handlers
from app.handlers.cluster import register_cluster_handlers from app.handlers.cluster import register_cluster_handlers
from app.handlers.cluster_config import register_cluster_config_handlers from app.handlers.cluster_config import register_cluster_config_handlers
from app.handlers.cluster_extra import register_cluster_extra_handlers from app.handlers.cluster_extra import register_cluster_extra_handlers
from app.handlers.common import require_node, subdirs from app.handlers.common import cluster_metadata, require_node, subdirs
from app.handlers.firewall import register_firewall_handlers from app.handlers.firewall import register_firewall_handlers
from app.handlers.ha import register_ha_handlers from app.handlers.ha import register_ha_handlers
from app.handlers.legacy_aliases import register_legacy_aliases from app.handlers.legacy_aliases import register_legacy_aliases
@@ -60,52 +60,360 @@ def build_core_handlers(settings: Settings) -> HandlerRegistry:
raise ApiError(401, "authentication failure") raise ApiError(401, "authentication failure")
key = settings.ticket_signing_key.get_secret_value().encode() key = settings.ticket_signing_key.get_secret_value().encode()
ticket = issue_ticket(username, key) ticket = issue_ticket(username, key)
cluster_name = await _database(request).pool.fetchval(
"SELECT name FROM clusters ORDER BY created_at LIMIT 1"
)
return { return {
"username": username, "username": username,
"ticket": ticket, "ticket": ticket,
"CSRFPreventionToken": csrf_token(ticket, key), "CSRFPreventionToken": csrf_token(ticket, key),
"clustername": str(cluster_name or "pve-simulator"),
"cap": {"vms": {"VM.Audit": 1, "VM.PowerMgmt": 1}}, "cap": {"vms": {"VM.Audit": 1, "VM.PowerMgmt": 1}},
} }
async def nodes(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def nodes(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
from app.handlers.nodes import load_node_ops
rows = await _database(request).pool.fetch( rows = await _database(request).pool.fetch(
"SELECT name AS node, status FROM nodes ORDER BY name" "SELECT name AS node, status FROM nodes ORDER BY name"
) )
return [{"node": str(row["node"]), "status": str(row["status"])} for row in rows] result: list[dict[str, Any]] = []
for row in rows:
name = str(row["node"])
ops = await load_node_ops(request, name)
status_payload = ops.get("status")
status_dict = dict(status_payload) if isinstance(status_payload, dict) else {}
fingerprint = status_dict.get("ssl_fingerprint") or status_dict.get("fingerprint")
if fingerprint in (None, "", 0) or isinstance(fingerprint, dict | list):
from app.simulation.seed import _seed_fingerprint
fingerprint = _seed_fingerprint(name)
def _as_float(value: object, default: float) -> float:
if isinstance(value, bool) or value is None or isinstance(value, dict | list):
return default
try:
return float(str(value))
except (TypeError, ValueError):
return default
def _as_int(value: object, default: int) -> int:
if isinstance(value, bool) or value is None or isinstance(value, dict | list):
return default
try:
return int(float(str(value)))
except (TypeError, ValueError):
return default
item: dict[str, Any] = {
"node": name,
"status": str(row["status"]),
"type": "node",
"ssl_fingerprint": str(fingerprint),
"cpu": _as_float(status_dict.get("cpu"), 0.0),
"maxcpu": _as_int(status_dict.get("maxcpu"), 4),
"mem": _as_int(status_dict.get("mem"), _as_int(status_dict.get("memory"), 0)),
"maxmem": _as_int(status_dict.get("maxmem"), 8 * 1024**3),
"uptime": _as_int(status_dict.get("uptime"), 0),
"level": str(status_dict.get("level") or ""),
}
result.append(item)
return result
async def node_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
from app.handlers.nodes import load_node_ops
node = str(cast(dict[str, Any], inputs["values"])["node"]) node = str(cast(dict[str, Any], inputs["values"])["node"])
row = await _database(request).pool.fetchrow( row = await _database(request).pool.fetchrow(
"SELECT name, status FROM nodes WHERE name=$1", node "SELECT name, status FROM nodes WHERE name=$1", node
) )
if row is None: if row is None:
raise ApiError(404, "node does not exist") raise ApiError(404, "node does not exist")
ops = await load_node_ops(request, node)
status = ops.get("status")
payload = dict(status) if isinstance(status, dict) else {}
mem_used = int(payload.get("mem") or 0)
mem_total = int(payload.get("maxmem") or 8 * 1024**3)
memory = payload.get("memory")
if isinstance(memory, dict):
mem_used = int(memory.get("used") or mem_used)
mem_total = int(memory.get("total") or mem_total)
rootfs = payload.get("rootfs")
if not isinstance(rootfs, dict):
rootfs = {
"used": int(payload.get("rootfs_used") or mem_used // 8),
"total": int(payload.get("rootfs_total") or 100 * 1024**3),
"avail": int(payload.get("rootfs_avail") or 80 * 1024**3),
"free": int(payload.get("rootfs_free") or 80 * 1024**3),
}
cpuinfo = payload.get("cpuinfo")
if not isinstance(cpuinfo, dict):
cpuinfo = {
"cpus": int(payload.get("maxcpu") or 4),
"cores": int(payload.get("maxcpu") or 4),
"sockets": 1,
"model": str(payload.get("cpu_model") or "QEMU Virtual CPU"),
"flags": "",
}
boot_info = payload.get("boot-info")
if not isinstance(boot_info, dict):
boot_info = {"mode": "efi", "secureboot": 0}
current_kernel = payload.get("current-kernel")
if not isinstance(current_kernel, dict):
kversion = str(payload.get("kversion") or "6.8.12-1-pve")
current_kernel = {
"sysname": "Linux",
"release": kversion,
"version": f"#1 SMP PREEMPT_DYNAMIC {kversion}",
"machine": "x86_64",
}
return { return {
"status": str(row["status"]), "status": str(row["status"]),
"node": str(row["name"]), "uptime": int(payload.get("uptime") or 0),
"uptime": 0, "wait": float(payload.get("wait") or 0.0),
"cpu": 0.0, "idle": float(payload.get("idle") or 0.95),
"memory": {"used": 0, "total": 0}, "cpu": float(payload.get("cpu") or 0.0),
"loadavg": list(payload.get("loadavg") or ["0.00", "0.00", "0.00"]),
"memory": {
"used": mem_used,
"total": mem_total,
"free": max(mem_total - mem_used, 0),
"available": max(mem_total - mem_used, 0),
},
"rootfs": rootfs,
"swap": payload.get("swap")
if isinstance(payload.get("swap"), dict)
else {"used": 0, "total": 0, "free": 0},
"cpuinfo": cpuinfo,
"kversion": str(payload.get("kversion") or current_kernel.get("release") or ""),
"pveversion": str(payload.get("pveversion") or "pve-manager/9.2.3"),
"current-kernel": current_kernel,
"boot-info": boot_info,
} }
async def resources(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def resources(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
rows = await _database(request).pool.fetch( """Cluster-wide inventory in Proxmox ``/cluster/resources`` shape.
"""SELECT r.kind AS type, r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id Do not dump guest ``state`` / config blobs: QEMU ``cpu`` is a model
ORDER BY r.kind, r.external_id""" string (e.g. ``qemu64``), while this endpoint's ``cpu`` is utilization
) (float). Bridged clients (bpg / pulumi-proxmoxve) decode strictly.
result: list[dict[str, Any]] = [] """
for row in rows:
raw_state = row["state"] def _as_dict(raw: object) -> dict[str, Any]:
state = json.loads(raw_state) if isinstance(raw_state, str) else dict(raw_state) if isinstance(raw, str):
result.append( loaded = json.loads(raw)
{ return dict(loaded) if isinstance(loaded, dict) else {}
"type": str(row["type"]), return dict(raw) if isinstance(raw, dict) else {}
"id": f"{row['type']}/{row['external_id']}",
"node": str(row["node"]), def _num(value: object, default: float | int) -> float | int:
**state, if isinstance(value, bool) or value is None or isinstance(value, dict | list):
} return default
try:
if isinstance(default, float):
return float(str(value))
return int(float(str(value)))
except (TypeError, ValueError):
return default
def _memory_bytes(state: dict[str, Any]) -> int:
raw = state.get("maxmem", state.get("memory"))
if raw in (None, ""):
return 0
value = _num(raw, 0)
# QEMU config stores memory in MiB; cluster resources use bytes.
if isinstance(raw, str) or (isinstance(value, int) and 0 < value < 10_000_000):
return int(value) * 1024 * 1024
return int(value)
def _maxcpu(state: dict[str, Any]) -> int:
cores = int(_num(state.get("cores", state.get("cpus", 1)), 1))
sockets = int(_num(state.get("sockets", 1), 1))
return max(cores * sockets, 1)
def _cpu_util(state: dict[str, Any], *, running: bool) -> float:
if not running:
return 0.0
samples = state.get("rrddata")
if isinstance(samples, list) and samples:
last = samples[-1]
if isinstance(last, dict):
return float(_num(last.get("cpu"), 0.0))
return (
float(_num(state.get("cpu"), 0.0)) if not isinstance(state.get("cpu"), str) else 0.0
) )
type_filter = cast(dict[str, Any], inputs.get("values") or {}).get("type")
result: list[dict[str, Any]] = []
if type_filter in (None, "node"):
node_rows = await _database(request).pool.fetch(
"SELECT name AS node, status FROM nodes ORDER BY name"
)
from app.handlers.nodes import load_node_ops
for row in node_rows:
name = str(row["node"])
ops = await load_node_ops(request, name)
status_payload = ops.get("status")
status_dict = dict(status_payload) if isinstance(status_payload, dict) else {}
mem_raw = status_dict.get("mem", status_dict.get("memory"))
if isinstance(mem_raw, dict):
mem_val = int(_num(mem_raw.get("used"), 0))
maxmem_val = int(
_num(status_dict.get("maxmem", mem_raw.get("total")), 8 * 1024**3)
)
else:
mem_val = int(_num(mem_raw, 0))
maxmem_val = int(_num(status_dict.get("maxmem"), 8 * 1024**3))
result.append(
{
"type": "node",
"id": f"node/{name}",
"node": name,
"status": str(row["status"]),
"cpu": float(_num(status_dict.get("cpu"), 0.0)),
"maxcpu": int(_num(status_dict.get("maxcpu"), 4)),
"mem": mem_val,
"maxmem": maxmem_val,
"uptime": int(_num(status_dict.get("uptime"), 0)),
"level": str(status_dict.get("level") or ""),
}
)
kind_filter: tuple[str, ...] | None
if type_filter == "vm":
kind_filter = ("qemu", "lxc")
elif type_filter == "storage":
kind_filter = ("storage",)
elif type_filter in (None,):
kind_filter = ("qemu", "lxc", "storage", "pool")
elif type_filter in {"qemu", "lxc", "storage", "pool", "sdn"}:
kind_filter = (str(type_filter),)
else:
kind_filter = ()
ha_by_guest: dict[str, str] = {}
if kind_filter and ("qemu" in kind_filter or "lxc" in kind_filter):
ha_rows = await _database(request).pool.fetch(
"SELECT external_id, state FROM resources WHERE kind='ha'"
)
for ha_row in ha_rows:
sid = str(ha_row["external_id"])
ha_state = _as_dict(ha_row["state"])
ha_by_guest[sid] = str(ha_state.get("state") or "started")
pool_by_vmid: dict[str, str] = {}
if kind_filter and ("qemu" in kind_filter or "lxc" in kind_filter or "pool" in kind_filter):
membership = await _database(request).pool.fetch(
"""SELECT p.pool_id AS pool, r.external_id AS vmid
FROM pool_members pm
JOIN pools p ON p.id=pm.pool_id
JOIN resources r ON r.id=pm.resource_id"""
)
for row in membership:
pool_by_vmid[str(row["vmid"])] = str(row["pool"])
node_names = [
str(item["node"])
for item in await _database(request).pool.fetch(
"SELECT name AS node FROM nodes ORDER BY name"
)
]
if kind_filter:
rows = await _database(request).pool.fetch(
"""SELECT r.kind AS type, r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind = ANY($1::text[])
ORDER BY r.kind, r.external_id""",
list(kind_filter),
)
for row in rows:
kind = str(row["type"])
external_id = str(row["external_id"])
node = str(row["node"])
state = _as_dict(row["state"])
if kind in {"qemu", "lxc"}:
status = str(state.get("status") or "stopped")
running = status in {"running", "paused"}
vmid = int(external_id)
sid = f"{'vm' if kind == 'qemu' else 'ct'}:{external_id}"
item: dict[str, Any] = {
"type": kind,
"id": f"{kind}/{external_id}",
"node": node,
"vmid": vmid,
"name": str(state.get("name") or f"{kind}-{external_id}"),
"status": status,
"template": bool(state.get("template") in (True, "1", 1)),
"cpu": _cpu_util(state, running=running),
"maxcpu": _maxcpu(state),
"mem": int(_num(state.get("mem"), 0)) if running else 0,
"maxmem": _memory_bytes(state),
"disk": int(_num(state.get("disk"), 0)),
"maxdisk": int(_num(state.get("maxdisk"), 0)),
"uptime": int(_num(state.get("uptime"), 0)) if running else 0,
}
if sid in ha_by_guest:
item["hastate"] = ha_by_guest[sid]
if external_id in pool_by_vmid:
item["pool"] = pool_by_vmid[external_id]
result.append(item)
elif kind == "storage":
content = state.get("content")
if isinstance(content, list):
content_text = ",".join(str(part) for part in content)
else:
content_text = str(content or "")
shared = bool(_num(state.get("shared"), 0))
# Schema allows one storages row per storage_id; expand to every
# node like a real PVE dump (local + shared storages).
targets = node_names if (shared or len(node_names) > 1) else [node]
if not shared and external_id not in {"local", "local-lvm", "ceph"}:
targets = [node]
for target in targets:
result.append(
{
"type": "storage",
"id": f"storage/{target}/{external_id}",
"node": target,
"storage": external_id,
"status": str(state.get("status") or "available"),
"content": content_text,
"disk": int(_num(state.get("disk"), 0)),
"maxdisk": int(_num(state.get("maxdisk"), 1 * 1024**3)),
"shared": shared,
"plugintype": str(
state.get("plugintype") or state.get("type") or "dir"
),
}
)
elif kind == "pool":
result.append(
{
"type": "pool",
"id": f"pool/{external_id}",
"pool": external_id,
"comment": str(state.get("comment") or ""),
}
)
if type_filter in (None, "sdn"):
metadata = await cluster_metadata(request)
sdn = metadata.get("sdn")
zones = sdn.get("zones") if isinstance(sdn, dict) else None
if isinstance(zones, dict):
for zone_name, zone in sorted(zones.items()):
if not isinstance(zone, dict):
continue
result.append(
{
"type": "sdn",
"id": f"sdn/{zone_name}",
"sdn": zone_name,
"status": str(zone.get("status") or "ok"),
}
)
return result return result
async def node_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]: async def node_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
+73 -41
View File
@@ -13,20 +13,6 @@ from app.api.registry import HandlerRegistry
from app.handlers.common import database, require_node, state, subdirs, values from app.handlers.common import database, require_node, state, subdirs, values
from app.simulation.seed import CLUSTER_ID from app.simulation.seed import CLUSTER_ID
DEFAULT_OPTIONS = {
"enable": 1,
"policy_in": "DROP",
"policy_out": "ACCEPT",
"log_level_in": "nolog",
"log_level_out": "nolog",
}
DEFAULT_MACROS = [
{"macro": "SSH", "descr": "Secure Shell"},
{"macro": "HTTPS", "descr": "Secure web server"},
{"macro": "HTTP", "descr": "Web server"},
]
ScopeFn = Callable[[dict[str, Any]], str] ScopeFn = Callable[[dict[str, Any]], str]
@@ -50,19 +36,36 @@ async def _save_firewall(request: Request, firewall: dict[str, Any]) -> None:
) )
def _scope_data(firewall: dict[str, Any], scope: str) -> dict[str, Any]: def _empty_scope() -> dict[str, Any]:
return {
"options": {},
"rules": [],
"aliases": {},
"ipset": {},
"groups": {},
"log": [],
}
def _get_scope(firewall: dict[str, Any], scope: str) -> dict[str, Any] | None:
scopes = firewall.get("scopes")
if not isinstance(scopes, dict):
return None
section = scopes.get(scope)
return cast(dict[str, Any], section) if isinstance(section, dict) else None
def _ensure_scope(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
"""Create an empty durable scope on mutation; never injects catalog defaults."""
scopes = firewall.setdefault("scopes", {}) scopes = firewall.setdefault("scopes", {})
if scope not in scopes or not isinstance(scopes[scope], dict): if not isinstance(scopes, dict):
scopes[scope] = { scopes = {}
"options": dict(DEFAULT_OPTIONS), firewall["scopes"] = scopes
"rules": [], section = scopes.get(scope)
"aliases": {}, if not isinstance(section, dict):
"ipset": {}, section = _empty_scope()
"groups": {}, scopes[scope] = section
"log": [], section.setdefault("options", {})
}
section = scopes[scope]
section.setdefault("options", dict(DEFAULT_OPTIONS))
section.setdefault("rules", []) section.setdefault("rules", [])
section.setdefault("aliases", {}) section.setdefault("aliases", {})
section.setdefault("ipset", {}) section.setdefault("ipset", {})
@@ -71,6 +74,11 @@ def _scope_data(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
return cast(dict[str, Any], section) return cast(dict[str, Any], section)
def _scope_data(firewall: dict[str, Any], scope: str) -> dict[str, Any]:
"""Mutation helper — ensure scope exists without template defaults."""
return _ensure_scope(firewall, scope)
def register_firewall_handlers(registry: HandlerRegistry) -> None: def register_firewall_handlers(registry: HandlerRegistry) -> None:
def register_scope( def register_scope(
base: str, base: str,
@@ -98,25 +106,31 @@ def register_firewall_handlers(registry: HandlerRegistry) -> None:
async def options_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def options_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = await _ready(request, inputs) payload = await _ready(request, inputs)
firewall = await _load_firewall(request) firewall = await _load_firewall(request)
return dict(_scope_data(firewall, scope_fn(payload)).get("options", DEFAULT_OPTIONS)) section = _get_scope(firewall, scope_fn(payload))
if section is None:
return {}
options = section.get("options", {})
return dict(options) if isinstance(options, dict) else {}
async def options_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def options_put(request: Request, inputs: dict[str, Any]) -> None:
payload = await _ready(request, inputs) payload = await _ready(request, inputs)
firewall = await _load_firewall(request) firewall = await _load_firewall(request)
section = _scope_data(firewall, scope_fn(payload)) section = _ensure_scope(firewall, scope_fn(payload))
current = dict(section.get("options", DEFAULT_OPTIONS)) current = dict(section.get("options") or {})
for key, value in payload.items(): for key, value in payload.items():
if key in {"node", "vmid", "delete", "digest"}: if key in {"node", "vmid", "delete", "digest"}:
continue continue
current[key] = value current[key] = value
section["options"] = current section["options"] = current
await _save_firewall(request, firewall) await _save_firewall(request, firewall)
return current
async def rules_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def rules_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs) payload = await _ready(request, inputs)
firewall = await _load_firewall(request) firewall = await _load_firewall(request)
rules = _scope_data(firewall, scope_fn(payload)).get("rules", []) section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
rules = section.get("rules", [])
return list(rules) if isinstance(rules, list) else [] return list(rules) if isinstance(rules, list) else []
async def rules_create(request: Request, inputs: dict[str, Any]) -> None: async def rules_create(request: Request, inputs: dict[str, Any]) -> None:
@@ -169,7 +183,8 @@ def register_firewall_handlers(registry: HandlerRegistry) -> None:
async def aliases_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def aliases_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs) payload = await _ready(request, inputs)
firewall = await _load_firewall(request) firewall = await _load_firewall(request)
aliases = _scope_data(firewall, scope_fn(payload)).get("aliases", {}) section = _get_scope(firewall, scope_fn(payload))
aliases = section.get("aliases", {}) if section else {}
if not isinstance(aliases, dict): if not isinstance(aliases, dict):
return [] return []
return [ return [
@@ -236,7 +251,8 @@ def register_firewall_handlers(registry: HandlerRegistry) -> None:
async def ipset_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def ipset_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs) payload = await _ready(request, inputs)
firewall = await _load_firewall(request) firewall = await _load_firewall(request)
ipsets = _scope_data(firewall, scope_fn(payload)).get("ipset", {}) section = _get_scope(firewall, scope_fn(payload))
ipsets = section.get("ipset", {}) if section else {}
if not isinstance(ipsets, dict): if not isinstance(ipsets, dict):
return [] return []
return [ return [
@@ -352,27 +368,43 @@ def register_firewall_handlers(registry: HandlerRegistry) -> None:
async def refs_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def refs_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs) payload = await _ready(request, inputs)
firewall = await _load_firewall(request) firewall = await _load_firewall(request)
section = _scope_data(firewall, scope_fn(payload)) section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
refs: list[dict[str, Any]] = [] refs: list[dict[str, Any]] = []
for name in section.get("aliases", {}): for name in section.get("aliases", {}) or {}:
refs.append({"type": "alias", "name": name}) refs.append({"type": "alias", "name": name})
for name in section.get("ipset", {}): for name in section.get("ipset", {}) or {}:
refs.append({"type": "ipset", "name": name}) refs.append({"type": "ipset", "name": name})
return refs return refs
async def log_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def log_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs) payload = await _ready(request, inputs)
firewall = await _load_firewall(request) firewall = await _load_firewall(request)
log = _scope_data(firewall, scope_fn(payload)).get("log", []) section = _get_scope(firewall, scope_fn(payload))
if section is None:
return []
log = section.get("log", [])
return list(log) if isinstance(log, list) else [] return list(log) if isinstance(log, list) else []
async def macros_list(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def macros_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(DEFAULT_MACROS) from app.handlers.common import cluster_metadata
metadata = await cluster_metadata(request)
macros = metadata.get("firewall_macros")
if isinstance(macros, list):
return [dict(item) for item in macros if isinstance(item, dict)]
firewall = await _load_firewall(request)
nested = firewall.get("macros")
if isinstance(nested, list):
return [dict(item) for item in nested if isinstance(item, dict)]
return []
async def groups_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def groups_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = await _ready(request, inputs) payload = await _ready(request, inputs)
firewall = await _load_firewall(request) firewall = await _load_firewall(request)
groups = _scope_data(firewall, scope_fn(payload)).get("groups", {}) section = _get_scope(firewall, scope_fn(payload))
groups = section.get("groups", {}) if section else {}
if not isinstance(groups, dict): if not isinstance(groups, dict):
return [] return []
return [ return [
+102 -62
View File
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
import hashlib
import json import json
from typing import Any from typing import Any
@@ -27,6 +28,13 @@ def _ha_groups(metadata: dict[str, Any]) -> dict[str, dict[str, Any]]:
return {str(key): dict(value) for key, value in groups.items() if isinstance(value, dict)} return {str(key): dict(value) for key, value in groups.items() if isinstance(value, dict)}
def _ha_digest(payload: dict[str, Any]) -> str:
material = {key: value for key, value in payload.items() if key != "digest"}
return hashlib.sha1( # noqa: S324 - Proxmox HA digests use SHA-1
json.dumps(material, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
def register_ha_handlers(registry: HandlerRegistry) -> None: def register_ha_handlers(registry: HandlerRegistry) -> None:
async def ha_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]: async def ha_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("groups", "resources", "rules", "status") return subdirs("groups", "resources", "rules", "status")
@@ -41,17 +49,17 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
for row in rows: for row in rows:
payload = state(row["state"]) payload = state(row["state"])
sid = str(row["external_id"]) sid = str(row["external_id"])
result.append( entry = {
{ "sid": sid,
"sid": sid, "type": "vm" if sid.startswith("vm:") else "ct",
"type": "vm" if sid.startswith("vm:") else "ct", "state": payload.get("state", "started"),
"state": payload.get("state", "started"), "group": payload.get("group"),
"group": payload.get("group"), "node": str(row["node"]),
"node": str(row["node"]), "max_relocate": payload.get("max_relocate", 1),
"max_relocate": payload.get("max_relocate", 1), "max_restart": payload.get("max_restart", 1),
"max_restart": payload.get("max_restart", 1), }
} entry["digest"] = _ha_digest(entry)
) result.append(entry)
return result return result
async def ha_resource_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def ha_resource_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -142,7 +150,7 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
async def ha_groups(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def ha_groups(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
configured = _ha_groups(metadata) configured = _ha_groups(metadata)
result = [ return [
{ {
"group": group_id, "group": group_id,
"nodes": str(payload.get("nodes", "")), "nodes": str(payload.get("nodes", "")),
@@ -153,26 +161,6 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
} }
for group_id, payload in sorted(configured.items()) for group_id, payload in sorted(configured.items())
] ]
if result:
return result
rows = await database(request).pool.fetch(
"""SELECT DISTINCT state->>'group' AS group_id
FROM resources WHERE kind='ha' AND state ? 'group'
ORDER BY 1"""
)
node_names = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
nodes = ",".join(str(row["name"]) for row in node_names) or "pve01"
return [
{
"group": str(row["group_id"]),
"nodes": nodes,
"nofailback": 0,
"restricted": 0,
"type": "group",
}
for row in rows
if row["group_id"]
]
async def ha_group_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def ha_group_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
group = str(values(inputs)["group"]) group = str(values(inputs)["group"])
@@ -228,50 +216,97 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
async def ha_status(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]: async def ha_status(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("current", "manager_status") return subdirs("current", "manager_status")
async def ha_status_current(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]: async def ha_status_current(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
row = await database(request).pool.fetchrow(
"""SELECT count(*) FILTER (WHERE state->>'state' = 'started') AS started,
count(*) AS total
FROM resources WHERE kind='ha'"""
)
master = await database(request).pool.fetchval( master = await database(request).pool.fetchval(
"SELECT name FROM nodes WHERE status='online' ORDER BY name LIMIT 1" "SELECT name FROM nodes WHERE status='online' ORDER BY name LIMIT 1"
) )
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
ha = metadata.get("ha", {}) if isinstance(metadata.get("ha"), dict) else {} ha_raw = metadata.get("ha")
return { ha: dict[str, Any] = dict(ha_raw) if isinstance(ha_raw, dict) else {}
"quorate": 1, armed = bool(ha.get("armed")) if "armed" in ha else False
"mode": "active" if ha.get("armed", True) else "disabled", status_raw = ha.get("status_current")
"master_node": str(master or "pve01"), status: dict[str, Any] = dict(status_raw) if isinstance(status_raw, dict) else {}
"ha_started": int(row["started"] or 0), quorate = bool(int(status.get("quorate", metadata.get("quorate", 1)) or 1))
"ha_total": int(row["total"] or 0), master_node = str(status.get("master_node") or master or "")
"armed": 1 if ha.get("armed", True) else 0, rows = await database(request).pool.fetch(
} """SELECT r.external_id, r.state, n.name AS node
FROM resources r JOIN nodes n ON n.id=r.node_id
WHERE r.kind='ha' ORDER BY r.external_id"""
)
nodes = await database(request).pool.fetch("SELECT name FROM nodes ORDER BY name")
result: list[dict[str, Any]] = [
{
"id": "quorum",
"type": "quorum",
"node": master_node,
"status": "OK" if quorate else "offline",
"quorate": quorate,
},
{
"id": "master",
"type": "master",
"node": master_node,
"status": "active" if armed else "disabled",
},
{
"id": "fencing",
"type": "fencing",
"node": master_node,
"status": "armed" if armed else "disarmed",
"armed-state": "armed" if armed else "disarmed",
},
]
for node_row in nodes:
node_name = str(node_row["name"])
result.append(
{
"id": f"lrm:{node_name}",
"type": "lrm",
"node": node_name,
"status": "active" if quorate else "offline",
}
)
for row in rows:
payload = state(row["state"])
sid = str(row["external_id"])
crm = str(payload.get("state") or "started")
result.append(
{
"id": f"service:{sid}",
"type": "service",
"sid": sid,
"node": str(row["node"]),
"crm_state": crm,
"state": crm,
"status": crm,
"request_state": str(payload.get("request_state") or crm),
"max_relocate": int(payload.get("max_relocate") or 1),
"max_restart": int(payload.get("max_restart") or 1),
}
)
return result
async def ha_manager_status(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]: async def ha_manager_status(request: Request, _inputs: dict[str, Any]) -> dict[str, Any]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
ha = metadata.get("ha", {}) if isinstance(metadata.get("ha"), dict) else {} ha_raw = metadata.get("ha")
armed = bool(ha.get("armed", True)) ha: dict[str, Any] = dict(ha_raw) if isinstance(ha_raw, dict) else {}
manager_raw = ha.get("manager_status")
manager: dict[str, Any] = dict(manager_raw) if isinstance(manager_raw, dict) else {}
armed = bool(ha.get("armed")) if "armed" in ha else False
return { return {
"manager_status": "active" if armed else "disabled", "manager_status": manager.get("manager_status", "active" if armed else "disabled"),
"quorum": "OK", "quorum": manager.get("quorum", ""),
"armed": 1 if armed else 0, "armed": 1 if armed else 0,
} }
async def ha_rules(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def ha_rules(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
rules = metadata.get("ha_rules") rules = metadata.get("ha_rules")
if isinstance(rules, list) and rules: if isinstance(rules, list):
return [dict(item) for item in rules if isinstance(item, dict)] return [dict(item) for item in rules if isinstance(item, dict)]
defaults = [ return []
{"rule": "node-fencing", "type": "node", "action": "restart"},
{"rule": "service-ha", "type": "resource", "action": "failover"},
]
metadata["ha_rules"] = defaults
await save_cluster_metadata(request, metadata)
return list(defaults)
async def ha_relocate(request: Request, inputs: dict[str, Any]) -> None: async def ha_relocate(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
sid = str(payload["sid"]) sid = str(payload["sid"])
target = str(payload.get("node") or payload.get("target") or "") target = str(payload.get("node") or payload.get("target") or "")
@@ -311,9 +346,14 @@ def register_ha_handlers(registry: HandlerRegistry) -> None:
ha_row["id"], ha_row["id"],
json.dumps(current, sort_keys=True), json.dumps(current, sort_keys=True),
) )
return {
"sid": sid,
"requested-node": target,
"comigrated-resources": [],
}
async def ha_migrate(request: Request, inputs: dict[str, Any]) -> None: async def ha_migrate(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await ha_relocate(request, inputs) return await ha_relocate(request, inputs)
async def ha_arm(request: Request, _inputs: dict[str, Any]) -> None: async def ha_arm(request: Request, _inputs: dict[str, Any]) -> None:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
+1 -1
View File
@@ -19,7 +19,7 @@ def register_legacy_aliases(registry: HandlerRegistry) -> None:
alias("/access/tfa", "POST", "/access/tfa/{userid}", "POST") alias("/access/tfa", "POST", "/access/tfa/{userid}", "POST")
alias("/access/tfa", "PUT", "/access/tfa/{userid}/{id}", "PUT") alias("/access/tfa", "PUT", "/access/tfa/{userid}/{id}", "PUT")
alias("/cluster/backupinfo", "GET", "/cluster/backup-info", "GET") # /cluster/backupinfo is registered as a PVE 6 stub string in backup.py
alias( alias(
"/cluster/backupinfo/not_backed_up", "/cluster/backupinfo/not_backed_up",
"GET", "GET",
+118 -10
View File
@@ -38,27 +38,120 @@ def _state(value: object) -> dict[str, Any]:
return dict(cast(Mapping[str, Any], value)) return dict(cast(Mapping[str, Any], value))
# Runtime-only keys stored in ``resources.state``; never echo on GET /config.
_LXC_CONFIG_INTERNAL = frozenset(
{
"agent",
"cloudinit_dump",
"config",
"consoles",
"interfaces",
"migrate_preconditions",
"name",
"rrd",
"rrddata",
"status",
"volume_moves",
}
)
def _public_lxc_config(
config: dict[str, Any], state: dict[str, Any], *, vmid: int
) -> dict[str, Any]:
import hashlib
merged = {**state, **config}
payload: dict[str, Any] = {"vmid": vmid}
for key, value in merged.items():
if key in _LXC_CONFIG_INTERNAL:
continue
if isinstance(value, dict | list):
continue
payload[key] = value
digest_src = {key: value for key, value in payload.items() if key != "digest"}
payload["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(digest_src, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
return payload
def _lxc_index_item(vmid: int, vm_state: dict[str, Any], config: dict[str, Any]) -> dict[str, Any]:
status = str(vm_state.get("status", "stopped"))
running = status in {"running", "paused"}
memory_mb = int(config.get("memory", config.get("mem", 512)))
maxmem = memory_mb * 2**20
return {
"vmid": vmid,
"name": str(config.get("hostname", config.get("name", f"ct-{vmid}"))),
"status": status,
"cpus": int(config.get("cores", config.get("cpus", 1))),
"cpu": float(vm_state.get("cpu", 0.05 if running else 0.0)),
"maxmem": maxmem,
"mem": int(vm_state.get("mem", maxmem // 2 if running else 0)),
"maxdisk": int(vm_state.get("maxdisk", config.get("maxdisk", 8 * 1024**3))),
"disk": int(vm_state.get("disk", 0)),
"diskread": int(vm_state.get("diskread", 0)),
"diskwrite": int(vm_state.get("diskwrite", 0)),
"netin": int(vm_state.get("netin", 0)),
"netout": int(vm_state.get("netout", 0)),
"maxswap": int(config.get("swap", 512)) * 2**20,
"uptime": int(vm_state.get("uptime", 0)) if running else 0,
"lock": str(vm_state.get("lock", "")),
"tags": str(config.get("tags", "")),
"template": int(bool(vm_state.get("template", False))),
"pressurecpusome": float(vm_state.get("pressurecpusome", 0.0)),
"pressurecpufull": float(vm_state.get("pressurecpufull", 0.0)),
"pressureiosome": float(vm_state.get("pressureiosome", 0.0)),
"pressureiofull": float(vm_state.get("pressureiofull", 0.0)),
"pressurememorysome": float(vm_state.get("pressurememorysome", 0.0)),
"pressurememoryfull": float(vm_state.get("pressurememoryfull", 0.0)),
}
def register_lxc_handlers(registry: HandlerRegistry) -> None: def register_lxc_handlers(registry: HandlerRegistry) -> None:
async def lxc_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def lxc_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(_values(inputs)["node"]) node = str(_values(inputs)["node"])
rows = await _database(request).pool.fetch( rows = await _database(request).pool.fetch(
"""SELECT r.external_id::integer AS vmid, r.state """SELECT r.external_id::integer AS vmid, r.state, c.config
FROM resources r JOIN nodes n ON n.id=r.node_id FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN containers c ON c.resource_id=r.id
WHERE n.name=$1 AND r.kind='lxc' ORDER BY r.external_id::integer""", WHERE n.name=$1 AND r.kind='lxc' ORDER BY r.external_id::integer""",
node, node,
) )
return [{"vmid": int(row["vmid"]), **_state(row["state"])} for row in rows] return [
_lxc_index_item(int(row["vmid"]), _state(row["state"]), _state(row["config"]))
for row in rows
]
async def lxc_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def lxc_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"]) node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"])
row = await _lxc_resource(request, node, vmid) row = await _lxc_resource(request, node, vmid)
return {"vmid": int(vmid), **_state(row["config"]), **_state(row["state"])} return _public_lxc_config(_state(row["config"]), _state(row["state"]), vmid=int(vmid))
async def lxc_current(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def lxc_current(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await lxc_config(request, inputs) payload = _values(inputs)
node, vmid = str(payload["node"]), int(payload["vmid"])
row = await _lxc_resource(request, node, str(vmid))
vm_state = _state(row["state"])
config = _state(row["config"])
item = _lxc_index_item(vmid, vm_state, config)
item["ha"] = {"managed": int(vm_state.get("ha_managed", 0))}
return item
async def lxc_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def lxc_status(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
return await lxc_config(request, inputs) payload = _values(inputs)
await _lxc_resource(request, str(payload["node"]), str(payload["vmid"]))
return subdirs(
"current",
"reboot",
"resume",
"shutdown",
"start",
"stop",
"suspend",
)
async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str: async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
@@ -323,7 +416,7 @@ def register_lxc_handlers(registry: HandlerRegistry) -> None:
async def migrate_preconditions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def migrate_preconditions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs) values = _values(inputs)
await _lxc_resource(request, str(values["node"]), str(values["vmid"])) resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
target = values.get("target") target = values.get("target")
if target in {None, ""}: if target in {None, ""}:
raise ApiError(400, "parameter 'target' is required") raise ApiError(400, "parameter 'target' is required")
@@ -333,7 +426,13 @@ def register_lxc_handlers(registry: HandlerRegistry) -> None:
) )
if not exists: if not exists:
raise ApiError(404, "target node does not exist") raise ApiError(404, "target node does not exist")
return {"local_disks": [], "local_resources": [], "running": False} state = _state(resource["state"])
pre = state.get("migrate_preconditions")
payload = dict(pre) if isinstance(pre, dict) else {}
payload["running"] = str(state.get("status") or "") == "running"
payload.setdefault("local_disks", [])
payload.setdefault("local_resources", [])
return payload
async def migrate(request: Request, inputs: dict[str, Any]) -> str: async def migrate(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
@@ -415,7 +514,7 @@ def register_lxc_handlers(registry: HandlerRegistry) -> None:
} }
} }
async def lxc_resize(request: Request, inputs: dict[str, Any]) -> None: async def lxc_resize(request: Request, inputs: dict[str, Any]) -> str:
payload = _values(inputs) payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"]) node, vmid = str(payload["node"]), str(payload["vmid"])
disk = str(payload.get("disk") or "rootfs") disk = str(payload.get("disk") or "rootfs")
@@ -440,6 +539,13 @@ def register_lxc_handlers(registry: HandlerRegistry) -> None:
resource["id"], resource["id"],
json.dumps({disk: config[disk]}, sort_keys=True), json.dumps({disk: config[disk]}, sort_keys=True),
) )
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-resize",
payload={"resource_id": str(resource["id"]), "disk": disk, "size": size},
)
async def lxc_template(request: Request, inputs: dict[str, Any]) -> None: async def lxc_template(request: Request, inputs: dict[str, Any]) -> None:
payload = _values(inputs) payload = _values(inputs)
@@ -531,6 +637,8 @@ async def _create_task(
"lxc-clone": "pctclone", "lxc-clone": "pctclone",
"lxc-migrate": "pctmigrate", "lxc-migrate": "pctmigrate",
"lxc-remote-migrate": "pctremote", "lxc-remote-migrate": "pctremote",
"lxc-resize": "pctresize",
"lxc-move-volume": "pctmove",
}[task_type] }[task_type]
upid = str(Upid.allocate(node, worker_type, vmid, str(request.state.principal))) upid = str(Upid.allocate(node, worker_type, vmid, str(request.state.principal)))
try: try:
+21 -20
View File
@@ -39,16 +39,15 @@ def register_lxc_extra_handlers(registry: HandlerRegistry) -> None:
values = _values(inputs) values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"])) resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"]) state = _state(resource["state"])
ifaces = state.setdefault( ifaces = state.get("interfaces")
"interfaces", return [dict(item) for item in ifaces] if isinstance(ifaces, list) else []
[{"name": "eth0", "hwaddr": "02:00:00:00:00:11", "inet": "192.0.2.20/24"}],
)
await _save_state(request, resource["id"], state)
return list(ifaces) if isinstance(ifaces, list) else []
async def move_volume(request: Request, inputs: dict[str, Any]) -> str: async def move_volume(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.lxc import _create_task
values = _values(inputs) values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"])) node, vmid = str(values["node"]), str(values["vmid"])
resource = await _lxc_resource(request, node, vmid)
volume = str(values.get("volume") or values.get("disk") or "rootfs") volume = str(values.get("volume") or values.get("disk") or "rootfs")
storage = str(values.get("storage") or "local-lvm") storage = str(values.get("storage") or "local-lvm")
config = _state(resource["config"]) config = _state(resource["config"])
@@ -64,29 +63,31 @@ def register_lxc_extra_handlers(registry: HandlerRegistry) -> None:
moves = state["volume_moves"] = [] moves = state["volume_moves"] = []
moves.append({"volume": volume, "storage": storage}) moves.append({"volume": volume, "storage": storage})
await _save_state(request, resource["id"], state) await _save_state(request, resource["id"], state)
return f"UPID:{values['node']}:lxc-move-volume:{values['vmid']}" return await _create_task(
request,
node=node,
vmid=vmid,
task_type="lxc-move-volume",
payload={
"resource_id": str(resource["id"]),
"volume": volume,
"storage": storage,
},
)
async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs) values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"])) resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"]) state = _state(resource["state"])
rrd_state = state.setdefault("rrd", {"filename": f"pve-ct-{values['vmid']}.rrd"}) rrd_state = state.get("rrd")
await _save_state(request, resource["id"], state) return dict(rrd_state) if isinstance(rrd_state, dict) else {}
return dict(rrd_state)
async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs) values = _values(inputs)
resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"])) resource = await _lxc_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"]) state = _state(resource["state"])
series = state.setdefault( series = state.get("rrddata")
"rrddata", return [dict(item) for item in series] if isinstance(series, list) else []
[
{"time": 1_700_000_000, "cpu": 0.02, "mem": 64 * 1024 * 1024},
{"time": 1_700_000_060, "cpu": 0.03, "mem": 66 * 1024 * 1024},
],
)
await _save_state(request, resource["id"], state)
return list(series)
async def _console(request: Request, inputs: dict[str, Any], kind: str) -> dict[str, Any]: async def _console(request: Request, inputs: dict[str, Any], kind: str) -> dict[str, Any]:
values = _values(inputs) values = _values(inputs)
+16 -9
View File
@@ -12,13 +12,14 @@ from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs
def _mappings(metadata: dict[str, Any]) -> dict[str, Any]: def _mappings(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault("mapping", {"dir": {}, "pci": {}, "usb": {}}) current = metadata.get("mapping")
if not isinstance(current, dict): if not isinstance(current, dict):
current = {"dir": {}, "pci": {}, "usb": {}} return {"dir": {}, "pci": {}, "usb": {}}
metadata["mapping"] = current return {
for kind in ("dir", "pci", "usb"): "dir": dict(current["dir"]) if isinstance(current.get("dir"), dict) else {},
current.setdefault(kind, {}) "pci": dict(current["pci"]) if isinstance(current.get("pci"), dict) else {},
return current "usb": dict(current["usb"]) if isinstance(current.get("usb"), dict) else {},
}
def register_mapping_handlers(registry: HandlerRegistry) -> None: def register_mapping_handlers(registry: HandlerRegistry) -> None:
@@ -42,12 +43,14 @@ def register_mapping_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
item_id = str(payload["id"]) item_id = str(payload["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _mappings(metadata)[kind] mapping = _mappings(metadata)
store = mapping[kind]
if item_id in store: if item_id in store:
raise ApiError(400, f"{kind} mapping '{item_id}' already exists") raise ApiError(400, f"{kind} mapping '{item_id}' already exists")
store[item_id] = { store[item_id] = {
key: value for key, value in payload.items() if key not in {"delete", "digest"} key: value for key, value in payload.items() if key not in {"delete", "digest"}
} }
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -62,7 +65,8 @@ def register_mapping_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
item_id = str(payload["id"]) item_id = str(payload["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _mappings(metadata)[kind] mapping = _mappings(metadata)
store = mapping[kind]
if item_id not in store: if item_id not in store:
raise ApiError(404, f"{kind} mapping does not exist") raise ApiError(404, f"{kind} mapping does not exist")
current = dict(store[item_id]) current = dict(store[item_id])
@@ -76,15 +80,18 @@ def register_mapping_handlers(registry: HandlerRegistry) -> None:
current[key] = value current[key] = value
current["id"] = item_id current["id"] = item_id
store[item_id] = current store[item_id] = current
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def delete(request: Request, inputs: dict[str, Any]) -> None: async def delete(request: Request, inputs: dict[str, Any]) -> None:
item_id = str(values(inputs)["id"]) item_id = str(values(inputs)["id"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _mappings(metadata)[kind] mapping = _mappings(metadata)
store = mapping[kind]
if item_id not in store: if item_id not in store:
raise ApiError(404, f"{kind} mapping does not exist") raise ApiError(404, f"{kind} mapping does not exist")
del store[item_id] del store[item_id]
metadata["mapping"] = mapping
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
registry.register(base, "GET", list_items) registry.register(base, "GET", list_items)
+130 -126
View File
@@ -2,7 +2,6 @@
from __future__ import annotations from __future__ import annotations
import copy
from typing import Any, cast from typing import Any, cast
from fastapi import Request from fastapi import Request
@@ -20,92 +19,13 @@ from app.handlers.common import (
from app.tasks.repository import TaskRepository from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid from app.tasks.upid import Upid
DEFAULT_NODE_OPS: dict[str, Any] = {
"network": [
{
"iface": "vmbr0",
"type": "bridge",
"active": 1,
"method": "static",
"address": "10.0.0.10/24",
},
{
"iface": "vmbr1",
"type": "bridge",
"active": 1,
"method": "static",
"address": "10.10.0.10/24",
},
{"iface": "eno1", "type": "eth", "active": 1, "method": "manual"},
],
"disks": {
"list": [
{
"devpath": "/dev/sda",
"size": 1_000_000_000_000,
"model": "SIM-DISK-01",
"serial": "SIM0001",
"gpt": 1,
},
{
"devpath": "/dev/sdb",
"size": 2_000_000_000_000,
"model": "SIM-SSD-01",
"serial": "SIM0002",
"gpt": 0,
},
],
"directory": [],
"lvm": [],
"lvmthin": [],
"zfs": [],
"smart": {},
},
"services": {
"pveproxy": {"state": "running", "enabled": 1},
"pvedaemon": {"state": "running", "enabled": 1},
"pvestatd": {"state": "running", "enabled": 1},
"corosync": {"state": "running", "enabled": 1},
},
"apt": {
"packages": [
{
"Package": "pve-manager",
"Version": "9.2.3",
"OldVersion": "9.2.2",
"Status": "upgradable",
},
{"Package": "libpve-common-perl", "Version": "9.0.3", "Status": "installed"},
],
"repositories": [
{
"path": "/etc/apt/sources.list.d/pve-enterprise.list",
"enabled": 1,
"types": "deb",
"uri": "http://download.proxmox.com/debian/pve",
"suites": "bookworm",
"components": "pve-no-subscription",
}
],
"update": {"status": "stopped", "exitstatus": "OK"},
"changelogs": {},
},
}
def default_node_ops() -> dict[str, Any]:
return copy.deepcopy(DEFAULT_NODE_OPS)
async def load_node_ops(request: Request, node: str) -> dict[str, Any]: async def load_node_ops(request: Request, node: str) -> dict[str, Any]:
metadata = await node_metadata(request, node) metadata = await node_metadata(request, node)
ops = metadata.get("ops") ops = metadata.get("ops")
if isinstance(ops, dict) and ops: if isinstance(ops, dict):
return ops return ops
ops = default_node_ops() return {}
metadata["ops"] = ops
await save_node_metadata(request, node, metadata)
return ops
async def save_node_ops(request: Request, node: str, ops: dict[str, Any]) -> None: async def save_node_ops(request: Request, node: str, ops: dict[str, Any]) -> None:
@@ -123,31 +43,76 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
packages = ops.get("apt", {}).get("packages", []) packages = ops.get("apt", {}).get("packages", [])
return list(packages) if isinstance(packages, list) else [] if not isinstance(packages, list):
return []
result: list[dict[str, Any]] = []
for item in packages:
if not isinstance(item, dict):
continue
entry = dict(item)
status = str(entry.pop("Status", entry.get("CurrentState", "Installed")))
if status.lower() == "upgradable":
entry["CurrentState"] = "Installed"
entry.setdefault("OldVersion", entry.get("Version", ""))
else:
entry["CurrentState"] = "Installed" if status.lower() == "installed" else status
entry.setdefault("Package", entry.get("Package") or "unknown")
entry.setdefault("Version", entry.get("Version") or "")
entry.setdefault("Arch", "amd64")
entry.setdefault("Origin", "Proxmox")
result.append(entry)
return result
async def apt_repositories(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def apt_repositories(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
repositories = ops.get("apt", {}).get("repositories", []) repositories = ops.get("apt", {}).get("repositories")
return list(repositories) if isinstance(repositories, list) else [] if isinstance(repositories, dict) and "files" in repositories:
return {
"digest": str(repositories.get("digest") or "seedapt"),
"files": list(repositories.get("files") or []),
"errors": list(repositories.get("errors") or []),
"infos": list(repositories.get("infos") or []),
"standard-repos": list(repositories.get("standard-repos") or []),
}
legacy = list(repositories) if isinstance(repositories, list) else []
return {
"digest": "seedapt",
"files": [
{
"path": "/etc/apt/sources.list.d/pve-enterprise.list",
"file-type": "list",
"digest": "seedapt-file0",
"repositories": legacy,
}
],
"errors": [],
"infos": [],
"standard-repos": [
{
"handle": "no-subscription",
"name": "Proxmox VE No-Subscription Repository",
"status": 1,
}
],
}
async def apt_changelog(request: Request, inputs: dict[str, Any]) -> str: async def apt_changelog(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
name = str(values(inputs).get("name") or "pve-manager") name = str(values(inputs).get("name") or "pve-manager")
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
changelogs = ops.setdefault("apt", {}).setdefault("changelogs", {}) apt = ops.get("apt")
if name not in changelogs: changelogs = apt.get("changelogs") if isinstance(apt, dict) else None
changelogs[name] = f"simulated changelog for {name}\n\n * emulator build\n" if not isinstance(changelogs, dict):
await save_node_ops(request, node, ops) return ""
return str(changelogs[name]) return str(changelogs.get(name) or "")
async def apt_update_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def apt_update_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
update = ops.get("apt", {}).get("update", {"status": "stopped", "exitstatus": "OK"}) apt = ops.get("apt")
if isinstance(update, dict): update = apt.get("update") if isinstance(apt, dict) else None
return dict(update) return dict(update) if isinstance(update, dict) else {}
return {"status": "stopped", "exitstatus": "OK"}
async def apt_update_start(request: Request, inputs: dict[str, Any]) -> str: async def apt_update_start(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
@@ -170,7 +135,7 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
return item return item
raise ApiError(404, "interface does not exist") raise ApiError(404, "interface does not exist")
async def network_mutate(request: Request, inputs: dict[str, Any]) -> None: async def network_mutate(request: Request, inputs: dict[str, Any]) -> str | None:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
payload = values(inputs) payload = values(inputs)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
@@ -221,8 +186,14 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
raise ApiError(404, "interface does not exist") raise ApiError(404, "interface does not exist")
ops["network"] = updated ops["network"] = updated
else: else:
# PUT /nodes/{node}/network without iface → reload (returns UPID string).
ops["network_applied"] = True ops["network_applied"] = True
await save_node_ops(request, node, ops)
return await _node_task(
request, node=node, task_type="network-reload", worker="srvreload"
)
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return None
async def disks_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]: async def disks_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
@@ -230,11 +201,9 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
async def _disks(request: Request, node: str) -> dict[str, Any]: async def _disks(request: Request, node: str) -> dict[str, Any]:
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
disks = ops.setdefault("disks", default_node_ops()["disks"]) disks = ops.get("disks")
if not isinstance(disks, dict): if not isinstance(disks, dict):
disks = default_node_ops()["disks"] return {}
ops["disks"] = disks
await save_node_ops(request, node, ops)
return cast(dict[str, Any], disks) return cast(dict[str, Any], disks)
async def disks_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def disks_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
@@ -247,17 +216,9 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or "/dev/sda") disk = str(values(inputs).get("disk") or "/dev/sda")
disks = await _disks(request, node) disks = await _disks(request, node)
smart = disks.setdefault("smart", {}) smart = disks.get("smart")
if disk not in smart: if not isinstance(smart, dict) or disk not in smart:
smart[disk] = { return {}
"health": "PASSED",
"type": "scsi",
"model": "SIM-DISK",
"serial": disk.rsplit("/", 1)[-1],
}
ops = await load_node_ops(request, node)
ops["disks"] = disks
await save_node_ops(request, node, ops)
return dict(smart[disk]) return dict(smart[disk])
async def disks_collection( async def disks_collection(
@@ -280,13 +241,16 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
async def disks_zfs(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def disks_zfs(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
return await disks_collection(request, inputs, "zfs") return await disks_collection(request, inputs, "zfs")
async def disks_initgpt(request: Request, inputs: dict[str, Any]) -> None: async def disks_initgpt(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or values(inputs).get("device") or "") disk = str(values(inputs).get("disk") or values(inputs).get("device") or "")
if not disk: if not disk:
raise ApiError(400, "parameter verification failed - 'disk' missing") raise ApiError(400, "parameter verification failed - 'disk' missing")
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
disks = ops.setdefault("disks", default_node_ops()["disks"]) disks = ops.get("disks")
if not isinstance(disks, dict):
disks = {}
ops["disks"] = disks
items = list(disks.get("list") or []) items = list(disks.get("list") or [])
found = False found = False
for item in items: for item in items:
@@ -307,14 +271,17 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
disks["list"] = items disks["list"] = items
ops["disks"] = disks ops["disks"] = disks
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return await _node_task(request, node=node, task_type="disk-initgpt", worker="diskinit")
async def disks_wipedisk(request: Request, inputs: dict[str, Any]) -> None: async def disks_wipedisk(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
disk = str(values(inputs).get("disk") or values(inputs).get("device") or "") disk = str(values(inputs).get("disk") or values(inputs).get("device") or "")
if not disk: if not disk:
raise ApiError(400, "parameter verification failed - 'disk' missing") raise ApiError(400, "parameter verification failed - 'disk' missing")
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
disks = ops.setdefault("disks", default_node_ops()["disks"]) disks = ops.get("disks")
if not isinstance(disks, dict):
raise ApiError(404, "disk does not exist")
items = list(disks.get("list") or []) items = list(disks.get("list") or [])
for item in items: for item in items:
if item.get("devpath") == disk: if item.get("devpath") == disk:
@@ -324,18 +291,35 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
else: else:
raise ApiError(404, "disk does not exist") raise ApiError(404, "disk does not exist")
disks["list"] = items disks["list"] = items
smart = disks.setdefault("smart", {}) smart = disks.get("smart")
smart.pop(disk, None) if isinstance(smart, dict):
smart.pop(disk, None)
ops["disks"] = disks ops["disks"] = disks
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return await _node_task(request, node=node, task_type="disk-wipe", worker="diskwipe")
async def services_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]: async def services_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
services = ops.get("services") or {} services = ops.get("services") or {}
return [{"subdir": name} for name in sorted(services)] result: list[dict[str, Any]] = []
for name in sorted(services):
payload = dict(services[name]) if isinstance(services[name], dict) else {}
state_name = str(payload.get("state", "stopped"))
enabled = bool(int(payload.get("enabled", 0) or 0))
result.append(
{
"service": name,
"name": f"{name}.service",
"desc": str(payload.get("desc") or name),
"state": state_name,
"active-state": "active" if state_name == "running" else "inactive",
"unit-state": "enabled" if enabled else "disabled",
}
)
return result
async def service_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def service_get(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
service = str(values(inputs)["service"]) service = str(values(inputs)["service"])
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
@@ -343,12 +327,25 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
if service not in services: if service not in services:
services[service] = {"state": "stopped", "enabled": 0} services[service] = {"state": "stopped", "enabled": 0}
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
payload = dict(services[service]) return subdirs("state", "start", "stop", "restart", "reload")
payload["service"] = service
return payload
async def service_state(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def service_state(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
return await service_get(request, inputs) node = str(values(inputs)["node"])
service = str(values(inputs)["service"])
ops = await load_node_ops(request, node)
services = ops.setdefault("services", {})
if service not in services:
services[service] = {"state": "stopped", "enabled": 0}
await save_node_ops(request, node, ops)
payload = dict(services[service]) if isinstance(services[service], dict) else {}
state_name = str(payload.get("state", "stopped"))
enabled = bool(int(payload.get("enabled", 0) or 0))
return {
"service": service,
"state": state_name,
"active-state": "active" if state_name == "running" else "inactive",
"unit-state": "enabled" if enabled else "disabled",
}
async def service_action(request: Request, inputs: dict[str, Any]) -> str: async def service_action(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
@@ -371,7 +368,12 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
services[service] = current services[service] = current
ops["services"] = services ops["services"] = services
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return "OK" return await _node_task(
request,
node=node,
task_type=f"service-{action}",
worker=f"srv{action}",
)
registry.register("/nodes/{node}/apt", "GET", apt_index) registry.register("/nodes/{node}/apt", "GET", apt_index)
registry.register("/nodes/{node}/apt/versions", "GET", apt_versions) registry.register("/nodes/{node}/apt/versions", "GET", apt_versions)
@@ -404,6 +406,8 @@ def register_node_ops_handlers(registry: HandlerRegistry) -> None:
async def _node_task(request: Request, *, node: str, task_type: str, worker: str) -> str: async def _node_task(request: Request, *, node: str, task_type: str, worker: str) -> str:
import secrets
from app.api.errors import ApiError from app.api.errors import ApiError
from app.db.primitives import ConflictError from app.db.primitives import ConflictError
@@ -414,7 +418,7 @@ async def _node_task(request: Request, *, node: str, task_type: str, worker: str
upid=upid, upid=upid,
task_type=task_type, task_type=task_type,
payload={"node": node}, payload={"node": node},
resource_key=f"node:{node}", resource_key=f"node:{node}:{task_type}:{secrets.token_hex(4)}",
) )
except ConflictError as error: except ConflictError as error:
raise ApiError(409, str(error)) from error raise ApiError(409, str(error)) from error
+310 -266
View File
@@ -2,7 +2,6 @@
from __future__ import annotations from __future__ import annotations
import copy
import json import json
import secrets import secrets
import time import time
@@ -13,128 +12,49 @@ from fastapi import Request
from app.api.errors import ApiError from app.api.errors import ApiError
from app.api.registry import HandlerRegistry from app.api.registry import HandlerRegistry
from app.handlers.common import database, require_node, subdirs, values from app.handlers.common import database, require_node, subdirs, values
from app.handlers.nodes import default_node_ops, load_node_ops, save_node_ops from app.handlers.nodes import load_node_ops, save_node_ops
from app.tasks.repository import TaskRepository from app.tasks.repository import TaskRepository
from app.tasks.upid import Upid from app.tasks.upid import Upid
DEFAULT_HARDWARE: dict[str, Any] = {
"pci": [
{
"id": "0000:00:1f.2",
"vendor_name": "Intel Corporation",
"device_name": "SATA Controller",
"iommugroup": 0,
},
{
"id": "0000:01:00.0",
"vendor_name": "NVIDIA Corporation",
"device_name": "GP102 [GeForce GTX 1080 Ti]",
"iommugroup": 1,
"mdev": 1,
},
],
"usb": [
{"busnum": 1, "devnum": 1, "level": 0, "port": "1", "prodid": "0002", "vendid": "1d6b"},
{"busnum": 2, "devnum": 2, "level": 1, "port": "2", "prodid": "5591", "vendid": "0781"},
],
"mdev": {
"0000:01:00.0": [
{"type": "nvidia-11", "available": 4, "description": "GRID profile"},
]
},
}
DEFAULT_SCAN: dict[str, list[dict[str, Any]]] = {
"cifs": [{"server": "files.local", "share": "backups"}],
"iscsi": [{"portal": "10.0.0.50:3260", "target": "iqn.2024-01.local:storage"}],
"lvm": [{"vg": "pve", "size": 500_000_000_000, "free": 100_000_000_000}],
"lvmthin": [{"lv": "data", "vg": "pve", "lv_size": 400_000_000_000}],
"nfs": [{"server": "nfs.local", "path": "/export/pve", "options": "vers=4"}],
"pbs": [{"server": "pbs.local", "datastore": "store1"}],
"zfs": [{"pool": "rpool", "name": "rpool/data", "size": 800_000_000_000}],
}
DEFAULT_SUBSCRIPTION: dict[str, Any] = {
"status": "notfound",
"message": "There is no subscription key",
"serverid": "SIMULATOR",
"sockets": 1,
"productname": "Proxmox VE",
"url": "https://www.proxmox.com/en/proxmox-virtual-environment/pricing",
}
DEFAULT_CONFIG: dict[str, Any] = {
"description": "Simulator node",
"startall-onboot-delay": 0,
"wakeonlan": "",
}
DEFAULT_DNS: dict[str, Any] = {
"search": "local",
"dns1": "1.1.1.1",
"dns2": "8.8.8.8",
"dns3": "",
}
DEFAULT_TIME: dict[str, Any] = {
"timezone": "UTC",
"time": 0,
"localtime": 0,
}
def _certificates(ops: dict[str, Any]) -> dict[str, Any]: def _certificates(ops: dict[str, Any]) -> dict[str, Any]:
certs = ops.setdefault( certs = ops.get("certificates")
"certificates",
{
"custom": None,
"acme": {"account": "default", "domains": [], "certificate": None},
"info": [],
},
)
if not isinstance(certs, dict): if not isinstance(certs, dict):
certs = {"custom": None, "acme": {}, "info": []} return {"custom": None, "acme": {}, "info": []}
ops["certificates"] = certs
certs.setdefault("acme", {"account": "default", "domains": [], "certificate": None})
certs.setdefault("info", [])
return certs return certs
def _hardware(ops: dict[str, Any]) -> dict[str, Any]: def _hardware(ops: dict[str, Any]) -> dict[str, Any]:
hardware = ops.get("hardware") hardware = ops.get("hardware")
if not isinstance(hardware, dict) or not hardware: if not isinstance(hardware, dict):
hardware = copy.deepcopy(DEFAULT_HARDWARE) return {"pci": [], "usb": [], "mdev": {}}
ops["hardware"] = hardware return {
hardware.setdefault("pci", copy.deepcopy(DEFAULT_HARDWARE["pci"])) "pci": list(hardware.get("pci") or []) if isinstance(hardware.get("pci"), list) else [],
hardware.setdefault("usb", copy.deepcopy(DEFAULT_HARDWARE["usb"])) "usb": list(hardware.get("usb") or []) if isinstance(hardware.get("usb"), list) else [],
hardware.setdefault("mdev", copy.deepcopy(DEFAULT_HARDWARE["mdev"])) "mdev": dict(hardware.get("mdev") or {}) if isinstance(hardware.get("mdev"), dict) else {},
return hardware }
def _scan_cache(ops: dict[str, Any]) -> dict[str, Any]: def _scan_cache(ops: dict[str, Any]) -> dict[str, Any]:
scan = ops.get("scan") scan = ops.get("scan")
if not isinstance(scan, dict) or not scan: if not isinstance(scan, dict):
scan = copy.deepcopy(DEFAULT_SCAN) return {}
ops["scan"] = scan
for key, value in DEFAULT_SCAN.items():
scan.setdefault(key, copy.deepcopy(value))
return scan return scan
def _subscription(ops: dict[str, Any]) -> dict[str, Any]: def _subscription(ops: dict[str, Any]) -> dict[str, Any]:
subscription = ops.get("subscription") subscription = ops.get("subscription")
if not isinstance(subscription, dict) or not subscription: if not isinstance(subscription, dict):
subscription = copy.deepcopy(DEFAULT_SUBSCRIPTION) return {}
ops["subscription"] = subscription
return subscription return subscription
def _disk_items(ops: dict[str, Any], kind: str) -> list[dict[str, Any]]: def _disk_items(ops: dict[str, Any], kind: str) -> list[dict[str, Any]]:
disks = ops.setdefault("disks", default_node_ops()["disks"]) disks = ops.get("disks")
if not isinstance(disks, dict): if not isinstance(disks, dict):
disks = default_node_ops()["disks"] disks = {}
ops["disks"] = disks ops["disks"] = disks
items = disks.setdefault(kind, []) items = disks.get(kind)
if not isinstance(items, list): if not isinstance(items, list):
items = [] items = []
disks[kind] = items disks[kind] = items
@@ -188,7 +108,9 @@ async def _migrate_guests(request: Request, node: str, target: str) -> None:
def register_nodes_extra_handlers(registry: HandlerRegistry) -> None: def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
async def disks_create(request: Request, inputs: dict[str, Any], kind: str) -> dict[str, Any]: async def disks_create(request: Request, inputs: dict[str, Any], kind: str) -> str:
from app.handlers.nodes import _node_task
payload = values(inputs) payload = values(inputs)
node = str(payload["node"]) node = str(payload["node"])
await require_node(request, node) await require_node(request, node)
@@ -208,11 +130,19 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
} }
entry["name"] = name entry["name"] = name
items.append(entry) items.append(entry)
ops.setdefault("disks", default_node_ops()["disks"])[kind] = items disks = ops.get("disks")
if not isinstance(disks, dict):
disks = {}
ops["disks"] = disks
disks[kind] = items
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return entry return await _node_task(
request, node=node, task_type=f"disk-{kind}-create", worker=f"disk{kind}"
)
async def disks_delete(request: Request, inputs: dict[str, Any], kind: str) -> str:
from app.handlers.nodes import _node_task
async def disks_delete(request: Request, inputs: dict[str, Any], kind: str) -> None:
payload = values(inputs) payload = values(inputs)
node = str(payload["node"]) node = str(payload["node"])
name = str(payload["name"]) name = str(payload["name"])
@@ -224,6 +154,9 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
raise ApiError(404, f"{kind} does not exist") raise ApiError(404, f"{kind} does not exist")
ops.setdefault("disks", {})[kind] = remaining ops.setdefault("disks", {})[kind] = remaining
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return await _node_task(
request, node=node, task_type=f"disk-{kind}-delete", worker=f"diskdel{kind}"
)
async def disks_zfs_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def disks_zfs_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
@@ -270,11 +203,11 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
certs["acme"] = acme certs["acme"] = acme
ops["certificates"] = certs ops["certificates"] = certs
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
if method == "DELETE":
return None
return await _node_task(request, node=node, task_type="acme", worker="acme") return await _node_task(request, node=node, task_type="acme", worker="acme")
async def certificates_custom(request: Request, inputs: dict[str, Any]) -> None: async def certificates_custom(
request: Request, inputs: dict[str, Any]
) -> dict[str, Any] | None:
payload = values(inputs) payload = values(inputs)
node = str(payload["node"]) node = str(payload["node"])
await require_node(request, node) await require_node(request, node)
@@ -282,35 +215,40 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
certs = _certificates(ops) certs = _certificates(ops)
if request.method.upper() == "DELETE": if request.method.upper() == "DELETE":
certs["custom"] = None certs["custom"] = None
else: ops["certificates"] = certs
certificates = str(payload.get("certificates") or payload.get("cert") or "") await save_node_ops(request, node, ops)
if not certificates: return None
raise ApiError(400, "parameter verification failed - 'certificates' missing") certificates = str(payload.get("certificates") or payload.get("cert") or "")
key = str(payload.get("key") or payload.get("private-key") or "") if not certificates:
certs["custom"] = { raise ApiError(400, "parameter verification failed - 'certificates' missing")
"certificates": certificates, key = str(payload.get("key") or payload.get("private-key") or "")
"key": key, filename = str(payload.get("filename") or "pveproxy-ssl.pem")
"restart": int(payload.get("restart") or 0), fingerprint = ":".join(secrets.token_hex(1) for _ in range(32)) # SHA-256 fingerprint style
"filename": str(payload.get("filename") or "pveproxy-ssl.pem"), summary = {
} "filename": filename,
info = list(certs.get("info") or []) "fingerprint": fingerprint,
info = [item for item in info if item.get("filename") != certs["custom"]["filename"]] "issuer": "CN=Simulator",
info.append( "subject": "CN=pve.local",
{ "notbefore": int(time.time()) - 86_400,
"filename": certs["custom"]["filename"], "notafter": int(time.time()) + 365 * 86_400,
"fingerprint": secrets.token_hex(20), "san": ["DNS:pve.local"],
"issuer": "CN=Simulator", "public-key-type": "rsa",
"subject": "CN=pve.local", "public-key-bits": 2048,
"notbefore": int(time.time()) - 86_400, "pem": certificates,
"notafter": int(time.time()) + 365 * 86_400, }
"san": ["DNS:pve.local"], certs["custom"] = {
"public-key-type": "rsa", "certificates": certificates,
"public-key-bits": 2048, "key": key,
} "restart": int(payload.get("restart") or 0),
) "filename": filename,
certs["info"] = info }
info = list(certs.get("info") or [])
info = [item for item in info if item.get("filename") != filename]
info.append({key: value for key, value in summary.items() if key != "pem"})
certs["info"] = info
ops["certificates"] = certs ops["certificates"] = certs
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return summary
async def certificates_info(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def certificates_info(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
@@ -333,7 +271,39 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
certs["info"] = info certs["info"] = info
ops["certificates"] = certs ops["certificates"] = certs
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return [dict(item) for item in info] if not info:
from app.simulation.seed import _seed_fingerprint
info = [
{
"filename": "pve-ssl.pem",
"fingerprint": _seed_fingerprint(node),
"issuer": f"CN={node}.proxmox.local",
"subject": f"CN={node}.proxmox.local",
"notbefore": 1_700_000_000,
"notafter": 2_000_000_000,
"pem": (
"-----BEGIN CERTIFICATE-----\n"
"SIMULATOR-SEED-CERT\n"
"-----END CERTIFICATE-----\n"
),
"san": [f"{node}.proxmox.local"],
"public-key-type": "rsa",
"public-key-bits": 4096,
}
]
certs["info"] = info
ops["certificates"] = certs
await save_node_ops(request, node, ops)
result: list[dict[str, Any]] = []
for item in info:
entry = dict(item)
entry.setdefault("pem", "-----BEGIN CERTIFICATE-----\nSIM\n-----END CERTIFICATE-----\n")
entry.setdefault("san", [str(entry.get("subject", "CN=pve.local")).removeprefix("CN=")])
entry.setdefault("public-key-type", "rsa")
entry.setdefault("public-key-bits", 4096)
result.append(entry)
return result
async def scan_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]: async def scan_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
@@ -360,35 +330,40 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
return subdirs("cpu", "cpu-flags", "machines", "migration") return subdirs("cpu", "cpu-flags", "machines", "migration")
async def capabilities_cpu(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def capabilities_cpu(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) node = str(values(inputs)["node"])
return [ await require_node(request, node)
{"name": "host", "vendor": "QEMU", "custom": 0}, ops = await load_node_ops(request, node)
{"name": "x86-64-v2-AES", "vendor": "QEMU", "custom": 0}, caps = ops.get("capabilities")
{"name": "kvm64", "vendor": "QEMU", "custom": 0}, items = caps.get("cpu") if isinstance(caps, dict) else None
] return [dict(item) for item in items] if isinstance(items, list) else []
async def capabilities_cpu_flags( async def capabilities_cpu_flags(
request: Request, inputs: dict[str, Any] request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) node = str(values(inputs)["node"])
return [ await require_node(request, node)
{"name": "aes", "introduces": "Westmere"}, ops = await load_node_ops(request, node)
{"name": "avx", "introduces": "SandyBridge"}, caps = ops.get("capabilities")
{"name": "avx2", "introduces": "Haswell"}, items = caps.get("cpu_flags") if isinstance(caps, dict) else None
] return [dict(item) for item in items] if isinstance(items, list) else []
async def capabilities_machines( async def capabilities_machines(
request: Request, inputs: dict[str, Any] request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) node = str(values(inputs)["node"])
return [ await require_node(request, node)
{"id": "pc-i440fx-9.0", "type": "i440fx", "version": "9.0"}, ops = await load_node_ops(request, node)
{"id": "pc-q35-9.0", "type": "q35", "version": "9.0"}, caps = ops.get("capabilities")
] items = caps.get("machines") if isinstance(caps, dict) else None
return [dict(item) for item in items] if isinstance(items, list) else []
async def capabilities_migration(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def capabilities_migration(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(request, str(values(inputs)["node"])) node = str(values(inputs)["node"])
return {"network": "", "type": "secure", "enabled": 1} await require_node(request, node)
ops = await load_node_ops(request, node)
caps = ops.get("capabilities")
migration = caps.get("migration") if isinstance(caps, dict) else None
return dict(migration) if isinstance(migration, dict) else {}
async def hardware_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]: async def hardware_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
@@ -399,8 +374,6 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
hardware = _hardware(ops) hardware = _hardware(ops)
ops["hardware"] = hardware
await save_node_ops(request, node, ops)
return [dict(item) for item in hardware.get("pci", [])] return [dict(item) for item in hardware.get("pci", [])]
async def hardware_pci_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def hardware_pci_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -430,8 +403,6 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
hardware = _hardware(ops) hardware = _hardware(ops)
ops["hardware"] = hardware
await save_node_ops(request, node, ops)
return [dict(item) for item in hardware.get("usb", [])] return [dict(item) for item in hardware.get("usb", [])]
async def subscription_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def subscription_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -452,7 +423,10 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
current = _subscription(ops) current = _subscription(ops)
method = request.method.upper() method = request.method.upper()
if method == "DELETE": if method == "DELETE":
ops["subscription"] = copy.deepcopy(DEFAULT_SUBSCRIPTION) ops["subscription"] = {
"status": "notfound",
"message": "There is no subscription key",
}
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
return None return None
if method == "POST": if method == "POST":
@@ -481,17 +455,8 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
items = ops.get("aplinfo") items = ops.get("aplinfo")
if not isinstance(items, list): if not isinstance(items, list):
items = [ return []
{ return [dict(item) for item in items if isinstance(item, dict)]
"package": "alpine-3-standard",
"section": "system",
"type": "lxc",
"version": "3.20",
}
]
ops["aplinfo"] = items
await save_node_ops(request, node, ops)
return [dict(item) for item in items]
async def aplinfo_download(request: Request, inputs: dict[str, Any]) -> str: async def aplinfo_download(request: Request, inputs: dict[str, Any]) -> str:
payload = values(inputs) payload = values(inputs)
@@ -514,45 +479,140 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
node = str(payload["node"]) node = str(payload["node"])
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
apt = ops.setdefault("apt", copy.deepcopy(default_node_ops()["apt"])) apt = ops.get("apt")
repositories = list(apt.get("repositories") or []) if not isinstance(apt, dict):
apt = {}
ops["apt"] = apt
envelope = apt.get("repositories")
if isinstance(envelope, dict) and "files" in envelope:
files = [
dict(item) for item in list(envelope.get("files") or []) if isinstance(item, dict)
]
standard = [
dict(item)
for item in list(envelope.get("standard-repos") or [])
if isinstance(item, dict)
]
else:
legacy = list(envelope) if isinstance(envelope, list) else []
files = [
{
"path": "/etc/apt/sources.list.d/pve-enterprise.list",
"file-type": "list",
"digest": "seedapt-file0",
"repositories": legacy,
}
]
standard = []
method = request.method.upper() method = request.method.upper()
if method == "POST": handle = payload.get("handle")
entry = { if handle is not None:
updated_standard: list[dict[str, Any]] = []
found = False
for item in standard:
if item.get("handle") == handle:
found = True
merged = dict(item)
if "enabled" in payload:
merged["status"] = int(bool(payload.get("enabled")))
updated_standard.append(merged)
else:
updated_standard.append(item)
if not found:
updated_standard.append(
{
"handle": str(handle),
"name": str(payload.get("name") or handle),
"status": int(bool(payload.get("enabled", 1))),
}
)
standard = updated_standard
elif method == "POST":
repo_entry = {
key: value key: value
for key, value in payload.items() for key, value in payload.items()
if key not in {"node", "delete", "digest"} if key not in {"node", "delete", "digest", "path", "handle", "index", "file-type"}
} }
entry.setdefault("path", f"/etc/apt/sources.list.d/sim-{secrets.token_hex(2)}.list") path = str(
entry.setdefault("enabled", 1) payload.get("path") or f"/etc/apt/sources.list.d/sim-{secrets.token_hex(2)}.list"
repositories.append(entry) )
else: matched = False
path = payload.get("path") for file_entry in files:
handle = payload.get("handle") if file_entry.get("path") == path:
index = payload.get("index") repos = list(file_entry.get("repositories") or [])
updated: list[dict[str, Any]] = [] repos.append(repo_entry)
for idx, item in enumerate(repositories): file_entry["repositories"] = repos
match = False matched = True
if path is not None and item.get("path") == path: break
match = True if not matched:
if handle is not None and item.get("handle") == handle: files.append(
match = True {
if index is not None and idx == int(index): "path": path,
match = True "file-type": str(payload.get("file-type") or "list"),
if match or (path is None and handle is None and index is None and idx == 0): "digest": secrets.token_hex(4),
merged = { "repositories": [repo_entry],
**item,
**{
key: value
for key, value in payload.items()
if key not in {"node", "delete", "digest", "path", "handle", "index"}
},
} }
updated.append(merged) )
else: else:
updated.append(item) target_path = payload.get("path")
repositories = updated index = payload.get("index")
apt["repositories"] = repositories for file_idx, file_entry in enumerate(files):
if target_path is not None and file_entry.get("path") != target_path:
continue
repos = [
dict(item)
for item in list(file_entry.get("repositories") or [])
if isinstance(item, dict)
]
if index is not None:
idx = int(index)
if 0 <= idx < len(repos):
repos[idx] = {
**repos[idx],
**{
key: value
for key, value in payload.items()
if key
not in {
"node",
"delete",
"digest",
"path",
"handle",
"index",
"file-type",
}
},
}
elif target_path is not None or file_idx == 0:
if repos:
repos[0] = {
**repos[0],
**{
key: value
for key, value in payload.items()
if key
not in {
"node",
"delete",
"digest",
"path",
"handle",
"index",
"file-type",
}
},
}
file_entry["repositories"] = repos
if target_path is not None:
break
apt["repositories"] = {
"digest": secrets.token_hex(4),
"errors": [],
"infos": [],
"standard-repos": standard,
"files": files,
}
ops["apt"] = apt ops["apt"] = apt
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
@@ -561,18 +621,14 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
config = ops.get("config") config = ops.get("config")
if not isinstance(config, dict): return dict(config) if isinstance(config, dict) else {}
config = copy.deepcopy(DEFAULT_CONFIG)
ops["config"] = config
await save_node_ops(request, node, ops)
return dict(config)
async def node_config_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_config_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
node = str(payload["node"]) node = str(payload["node"])
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
config = dict(ops.get("config") or DEFAULT_CONFIG) config = dict(ops.get("config") or {})
config.update( config.update(
{ {
key: value key: value
@@ -589,18 +645,14 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
dns = ops.get("dns") dns = ops.get("dns")
if not isinstance(dns, dict): return dict(dns) if isinstance(dns, dict) else {}
dns = copy.deepcopy(DEFAULT_DNS)
ops["dns"] = dns
await save_node_ops(request, node, ops)
return dict(dns)
async def dns_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def dns_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
node = str(payload["node"]) node = str(payload["node"])
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
dns = dict(ops.get("dns") or DEFAULT_DNS) dns = dict(ops.get("dns") or {})
dns.update( dns.update(
{ {
key: value key: value
@@ -616,13 +668,10 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
current = dict(ops.get("time") or DEFAULT_TIME) current = dict(ops.get("time") or {})
now = int(time.time()) now = int(time.time())
current["time"] = now current["time"] = now
current["localtime"] = now current["localtime"] = now
current.setdefault("timezone", "UTC")
ops["time"] = current
await save_node_ops(request, node, ops)
return current return current
async def time_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def time_put(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -630,7 +679,7 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
node = str(payload["node"]) node = str(payload["node"])
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
current = dict(ops.get("time") or DEFAULT_TIME) current = dict(ops.get("time") or {})
if "timezone" in payload: if "timezone" in payload:
current["timezone"] = str(payload["timezone"]) current["timezone"] = str(payload["timezone"])
now = int(time.time()) now = int(time.time())
@@ -668,12 +717,7 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
hosts = ops.get("hosts") hosts = ops.get("hosts")
if not isinstance(hosts, dict): if not isinstance(hosts, dict):
hosts = { return {"data": "", "digest": ""}
"data": f"127.0.0.1 localhost\n10.0.0.10 {node}\n",
"digest": secrets.token_hex(8),
}
ops["hosts"] = hosts
await save_node_ops(request, node, ops)
return {"data": str(hosts.get("data", "")), "digest": str(hosts.get("digest", ""))} return {"data": str(hosts.get("data", "")), "digest": str(hosts.get("digest", ""))}
async def hosts_post(request: Request, inputs: dict[str, Any]) -> None: async def hosts_post(request: Request, inputs: dict[str, Any]) -> None:
@@ -692,46 +736,50 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node) await require_node(request, node)
start = int(values(inputs).get("startcursor") or values(inputs).get("start") or 0) start = int(values(inputs).get("startcursor") or values(inputs).get("start") or 0)
limit = int(values(inputs).get("limit") or 50) limit = int(values(inputs).get("limit") or 50)
lines = [ ops = await load_node_ops(request, node)
f"{index}: {node} systemd[1]: Started simulated service {index}." lines = ops.get("journal")
for index in range(start, start + limit) if not isinstance(lines, list):
] return []
return lines sliced = lines[start : start + limit]
return [str(item) for item in sliced]
async def syslog(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def syslog(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
await require_node(request, node) await require_node(request, node)
limit = int(values(inputs).get("limit") or 50) limit = int(values(inputs).get("limit") or 50)
return [ ops = await load_node_ops(request, node)
{"n": index, "t": f"{node} kernel: simulated syslog line {index}"} lines = ops.get("syslog")
for index in range(limit) if not isinstance(lines, list):
] return []
return [dict(item) for item in lines[:limit] if isinstance(item, dict)]
async def netstat(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def netstat(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
await require_node(request, node) await require_node(request, node)
return [ ops = await load_node_ops(request, node)
{"in": 1_000_000, "out": 900_000, "vnet": "vmbr0", "hwaddr": "bc:24:11:00:00:01"}, items = ops.get("netstat")
{"in": 500_000, "out": 450_000, "vnet": "vmbr1", "hwaddr": "bc:24:11:00:00:02"}, return [dict(item) for item in items] if isinstance(items, list) else []
]
async def report(request: Request, inputs: dict[str, Any]) -> str: async def report(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
await require_node(request, node) await require_node(request, node)
return f"==== Proxmox node report for {node} ====\nuptime: simulated\n" ops = await load_node_ops(request, node)
report_text = ops.get("report")
return str(report_text) if report_text is not None else ""
async def rrd(_request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(_request, str(values(inputs)["node"])) node = str(values(inputs)["node"])
return {"filename": "/var/lib/rrdcached/db/pve-node.rrd"} await require_node(request, node)
ops = await load_node_ops(request, node)
rrd_state = ops.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def rrddata(_request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(_request, str(values(inputs)["node"])) node = str(values(inputs)["node"])
now = int(time.time()) await require_node(request, node)
return [ ops = await load_node_ops(request, node)
{"time": now - 120, "cpu": 0.05, "memused": 1_000_000_000}, series = ops.get("rrddata")
{"time": now - 60, "cpu": 0.07, "memused": 1_100_000_000}, return [dict(item) for item in series] if isinstance(series, list) else []
{"time": now, "cpu": 0.04, "memused": 1_050_000_000},
]
async def startall(request: Request, inputs: dict[str, Any]) -> str: async def startall(request: Request, inputs: dict[str, Any]) -> str:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
@@ -809,7 +857,7 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node) await require_node(request, node)
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
if not isinstance(ops.get("network"), list): if not isinstance(ops.get("network"), list):
ops["network"] = copy.deepcopy(default_node_ops()["network"]) ops["network"] = []
ops["network_applied"] = False ops["network_applied"] = False
await save_node_ops(request, node, ops) await save_node_ops(request, node, ops)
@@ -818,28 +866,24 @@ def register_nodes_extra_handlers(registry: HandlerRegistry) -> None:
await require_node(request, node) await require_node(request, node)
repo = str(values(inputs).get("repo") or "library/alpine") repo = str(values(inputs).get("repo") or "library/alpine")
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
cache = ops.setdefault("oci_tags", {}) cache = ops.get("oci_tags")
if repo not in cache: if not isinstance(cache, dict):
cache[repo] = [{"tag": "latest"}, {"tag": "3.20"}] return []
ops["oci_tags"] = cache items = cache.get(repo)
await save_node_ops(request, node, ops) if not isinstance(items, list):
return [dict(item) for item in cache[repo]] return []
return [dict(item) for item in items if isinstance(item, dict)]
async def query_url_metadata(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def query_url_metadata(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
await require_node(request, node) await require_node(request, node)
url = str(values(inputs).get("url") or "") url = str(values(inputs).get("url") or "")
ops = await load_node_ops(request, node) ops = await load_node_ops(request, node)
cache = ops.setdefault("url_metadata", {}) cache = ops.get("url_metadata")
if url not in cache: if not isinstance(cache, dict):
cache[url] = { return {}
"filename": url.rsplit("/", 1)[-1] or "download.bin", payload = cache.get(url)
"mimetype": "application/octet-stream", return dict(payload) if isinstance(payload, dict) else {}
"size": 1024,
}
ops["url_metadata"] = cache
await save_node_ops(request, node, ops)
return dict(cache[url])
async def vncwebsocket(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def vncwebsocket(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
+57 -50
View File
@@ -13,53 +13,40 @@ from app.handlers.common import cluster_metadata, save_cluster_metadata, subdirs
_SECRET_KEYS = frozenset({"token", "password", "secret"}) _SECRET_KEYS = frozenset({"token", "password", "secret"})
DEFAULT_MATCHER_FIELDS = [
{"name": "type", "type": "string"},
{"name": "hostname", "type": "string"},
{"name": "job-id", "type": "string"},
{"name": "severity", "type": "string"},
]
DEFAULT_MATCHER_FIELD_VALUES = [
{"field": "type", "value": "fencing"},
{"field": "type", "value": "package-updates"},
{"field": "type", "value": "replication"},
{"field": "type", "value": "system-mail"},
]
def _public(endpoint: dict[str, Any]) -> dict[str, Any]: def _public(endpoint: dict[str, Any]) -> dict[str, Any]:
return {key: value for key, value in endpoint.items() if key not in _SECRET_KEYS} return {key: value for key, value in endpoint.items() if key not in _SECRET_KEYS}
def _notifications(metadata: dict[str, Any]) -> dict[str, Any]: def _notifications(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault( current = metadata.get("notifications")
"notifications",
{
"endpoints": {
"gotify": {},
"sendmail": {},
"smtp": {},
"webhook": {},
},
"matchers": {},
"tests": [],
},
)
if not isinstance(current, dict): if not isinstance(current, dict):
current = { return {
"endpoints": {"gotify": {}, "sendmail": {}, "smtp": {}, "webhook": {}}, "endpoints": {"gotify": {}, "sendmail": {}, "smtp": {}, "webhook": {}},
"matchers": {}, "matchers": {},
"tests": [], "tests": [],
} }
metadata["notifications"] = current endpoints = current.get("endpoints")
current.setdefault( if not isinstance(endpoints, dict):
"endpoints", endpoints = {}
{"gotify": {}, "sendmail": {}, "smtp": {}, "webhook": {}}, result = {
key: value
for key, value in current.items()
if key not in {"endpoints", "matchers", "tests"}
}
result["endpoints"] = {
"gotify": dict(endpoints["gotify"]) if isinstance(endpoints.get("gotify"), dict) else {},
"sendmail": (
dict(endpoints["sendmail"]) if isinstance(endpoints.get("sendmail"), dict) else {}
),
"smtp": dict(endpoints["smtp"]) if isinstance(endpoints.get("smtp"), dict) else {},
"webhook": dict(endpoints["webhook"]) if isinstance(endpoints.get("webhook"), dict) else {},
}
result["matchers"] = (
dict(current["matchers"]) if isinstance(current.get("matchers"), dict) else {}
) )
current.setdefault("matchers", {}) result["tests"] = list(current["tests"]) if isinstance(current.get("tests"), list) else []
current.setdefault("tests", []) return result
return current
def register_notifications_handlers(registry: HandlerRegistry) -> None: def register_notifications_handlers(registry: HandlerRegistry) -> None:
@@ -80,26 +67,28 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
async def list_endpoints(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def list_endpoints(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {}) store = _notifications(metadata)["endpoints"].get(kind) or {}
return [_public({"name": name, **item}) for name, item in sorted(store.items())] return [_public({"name": name, **item}) for name, item in sorted(store.items())]
async def create(request: Request, inputs: dict[str, Any]) -> None: async def create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
name = str(payload["name"]) name = str(payload["name"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {}) notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name in store: if name in store:
raise ApiError(400, f"{kind} endpoint '{name}' already exists") raise ApiError(400, f"{kind} endpoint '{name}' already exists")
entry = {key: payload[key] for key in create_keys if key in payload} entry = {key: payload[key] for key in create_keys if key in payload}
entry["name"] = name entry["name"] = name
entry.setdefault("disable", 0) entry.setdefault("disable", 0)
store[name] = entry store[name] = entry
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
name = str(values(inputs)["name"]) name = str(values(inputs)["name"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {}) store = _notifications(metadata)["endpoints"].get(kind) or {}
if name not in store: if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist") raise ApiError(404, f"{kind} endpoint does not exist")
return _public({"name": name, **store[name]}) return _public({"name": name, **store[name]})
@@ -108,7 +97,8 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
name = str(payload["name"]) name = str(payload["name"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {}) notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name not in store: if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist") raise ApiError(404, f"{kind} endpoint does not exist")
current = dict(store[name]) current = dict(store[name])
@@ -123,15 +113,18 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
current[key] = value current[key] = value
current["name"] = name current["name"] = name
store[name] = current store[name] = current
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def delete(request: Request, inputs: dict[str, Any]) -> None: async def delete(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["name"]) name = str(values(inputs)["name"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _notifications(metadata)["endpoints"].setdefault(kind, {}) notifications = _notifications(metadata)
store = notifications["endpoints"].setdefault(kind, {})
if name not in store: if name not in store:
raise ApiError(404, f"{kind} endpoint does not exist") raise ApiError(404, f"{kind} endpoint does not exist")
del store[name] del store[name]
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
registry.register(base, "GET", list_endpoints) registry.register(base, "GET", list_endpoints)
@@ -149,12 +142,14 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
name = str(payload["name"]) name = str(payload["name"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _notifications(metadata)["matchers"] notifications = _notifications(metadata)
store = notifications["matchers"]
if name in store: if name in store:
raise ApiError(400, f"matcher '{name}' already exists") raise ApiError(400, f"matcher '{name}' already exists")
store[name] = { store[name] = {
key: value for key, value in payload.items() if key not in {"delete", "digest"} key: value for key, value in payload.items() if key not in {"delete", "digest"}
} }
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def matchers_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def matchers_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -169,7 +164,8 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
name = str(payload["name"]) name = str(payload["name"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _notifications(metadata)["matchers"] notifications = _notifications(metadata)
store = notifications["matchers"]
if name not in store: if name not in store:
raise ApiError(404, "matcher does not exist") raise ApiError(404, "matcher does not exist")
current = dict(store[name]) current = dict(store[name])
@@ -183,24 +179,35 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
current[key] = value current[key] = value
current["name"] = name current["name"] = name
store[name] = current store[name] = current
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def matchers_delete(request: Request, inputs: dict[str, Any]) -> None: async def matchers_delete(request: Request, inputs: dict[str, Any]) -> None:
name = str(values(inputs)["name"]) name = str(values(inputs)["name"])
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
store = _notifications(metadata)["matchers"] notifications = _notifications(metadata)
store = notifications["matchers"]
if name not in store: if name not in store:
raise ApiError(404, "matcher does not exist") raise ApiError(404, "matcher does not exist")
del store[name] del store[name]
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def matcher_fields(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def matcher_fields(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
return list(DEFAULT_MATCHER_FIELDS) metadata = await cluster_metadata(request)
fields = _notifications(metadata).get("matcher_fields")
if isinstance(fields, list):
return [dict(item) for item in fields if isinstance(item, dict)]
return []
async def matcher_field_values( async def matcher_field_values(
_request: Request, _inputs: dict[str, Any] request: Request, _inputs: dict[str, Any]
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
return list(DEFAULT_MATCHER_FIELD_VALUES) metadata = await cluster_metadata(request)
values_list = _notifications(metadata).get("matcher_field_values")
if isinstance(values_list, list):
return [dict(item) for item in values_list if isinstance(item, dict)]
return []
async def targets(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def targets(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
@@ -231,10 +238,10 @@ def register_notifications_handlers(registry: HandlerRegistry) -> None:
break break
if not found: if not found:
raise ApiError(404, "notification target does not exist") raise ApiError(404, "notification target does not exist")
tests = notifications.setdefault("tests", []) tests = list(notifications.get("tests") or [])
if not isinstance(tests, list):
tests = notifications["tests"] = []
tests.append({"name": name, "tested_at": int(time.time()), "ok": True}) tests.append({"name": name, "tested_at": int(time.time()), "ok": True})
notifications["tests"] = tests
metadata["notifications"] = notifications
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
registry.register("/cluster/notifications", "GET", index) registry.register("/cluster/notifications", "GET", index)
+37 -11
View File
@@ -10,18 +10,35 @@ from fastapi import Request
from app.api.errors import ApiError from app.api.errors import ApiError
from app.api.registry import HandlerRegistry from app.api.registry import HandlerRegistry
from app.handlers.common import database, state, values from app.handlers.common import database, require_value, state, values
from app.simulation.seed import CLUSTER_ID, stable_id from app.simulation.seed import CLUSTER_ID, stable_id
async def _pool_members(request: Request, pool_id: uuid.UUID) -> list[str]: async def _pool_members(request: Request, pool_id: uuid.UUID) -> list[dict[str, Any]]:
rows = await database(request).pool.fetch( rows = await database(request).pool.fetch(
"""SELECT r.external_id FROM pool_members pm """SELECT r.external_id, r.kind, r.state, n.name AS node
FROM pool_members pm
JOIN resources r ON r.id = pm.resource_id JOIN resources r ON r.id = pm.resource_id
WHERE pm.pool_id=$1 ORDER BY r.external_id::integer""", JOIN nodes n ON n.id = r.node_id
WHERE pm.pool_id=$1
ORDER BY r.external_id::integer""",
pool_id, pool_id,
) )
return [str(row["external_id"]) for row in rows] members: list[dict[str, Any]] = []
for row in rows:
guest_state = state(row["state"])
kind = str(row["kind"])
vmid = int(row["external_id"])
members.append(
{
"id": f"{kind}/{vmid}",
"node": str(row["node"]),
"type": kind,
"vmid": vmid,
"name": str(guest_state.get("name") or f"{kind}-{vmid}"),
}
)
return members
def register_pool_handlers(registry: HandlerRegistry) -> None: def register_pool_handlers(registry: HandlerRegistry) -> None:
@@ -39,16 +56,25 @@ def register_pool_handlers(registry: HandlerRegistry) -> None:
metadata = state(row["metadata"]) metadata = state(row["metadata"])
members = await _pool_members(request, row["id"]) members = await _pool_members(request, row["id"])
if not members and isinstance(metadata.get("members"), list): if not members and isinstance(metadata.get("members"), list):
members = [str(item) for item in metadata["members"]] # Fallback for metadata-only membership before pool_members rows exist.
item: dict[str, Any] = { for item in metadata["members"]:
members.append(
{
"id": f"qemu/{item}",
"node": "",
"type": "qemu",
"vmid": int(item) if str(item).isdigit() else 0,
}
)
pool_item: dict[str, Any] = {
"poolid": str(row["pool_id"]), "poolid": str(row["pool_id"]),
"members": members, "members": members,
} }
if row["comment"] is not None: if row["comment"] is not None:
item["comment"] = str(row["comment"]) pool_item["comment"] = str(row["comment"])
elif metadata.get("comment"): elif metadata.get("comment"):
item["comment"] = str(metadata["comment"]) pool_item["comment"] = str(metadata["comment"])
result.append(item) result.append(pool_item)
return result return result
async def pool_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def pool_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -117,7 +143,7 @@ def register_pool_handlers(registry: HandlerRegistry) -> None:
) )
async def pool_delete(request: Request, inputs: dict[str, Any]) -> None: async def pool_delete(request: Request, inputs: dict[str, Any]) -> None:
poolid = str(values(inputs)["poolid"]) poolid = str(require_value(values(inputs), "poolid"))
status = await database(request).pool.execute( status = await database(request).pool.execute(
"DELETE FROM pools WHERE pool_id=$1", "DELETE FROM pools WHERE pool_id=$1",
poolid, poolid,
+225 -76
View File
@@ -33,16 +33,124 @@ def _state(value: object) -> dict[str, Any]:
return dict(cast(Mapping[str, Any], value)) return dict(cast(Mapping[str, Any], value))
# Runtime-only keys stored in ``resources.state``; never echo on GET /config.
_QEMU_CONFIG_INTERNAL = frozenset(
{
"agent",
"cloudinit_dump",
"config",
"interfaces",
"migrate_preconditions",
"rrd",
"rrddata",
"sendkey",
"status",
}
)
def _agent_config_string(value: object) -> str:
"""Proxmox QEMU config ``agent`` is a string (e.g. ``1`` / ``0``), not an object."""
if isinstance(value, dict):
return "1" if value.get("enabled", True) else "0"
if isinstance(value, bool):
return "1" if value else "0"
if value in (None, ""):
return "0"
text = str(value).strip()
return text or "0"
def _public_qemu_config(config: dict[str, Any], state: dict[str, Any]) -> dict[str, Any]:
"""Wire-format QEMU config for clients (bpg / pulumi-proxmoxve)."""
import hashlib
merged = {**state, **config}
payload: dict[str, Any] = {}
for key, value in merged.items():
if key in _QEMU_CONFIG_INTERNAL:
continue
if isinstance(value, dict | list):
continue
payload[key] = value
agent = config.get("agent", state.get("agent", "0"))
payload["agent"] = _agent_config_string(agent)
digest_src = {key: value for key, value in payload.items() if key != "digest"}
payload["digest"] = hashlib.sha1( # noqa: S324 - Proxmox config digests use SHA-1
json.dumps(digest_src, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
return payload
def _cpu_utilization(vm_state: Mapping[str, Any], *, running: bool, idle: float = 0.1) -> float:
"""Status ``cpu`` is utilization; config may store model strings like ``qemu64``."""
raw = vm_state.get("cpu")
if isinstance(raw, (int, float)):
return float(raw)
if isinstance(raw, str):
try:
return float(raw)
except ValueError:
pass
return idle if running else 0.0
def _qemu_index_item(vmid: int, vm_state: dict[str, Any], config: dict[str, Any]) -> dict[str, Any]:
status = str(vm_state.get("status", "stopped"))
running = status in {"running", "paused"}
memory_mb = int(config.get("memory", config.get("mem", 2048)))
maxmem = memory_mb * 2**20
mem = int(vm_state.get("mem", maxmem // 2 if running else 0))
return {
"vmid": vmid,
"name": str(config.get("name", f"vm-{vmid}")),
"status": status,
"qmpstatus": status if running else "stopped",
"cpus": int(config.get("cores", config.get("cpus", 1))),
"cpu": _cpu_utilization(vm_state, running=running),
"maxmem": maxmem,
"mem": mem,
"memhost": int(vm_state.get("memhost", mem)),
"maxdisk": int(vm_state.get("maxdisk", config.get("maxdisk", 32 * 1024**3))),
"disk": int(vm_state.get("disk", 0)),
"diskread": int(vm_state.get("diskread", 0)),
"diskwrite": int(vm_state.get("diskwrite", 0)),
"netin": int(vm_state.get("netin", 0)),
"netout": int(vm_state.get("netout", 0)),
"uptime": int(vm_state.get("uptime", 0)) if running else 0,
"pid": int(vm_state.get("pid", 12_345 if running else 0)),
"lock": str(vm_state.get("lock", "")),
"tags": str(config.get("tags", "")),
"template": int(bool(vm_state.get("template", False))),
"running-qemu": str(vm_state.get("running-qemu", "9.2.0")) if running else "",
"running-machine": str(vm_state.get("running-machine", "pc-i440fx-9.2")) if running else "",
"pressurecpusome": float(vm_state.get("pressurecpusome", 0.0)),
"pressurecpufull": float(vm_state.get("pressurecpufull", 0.0)),
"pressureiosome": float(vm_state.get("pressureiosome", 0.0)),
"pressureiofull": float(vm_state.get("pressureiofull", 0.0)),
"pressurememorysome": float(vm_state.get("pressurememorysome", 0.0)),
"pressurememoryfull": float(vm_state.get("pressurememoryfull", 0.0)),
}
def register_qemu_handlers(registry: HandlerRegistry) -> None: def register_qemu_handlers(registry: HandlerRegistry) -> None:
async def qemu_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def qemu_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
node = str(_values(inputs)["node"]) node = str(_values(inputs)["node"])
rows = await _database(request).pool.fetch( rows = await _database(request).pool.fetch(
"""SELECT r.external_id::integer AS vmid, r.state """SELECT r.external_id::integer AS vmid, r.state, v.config
FROM resources r JOIN nodes n ON n.id=r.node_id FROM resources r
JOIN nodes n ON n.id=r.node_id
JOIN virtual_machines v ON v.resource_id=r.id
WHERE n.name=$1 AND r.kind='qemu' ORDER BY r.external_id::integer""", WHERE n.name=$1 AND r.kind='qemu' ORDER BY r.external_id::integer""",
node, node,
) )
return [{"vmid": int(row["vmid"]), **_state(row["state"])} for row in rows] return [
_qemu_index_item(int(row["vmid"]), _state(row["state"]), _state(row["config"]))
for row in rows
]
async def qemu_status_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]: async def qemu_status_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs) payload = _values(inputs)
@@ -82,28 +190,28 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
else 0, else 0,
) )
) )
return { agent_enabled = str(config.get("agent", "0")).startswith("1")
"vmid": vmid, item = _qemu_index_item(vmid, vm_state, config)
"name": str(config.get("name", f"vm-{vmid}")), item.update(
"status": status, {
"qmpstatus": status if running else "stopped", "uptime": uptime,
"lock": str(vm_state.get("lock", "")), "mem": mem_used,
"pid": int(vm_state.get("pid", 12_345 if running else 0)), "balloon": int(vm_state.get("balloon", 0)),
"cpus": int(config.get("cores", config.get("cpus", 1))), "ballooninfo": {
"maxmem": maxmem, "actual": mem_used,
"mem": mem_used, "max_mem": maxmem,
"balloon": int(vm_state.get("balloon", 0)), "mem_swapped_in": 0,
"ballooninfo": { "mem_swapped_out": 0,
"actual": mem_used, },
"max_mem": maxmem, "template": bool(vm_state.get("template", False)),
"mem_swapped_in": 0, "ha": {"managed": int(vm_state.get("ha_managed", 0))},
"mem_swapped_out": 0, "agent": bool(running and agent_enabled),
}, "clipboard": str(vm_state.get("clipboard", "")),
"uptime": uptime, "spice": bool(vm_state.get("spice", False)),
"template": int(bool(vm_state.get("template", False))), "serial": bool(vm_state.get("serial", False)),
"ha": {"managed": int(vm_state.get("ha_managed", 0))}, }
"agent": 1 if running and str(config.get("agent", "0")).startswith("1") else 0, )
} return item
async def qemu_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def qemu_config(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"]) node, vmid = str(_values(inputs)["node"]), str(_values(inputs)["vmid"])
@@ -117,7 +225,10 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
) )
if row is None: if row is None:
raise ApiError(404, "virtual machine does not exist") raise ApiError(404, "virtual machine does not exist")
return {"vmid": int(vmid), **_state(row["config"]), **_state(row["state"])} return {
"vmid": int(vmid),
**_public_qemu_config(_state(row["config"]), _state(row["state"])),
}
async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str: async def mutate(operation: str, request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
@@ -291,24 +402,30 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
values = _values(inputs) values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"])) resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
rows = await _database(request).pool.fetch( rows = await _database(request).pool.fetch(
"""SELECT name, parent_name, description, created_at FROM snapshots """SELECT name, parent_name, description, created_at, state FROM snapshots
WHERE resource_id=$1 ORDER BY created_at, name""", WHERE resource_id=$1 ORDER BY created_at, name""",
resource["id"], resource["id"],
) )
return [ result: list[dict[str, Any]] = []
{ for row in rows:
"name": row["name"], raw_state = row["state"] if "state" in row else None
"parent": row["parent_name"], snap_state = _state(raw_state) if raw_state is not None else {}
"description": row["description"] or "", result.append(
"snaptime": int(row["created_at"].timestamp()), {
} "name": row["name"],
for row in rows "parent": row["parent_name"],
] "description": row["description"] or "",
"snaptime": int(row["created_at"].timestamp()),
"vmstate": int(bool(snap_state.get("vmstate"))),
}
)
return result
async def snapshot_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def snapshot_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs) values = _values(inputs)
row = await _snapshot(request, values) row = await _snapshot(request, values)
state = _state(row["state"]) raw_state = row["state"] if "state" in row else None
state = _state(raw_state) if raw_state is not None else {}
return { return {
"name": row["name"], "name": row["name"],
"parent": row["parent_name"], "parent": row["parent_name"],
@@ -406,7 +523,7 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
async def migrate_preconditions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def migrate_preconditions(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs) values = _values(inputs)
await _qemu_resource(request, str(values["node"]), str(values["vmid"])) resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
target = values.get("target") target = values.get("target")
if target in {None, ""}: if target in {None, ""}:
raise ApiError(400, "parameter 'target' is required") raise ApiError(400, "parameter 'target' is required")
@@ -416,7 +533,13 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
) )
if not exists: if not exists:
raise ApiError(404, "target node does not exist") raise ApiError(404, "target node does not exist")
return {"local_disks": [], "local_resources": [], "running": False} state = _state(resource["state"])
pre = state.get("migrate_preconditions")
payload = dict(pre) if isinstance(pre, dict) else {}
payload["running"] = str(state.get("status") or "") == "running"
payload.setdefault("local_disks", [])
payload.setdefault("local_resources", [])
return payload
async def migrate(request: Request, inputs: dict[str, Any]) -> str: async def migrate(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
@@ -474,7 +597,7 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
}, },
) )
async def resize(request: Request, inputs: dict[str, Any]) -> None: async def resize(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
node, vmid, disk = str(values["node"]), str(values["vmid"]), str(values["disk"]) node, vmid, disk = str(values["node"]), str(values["vmid"]), str(values["disk"])
resource = await _qemu_resource(request, node, vmid) resource = await _qemu_resource(request, node, vmid)
@@ -507,6 +630,13 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
str(config[disk]).split(":", 1)[0], str(config[disk]).split(":", 1)[0],
size, size,
) )
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-resize",
payload={"resource_id": str(resource["id"]), "disk": disk, "size": size},
)
async def move_disk(request: Request, inputs: dict[str, Any]) -> str: async def move_disk(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
@@ -544,39 +674,16 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
command: str, request: Request, inputs: dict[str, Any] command: str, request: Request, inputs: dict[str, Any]
) -> dict[str, Any]: ) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs)) resource = await _agent_resource(request, _values(inputs))
config = _state(resource["config"])
vmid = str(_values(inputs)["vmid"])
results: dict[str, Any] = {
"info": {
"version": "9.2.0-simulator",
"supported_commands": [
{"name": name, "enabled": True, "success-response": True}
for name in ("guest-ping", "guest-info", "guest-get-osinfo")
],
},
"get-osinfo": {
"name": str(config.get("ostype", "linux")),
"pretty-name": "Proxmox Simulator Guest",
"version": "1.0",
"machine": "x86_64",
},
"get-host-name": {"host-name": str(config.get("name", f"vm-{vmid}"))},
"network-get-interfaces": [
{
"name": "eth0",
"hardware-address": "02:00:00:00:00:01",
"ip-addresses": [
{"ip-address": "192.0.2.10", "ip-address-type": "ipv4", "prefix": 24}
],
}
],
"ping": {},
}
if command == "get-time": if command == "get-time":
seconds = int( seconds = int(
await _database(request).pool.fetchval("SELECT extract(epoch from now())::bigint") await _database(request).pool.fetchval("SELECT extract(epoch from now())::bigint")
) )
return {"result": {"seconds": seconds, "nanoseconds": 0}} return {"result": {"seconds": seconds, "nanoseconds": 0}}
state = _state(resource["state"])
agent = state.get("agent")
results = agent.get("results") if isinstance(agent, dict) else None
if not isinstance(results, dict) or command not in results:
raise ApiError(404, f"agent command '{command}' has no stored result")
return {"result": results[command]} return {"result": results[command]}
async def agent_info(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def agent_info(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -586,6 +693,12 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
return await agent_result("get-osinfo", request, inputs) return await agent_result("get-osinfo", request, inputs)
async def agent_hostname(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def agent_hostname(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"])
config = _state(resource["config"])
name = state.get("name") or config.get("name")
if name:
return {"result": {"host-name": str(name)}}
return await agent_result("get-host-name", request, inputs) return await agent_result("get-host-name", request, inputs)
async def agent_network(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def agent_network(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -598,12 +711,32 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
return await agent_result("ping", request, inputs) return await agent_result("ping", request, inputs)
async def task_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def task_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
tasks = await TaskRepository(_database(request).pool).list_for_node( from app.tasks.upid import Upid
str(_values(inputs)["node"])
) node = str(_values(inputs)["node"])
return [ tasks = await TaskRepository(_database(request).pool).list_for_node(node)
{"upid": task.upid, "status": task.status, "type": task.task_type} for task in tasks result: list[dict[str, Any]] = []
] for task in tasks:
try:
parsed = Upid.parse(task.upid)
except ValueError:
continue
finished = task.status in {"success", "error", "cancelled"}
entry: dict[str, Any] = {
"upid": task.upid,
"id": parsed.task_id,
"node": parsed.node,
"pid": parsed.pid,
"pstart": parsed.process_start,
"starttime": parsed.start_time,
"type": parsed.task_type,
"user": parsed.user,
}
if finished:
entry["status"] = "OK" if task.status == "success" else str(task.status).upper()
entry["endtime"] = parsed.start_time + max(task.progress, 1)
result.append(entry)
return result
async def task_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def task_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
task = await TaskRepository(_database(request).pool).get_by_upid( task = await TaskRepository(_database(request).pool).get_by_upid(
@@ -644,7 +777,7 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
} }
} }
async def qemu_template(request: Request, inputs: dict[str, Any]) -> None: async def qemu_template(request: Request, inputs: dict[str, Any]) -> str:
payload = _values(inputs) payload = _values(inputs)
node, vmid = str(payload["node"]), str(payload["vmid"]) node, vmid = str(payload["node"]), str(payload["vmid"])
resource = await _qemu_resource(request, node, vmid) resource = await _qemu_resource(request, node, vmid)
@@ -661,6 +794,13 @@ def register_qemu_handlers(registry: HandlerRegistry) -> None:
resource["id"], resource["id"],
json.dumps(state, sort_keys=True), json.dumps(state, sort_keys=True),
) )
return await _create_task(
request,
node=node,
vmid=vmid,
task_type="qemu-template",
payload={"resource_id": str(resource["id"])},
)
async def qemu_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]: async def qemu_index(request: Request, inputs: dict[str, Any]) -> list[dict[str, str]]:
payload = _values(inputs) payload = _values(inputs)
@@ -778,6 +918,8 @@ async def _create_task(
"qemu-migrate": "qmigrate", "qemu-migrate": "qmigrate",
"qemu-remote-migrate": "qmremote", "qemu-remote-migrate": "qmremote",
"qemu-move-disk": "qmmove", "qemu-move-disk": "qmmove",
"qemu-resize": "qmresize",
"qemu-template": "qmtemplate",
}[task_type] }[task_type]
upid = str(Upid.allocate(node, worker_type, vmid, str(request.state.principal))) upid = str(Upid.allocate(node, worker_type, vmid, str(request.state.principal)))
try: try:
@@ -825,7 +967,14 @@ async def _agent_resource(request: Request, values: dict[str, Any]) -> Any:
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"])) resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
config = _state(resource["config"]) config = _state(resource["config"])
state = _state(resource["state"]) state = _state(resource["state"])
if str(config.get("agent", "0")).split(",", 1)[0].lower() not in {"1", "true", "yes"}: agent = state.get("agent")
if not isinstance(agent, dict):
agent = config.get("agent")
if isinstance(agent, dict):
enabled = bool(agent.get("enabled", True))
else:
enabled = str(config.get("agent", "0")).split(",", 1)[0].lower() in {"1", "true", "yes"}
if not enabled:
raise ApiError(409, "QEMU guest agent is not enabled") raise ApiError(409, "QEMU guest agent is not enabled")
if state.get("status") != "running": if state.get("status") != "running":
raise ApiError(409, "QEMU guest agent is not running") raise ApiError(409, "QEMU guest agent is not running")
+23 -49
View File
@@ -79,20 +79,10 @@ def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
async def _agent_blob(command: str, request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def _agent_blob(command: str, request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs)) resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"]) state = _state(resource["state"])
agent = state.setdefault("agent", {}) agent = state.get("agent")
blobs = agent.setdefault("results", {}) blobs = agent.get("results") if isinstance(agent, dict) else None
defaults: dict[str, Any] = { if not isinstance(blobs, dict) or command not in blobs:
"get-users": [{"user": "root", "login-time": 0}], raise ApiError(404, f"agent command '{command}' has no stored result")
"get-fsinfo": [{"name": "/", "type": "ext4", "total-bytes": 32 * 1024**3}],
"get-memory-block-info": {"size": 1024**3},
"get-memory-blocks": [{"start": 0, "size": 1024**3}],
"get-timezone": {"zone": "UTC", "offset": 0},
"get-vcpus": [{"online": True, "can-offline": False}],
"fsfreeze-status": "thawed",
}
if command not in blobs:
blobs[command] = defaults.get(command, {})
await _save_guest_state(request, resource["id"], state)
return {"result": blobs[command]} return {"result": blobs[command]}
async def agent_users(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def agent_users(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
@@ -146,10 +136,10 @@ def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
resource = await _agent_resource(request, payload) resource = await _agent_resource(request, payload)
path = str(payload.get("file") or payload.get("path") or "/etc/hostname") path = str(payload.get("file") or payload.get("path") or "/etc/hostname")
state = _state(resource["state"]) state = _state(resource["state"])
files = state.setdefault("agent", {}).setdefault("files", {}) agent = state.get("agent")
if path not in files: files = agent.get("files") if isinstance(agent, dict) else None
files[path] = f"simulated:{path}\n" if not isinstance(files, dict) or path not in files:
await _save_guest_state(request, resource["id"], state) raise ApiError(404, "file does not exist")
content = str(files[path]) content = str(files[path])
return {"result": {"content": content, "truncated": True}} return {"result": {"content": content, "truncated": True}}
@@ -188,9 +178,15 @@ def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
async def agent_fstrim(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def agent_fstrim(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
resource = await _agent_resource(request, _values(inputs)) resource = await _agent_resource(request, _values(inputs))
state = _state(resource["state"]) state = _state(resource["state"])
state.setdefault("agent", {})["last_fstrim"] = True agent = dict(state.get("agent") or {})
agent["last_fstrim"] = True
state["agent"] = agent
await _save_guest_state(request, resource["id"], state) await _save_guest_state(request, resource["id"], state)
return {"result": {"paths": [{"path": "/", "trimmed": 0}]}} results = agent.get("results") if isinstance(agent.get("results"), dict) else {}
fstrim = results.get("fstrim") if isinstance(results, dict) else None
if isinstance(fstrim, dict):
return {"result": fstrim}
return {"result": {}}
async def agent_set_password(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def agent_set_password(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = _values(inputs) payload = _values(inputs)
@@ -258,45 +254,23 @@ def register_qemu_extra_handlers(registry: HandlerRegistry) -> None:
async def cloudinit_dump(request: Request, inputs: dict[str, Any]) -> str: async def cloudinit_dump(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"])) resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
config = _state(resource["config"]) state = _state(resource["state"])
return ( dump = state.get("cloudinit_dump")
f"#cloud-config\nhostname: {config.get('name', values['vmid'])}\n" return str(dump) if dump is not None else ""
f"manage_etc_hosts: true\n"
)
async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
values = _values(inputs) values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"])) resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"]) state = _state(resource["state"])
rrd_state = state.setdefault("rrd", {"filename": f"pve-vm-{values['vmid']}.rrd"}) rrd_state = state.get("rrd")
await _save_guest_state(request, resource["id"], state) return dict(rrd_state) if isinstance(rrd_state, dict) else {}
return dict(rrd_state)
async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
values = _values(inputs) values = _values(inputs)
resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"])) resource = await _qemu_resource(request, str(values["node"]), str(values["vmid"]))
state = _state(resource["state"]) state = _state(resource["state"])
series = state.setdefault( series = state.get("rrddata")
"rrddata", return [dict(item) for item in series] if isinstance(series, list) else []
[
{
"time": 1_700_000_000,
"cpu": 0.05,
"mem": 256 * 1024 * 1024,
"netin": 0,
"netout": 0,
},
{
"time": 1_700_000_060,
"cpu": 0.08,
"mem": 260 * 1024 * 1024,
"netin": 100,
"netout": 80,
},
],
)
await _save_guest_state(request, resource["id"], state)
return list(series)
async def monitor(request: Request, inputs: dict[str, Any]) -> str: async def monitor(request: Request, inputs: dict[str, Any]) -> str:
values = _values(inputs) values = _values(inputs)
+180 -122
View File
@@ -21,65 +21,77 @@ _SECRET_KEYS = frozenset({"key", "token", "fingerprint"})
def _sdn(metadata: dict[str, Any]) -> dict[str, Any]: def _sdn(metadata: dict[str, Any]) -> dict[str, Any]:
current = metadata.setdefault( current = metadata.get("sdn")
"sdn", return current if isinstance(current, dict) else {}
{
"zones": {},
"vnets": {}, _VNET_LIST_KEYS = frozenset(
"controllers": {}, {
"dns": {}, "vnet",
"ipams": {}, "type",
"fabrics": {}, "zone",
"fabric_nodes": {}, "alias",
"prefix_lists": {}, "tag",
"route_maps": {}, "vlanaware",
"lock": None, "digest",
"pending": False, "isolate-ports",
"running_version": 1, "pending",
}, "state",
) }
if not isinstance(current, dict): )
current = {
"zones": {}, _ZONE_LIST_KEYS = frozenset(
"vnets": {}, {
"controllers": {}, "zone",
"dns": {}, "type",
"ipams": {}, "bridge",
"fabrics": {}, "mtu",
"fabric_nodes": {}, "nodes",
"prefix_lists": {}, "ipam",
"route_maps": {},
"lock": None,
"pending": False,
"running_version": 1,
}
metadata["sdn"] = current
for key in (
"zones",
"vnets",
"controllers",
"dns", "dns",
"ipams", "reversedns",
"fabrics", "vlan-protocol",
"fabric_nodes", "digest",
"prefix_lists", "pending",
"route_maps", "state",
): "comment",
current.setdefault(key, {}) }
return current )
def _public(item: dict[str, Any]) -> dict[str, Any]: def _public(item: dict[str, Any]) -> dict[str, Any]:
return {key: value for key, value in item.items() if key not in _SECRET_KEYS} return {key: value for key, value in item.items() if key not in _SECRET_KEYS}
def _vnet_list_item(name: str, item: dict[str, Any]) -> dict[str, Any]:
payload = {key: value for key, value in item.items() if key in _VNET_LIST_KEYS}
payload["vnet"] = str(payload.get("vnet") or name)
payload.setdefault("type", "vnet")
return _public(payload)
def _zone_list_item(name: str, item: dict[str, Any]) -> dict[str, Any]:
payload = {key: value for key, value in item.items() if key in _ZONE_LIST_KEYS}
payload["zone"] = str(payload.get("zone") or name)
return _public(payload)
def _store_list(store: dict[str, Any], *, id_key: str) -> list[dict[str, Any]]: def _store_list(store: dict[str, Any], *, id_key: str) -> list[dict[str, Any]]:
if id_key == "vnet":
return [_vnet_list_item(name, item) for name, item in sorted(store.items())]
if id_key == "zone":
return [_zone_list_item(name, item) for name, item in sorted(store.items())]
return [_public({id_key: name, **item}) for name, item in sorted(store.items())] return [_public({id_key: name, **item}) for name, item in sorted(store.items())]
async def _load(request: Request) -> tuple[dict[str, Any], dict[str, Any]]: async def _load(
request: Request, *, for_write: bool = False
) -> tuple[dict[str, Any], dict[str, Any]]:
metadata = await cluster_metadata(request) metadata = await cluster_metadata(request)
return metadata, _sdn(metadata) sdn = _sdn(metadata)
if for_write and metadata.get("sdn") is not sdn:
metadata["sdn"] = sdn
return metadata, sdn
def register_sdn_handlers(registry: HandlerRegistry) -> None: def register_sdn_handlers(registry: HandlerRegistry) -> None:
@@ -98,9 +110,11 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
"zones", "zones",
) )
async def apply(request: Request, inputs: dict[str, Any]) -> None: async def apply(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.cluster_extra import _cluster_task
payload = values(inputs) payload = values(inputs)
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
lock = sdn.get("lock") lock = sdn.get("lock")
token = payload.get("lock-token") token = payload.get("lock-token")
if lock and token and lock.get("token") != token: if lock and token and lock.get("token") != token:
@@ -110,19 +124,20 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
if payload.get("release-lock"): if payload.get("release-lock"):
sdn["lock"] = None sdn["lock"] = None
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return await _cluster_task(request, task_type="sdn-apply", worker="sdnapply")
async def lock_create(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def lock_create(request: Request, inputs: dict[str, Any]) -> str:
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
if sdn.get("lock") and not values(inputs).get("allow-pending"): if sdn.get("lock") and not values(inputs).get("allow-pending"):
raise ApiError(400, "SDN is already locked") raise ApiError(400, "SDN is already locked")
token = secrets.token_hex(8) token = secrets.token_hex(8)
sdn["lock"] = {"token": token} sdn["lock"] = {"token": token}
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
return {"digest": token, "token": token} return token
async def lock_delete(request: Request, inputs: dict[str, Any]) -> None: async def lock_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
lock = sdn.get("lock") lock = sdn.get("lock")
if lock is None: if lock is None:
return None return None
@@ -133,14 +148,14 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def rollback(request: Request, inputs: dict[str, Any]) -> None: async def rollback(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
sdn["pending"] = False sdn["pending"] = False
if payload.get("release-lock"): if payload.get("release-lock"):
sdn["lock"] = None sdn["lock"] = None
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def dry_run(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def dry_run(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
return [ return [
{"type": "zone", "name": name, "action": "noop"} {"type": "zone", "name": name, "action": "noop"}
for name in sorted(sdn.get("zones") or {}) for name in sorted(sdn.get("zones") or {})
@@ -154,7 +169,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
create_required: str | None = None, create_required: str | None = None,
) -> None: ) -> None:
async def list_items(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def list_items(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
items = _store_list(sdn.get(store_key) or {}, id_key=id_param) items = _store_list(sdn.get(store_key) or {}, id_key=id_param)
type_filter = values(inputs).get("type") type_filter = values(inputs).get("type")
if type_filter: if type_filter:
@@ -164,7 +179,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def create(request: Request, inputs: dict[str, Any]) -> None: async def create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
item_id = str(payload[create_required or id_param]) item_id = str(payload[create_required or id_param])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault(store_key, {}) store = sdn.setdefault(store_key, {})
if item_id in store: if item_id in store:
raise ApiError(400, f"{store_key} '{item_id}' already exists") raise ApiError(400, f"{store_key} '{item_id}' already exists")
@@ -179,7 +194,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
item_id = str(values(inputs)[id_param]) item_id = str(values(inputs)[id_param])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = (sdn.get(store_key) or {}).get(item_id) item = (sdn.get(store_key) or {}).get(item_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, f"{store_key} entry does not exist") raise ApiError(404, f"{store_key} entry does not exist")
@@ -188,7 +203,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def update(request: Request, inputs: dict[str, Any]) -> None: async def update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
item_id = str(payload[id_param]) item_id = str(payload[id_param])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault(store_key, {}) store = sdn.setdefault(store_key, {})
if item_id not in store: if item_id not in store:
raise ApiError(404, f"{store_key} entry does not exist") raise ApiError(404, f"{store_key} entry does not exist")
@@ -208,7 +223,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def delete(request: Request, inputs: dict[str, Any]) -> None: async def delete(request: Request, inputs: dict[str, Any]) -> None:
item_id = str(values(inputs)[id_param]) item_id = str(values(inputs)[id_param])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault(store_key, {}) store = sdn.setdefault(store_key, {})
if item_id not in store: if item_id not in store:
raise ApiError(404, f"{store_key} entry does not exist") raise ApiError(404, f"{store_key} entry does not exist")
@@ -230,20 +245,20 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def ipam_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def ipam_status(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
ipam = str(values(inputs)["ipam"]) ipam = str(values(inputs)["ipam"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
if ipam not in (sdn.get("ipams") or {}): if ipam not in (sdn.get("ipams") or {}):
raise ApiError(404, "ipam does not exist") raise ApiError(404, "ipam does not exist")
return {"status": "ok", "ipam": ipam} return {"status": "ok", "ipam": ipam}
# vnets + nested # vnets + nested
async def vnets_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def vnets_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
return _store_list(sdn.get("vnets") or {}, id_key="vnet") return _store_list(sdn.get("vnets") or {}, id_key="vnet")
async def vnets_create(request: Request, inputs: dict[str, Any]) -> None: async def vnets_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("vnets", {}) store = sdn.setdefault("vnets", {})
if vnet in store: if vnet in store:
raise ApiError(400, f"vnet '{vnet}' already exists") raise ApiError(400, f"vnet '{vnet}' already exists")
@@ -259,7 +274,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def vnet_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def vnet_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
vnet = str(values(inputs)["vnet"]) vnet = str(values(inputs)["vnet"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("vnets") or {}).get(vnet) item = (sdn.get("vnets") or {}).get(vnet)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "vnet does not exist") raise ApiError(404, "vnet does not exist")
@@ -268,7 +283,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def vnet_update(request: Request, inputs: dict[str, Any]) -> None: async def vnet_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("vnets", {}) store = sdn.setdefault("vnets", {})
if vnet not in store: if vnet not in store:
raise ApiError(404, "vnet does not exist") raise ApiError(404, "vnet does not exist")
@@ -288,7 +303,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def vnet_delete(request: Request, inputs: dict[str, Any]) -> None: async def vnet_delete(request: Request, inputs: dict[str, Any]) -> None:
vnet = str(values(inputs)["vnet"]) vnet = str(values(inputs)["vnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("vnets", {}) store = sdn.setdefault("vnets", {})
if vnet not in store: if vnet not in store:
raise ApiError(404, "vnet does not exist") raise ApiError(404, "vnet does not exist")
@@ -307,7 +322,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def subnets_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def subnets_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
vnet = str(values(inputs)["vnet"]) vnet = str(values(inputs)["vnet"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
return _store_list(item.get("subnets") or {}, id_key="subnet") return _store_list(item.get("subnets") or {}, id_key="subnet")
@@ -315,7 +330,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
subnet = str(payload["subnet"]) subnet = str(payload["subnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
subnets = item.setdefault("subnets", {}) subnets = item.setdefault("subnets", {})
if subnet in subnets: if subnet in subnets:
@@ -331,7 +346,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
subnet = str(payload["subnet"]) subnet = str(payload["subnet"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
data = (item.get("subnets") or {}).get(subnet) data = (item.get("subnets") or {}).get(subnet)
if not isinstance(data, dict): if not isinstance(data, dict):
@@ -342,7 +357,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
subnet = str(payload["subnet"]) subnet = str(payload["subnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
subnets = item.setdefault("subnets", {}) subnets = item.setdefault("subnets", {})
if subnet not in subnets: if subnet not in subnets:
@@ -365,7 +380,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
subnet = str(payload["subnet"]) subnet = str(payload["subnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
subnets = item.setdefault("subnets", {}) subnets = item.setdefault("subnets", {})
if subnet not in subnets: if subnet not in subnets:
@@ -377,7 +392,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def ips_create(request: Request, inputs: dict[str, Any]) -> None: async def ips_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
ips = item.setdefault("ips", []) ips = item.setdefault("ips", [])
if not isinstance(ips, list): if not isinstance(ips, list):
@@ -398,7 +413,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def ips_delete(request: Request, inputs: dict[str, Any]) -> None: async def ips_delete(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
ips = item.setdefault("ips", []) ips = item.setdefault("ips", [])
if not isinstance(ips, list): if not isinstance(ips, list):
@@ -416,14 +431,14 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fw_options_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def fw_options_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
vnet = str(values(inputs)["vnet"]) vnet = str(values(inputs)["vnet"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
return dict(item.get("firewall", {}).get("options") or {"enable": 0}) return dict(item.get("firewall", {}).get("options") or {"enable": 0})
async def fw_options_put(request: Request, inputs: dict[str, Any]) -> None: async def fw_options_put(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
options = dict(item.setdefault("firewall", {}).setdefault("options", {"enable": 0})) options = dict(item.setdefault("firewall", {}).setdefault("options", {"enable": 0}))
for key, value in payload.items(): for key, value in payload.items():
@@ -435,7 +450,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fw_rules_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def fw_rules_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
vnet = str(values(inputs)["vnet"]) vnet = str(values(inputs)["vnet"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
rules = item.get("firewall", {}).get("rules") or [] rules = item.get("firewall", {}).get("rules") or []
return list(rules) if isinstance(rules, list) else [] return list(rules) if isinstance(rules, list) else []
@@ -443,7 +458,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fw_rules_create(request: Request, inputs: dict[str, Any]) -> None: async def fw_rules_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
rules = item.setdefault("firewall", {}).setdefault("rules", []) rules = item.setdefault("firewall", {}).setdefault("rules", [])
if not isinstance(rules, list): if not isinstance(rules, list):
@@ -464,7 +479,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
pos = int(payload["pos"]) pos = int(payload["pos"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
rules = item.setdefault("firewall", {}).setdefault("rules", []) rules = item.setdefault("firewall", {}).setdefault("rules", [])
if not isinstance(rules, list) or pos < 0 or pos >= len(rules): if not isinstance(rules, list) or pos < 0 or pos >= len(rules):
@@ -479,7 +494,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
vnet = str(payload["vnet"]) vnet = str(payload["vnet"])
pos = int(payload["pos"]) pos = int(payload["pos"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = await _vnet(sdn, vnet) item = await _vnet(sdn, vnet)
rules = item.setdefault("firewall", {}).setdefault("rules", []) rules = item.setdefault("firewall", {}).setdefault("rules", [])
if not isinstance(rules, list) or pos < 0 or pos >= len(rules): if not isinstance(rules, list) or pos < 0 or pos >= len(rules):
@@ -492,7 +507,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
return subdirs("all", "fabric", "node") return subdirs("all", "fabric", "node")
async def fabrics_all(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def fabrics_all(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
return _store_list(sdn.get("fabrics") or {}, id_key="id") return _store_list(sdn.get("fabrics") or {}, id_key="id")
async def fabric_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def fabric_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
@@ -501,7 +516,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fabric_create(request: Request, inputs: dict[str, Any]) -> None: async def fabric_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
fabric_id = str(payload["id"]) fabric_id = str(payload["id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabrics", {}) store = sdn.setdefault("fabrics", {})
if fabric_id in store: if fabric_id in store:
raise ApiError(400, f"fabric '{fabric_id}' already exists") raise ApiError(400, f"fabric '{fabric_id}' already exists")
@@ -514,7 +529,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fabric_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def fabric_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
fabric_id = str(values(inputs)["id"]) fabric_id = str(values(inputs)["id"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("fabrics") or {}).get(fabric_id) item = (sdn.get("fabrics") or {}).get(fabric_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "fabric does not exist") raise ApiError(404, "fabric does not exist")
@@ -523,7 +538,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fabric_update(request: Request, inputs: dict[str, Any]) -> None: async def fabric_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
fabric_id = str(payload["id"]) fabric_id = str(payload["id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabrics", {}) store = sdn.setdefault("fabrics", {})
if fabric_id not in store: if fabric_id not in store:
raise ApiError(404, "fabric does not exist") raise ApiError(404, "fabric does not exist")
@@ -543,7 +558,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def fabric_delete(request: Request, inputs: dict[str, Any]) -> None: async def fabric_delete(request: Request, inputs: dict[str, Any]) -> None:
fabric_id = str(values(inputs)["id"]) fabric_id = str(values(inputs)["id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabrics", {}) store = sdn.setdefault("fabrics", {})
if fabric_id not in store: if fabric_id not in store:
raise ApiError(404, "fabric does not exist") raise ApiError(404, "fabric does not exist")
@@ -554,7 +569,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
await save_cluster_metadata(request, metadata) await save_cluster_metadata(request, metadata)
async def fabric_nodes_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def fabric_nodes_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
fabric_id = values(inputs).get("fabric_id") fabric_id = values(inputs).get("fabric_id")
nodes = sdn.get("fabric_nodes") or {} nodes = sdn.get("fabric_nodes") or {}
result: list[dict[str, Any]] = [] result: list[dict[str, Any]] = []
@@ -571,7 +586,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
fabric_id = str(payload["fabric_id"]) fabric_id = str(payload["fabric_id"])
node_id = str(payload["node_id"]) node_id = str(payload["node_id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
if fabric_id not in (sdn.get("fabrics") or {}): if fabric_id not in (sdn.get("fabrics") or {}):
raise ApiError(404, "fabric does not exist") raise ApiError(404, "fabric does not exist")
store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {}) store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {})
@@ -588,7 +603,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
fabric_id = str(payload["fabric_id"]) fabric_id = str(payload["fabric_id"])
node_id = str(payload["node_id"]) node_id = str(payload["node_id"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = ((sdn.get("fabric_nodes") or {}).get(fabric_id) or {}).get(node_id) item = ((sdn.get("fabric_nodes") or {}).get(fabric_id) or {}).get(node_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "fabric node does not exist") raise ApiError(404, "fabric node does not exist")
@@ -598,7 +613,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
fabric_id = str(payload["fabric_id"]) fabric_id = str(payload["fabric_id"])
node_id = str(payload["node_id"]) node_id = str(payload["node_id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {}) store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {})
if node_id not in store: if node_id not in store:
raise ApiError(404, "fabric node does not exist") raise ApiError(404, "fabric node does not exist")
@@ -620,7 +635,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
fabric_id = str(payload["fabric_id"]) fabric_id = str(payload["fabric_id"])
node_id = str(payload["node_id"]) node_id = str(payload["node_id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {}) store = sdn.setdefault("fabric_nodes", {}).setdefault(fabric_id, {})
if node_id not in store: if node_id not in store:
raise ApiError(404, "fabric node does not exist") raise ApiError(404, "fabric node does not exist")
@@ -630,13 +645,13 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
# prefix lists # prefix lists
async def prefix_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]: async def prefix_list(request: Request, _inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
return _store_list(sdn.get("prefix_lists") or {}, id_key="id") return _store_list(sdn.get("prefix_lists") or {}, id_key="id")
async def prefix_create(request: Request, inputs: dict[str, Any]) -> None: async def prefix_create(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
list_id = str(payload["id"]) list_id = str(payload["id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("prefix_lists", {}) store = sdn.setdefault("prefix_lists", {})
if list_id in store: if list_id in store:
raise ApiError(400, f"prefix-list '{list_id}' already exists") raise ApiError(400, f"prefix-list '{list_id}' already exists")
@@ -654,7 +669,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def prefix_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def prefix_get(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
list_id = str(values(inputs)["id"]) list_id = str(values(inputs)["id"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id) item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist") raise ApiError(404, "prefix-list does not exist")
@@ -663,7 +678,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def prefix_update(request: Request, inputs: dict[str, Any]) -> None: async def prefix_update(request: Request, inputs: dict[str, Any]) -> None:
payload = values(inputs) payload = values(inputs)
list_id = str(payload["id"]) list_id = str(payload["id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("prefix_lists", {}) store = sdn.setdefault("prefix_lists", {})
if list_id not in store: if list_id not in store:
raise ApiError(404, "prefix-list does not exist") raise ApiError(404, "prefix-list does not exist")
@@ -676,7 +691,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def prefix_delete(request: Request, inputs: dict[str, Any]) -> None: async def prefix_delete(request: Request, inputs: dict[str, Any]) -> None:
list_id = str(values(inputs)["id"]) list_id = str(values(inputs)["id"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
store = sdn.setdefault("prefix_lists", {}) store = sdn.setdefault("prefix_lists", {})
if list_id not in store: if list_id not in store:
raise ApiError(404, "prefix-list does not exist") raise ApiError(404, "prefix-list does not exist")
@@ -686,7 +701,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def prefix_entries(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def prefix_entries(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
list_id = str(values(inputs)["id"]) list_id = str(values(inputs)["id"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id) item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist") raise ApiError(404, "prefix-list does not exist")
@@ -699,7 +714,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
list_id = str(payload["id"]) list_id = str(payload["id"])
seq = str(payload.get("seq") or secrets.randbelow(10000)) seq = str(payload.get("seq") or secrets.randbelow(10000))
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id) item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist") raise ApiError(404, "prefix-list does not exist")
@@ -720,7 +735,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
list_id = str(payload["id"]) list_id = str(payload["id"])
seq = str(payload["url_seq"]) seq = str(payload["url_seq"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id) item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist") raise ApiError(404, "prefix-list does not exist")
@@ -733,7 +748,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
list_id = str(payload["id"]) list_id = str(payload["id"])
seq = str(payload["url_seq"]) seq = str(payload["url_seq"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id) item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist") raise ApiError(404, "prefix-list does not exist")
@@ -752,7 +767,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
list_id = str(payload["id"]) list_id = str(payload["id"])
seq = str(payload["url_seq"]) seq = str(payload["url_seq"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("prefix_lists") or {}).get(list_id) item = (sdn.get("prefix_lists") or {}).get(list_id)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "prefix-list does not exist") raise ApiError(404, "prefix-list does not exist")
@@ -767,16 +782,44 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def route_maps_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]: async def route_maps_index(_request: Request, _inputs: dict[str, Any]) -> list[dict[str, str]]:
return subdirs("entries") return subdirs("entries")
def _route_map_entries(item: object) -> dict[str, Any]:
if not isinstance(item, dict):
return {}
nested = item.get("entries")
if isinstance(nested, dict):
return {
str(key): value
for key, value in nested.items()
if isinstance(value, dict) and str(key).isdigit()
}
return {
str(key): value
for key, value in item.items()
if isinstance(value, dict) and str(key).isdigit()
}
def _route_map_entries_mutable(sdn: dict[str, Any], map_id: str) -> dict[str, Any]:
bucket = sdn.setdefault("route_maps", {}).setdefault(map_id, {})
if not isinstance(bucket, dict):
bucket = {}
sdn["route_maps"][map_id] = bucket
if "entries" in bucket or "name" in bucket:
entries = bucket.setdefault("entries", {})
if not isinstance(entries, dict):
entries = {}
bucket["entries"] = entries
return entries
return bucket
async def route_entries_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def route_entries_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
maps = sdn.get("route_maps") or {} maps = sdn.get("route_maps") or {}
route_map_id = values(inputs).get("route-map-id") route_map_id = values(inputs).get("route-map-id")
result: list[dict[str, Any]] = [] result: list[dict[str, Any]] = []
for map_id, entries in sorted(maps.items()): for map_id, raw_entries in sorted(maps.items()):
if route_map_id and map_id != route_map_id: if route_map_id and map_id != route_map_id:
continue continue
if not isinstance(entries, dict): entries = _route_map_entries(raw_entries)
continue
for order, entry in sorted(entries.items(), key=lambda pair: int(pair[0])): for order, entry in sorted(entries.items(), key=lambda pair: int(pair[0])):
result.append({"route-map-id": map_id, "order": int(order), **entry}) result.append({"route-map-id": map_id, "order": int(order), **entry})
return result return result
@@ -785,8 +828,8 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
map_id = str(payload["route-map-id"]) map_id = str(payload["route-map-id"])
order = str(payload.get("order") or 10) order = str(payload.get("order") or 10)
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
entries = sdn.setdefault("route_maps", {}).setdefault(map_id, {}) entries = _route_map_entries_mutable(sdn, map_id)
if order in entries: if order in entries:
raise ApiError(400, f"route-map entry '{order}' already exists") raise ApiError(400, f"route-map entry '{order}' already exists")
entries[order] = { entries[order] = {
@@ -809,8 +852,8 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
map_id = str(payload["route-map-id"]) map_id = str(payload["route-map-id"])
order = str(payload["order"]) order = str(payload["order"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
entry = ((sdn.get("route_maps") or {}).get(map_id) or {}).get(order) entry = _route_map_entries((sdn.get("route_maps") or {}).get(map_id)).get(order)
if not isinstance(entry, dict): if not isinstance(entry, dict):
raise ApiError(404, "route-map entry does not exist") raise ApiError(404, "route-map entry does not exist")
return {"route-map-id": map_id, "order": int(order), **entry} return {"route-map-id": map_id, "order": int(order), **entry}
@@ -819,8 +862,8 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
map_id = str(payload["route-map-id"]) map_id = str(payload["route-map-id"])
order = str(payload["order"]) order = str(payload["order"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
entries = sdn.setdefault("route_maps", {}).setdefault(map_id, {}) entries = _route_map_entries_mutable(sdn, map_id)
if order not in entries: if order not in entries:
raise ApiError(404, "route-map entry does not exist") raise ApiError(404, "route-map entry does not exist")
current = dict(entries[order]) current = dict(entries[order])
@@ -837,8 +880,8 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
map_id = str(payload["route-map-id"]) map_id = str(payload["route-map-id"])
order = str(payload["order"]) order = str(payload["order"])
metadata, sdn = await _load(request) metadata, sdn = await _load(request, for_write=True)
entries = sdn.setdefault("route_maps", {}).setdefault(map_id, {}) entries = _route_map_entries_mutable(sdn, map_id)
if order not in entries: if order not in entries:
raise ApiError(404, "route-map entry does not exist") raise ApiError(404, "route-map entry does not exist")
del entries[order] del entries[order]
@@ -852,13 +895,13 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def node_zones(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def node_zones(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
return _store_list(sdn.get("zones") or {}, id_key="zone") return _store_list(sdn.get("zones") or {}, id_key="zone")
async def node_zone(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_zone(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
zone = str(values(inputs)["zone"]) zone = str(values(inputs)["zone"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
item = (sdn.get("zones") or {}).get(zone) item = (sdn.get("zones") or {}).get(zone)
if not isinstance(item, dict): if not isinstance(item, dict):
raise ApiError(404, "zone does not exist") raise ApiError(404, "zone does not exist")
@@ -866,13 +909,16 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def node_zone_bridges(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def node_zone_bridges(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
zone = await node_zone(request, inputs) zone = await node_zone(request, inputs)
bridge = zone.get("bridge") or f"vmbr-{zone.get('zone')}" bridges = zone.get("bridges")
return [{"iface": bridge, "active": 1}] if isinstance(bridges, list):
return [dict(item) for item in bridges if isinstance(item, dict)]
bridge = zone.get("bridge")
return [{"iface": bridge, "active": 1}] if bridge else []
async def node_zone_content(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def node_zone_content(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
zone = str(values(inputs)["zone"]) zone = str(values(inputs)["zone"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
return [ return [
{"vnet": name, **item} {"vnet": name, **item}
for name, item in sorted((sdn.get("vnets") or {}).items()) for name, item in sorted((sdn.get("vnets") or {}).items())
@@ -881,7 +927,11 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def node_zone_ip_vrf(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_zone_ip_vrf(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
zone = await node_zone(request, inputs) zone = await node_zone(request, inputs)
return {"zone": zone.get("zone"), "vrf": f"vrf-{zone.get('zone')}", "table": 100} return {
"zone": zone.get("zone"),
"vrf": zone.get("vrf") or "",
"table": zone.get("table") if zone.get("table") is not None else 0,
}
async def node_vnet(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_vnet(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
@@ -889,7 +939,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
async def node_vnet_mac_vrf(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_vnet_mac_vrf(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
vnet = await node_vnet(request, inputs) vnet = await node_vnet(request, inputs)
return {"vnet": vnet.get("vnet"), "mac-vrf": f"macvrf-{vnet.get('vnet')}"} return {"vnet": vnet.get("vnet"), "mac-vrf": vnet.get("mac-vrf") or ""}
async def node_fabric(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_fabric(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
@@ -901,7 +951,7 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
fabric = str(values(inputs)["fabric"]) fabric = str(values(inputs)["fabric"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request, for_write=True)
nodes = (sdn.get("fabric_nodes") or {}).get(fabric) or {} nodes = (sdn.get("fabric_nodes") or {}).get(fabric) or {}
result = [] result = []
for node_id, item in nodes.items(): for node_id, item in nodes.items():
@@ -919,15 +969,23 @@ def register_sdn_handlers(registry: HandlerRegistry) -> None:
fabric = str(values(inputs)["fabric"]) fabric = str(values(inputs)["fabric"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request)
nodes = (sdn.get("fabric_nodes") or {}).get(fabric) or {} nodes = (sdn.get("fabric_nodes") or {}).get(fabric) or {}
return [{"node": node_id, "state": "up"} for node_id in sorted(nodes)] return [
{
"node": node_id,
"state": item.get("state") if isinstance(item, dict) else "",
}
for node_id, item in sorted(nodes.items())
]
async def node_fabric_routes(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def node_fabric_routes(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
await require_node(request, str(values(inputs)["node"])) await require_node(request, str(values(inputs)["node"]))
fabric = str(values(inputs)["fabric"]) fabric = str(values(inputs)["fabric"])
_metadata, sdn = await _load(request) _metadata, sdn = await _load(request)
item = (sdn.get("fabrics") or {}).get(fabric) or {} item = (sdn.get("fabrics") or {}).get(fabric) or {}
prefix = item.get("ip_prefix") or "10.0.0.0/24" routes = item.get("routes") if isinstance(item, dict) else None
return [{"dst": prefix, "protocol": item.get("protocol") or "ospf"}] if isinstance(routes, list):
return [dict(route) for route in routes if isinstance(route, dict)]
return []
# registrations # registrations
registry.register("/cluster/sdn", "GET", index) registry.register("/cluster/sdn", "GET", index)
+110 -27
View File
@@ -9,7 +9,15 @@ from fastapi import Request
from app.api.errors import ApiError from app.api.errors import ApiError
from app.api.registry import HandlerRegistry from app.api.registry import HandlerRegistry
from app.handlers.common import database, require_node, state, storage_payload, subdirs, values from app.handlers.common import (
database,
require_node,
require_value,
state,
storage_payload,
subdirs,
values,
)
from app.simulation.seed import CLUSTER_ID, stable_id from app.simulation.seed import CLUSTER_ID, stable_id
@@ -140,6 +148,7 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
JOIN resources r ON r.id = s.resource_id JOIN resources r ON r.id = s.resource_id
JOIN nodes n ON n.id = r.node_id JOIN nodes n ON n.id = r.node_id
WHERE n.name=$1 OR s.shared = true WHERE n.name=$1 OR s.shared = true
OR s.storage_id IN ('local', 'local-lvm')
ORDER BY s.storage_id""", ORDER BY s.storage_id""",
node, node,
) )
@@ -286,7 +295,55 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
if status != "DELETE 1": if status != "DELETE 1":
raise ApiError(404, "volume does not exist") raise ApiError(404, "volume does not exist")
async def node_storage_upload(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def node_storage_allocate(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import _node_task
node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"])
await require_node(request, node)
await _storage_row(request, None, storage_id)
payload = values(inputs)
filename = str(payload.get("filename") or f"vm-{payload.get('vmid', '0')}-disk-0")
content_type = str(payload.get("content") or "images")
size_raw = str(payload.get("size") or "1G")
# Accept either integer bytes or Proxmox size strings like 32G.
try:
size = int(size_raw)
except ValueError:
from app.handlers.common import parse_size_bytes
try:
size = parse_size_bytes(size_raw)
except ValueError as error:
raise ApiError(400, f"invalid size '{size_raw}'") from error
format_name = str(payload.get("format") or "raw")
volume_id = f"{storage_id}:{content_type}/{filename}"
resource_id = await database(request).pool.fetchval(
"SELECT resource_id FROM storages WHERE storage_id=$1",
storage_id,
)
await database(request).pool.execute(
"""INSERT INTO storage_contents(
id, storage_resource_id, volume_id, content_type, size_bytes, metadata
) VALUES(gen_random_uuid(), $1, $2, $3, $4, $5::jsonb)
ON CONFLICT (storage_resource_id, volume_id) DO UPDATE
SET size_bytes=EXCLUDED.size_bytes,
content_type=EXCLUDED.content_type,
metadata=EXCLUDED.metadata""",
resource_id,
volume_id,
content_type,
size,
json.dumps(
{"filename": filename, "format": format_name, "source": "allocate"},
sort_keys=True,
),
)
return await _node_task(request, node=node, task_type="storage-allocate", worker="imgcopy")
async def node_storage_upload(request: Request, inputs: dict[str, Any]) -> str:
from app.handlers.nodes import _node_task
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"]) storage_id = str(values(inputs)["storage"])
await require_node(request, node) await require_node(request, node)
@@ -318,11 +375,11 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
size, size,
json.dumps({"filename": filename, "source": "upload"}, sort_keys=True), json.dumps({"filename": filename, "source": "upload"}, sort_keys=True),
) )
return {"uploadid": volume_id, "filename": filename, "size": size, "volid": volume_id} return await _node_task(request, node=node, task_type="storage-upload", worker="download")
async def node_storage_prunebackups( async def node_storage_prunebackups(
request: Request, inputs: dict[str, Any] request: Request, inputs: dict[str, Any]
) -> list[dict[str, Any]] | None: ) -> list[dict[str, Any]] | str:
node = str(values(inputs)["node"]) node = str(values(inputs)["node"])
storage_id = str(values(inputs)["storage"]) storage_id = str(values(inputs)["storage"])
await require_node(request, node) await require_node(request, node)
@@ -332,6 +389,8 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
storage_id, storage_id,
) )
if request.method == "DELETE": if request.method == "DELETE":
from app.handlers.nodes import _node_task
keep = int(values(inputs).get("keep-last") or values(inputs).get("keep_last") or 1) keep = int(values(inputs).get("keep-last") or values(inputs).get("keep_last") or 1)
await database(request).pool.execute( await database(request).pool.execute(
"""DELETE FROM backups """DELETE FROM backups
@@ -344,7 +403,9 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
resource_id, resource_id,
keep, keep,
) )
return None return await _node_task(
request, node=node, task_type="prune-backups", worker="prunebackups"
)
rows = await database(request).pool.fetch( rows = await database(request).pool.fetch(
"""SELECT volume_id, size_bytes, created_at FROM backups """SELECT volume_id, size_bytes, created_at FROM backups
WHERE storage_resource_id=$1 ORDER BY created_at DESC""", WHERE storage_resource_id=$1 ORDER BY created_at DESC""",
@@ -485,18 +546,29 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
async def file_restore_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def file_restore_list(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = values(inputs) payload = values(inputs)
await require_node(request, str(payload["node"])) await require_node(request, str(payload["node"]))
await _storage_row(request, None, str(payload["storage"])) row = await _storage_row(request, None, str(payload["storage"]))
filepath = str(payload.get("filepath") or "/") filepath = str(payload.get("filepath") or "/")
return [{"filepath": filepath.rstrip("/") + "/etc", "type": "d", "text": "etc"}] config = state(row["config"])
restore = config.get("file_restore")
if not isinstance(restore, dict):
return []
items = restore.get(filepath) or restore.get(filepath.rstrip("/") or "/")
return [dict(item) for item in items] if isinstance(items, list) else []
async def file_restore_download(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def file_restore_download(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
await require_node(request, str(payload["node"])) await require_node(request, str(payload["node"]))
await _storage_row(request, None, str(payload["storage"])) row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
downloads = config.get("file_restore_downloads")
filepath = str(payload.get("filepath") or "/")
if isinstance(downloads, dict) and filepath in downloads:
entry = downloads[filepath]
return dict(entry) if isinstance(entry, dict) else {"filepath": filepath}
return { return {
"download-url": f"/api2/json/nodes/{payload['node']}/storage/" "download-url": f"/api2/json/nodes/{payload['node']}/storage/"
f"{payload['storage']}/file-restore/download", f"{payload['storage']}/file-restore/download",
"filepath": payload.get("filepath") or "/", "filepath": filepath,
"volume": payload.get("volume"), "volume": payload.get("volume"),
} }
@@ -504,40 +576,51 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
payload = values(inputs) payload = values(inputs)
await require_node(request, str(payload["node"])) await require_node(request, str(payload["node"]))
row = await _storage_row(request, None, str(payload["storage"])) row = await _storage_row(request, None, str(payload["storage"]))
config = state(row["config"])
identity = config.get("identity")
if isinstance(identity, dict):
return {
"storage": str(row["storage_id"]),
"type": str(identity.get("type") or row["storage_type"]),
"fingerprint": str(identity.get("fingerprint") or ""),
}
return { return {
"storage": str(row["storage_id"]), "storage": str(row["storage_id"]),
"type": str(row["storage_type"]), "type": str(row["storage_type"]),
"fingerprint": f"sim-{row['storage_id']}", "fingerprint": "",
} }
async def import_metadata(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def import_metadata(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
await require_node(request, str(payload["node"])) await require_node(request, str(payload["node"]))
storage_id = str(payload["storage"]) storage_id = str(payload["storage"])
volume = str(payload["volume"]) volume = str(require_value(payload, "volume"))
await _storage_row(request, None, storage_id) row = await _storage_row(request, None, storage_id)
return { config = state(row["config"])
"type": "qemu", by_volume = config.get("import_metadata_by_volume")
"source": volume, if isinstance(by_volume, dict) and volume in by_volume:
"disks": {"scsi0": f"{storage_id}:0/vm-import.raw"}, meta = by_volume[volume]
"net0": "virtio,bridge=vmbr0", return dict(meta) if isinstance(meta, dict) else {}
} meta = config.get("import_metadata")
if isinstance(meta, dict):
return {"source": volume, **meta}
return {}
async def storage_rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]: async def storage_rrd(request: Request, inputs: dict[str, Any]) -> dict[str, Any]:
payload = values(inputs) payload = values(inputs)
await require_node(request, str(payload["node"])) await require_node(request, str(payload["node"]))
storage_id = str(payload["storage"]) row = await _storage_row(request, None, str(payload["storage"]))
await _storage_row(request, None, storage_id) config = state(row["config"])
return {"filename": f"pve-storage-{storage_id}.rrd"} rrd_state = config.get("rrd")
return dict(rrd_state) if isinstance(rrd_state, dict) else {}
async def storage_rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]: async def storage_rrddata(request: Request, inputs: dict[str, Any]) -> list[dict[str, Any]]:
payload = values(inputs) payload = values(inputs)
await require_node(request, str(payload["node"])) await require_node(request, str(payload["node"]))
await _storage_row(request, None, str(payload["storage"])) row = await _storage_row(request, None, str(payload["storage"]))
return [ config = state(row["config"])
{"time": 1_700_000_000, "used": 10, "total": 100}, series = config.get("rrddata")
{"time": 1_700_000_060, "used": 12, "total": 100}, return [dict(item) for item in series] if isinstance(series, list) else []
]
registry.register("/storage", "GET", storage_ids) registry.register("/storage", "GET", storage_ids)
registry.register("/storage", "POST", storage_create) registry.register("/storage", "POST", storage_create)
@@ -548,7 +631,7 @@ def register_storage_handlers(registry: HandlerRegistry) -> None:
registry.register("/nodes/{node}/storage/{storage}", "GET", node_storage_index) registry.register("/nodes/{node}/storage/{storage}", "GET", node_storage_index)
registry.register("/nodes/{node}/storage/{storage}/status", "GET", node_storage_status) registry.register("/nodes/{node}/storage/{storage}/status", "GET", node_storage_status)
registry.register("/nodes/{node}/storage/{storage}/content", "GET", node_storage_content) registry.register("/nodes/{node}/storage/{storage}/content", "GET", node_storage_content)
registry.register("/nodes/{node}/storage/{storage}/content", "POST", node_storage_upload) registry.register("/nodes/{node}/storage/{storage}/content", "POST", node_storage_allocate)
registry.register( registry.register(
"/nodes/{node}/storage/{storage}/content/{volume}", "GET", node_storage_content_get "/nodes/{node}/storage/{storage}/content/{volume}", "GET", node_storage_content_get
) )
+28 -1
View File
@@ -25,6 +25,7 @@ from app.tasks.lxc import lxc_handler
from app.tasks.qemu import qemu_handler from app.tasks.qemu import qemu_handler
from app.tasks.repository import TaskRepository from app.tasks.repository import TaskRepository
from app.tasks.worker import TaskWorker from app.tasks.worker import TaskWorker
from app.version import get_app_version
from app.web.routes import router as web_router from app.web.routes import router as web_router
@@ -68,12 +69,15 @@ def create_app(
"qemu-stop": qemu, "qemu-stop": qemu,
"qemu-suspend": qemu, "qemu-suspend": qemu,
"qemu-update": qemu, "qemu-update": qemu,
"qemu-resize": qemu,
"qemu-template": qemu,
"lxc-clone": lxc, "lxc-clone": lxc,
"lxc-create": lxc, "lxc-create": lxc,
"lxc-delete": lxc, "lxc-delete": lxc,
"lxc-migrate": lxc, "lxc-migrate": lxc,
"lxc-reboot": lxc, "lxc-reboot": lxc,
"lxc-resume": lxc, "lxc-resume": lxc,
"lxc-resize": lxc,
"lxc-shutdown": lxc, "lxc-shutdown": lxc,
"lxc-snapshot-create": lxc, "lxc-snapshot-create": lxc,
"lxc-snapshot-delete": lxc, "lxc-snapshot-delete": lxc,
@@ -83,6 +87,29 @@ def create_app(
"lxc-suspend": lxc, "lxc-suspend": lxc,
"vzdump": backup, "vzdump": backup,
"aptupdate": backup, "aptupdate": backup,
"network-reload": backup,
"disk-initgpt": backup,
"disk-wipe": backup,
"disk-directory-create": backup,
"disk-directory-delete": backup,
"disk-lvm-create": backup,
"disk-lvm-delete": backup,
"disk-lvmthin-create": backup,
"disk-lvmthin-delete": backup,
"disk-zfs-create": backup,
"disk-zfs-delete": backup,
"service-start": backup,
"service-stop": backup,
"service-restart": backup,
"service-reload": backup,
"storage-allocate": backup,
"storage-upload": backup,
"prune-backups": backup,
"acme": backup,
"sdn-apply": backup,
"ceph-flags": backup,
"cluster-create": backup,
"cluster-join": backup,
}, },
concurrency=resolved.task_worker_concurrency, concurrency=resolved.task_worker_concurrency,
lease_seconds=resolved.task_lease_seconds, lease_seconds=resolved.task_lease_seconds,
@@ -91,7 +118,7 @@ def create_app(
resolved_workers = (task_worker,) resolved_workers = (task_worker,)
app = FastAPI( app = FastAPI(
title=resolved.app_name, title=resolved.app_name,
version="0.1.0", version=get_app_version(),
openapi_tags=openapi_tag_metadata(), openapi_tags=openapi_tag_metadata(),
lifespan=create_lifespan(resolved, database_factory, resolved_workers or ()), lifespan=create_lifespan(resolved, database_factory, resolved_workers or ()),
) )
+25
View File
@@ -0,0 +1,25 @@
"""Canonical Ceph cluster flag names (PVE /cluster/ceph/flags)."""
from __future__ import annotations
# Keep in sync with Proxmox `ceph_flags` API (majors 6-9).
CEPH_FLAG_DESCRIPTIONS: dict[str, str] = {
"nobackfill": "Pause backfill operations",
"nodeep-scrub": "Disable deep scrubbing",
"nodown": "Do not mark OSDs down",
"noin": "Do not mark OSDs in",
"noout": "Do not mark OSDs out",
"norebalance": "Pause rebalance",
"norecover": "Pause recovery",
"noscrub": "Disable scrubbing",
"notieragent": "Pause tiering agent",
"noup": "Do not mark OSDs up",
"pause": "Pause I/O",
}
def default_ceph_flags(*, enabled: frozenset[str] | None = None) -> dict[str, int]:
"""Return the full flag map; unset flags are ``0`` (PVE GET → ``false``)."""
active = enabled or frozenset()
return {name: (1 if name in active else 0) for name in CEPH_FLAG_DESCRIPTIONS}
+2020 -93
View File
File diff suppressed because it is too large Load Diff
+7 -7
View File
@@ -27,7 +27,7 @@ from app.contracts.model import Method, Snapshot
from app.db.migrations import migrate from app.db.migrations import migrate
from app.db.pool import AsyncpgDatabase from app.db.pool import AsyncpgDatabase
from app.main import create_app from app.main import create_app
from app.simulation.seed import apply_seed, small_profile from app.simulation.seed import apply_seed, lab_profile
from app.web.contract_catalog import get_major_releases from app.web.contract_catalog import get_major_releases
_BUNDLED_9 = Path( _BUNDLED_9 = Path(
@@ -119,7 +119,7 @@ async def prepare_db(url: str) -> None:
connection = await asyncpg.connect(url) connection = await asyncpg.connect(url)
try: try:
await migrate(connection) await migrate(connection)
await apply_seed(connection, small_profile()) await apply_seed(connection, lab_profile())
finally: finally:
await connection.close() await connection.close()
@@ -159,16 +159,16 @@ async def probe_major(
verb = method.verb.upper() verb = method.verb.upper()
url = f"/api2/json{render_path(path_template)}" url = f"/api2/json{render_path(path_template)}"
headers = {"CSRFPreventionToken": csrf} if verb != "GET" else {} headers = {"CSRFPreventionToken": csrf} if verb != "GET" else {}
body = body_for(method, path_template) if verb in {"PUT", "POST"} else None payload = body_for(method, path_template)
try: try:
if verb == "GET": if verb == "GET":
response = await client.get(url, headers=headers) response = await client.get(url, headers=headers, params=payload or None)
elif verb == "PUT": elif verb == "PUT":
response = await client.put(url, data=body or {}, headers=headers) response = await client.put(url, data=payload or {}, headers=headers)
elif verb == "POST": elif verb == "POST":
response = await client.post(url, data=body or {}, headers=headers) response = await client.post(url, data=payload or {}, headers=headers)
elif verb == "DELETE": elif verb == "DELETE":
response = await client.delete(url, headers=headers) response = await client.request("DELETE", url, data=payload or {}, headers=headers)
else: else:
continue continue
except Exception as exc: except Exception as exc:
+5
View File
@@ -43,6 +43,11 @@ def backup_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
await repository.append_log(task.id, "apt update finished") await repository.append_log(task.id, "apt update finished")
return {"status": "OK"} return {"status": "OK"}
# Node/cluster ops whose durable side effects were applied by the API handler.
if task.task_type != "vzdump":
await repository.append_log(task.id, f"{task.task_type} completed")
return {"status": "OK"}
node = str(task.payload["node"]) node = str(task.payload["node"])
vmids = [str(item) for item in task.payload.get("vmids", [])] vmids = [str(item) for item in task.payload.get("vmids", [])]
storage_id = str(task.payload.get("storage") or "nfs-backup") storage_id = str(task.payload.get("storage") or "nfs-backup")
+3
View File
@@ -29,6 +29,9 @@ def lxc_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
) )
if operation == "migrate" or operation == "remote-migrate": if operation == "migrate" or operation == "remote-migrate":
return await _migrate(repository, task, resource_id, clock) return await _migrate(repository, task, resource_id, clock)
if operation == "resize":
await repository.append_log(task.id, "container resize completed")
return {"status": "OK"}
async with repository.pool.acquire() as connection: async with repository.pool.acquire() as connection:
row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id) row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id)
if row is None: if row is None:
+7 -1
View File
@@ -33,6 +33,10 @@ def qemu_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
return await _migrate(repository, task, resource_id, clock) return await _migrate(repository, task, resource_id, clock)
if operation == "move-disk": if operation == "move-disk":
return await _move_disk(repository, task, resource_id) return await _move_disk(repository, task, resource_id)
if operation in {"resize", "template"}:
# Side effects already persisted by the API handler; complete the worker.
await repository.append_log(task.id, f"VM {operation} completed")
return {"status": "OK"}
async with repository.pool.acquire() as connection: async with repository.pool.acquire() as connection:
row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id) row = await connection.fetchrow("SELECT state FROM resources WHERE id=$1", resource_id)
if row is None: if row is None:
@@ -62,10 +66,12 @@ def qemu_handler(repository: TaskRepository, clock: Clock) -> TaskHandler:
async def _create(repository: TaskRepository, task: Task) -> dict[str, Any]: async def _create(repository: TaskRepository, task: Task) -> dict[str, Any]:
from app.simulation.seed import enrich_guest_state
node, vmid = str(task.payload["node"]), int(task.payload["vmid"]) node, vmid = str(task.payload["node"]), int(task.payload["vmid"])
config = dict(task.payload.get("config", {})) config = dict(task.payload.get("config", {}))
resource_id = uuid.uuid4() resource_id = uuid.uuid4()
state = {"status": "stopped", **config} state = enrich_guest_state({"status": "stopped", **config}, kind="qemu", vmid=str(vmid))
async with repository.pool.acquire() as connection: async with repository.pool.acquire() as connection:
async with connection.transaction(): async with connection.transaction():
node_row = await connection.fetchrow( node_row = await connection.fetchrow(
+130
View File
@@ -0,0 +1,130 @@
"""SemVer source of truth: VERSION file (synced to pyproject + Helm)."""
from __future__ import annotations
import re
from functools import lru_cache
from pathlib import Path
_VERSION_FILE = Path(__file__).resolve().parents[1] / "VERSION"
_PROJECT_ROOT = _VERSION_FILE.parent
_PYPROJECT_FILE = _PROJECT_ROOT / "pyproject.toml"
_CHART_FILE = _PROJECT_ROOT / "helm" / "proxmox-api-simulator" / "Chart.yaml"
_VALUES_FILE = _PROJECT_ROOT / "helm" / "proxmox-api-simulator" / "values.yaml"
_DEFAULT_VERSION = "0.1.0"
_VERSION_RE = re.compile(r"^(\d+)\.(\d+)\.(\d+)$")
def _read_version_file(path: Path) -> str | None:
try:
text = path.read_text(encoding="utf-8").strip()
except OSError:
return None
if _VERSION_RE.match(text):
return text
return None
@lru_cache
def get_app_version() -> str:
for path in (
Path("/app/VERSION"),
_VERSION_FILE,
Path.cwd() / "VERSION",
):
found = _read_version_file(path)
if found:
return found
try:
from importlib.metadata import version
meta = version("proxmox-api-simulator")
if _VERSION_RE.match(meta):
return meta
except Exception:
pass
return _DEFAULT_VERSION
def get_app_version_label() -> str:
clear_version_cache()
return f"v{get_app_version()}"
def parse_version(value: str) -> tuple[int, int, int]:
match = _VERSION_RE.match(value.strip())
if not match:
raise ValueError(f"Invalid semantic version: {value!r}")
return int(match.group(1)), int(match.group(2)), int(match.group(3))
def format_version(major: int, minor: int, patch: int) -> str:
return f"{major}.{minor}.{patch}"
def clear_version_cache() -> None:
get_app_version.cache_clear()
def write_project_version(version: str) -> None:
parse_version(version)
_VERSION_FILE.write_text(f"{version}\n", encoding="utf-8")
pyproject = _PYPROJECT_FILE.read_text(encoding="utf-8")
pyproject, count = re.subn(
r'^version = ".*"$',
f'version = "{version}"',
pyproject,
count=1,
flags=re.MULTILINE,
)
if count != 1:
raise RuntimeError("Failed to update pyproject.toml version")
_PYPROJECT_FILE.write_text(pyproject, encoding="utf-8")
chart = _CHART_FILE.read_text(encoding="utf-8")
chart, chart_count = re.subn(
r"^version: .*$",
f"version: {version}",
chart,
count=1,
flags=re.MULTILINE,
)
chart, app_count = re.subn(
r'^appVersion: ".*"$',
f'appVersion: "{version}"',
chart,
count=1,
flags=re.MULTILINE,
)
if chart_count != 1 or app_count != 1:
raise RuntimeError("Failed to update Chart.yaml version")
_CHART_FILE.write_text(chart, encoding="utf-8")
values = _VALUES_FILE.read_text(encoding="utf-8")
values, values_count = re.subn(
r'^(\s*tag:\s*).*$',
rf'\1"{version}"',
values,
count=1,
flags=re.MULTILINE,
)
if values_count != 1:
raise RuntimeError("Failed to update helm values.yaml image.tag")
_VALUES_FILE.write_text(values, encoding="utf-8")
clear_version_cache()
def bump_patch_version() -> str:
major, minor, patch = parse_version(get_app_version())
version = format_version(major, minor, patch + 1)
write_project_version(version)
return version
def bump_minor_version() -> str:
major, minor, patch = parse_version(get_app_version())
version = format_version(major, minor + 1, 0)
write_project_version(version)
return version
+140 -12
View File
@@ -11,7 +11,7 @@ from pathlib import Path
from app.api.openapi import contract_openapi_tag from app.api.openapi import contract_openapi_tag
from app.config import Settings from app.config import Settings
from app.contracts.examples import path_param_example, schema_example from app.contracts.examples import path_param_example, schema_example, wire_param_name
from app.contracts.importer import RemoteSourceImporter from app.contracts.importer import RemoteSourceImporter
from app.contracts.model import Method, Parameter, Snapshot from app.contracts.model import Method, Parameter, Snapshot
from app.contracts.normalize import normalize_snapshot from app.contracts.normalize import normalize_snapshot
@@ -102,8 +102,10 @@ def list_majors(
*, *,
runtime_version: str | None, runtime_version: str | None,
settings: Settings | None = None, settings: Settings | None = None,
app_version: str | None = None,
) -> dict[str, object]: ) -> dict[str, object]:
return { return {
"app_version": app_version,
"runtime_version": runtime_version, "runtime_version": runtime_version,
"majors": [ "majors": [
{ {
@@ -226,16 +228,25 @@ def _path_param_names(path: str) -> tuple[str, ...]:
return tuple(match.group(1) for match in _PATH_PARAM.finditer(path)) return tuple(match.group(1) for match in _PATH_PARAM.finditer(path))
def _parameter_payload(parameter: Parameter) -> dict[str, object]: def _parameter_payload(parameter: Parameter, *, wire_name: str | None = None) -> dict[str, object]:
schema = parameter.definition schema = parameter.definition
return { name = wire_name or parameter.name
"name": parameter.name, typetext = schema.extra.get("typetext")
payload: dict[str, object] = {
"name": name,
"type": schema.type, "type": schema.type,
"description": schema.description, "description": schema.description,
"optional": bool(schema.optional), "optional": bool(schema.optional),
"enum": list(schema.enum), "enum": list(schema.enum),
"example": schema_example(schema, name=parameter.name), "example": schema_example(schema, name=parameter.name),
} }
if wire_name and wire_name != parameter.name:
payload["template"] = parameter.name
if isinstance(schema.format, str):
payload["format"] = schema.format
if isinstance(typetext, str) and typetext:
payload["typetext"] = typetext
return payload
def method_payload( def method_payload(
@@ -274,15 +285,20 @@ def method_payload(
"example": path_param_example(name) or name, "example": path_param_example(name) or name,
} }
) )
body_fields = [ contract_body_fields = [
_parameter_payload(parameter) _parameter_payload(parameter)
for parameter in method.parameters for parameter in method.parameters
if parameter.name not in path_params and "[n]" not in parameter.name if parameter.name not in path_params and "[n]" not in parameter.name
] ]
# Contract uses foo[n]; the wire form is foo0..fooN — seed the console with foo0.
indexed_fields = [ indexed_fields = [
_parameter_payload(parameter) for parameter in method.parameters if "[n]" in parameter.name _parameter_payload(parameter, wire_name=wire_param_name(parameter.name))
for parameter in method.parameters
if "[n]" in parameter.name
] ]
body_example = _body_example(method, path_params) body_example = _body_example(method, path_params)
# Flatten nested scalars from body_example (and nested field examples) into PARAMS.
body_fields = _body_fields_with_nested(contract_body_fields, body_example)
resolved_path = _resolve_path(path, path_fields) resolved_path = _resolve_path(path, path_fields)
implemented = ( implemented = (
(path, method.verb.upper()) in implemented_methods (path, method.verb.upper()) in implemented_methods
@@ -316,21 +332,133 @@ def _resolve_path(path: str, path_fields: list[dict[str, object]]) -> str:
def _body_example(method: Method, path_params: tuple[str, ...]) -> dict[str, object]: def _body_example(method: Method, path_params: tuple[str, ...]) -> dict[str, object]:
"""Minimal request body: required parameters only, Proxmox wire names/values."""
body: dict[str, object] = {} body: dict[str, object] = {}
for parameter in method.parameters: for parameter in method.parameters:
if parameter.name in path_params: if parameter.name in path_params:
continue continue
if "[n]" in parameter.name:
concrete = parameter.name.replace("[n]", "0")
if not parameter.definition.optional:
body[concrete] = schema_example(parameter.definition, name=concrete)
continue
if parameter.definition.optional: if parameter.definition.optional:
continue continue
body[parameter.name] = schema_example(parameter.definition, name=parameter.name) wire = wire_param_name(parameter.name)
body[wire] = schema_example(parameter.definition, name=parameter.name)
return body return body
def _leaf_type(value: object) -> str:
if isinstance(value, bool):
return "boolean"
if isinstance(value, int) and not isinstance(value, bool):
return "integer"
if isinstance(value, float):
return "number"
return "string"
def _body_fields_from_example(body_example: dict[str, object] | None) -> list[dict[str, object]]:
"""PARAM inputs derived from body_example, including nested scalar paths."""
if not isinstance(body_example, dict) or not body_example:
return []
inner: object = body_example
# Soft unwrap a single root object envelope (Engine-style), never a scalar/list.
if len(body_example) == 1:
only = next(iter(body_example.values()))
if isinstance(only, dict):
inner = only
if not isinstance(inner, dict):
return []
fields: list[dict[str, object]] = []
def _walk(prefix: str, value: object) -> None:
if isinstance(value, dict):
for key, child in value.items():
path = f"{prefix}.{key}" if prefix else str(key)
_walk(path, child)
return
if isinstance(value, list):
for index, child in enumerate(value):
path = f"{prefix}.{index}" if prefix else str(index)
_walk(path, child)
return
fields.append(
{
"name": prefix,
"type": _leaf_type(value),
"description": prefix,
"optional": True,
"enum": [],
"example": value,
}
)
_walk("", inner)
return fields
def _body_fields_with_nested(
contract_fields: list[dict[str, object]],
body_example: dict[str, object],
) -> list[dict[str, object]]:
"""Merge contract PARAMS with nested scalar paths from body_example."""
example_fields = _body_fields_from_example(body_example)
nested_tops = {
str(item["name"]).split(".", 1)[0] for item in example_fields if "." in str(item["name"])
}
contract_by_name = {str(item["name"]): item for item in contract_fields}
merged: list[dict[str, object]] = []
seen: set[str] = set()
for item in example_fields:
name = str(item["name"])
top = name.split(".", 1)[0]
parent = contract_by_name.get(name) or contract_by_name.get(top)
row = dict(item)
if parent is not None:
row["optional"] = bool(parent.get("optional"))
description = parent.get("description")
if isinstance(description, str) and description:
row["description"] = description
typetext = parent.get("typetext")
if isinstance(typetext, str) and typetext:
row["typetext"] = typetext
fmt = parent.get("format")
if isinstance(fmt, str) and fmt:
row["format"] = fmt
else:
# body_example is required-only — treat uncovered leaves as required.
row["optional"] = False
merged.append(row)
seen.add(name)
for field in contract_fields:
name = str(field["name"])
if name in nested_tops or name in seen:
continue
example = field.get("example")
if isinstance(example, dict | list):
nested = _body_fields_from_example({name: example})
if nested:
for item in nested:
nested_name = str(item["name"])
if nested_name in seen:
continue
row = dict(item)
row["optional"] = bool(field.get("optional"))
description = field.get("description")
if isinstance(description, str) and description:
row["description"] = description
merged.append(row)
seen.add(nested_name)
continue
# Empty object/array examples stay as a top-level PARAM.
merged.append(field)
seen.add(name)
return merged
@lru_cache(maxsize=1) @lru_cache(maxsize=1)
def default_store_root() -> Path: def default_store_root() -> Path:
return _DEFAULT_STORE return _DEFAULT_STORE
+686 -87
View File
File diff suppressed because it is too large Load Diff
+26 -3
View File
@@ -16,6 +16,7 @@ from app.contracts.source import SourceError
from app.db.pool import AsyncpgDatabase from app.db.pool import AsyncpgDatabase
from app.dependencies import get_database from app.dependencies import get_database
from app.simulation.seed import apply_seed, build_profile, simulation_state_summary from app.simulation.seed import apply_seed, build_profile, simulation_state_summary
from app.version import get_app_version
from app.web.assets import console_html from app.web.assets import console_html
from app.web.compatibility_catalog import compatibility_payload from app.web.compatibility_catalog import compatibility_payload
from app.web.contract_catalog import catalog_payload, list_majors, load_snapshot, method_payload from app.web.contract_catalog import catalog_payload, list_majors, load_snapshot, method_payload
@@ -37,7 +38,13 @@ async def console() -> HTMLResponse:
async def ui_versions(request: Request) -> JSONResponse: async def ui_versions(request: Request) -> JSONResponse:
settings = _settings(request) settings = _settings(request)
runtime_version = _runtime_version(request) runtime_version = _runtime_version(request)
return JSONResponse(list_majors(runtime_version=runtime_version, settings=settings)) return JSONResponse(
list_majors(
runtime_version=runtime_version,
settings=settings,
app_version=get_app_version(),
)
)
@router.get("/ui/api/catalog", include_in_schema=False) @router.get("/ui/api/catalog", include_in_schema=False)
@@ -161,7 +168,23 @@ async def ui_demo_state(request: Request) -> JSONResponse:
@router.post("/ui/api/demo/load", include_in_schema=False) @router.post("/ui/api/demo/load", include_in_schema=False)
async def ui_demo_load(request: Request) -> JSONResponse: async def ui_demo_load(request: Request) -> JSONResponse:
pool = _database_pool(request) pool = _database_pool(request)
profile = build_profile("demo-cluster") requested = "small"
if request.query_params.get("profile"):
requested = str(request.query_params.get("profile"))
else:
try:
body = await request.json()
except Exception:
body = None
if isinstance(body, dict) and body.get("profile"):
requested = str(body["profile"])
allowed = {"small", "large", "big"}
if requested not in allowed:
raise HTTPException(
status_code=400,
detail=f"unknown cluster profile '{requested}' (expected: small, large, big)",
)
profile = build_profile(requested)
async with pool.acquire() as connection: async with pool.acquire() as connection:
await apply_seed(connection, profile) await apply_seed(connection, profile)
summary = await simulation_state_summary(connection) summary = await simulation_state_summary(connection)
@@ -179,7 +202,7 @@ async def ui_demo_unload(request: Request) -> JSONResponse:
summary = await simulation_state_summary(connection) summary = await simulation_state_summary(connection)
except Exception as error: except Exception as error:
raise HTTPException( raise HTTPException(
status_code=500, detail=f"failed to remove demo data: {error}" status_code=500, detail=f"failed to reset to minimal cluster: {error}"
) from error ) from error
return JSONResponse({"ok": True, "profile": profile.name, "summary": summary}) return JSONResponse({"ok": True, "profile": profile.name, "summary": summary})
+51 -2
View File
@@ -1,12 +1,28 @@
# Quick start with the published Docker Hub runtime image. # Quick start with the published Docker Hub runtime image.
# #
# WARNING: Laboratory / CI only.
# Default TICKET_SIGNING_KEY and PostgreSQL password are intentional lab
# defaults. Do NOT expose host :8006 to untrusted networks without replacing
# secrets and adding your own controls. See SECURITY.md and docs/security.md.
#
# Requires this repository checkout (compose file).
#
# docker compose -f docker-compose.release.yml up -d # docker compose -f docker-compose.release.yml up -d
# docker compose -f docker-compose.release.yml run --rm --entrypoint python simulator -m app.simulation.seed_cli # docker compose -f docker-compose.release.yml run --rm --entrypoint python \
# simulator -m app.simulation.seed_cli
# curl -sS http://localhost:8006/health/ready
# #
# Override the image tag: # Override the image tag:
# IMAGE_TAG=0.1.0 docker compose -f docker-compose.release.yml up -d # IMAGE_TAG=0.1.0 docker compose -f docker-compose.release.yml up -d
# #
# Change TICKET_SIGNING_KEY before exposing the stack beyond a local lab. # Rotate lab secrets before any shared or networked demo:
# TICKET_SIGNING_KEY=$(openssl rand -hex 32) \
# POSTGRES_PASSWORD=$(openssl rand -hex 16) \
# docker compose -f docker-compose.release.yml up -d
#
# Optional HTTPS for proxmoxer-style clients (separate port):
# docker compose -f docker-compose.release.yml --profile tls up -d
# curl -sk https://localhost:8443/health/ready
name: proxmox-api-simulator-release name: proxmox-api-simulator-release
@@ -18,6 +34,7 @@ x-app-env: &app-env
CONTRACT_SNAPSHOT: /app/contracts/pve-9.2.3.json CONTRACT_SNAPSHOT: /app/contracts/pve-9.2.3.json
COMPATIBILITY_EVIDENCE: /app/evidence/pve-9.2.3.json COMPATIBILITY_EVIDENCE: /app/evidence/pve-9.2.3.json
LOG_LEVEL: ${LOG_LEVEL:-INFO} LOG_LEVEL: ${LOG_LEVEL:-INFO}
# Lab default — replace for any shared or networked use.
TICKET_SIGNING_KEY: ${TICKET_SIGNING_KEY:-development-only-signing-key-change-me} TICKET_SIGNING_KEY: ${TICKET_SIGNING_KEY:-development-only-signing-key-change-me}
TASK_WORKER_CONCURRENCY: ${TASK_WORKER_CONCURRENCY:-2} TASK_WORKER_CONCURRENCY: ${TASK_WORKER_CONCURRENCY:-2}
SIMULATION_TIME_SCALE: ${SIMULATION_TIME_SCALE:-10} SIMULATION_TIME_SCALE: ${SIMULATION_TIME_SCALE:-10}
@@ -86,3 +103,35 @@ services:
start_period: 20s start_period: 20s
ports: ports:
- "${SIMULATOR_PORT:-8006}:8006" - "${SIMULATOR_PORT:-8006}:8006"
tls-gateway:
profiles: [tls]
image: nginx:1.28.0-alpine
restart: unless-stopped
networks: [simulator]
depends_on:
simulator:
condition: service_healthy
ports:
- "${TLS_GATEWAY_PORT:-8443}:8443"
volumes:
- ./docker/tls/gateway.conf:/etc/nginx/conf.d/default.conf:ro
- ./docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
- ./docker/tls/server.key:/etc/nginx/tls/server.key:ro
healthcheck:
test:
[
"CMD-SHELL",
"wget -qO- --no-check-certificate https://127.0.0.1:8443/health/live || exit 1",
]
interval: 10s
timeout: 3s
retries: 5
start_period: 5s
read_only: true
tmpfs:
- /var/cache/nginx
- /tmp
- /var/run
security_opt:
- no-new-privileges:true
+8 -4
View File
@@ -12,8 +12,9 @@ x-dev-env: &dev-env
TEST_DATABASE_URL: postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator TEST_DATABASE_URL: postgresql://proxmox:proxmox@postgres:5432/proxmox_simulator
CONTRACT_SNAPSHOT: /workspace/contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json CONTRACT_SNAPSHOT: /workspace/contracts/e61a893e996d05d376579226e7dfbedbcfce8b71787adacffbc557e6e35901c1/snapshot.json
COMPATIBILITY_EVIDENCE: /workspace/evidence/pve-9.2.3.json COMPATIBILITY_EVIDENCE: /workspace/evidence/pve-9.2.3.json
PROXMOXER_HOST: tls-gateway # proxmoxer is HTTPS-only — set when using --profile tls (see Makefile test-compatibility)
PROXMOXER_PORT: "8443" PROXMOXER_HOST: ${PROXMOXER_HOST:-}
PROXMOXER_PORT: ${PROXMOXER_PORT:-8443}
LOG_LEVEL: INFO LOG_LEVEL: INFO
TICKET_SIGNING_KEY: development-only-signing-key-change-me TICKET_SIGNING_KEY: development-only-signing-key-change-me
@@ -108,9 +109,12 @@ services:
retries: 8 retries: 8
start_period: 20s start_period: 20s
ports: ports:
- "8006:8006" - "${SIMULATOR_PORT:-8006}:8006"
# Optional HTTPS front for clients that cannot speak plain HTTP (proxmoxer).
# Default lab URL is http://localhost:8006/ — K8s TLS is Ingress-only.
tls-gateway: tls-gateway:
profiles: [tls]
image: nginx:1.28.0-alpine image: nginx:1.28.0-alpine
restart: unless-stopped restart: unless-stopped
networks: [simulator] networks: [simulator]
@@ -118,7 +122,7 @@ services:
simulator: simulator:
condition: service_healthy condition: service_healthy
ports: ports:
- "8007:8443" - "${TLS_GATEWAY_PORT:-8443}:8443"
volumes: volumes:
- ./docker/tls/gateway.conf:/etc/nginx/conf.d/default.conf:ro - ./docker/tls/gateway.conf:/etc/nginx/conf.d/default.conf:ro
- ./docker/tls/server.crt:/etc/nginx/tls/server.crt:ro - ./docker/tls/server.crt:/etc/nginx/tls/server.crt:ro
+23 -2
View File
@@ -1,5 +1,26 @@
# Development HTTPS gateway for proxmoxer and other TLS clients. # Optional / internal TLS terminator.
# Upstream hostnames are resolved at request time via Docker embedded DNS. #
# - Host lab default is plain HTTP :8006 on the simulator (no TLS).
# - Kubernetes HTTPS terminates at Ingress (cert-manager).
# - This gateway is for HTTPS-only clients (proxmoxer, pulumi-proxmoxve).
# - Also listen on :80: some bridged providers rewrite https://host:8443 → http://host
# on later requests (delete); without :80 those calls fail.
server {
listen 80;
server_name _;
resolver 127.0.0.11 valid=10s ipv6=off;
location / {
set $upstream simulator:8006;
proxy_pass http://$upstream;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-ID $request_id;
}
}
server { server {
listen 8443 ssl; listen 8443 ssl;
+39
View File
@@ -0,0 +1,39 @@
**Language / Язык:** [English](README.md) | [Русский](ru/README.md)
# Documentation
Guides for the Proxmox VE API simulator. Switch language with the header on each
page. Russian mirrors live under [`ru/`](ru/README.md).
![Web UI main](images/web-ui-main.png)
| Guide | Description |
|---|---|
| [Getting started](getting-started.md) | First successful lab session |
| [Configuration](configuration.md) | Environment variables and Compose |
| [Authentication](authentication.md) | Tickets, CSRF, API tokens, ACLs |
| [API versions](api-versions.md) | Contracts 6–9 and hot-swap |
| [Clients & examples](clients.md) | Python, Go, Java, Perl, Ansible, Terraform, Pulumi |
| [Seed profiles](seed-profiles.md) | Deterministic cluster fixtures |
| [API surface](api-surface.md) | Routing, handlers, fallbacks |
| [Domains](domains/README.md) | QEMU, LXC, storage, HA, SDN, … |
| [Web UI](web-ui.md) | Interactive console and catalogs |
| [Operations](operations.md) | Migrate, reseed, upgrade, Hub publish |
| [CI/CD (Jenkins)](ci-cd.md) | VERSION, Harbor/Hub push, deploy to proxmox.devops.org.ru |
| [Docker Hub overview](docker-hub-overview.md) | Paste-ready Hub repository description |
| [Kubernetes / Helm](kubernetes.md) | Hub image + Ingress + Let's Encrypt |
| [Security](security.md) | Lab threat model and credentials |
| [Observability](observability.md) | Health endpoints and logging |
| [Troubleshooting](troubleshooting.md) | Common failure modes |
| [FAQ](faq.md) | Short answers |
| [Architecture](architecture.md) | Component boundaries |
| [Compatibility](compatibility.md) | Evidence model and release matrix |
| [Testing](testing.md) | Suites layout, Make targets, latest results |
Runnable cookbooks: [`examples/`](../examples/README.md).
Integration suites: [`pulumi-tests/`](../pulumi-tests/README.md).
Test map and latest pass notes: [Testing](testing.md).
## See also
- [Web UI](web-ui.md) — interactive console, catalogs, DATA panel, and screenshots
+4 -1
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](api-surface.md) | [Русский](ru/api-surface.md)
# API surface # API surface
## Request path ## Request path
@@ -33,7 +35,8 @@ not part of the product contract. See the workspace durable-simulator rule.
- Interactive FastAPI docs: `/docs` - Interactive FastAPI docs: `/docs`
- Web UI method inspector: `/` → catalog → method - Web UI method inspector: `/` → catalog → method
- UI APIs: `/ui/api/catalog`, `/ui/api/method`, `/ui/api/compatibility` - UI APIs: `/ui/api/versions`, `/ui/api/catalog`, `/ui/api/method`,
`/ui/api/compatibility`, `/ui/api/contract/apply`, `/ui/api/demo/*`
## Compatibility endpoints ## Compatibility endpoints
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](api-versions.md) | [Русский](ru/api-versions.md)
# API versions (PVE 6–9) # API versions (PVE 6–9)
The simulator ships authoritative imported contracts for four Proxmox VE majors. The simulator ships authoritative imported contracts for four Proxmox VE majors.
+3 -1
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](architecture.md) | [Русский](ru/architecture.md)
# Architecture # Architecture
## Goals ## Goals
@@ -30,7 +32,7 @@ flowchart LR
Obs["Logs / Prometheus / OpenTelemetry"] Obs["Logs / Prometheus / OpenTelemetry"]
Client -->|"/api2/json"| API Client -->|"/api2/json"| API
Admin -->|"CLI and /_simulator"| API Admin -->|"CLI, Make/Helm, Web UI /ui/api"| API
Docs -->|"explicit import only"| Importer Docs -->|"explicit import only"| Importer
Importer --> Contract Importer --> Contract
Contract --> DB Contract --> DB
+5 -2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](authentication.md) | [Русский](ru/authentication.md)
# Authentication # Authentication
The simulator implements Proxmox-compatible ticket and API-token authentication The simulator implements Proxmox-compatible ticket and API-token authentication
@@ -54,8 +56,9 @@ beyond its owner.
## Seeded development principals ## Seeded development principals
Loaded by every standard seed profile (unless replaced by UI demo unload → Seeded for **every** profile — including `minimal` and after Web UI demo unload.
`minimal`): Unload shrinks guests/nodes/storages; lab principals and tokens are still
inserted by `apply_seed`:
| Principal | Password | Token | Notes | | Principal | Password | Token | Notes |
|---|---|---|---| |---|---|---|---|
+78
View File
@@ -0,0 +1,78 @@
**Language / Язык:** [English](ci-cd.md) | [Русский](ru/ci-cd.md)
# CI/CD (Jenkins + Gitea)
SemVer lives in [`VERSION`](../VERSION). Bump locally with `make bump-patch` /
`make bump-minor` (also syncs `pyproject.toml` and Helm chart/values). Jenkins
does **not** bump — it builds whatever is in the commit.
Public lab URL after deploy: **https://proxmox.devops.org.ru/**
## Pipeline
Single [`Jenkinsfile`](../Jenkinsfile) on `main` / `master`:
1. **Build & push** (agent `docker` / DinD) — multi-arch `linux/amd64,linux/arm64`
to Harbor and Docker Hub with tags **`:{VERSION}`** and **`:latest`** only.
2. **Deploy** — `kubectl set image` on `simulators/simulators-proxmox` (main
container `simulator` + initContainer `migrate`), then rollout status
(`--timeout=1200s`).
First cluster install (Ingress, Postgres, secrets) is **not** this job — use
`make addon-simulators` in DevOpsTools/K3S (`simulators_proxmox_host:
proxmox.devops.org.ru`).
### Images
| Registry | Repository |
|---|---|
| Harbor | `hub.antropoff.ru/devops-tools/proxmox-api-simulator` |
| Docker Hub | `inecs/proxmox-api-simulator` |
### Credentials (Jenkins Global)
| ID | Type | Use |
|---|---|---|
| `harbor-devops-tools-push-pull-access` | Username/password | Harbor robot push |
| `docker-hub` | Username/password | Docker Hub `inecs` |
| `k3s-kubeconfig` | Secret file | Kubeconfig for deploy |
| `ssh-gitea-key` | SSH private key | Gitea SCM checkout |
## Gitea setup
1. Ensure the repo is on Gitea (mirror or primary) with branch `main`.
2. Deploy key / Jenkins credential `ssh-gitea-key` can clone the repo
(`git@…:…/proxmox_api_simulator.git` or your path).
3. Optional: webhook from Gitea → Jenkins Multibranch (or rely on Jenkins SCM
polling / Organization Folder).
## Jenkins setup
1. **Credentials** — create the four IDs above if missing (same as Wrapped).
2. **Agent** — Kubernetes cloud pod template with label `docker` and a
`docker` container (DinD / buildx), same as Wrapped.
3. **Job** — Multibranch Pipeline (or Pipeline from SCM):
- Script Path: `Jenkinsfile`
- Branch discover: `main` (and `master` if needed)
- SCM: Gitea SSH URL + credential `ssh-gitea-key`
4. **Harbor project** — `devops-tools` must allow the robot to push
`proxmox-api-simulator` (create the repository on first push or pre-create).
5. **Docker Hub** — `inecs/proxmox-api-simulator` exists / push rights for
credential `docker-hub`.
6. **Cluster** — release `simulators` in namespace `simulators` already
installed (`make addon-simulators`). Deployment name must be
`simulators-proxmox`.
7. Run once on `main` after bumping `VERSION` and pushing. Confirm:
- Harbor/Hub tags `:0.x.y` and `:latest`
- `kubectl -n simulators get deploy,ing`
- Help → About shows badge `v0.x.y` on https://proxmox.devops.org.ru/
## Local release (without Jenkins)
```bash
make bump-patch # or bump-minor
make version-commit
git push origin HEAD
# or manual Hub-only:
make release # DOCKER_IMAGE=inecs/proxmox-api-simulator
```
+18 -27
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](clients.md) | [Русский](ru/clients.md)
# Clients # Clients
Use the simulator from common automation stacks. Each cookbook aims for the Use the simulator from common automation stacks. Each cookbook aims for the
@@ -13,39 +15,28 @@ same laboratory flow where the tool allows it:
## Connection matrix ## Connection matrix
| Stack | Transport | Notes | Docs | Code | Real Proxmox VE clients talk to **HTTPS `:8006`**. This lab’s Compose stack
publishes plain **HTTP `:8006`** (same port number). HTTPS belongs on
**Kubernetes Ingress** (cert-manager). Clients that cannot speak HTTP
(proxmoxer) use the optional profile: `docker compose --profile tls` →
`https://localhost:8443/` (see [Ports and TLS](configuration.md#ports-and-tls)).
| Stack | Compose transport | Notes | Docs | Code |
|---|---|---|---|---| |---|---|---|---|---|
| Python (proxmoxer) | HTTPS `:8007` | Unmodified library; `verify_ssl=False` for local cert | [guide](examples/python-proxmoxer.md) | [`examples/python`](../examples/python) | | Python (proxmoxer) | HTTPS `:8443` (`--profile tls`) | HTTPS-only library; `verify_ssl=False` for lab cert | [guide](examples/python-proxmoxer.md) | [`examples/python`](../examples/python) |
| Python (requests) | HTTP `:8006` | Raw `/api2/json` | [guide](examples/python-requests.md) | [`examples/python`](../examples/python) | | Python (requests) | HTTP `:8006` | Raw `/api2/json` | [guide](examples/python-requests.md) | [`examples/python`](../examples/python) |
| Go | HTTP `:8006` | stdlib `net/http` | [guide](examples/go.md) | [`examples/go`](../examples/go) | | Go | HTTP `:8006` | stdlib `net/http` | [guide](examples/go.md) | [`examples/go`](../examples/go) |
| Java | HTTP `:8006` | Java 11+ `HttpClient` | [guide](examples/java.md) | [`examples/java`](../examples/java) | | Java | HTTP `:8006` | Java 11+ `HttpClient` | [guide](examples/java.md) | [`examples/java`](../examples/java) |
| Perl | HTTP `:8006` | `HTTP::Tiny` + JSON | [guide](examples/perl.md) | [`examples/perl`](../examples/perl) | | Perl | HTTP `:8006` | `HTTP::Tiny` + JSON | [guide](examples/perl.md) | [`examples/perl`](../examples/perl) |
| Ansible | HTTP `:8006` | `uri` module cookbook | [guide](examples/ansible.md) | [`examples/ansible`](../examples/ansible) | | Ansible | HTTP `:8006` | `uri` module cookbook | [guide](examples/ansible.md) | [`examples/ansible`](../examples/ansible) |
| Terraform | HTTPS `:8007` | Provider + insecure TLS for local gateway | [guide](examples/terraform.md) | [`examples/terraform`](../examples/terraform) | | Terraform | HTTP `:8006` (or TLS `:8443`) | Prefer HTTP; use `insecure` only with `--profile tls` | [guide](examples/terraform.md) | [`examples/terraform`](../examples/terraform) |
| Pulumi | HTTPS `:8007` | Python program against the API | [guide](examples/pulumi.md) | [`examples/pulumi`](../examples/pulumi) | | Pulumi | HTTP `:8006` | `pulumi-proxmoxve` or HTTP cookbooks | [guide](examples/pulumi.md) | [`examples/pulumi`](../examples/pulumi) |
Shared prerequisites: [examples overview](examples/overview.md). On Kubernetes with Ingress + cert-manager, point every client at
`https://<your-host>/` instead.
## Credentials (seed) ## More
| Use | Value | - Cookbooks index: [examples/overview.md](examples/overview.md)
|---|---| - Troubleshooting: [examples/troubleshooting-clients.md](examples/troubleshooting-clients.md)
| User | `root@pam` | - Pulumi full suite: [`pulumi-tests/`](../pulumi-tests/README.md)
| Password | `secret` |
| Token | `root@pam!automation=automation-secret` |
| Default node (`small`) | `pve01` |
## API major
Pin the major before long runs:
- Cold start: `CONTRACT_SNAPSHOT`
- Runtime: Web UI apply or `POST /ui/api/contract/apply?major=N`
Confirm with `GET /api2/json/version`. Coverage is **100%** for declared methods
on majors 6–9.
## Troubleshooting clients
See [troubleshooting-clients](examples/troubleshooting-clients.md) and the
global [Troubleshooting](troubleshooting.md) guide.
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](compatibility-0.1.0.md) | [Русский](ru/compatibility-0.1.0.md)
# Compatibility report — 0.1.0 # Compatibility report — 0.1.0
This report records evidence for simulator release 0.1.0 against the bundled This report records evidence for simulator release 0.1.0 against the bundled
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](compatibility.md) | [Русский](ru/compatibility.md)
# Compatibility # Compatibility
This document explains how the simulator claims compatibility with Proxmox VE This document explains how the simulator claims compatibility with Proxmox VE
+32 -3
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](configuration.md) | [Русский](ru/configuration.md)
# Configuration # Configuration
Application settings are loaded from the environment (see `.env.example`). Application settings are loaded from the environment (see `.env.example`).
@@ -53,14 +55,41 @@ any accidental gap surfaces as HTTP 501.
| `SEED_LARGE_NODES` | Node count for `large` | | `SEED_LARGE_NODES` | Node count for `large` |
| `SEED_LARGE_RESOURCES` | Guest count for `large` (default 10 000) | | `SEED_LARGE_RESOURCES` | Guest count for `large` (default 10 000) |
| `TEST_DATABASE_URL` | Integration-test DSN | | `TEST_DATABASE_URL` | Integration-test DSN |
| `PROXMOXER_HOST` / `PROXMOXER_PORT` | Compatibility test client target (`tls-gateway` / `8443` in Compose) | | `PROXMOXER_HOST` / `PROXMOXER_PORT` | Compatibility test client target with `--profile tls` (`tls-gateway` / `8443`) |
## Ports and TLS ## Ports and TLS
### Real Proxmox VE (reference)
On a physical / production PVE node the management API listens on **HTTPS
`:8006`** (`/api2/json/...`). Related management ports (not separate REST APIs):
| Port | Protocol | Role |
|---|---|---|
| `8006` | TCP, HTTPS | Web UI + REST API |
| `3128` | TCP | SPICE proxy (graphical console) |
| `5900–5999` | TCP (WebSocket) | VNC web console |
| `22` | TCP | SSH / cluster actions |
| `5405–5412` | UDP | Corosync cluster traffic |
Port **`8007`** is **not** the PVE API — it is the usual Proxmox Backup Server
(PBS) management port. Do not point PVE clients at `:8007` on real hardware.
### Simulator lab endpoints
| Endpoint | Use | | Endpoint | Use |
|---|---| |---|---|
| `http://localhost:8006` | Direct HTTP (curl, browsers, most examples) | | `http://localhost:8006` | Primary client URL — simulator (curl, browsers, requests, Terraform, …) |
| `https://localhost:8007` | TLS gateway for TLS-assuming clients (proxmoxer, etc.) | | `https://localhost:8443` | Optional — `docker compose --profile tls` for proxmoxer-style HTTPS-only clients |
Compose publishes plain **HTTP on host `:8006`** (same port number as real PVE,
which uses HTTPS). The simulator process speaks HTTP on `:8006` inside the Docker
network as well. For Kubernetes, TLS terminates at Ingress (cert-manager). Host
**`:8007` is not used** for the lab API (on real hardware that port is typically
PBS, not PVE).
Optional Compose TLS: `docker compose --profile tls` starts an nginx gateway on
host `:8443` (requires `docker/tls/`). It proxies to `simulator:8006`.
The checked-in certificate under `docker/tls/` is disposable development The checked-in certificate under `docker/tls/` is disposable development
material. Never reuse it outside local labs. See [Security](security.md). material. Never reuse it outside local labs. See [Security](security.md).
+40
View File
@@ -0,0 +1,40 @@
# Docker Hub overview (paste into Hub)
Copy the block below into the **Full description** of
[`inecs/proxmox-api-simulator`](https://hub.docker.com/r/inecs/proxmox-api-simulator)
so Hub wording matches GitHub (stateful simulator — not a thin mock).
---
**proxmox-api-simulator** — stateful asynchronous Proxmox VE API simulator for
labs and CI. PostgreSQL-backed mutations, durable UPIDs, official API contracts
for PVE 6–9, and the same `/api2/json` surface clients already speak.
**Laboratory / CI only.** Default credentials and signing keys are intentional
lab defaults. Do **not** expose this image to the public Internet without
replacing secrets and adding your own network controls.
### Quick start
```bash
# from a git checkout (needs docker-compose.release.yml + docker/tls/)
docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \
simulator -m app.simulation.seed_cli
curl -sS http://localhost:8006/health/ready
curl -sS http://localhost:8006/api2/json/version
```
- HTTP API + Web UI: `http://localhost:8006/`
- Optional HTTPS for proxmoxer: `docker compose --profile tls` → `https://localhost:8443/`
- Seeded admin: `root@pam` / `secret`
- Source & docs: https://github.com/sergeyantropoff/proxmox-api-simulator
- Helm chart: `helm/proxmox-api-simulator` in the same repository
### Tags
| Tag | Meaning |
|---|---|
| `0.1.0`, `…` | Immutable release from `pyproject.toml` / `make release` |
| `latest` | Most recent `make release` (when `PUSH_LATEST=1`) |
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](README.md) | [Русский](../ru/domains/README.md)
# Domain guides # Domain guides
These pages summarize durable semantics by area. For exhaustive method lists, These pages summarize durable semantics by area. For exhaustive method lists,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](access.md) | [Русский](../ru/domains/access.md)
# Access # Access
Durable identity and authorization: users, groups, roles, ACL entries, realms, Durable identity and authorization: users, groups, roles, ACL entries, realms,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](ceph.md) | [Русский](../ru/domains/ceph.md)
# Ceph # Ceph
Ceph-related API paths persist simulated cluster, pool, OSD, and monitor state. Ceph-related API paths persist simulated cluster, pool, OSD, and monitor state.
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](cluster-extras.md) | [Русский](../ru/domains/cluster-extras.md)
# Cluster extras # Cluster extras
Additional cluster-scoped domains with durable handlers: Additional cluster-scoped domains with durable handlers:
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](core-cluster.md) | [Русский](../ru/domains/core-cluster.md)
# Core & cluster # Core & cluster
## Version ## Version
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](firewall.md) | [Русский](../ru/domains/firewall.md)
# Firewall # Firewall
Cluster, node, and guest firewall configuration — rules, aliases, IP sets, Cluster, node, and guest firewall configuration — rules, aliases, IP sets,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](ha.md) | [Русский](../ru/domains/ha.md)
# HA # HA
High-availability groups, resources, status, and rules persist in cluster High-availability groups, resources, status, and rules persist in cluster
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](lxc.md) | [Русский](../ru/domains/lxc.md)
# LXC # LXC
Container APIs mirror the QEMU lifecycle patterns where the contract declares Container APIs mirror the QEMU lifecycle patterns where the contract declares
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](pools.md) | [Русский](../ru/domains/pools.md)
# Pools # Pools
Pool CRUD and resource membership are fully covered and durable. The `medium` Pool CRUD and resource membership are fully covered and durable. The `medium`
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](qemu.md) | [Русский](../ru/domains/qemu.md)
# QEMU # QEMU
Full contract surface for QEMU guests on the active major, including: Full contract surface for QEMU guests on the active major, including:
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](sdn.md) | [Русский](../ru/domains/sdn.md)
# SDN # SDN
Software-defined networking handlers cover declared zones, VNets, subnets, Software-defined networking handlers cover declared zones, VNets, subnets,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](storage-backup.md) | [Русский](../ru/domains/storage-backup.md)
# Storage & backup # Storage & backup
## Storage ## Storage
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](tasks.md) | [Русский](../ru/domains/tasks.md)
# Tasks # Tasks
Long-running operations return a Proxmox-style **UPID**. Task rows, events, Long-running operations return a Proxmox-style **UPID**. Task rows, events,
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](ansible.md) | [Русский](../ru/examples/ansible.md)
# Ansible # Ansible
Playbook uses the `uri` module against HTTP `:8006` with token auth, then Playbook uses the `uri` module against HTTP `:8006` with token auth, then
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](go.md) | [Русский](../ru/examples/go.md)
# Go # Go
Uses the Go standard library against `http://localhost:8006` with API-token Uses the Go standard library against `http://localhost:8006` with API-token
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](java.md) | [Русский](../ru/examples/java.md)
# Java # Java
Java 11+ `HttpClient` cookbook using API-token auth against `:8006`. Java 11+ `HttpClient` cookbook using API-token auth against `:8006`.
+3 -1
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](overview.md) | [Русский](../ru/examples/overview.md)
# Client examples overview # Client examples overview
## Bring-up checklist ## Bring-up checklist
@@ -20,7 +22,7 @@ curl -s -X POST 'http://localhost:8006/ui/api/contract/apply?major=8'
| URL | When | | URL | When |
|---|---| |---|---|
| `http://localhost:8006` | curl, Go, Java, Perl, Ansible, requests | | `http://localhost:8006` | curl, Go, Java, Perl, Ansible, requests |
| `https://localhost:8007` | proxmoxer, many Terraform/Pulumi TLS clients | | `http://localhost:8006` | proxmoxer, many Terraform/Pulumi TLS clients |
## Auth quick reference ## Auth quick reference
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](perl.md) | [Русский](../ru/examples/perl.md)
# Perl # Perl
`HTTP::Tiny` + JSON cookbook with API-token auth. `HTTP::Tiny` + JSON cookbook with API-token auth.
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](pulumi.md) | [Русский](../ru/examples/pulumi.md)
# Pulumi # Pulumi
Python Pulumi program that drives the simulator over HTTPS using token auth via Python Pulumi program that drives the simulator over HTTPS using token auth via
+3 -1
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](python-proxmoxer.md) | [Русский](../ru/examples/python-proxmoxer.md)
# Python — proxmoxer # Python — proxmoxer
Canonical library path against the HTTPS gateway. Canonical library path against the HTTPS gateway.
@@ -11,7 +13,7 @@ python examples/python/proxmoxer_cookbook.py
``` ```
Environment overrides: `PVE_HOST` (default `localhost`), `PVE_PORT` (default Environment overrides: `PVE_HOST` (default `localhost`), `PVE_PORT` (default
`8007`), `PVE_USER`, `PVE_PASSWORD`, or token via `PVE_TOKEN_NAME` / `8006`), `PVE_USER`, `PVE_PASSWORD`, or token via `PVE_TOKEN_NAME` /
`PVE_TOKEN_VALUE`. `PVE_TOKEN_VALUE`.
## Notes ## Notes
+2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](python-requests.md) | [Русский](../ru/examples/python-requests.md)
# Python — requests # Python — requests
Raw HTTP against `:8006` without proxmoxer. Raw HTTP against `:8006` without proxmoxer.
+3 -1
View File
@@ -1,7 +1,9 @@
**Language / Язык:** [English](terraform.md) | [Русский](../ru/examples/terraform.md)
# Terraform # Terraform
Example uses a Proxmox provider pointed at the local HTTPS gateway Example uses a Proxmox provider pointed at the local HTTPS gateway
(`https://localhost:8007`) with `insecure = true` for the development (`http://localhost:8006`) with `insecure = true` for the development
certificate. certificate.
```bash ```bash
+3 -1
View File
@@ -1,8 +1,10 @@
**Language / Язык:** [English](troubleshooting-clients.md) | [Русский](../ru/examples/troubleshooting-clients.md)
# Troubleshooting clients # Troubleshooting clients
| Symptom | Fix | | Symptom | Fix |
|---|---| |---|---|
| TLS certificate errors | Use `:8007` with verify disabled **only** locally, or use HTTP `:8006` | | TLS certificate errors | Compose default is `http://localhost:8006` (no TLS). For proxmoxer use `docker compose --profile tls` and `https://localhost:8443` with verify disabled **only** locally (`curl -sk`, `verify_ssl=False`, `insecure=true`). On Kubernetes use your Ingress hostname with cert-manager TLS. |
| CSRF failure | Send `CSRFPreventionToken` with ticket mutations; prefer token auth in scripts | | CSRF failure | Send `CSRFPreventionToken` with ticket mutations; prefer token auth in scripts |
| Node not found | `small` seed uses `pve01` | | Node not found | `small` seed uses `pve01` |
| 403 on power | You may be using `auditor@pve` / readonly token — switch to root or operator | | 403 on power | You may be using `auditor@pve` / readonly token — switch to root or operator |
+19 -2
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](faq.md) | [Русский](ru/faq.md)
# FAQ # FAQ
## Is this a real Proxmox hypervisor? ## Is this a real Proxmox hypervisor?
@@ -14,7 +16,10 @@ See [API versions](api-versions.md) and [Compatibility](compatibility.md).
## Can I use this in CI for Terraform / Ansible / custom clients? ## Can I use this in CI for Terraform / Ansible / custom clients?
Yes. That is a primary use case. Pin the API major, seed a profile, and point Yes. That is a primary use case. Pin the API major, seed a profile, and point
clients at HTTP `:8006` or HTTPS `:8007`. See [Clients](clients.md). clients at **HTTP `:8006`** (Compose) or your Ingress **HTTPS** hostname on
Kubernetes. See [Clients](clients.md). For the
Pulumi surface suite see [`pulumi-tests/`](../pulumi-tests/README.md).
Full test map and latest recorded results: [Testing](testing.md).
## Why do some OpenID / LDAP / ACME / Ceph calls “succeed” without remotes? ## Why do some OpenID / LDAP / ACME / Ceph calls “succeed” without remotes?
@@ -40,4 +45,16 @@ Hub image. Ingress + cert-manager Let's Encrypt is supported — see
## Which node name does the small seed use? ## Which node name does the small seed use?
`pve01`. `pve01`. Profiles `medium` and `ha-demo` use **`pve1` / `pve2` / `pve3`**.
## What ports does real Proxmox VE use vs this simulator?
Real PVE serves the Web UI and REST API on **HTTPS `:8006`** only. Related
management ports include SPICE `:3128`, VNC `:5900–5999`, SSH `:22`, and
Corosync UDP `:5405–5412`. Port `:8007` on real hardware is typically
**Proxmox Backup Server**, not PVE.
This lab publishes plain **HTTP `:8006`** in Compose (same port number as real
PVE). HTTPS belongs on Kubernetes Ingress. Optional proxmoxer TLS:
`docker compose --profile tls` on `:8443`. Host `:8007` is **not** used. Details:
[Ports and TLS](configuration.md#ports-and-tls).
+24 -14
View File
@@ -1,3 +1,5 @@
**Language / Язык:** [English](getting-started.md) | [Русский](ru/getting-started.md)
# Getting started # Getting started
Bring up a local laboratory cluster, authenticate, and exercise a first Bring up a local laboratory cluster, authenticate, and exercise a first
@@ -17,15 +19,18 @@ Python toolchain for day-to-day use.
|---|---| |---|---|
| [Published image](#1a-published-image-docker-hub) | Fastest lab using `inecs/proxmox-api-simulator` | | [Published image](#1a-published-image-docker-hub) | Fastest lab using `inecs/proxmox-api-simulator` |
| [Helm / Kubernetes](kubernetes.md) | Cluster install with Ingress + Let's Encrypt | | [Helm / Kubernetes](kubernetes.md) | Cluster install with Ingress + Let's Encrypt |
| [Development checkout](#1b-development-checkout) | Contribute / bind-mount source / HTTPS gateway on `:8007` | | [Development checkout](#1b-development-checkout) | Contribute / bind-mount source / HTTP API on `:8006` |
## 1a. Published image (Docker Hub) ## 1a. Published image (Docker Hub)
Uses [`docker-compose.release.yml`](../docker-compose.release.yml) — PostgreSQL + Uses [`docker-compose.release.yml`](../docker-compose.release.yml) — PostgreSQL +
runtime simulator from Hub. No source build required. runtime simulator from Hub. No source build required.
> Laboratory / CI only — rotate `TICKET_SIGNING_KEY` and the DB password before
> any shared or networked demo. See [SECURITY.md](../SECURITY.md).
```bash ```bash
# from this repository, or download docker-compose.release.yml alone # from this repository (compose file + docker/tls/)
docker compose -f docker-compose.release.yml pull docker compose -f docker-compose.release.yml pull
docker compose -f docker-compose.release.yml up -d docker compose -f docker-compose.release.yml up -d
docker compose -f docker-compose.release.yml run --rm --entrypoint python \ docker compose -f docker-compose.release.yml run --rm --entrypoint python \
@@ -47,7 +52,7 @@ make release-seed PROFILE=small
| Host port | Service | | Host port | Service |
|---|---| |---|---|
| `8006` | HTTP API + Web UI | | `8006` | HTTP API + Web UI (same port as real PVE; real PVE uses HTTPS) |
| `5432` | PostgreSQL (localhost only) | | `5432` | PostgreSQL (localhost only) |
Migrations run automatically via the `migrate` one-shot service. Migrations run automatically via the `migrate` one-shot service.
@@ -65,17 +70,22 @@ Services:
| Host port | Service | | Host port | Service |
|---|---| |---|---|
| `8006` | HTTP API + Web UI | | `8006` | HTTP API + Web UI (same port as real PVE; real PVE uses HTTPS) |
| `8007` | HTTPS nginx gateway → simulator |
| `5432` | PostgreSQL (localhost only) | | `5432` | PostgreSQL (localhost only) |
On real Proxmox VE the REST API is **only** `https://<host>:8006/api2/json/...`.
The lab publishes plain **HTTP** on host `:8006`; see
[Ports and TLS](configuration.md#ports-and-tls). Optional HTTPS for proxmoxer:
`docker compose --profile tls` → `https://localhost:8443/`. Host `:8007` is
**not** used (on hardware it is typically PBS, not PVE API).
Migrations apply automatically before the simulator becomes ready. Migrations apply automatically before the simulator becomes ready.
## 2. Wait until ready ## 2. Wait until ready
```bash ```bash
curl http://localhost:8006/health/live curl -sS http://localhost:8006/health/live
curl http://localhost:8006/health/ready curl -sS http://localhost:8006/health/ready
``` ```
`/health/ready` returns HTTP 503 until PostgreSQL is reachable **and** the `/health/ready` returns HTTP 503 until PostgreSQL is reachable **and** the
@@ -94,7 +104,7 @@ local storages, and the standard development principals. See
## 4. Check the API version ## 4. Check the API version
```bash ```bash
curl -s http://localhost:8006/api2/json/version | jq . curl -sS http://localhost:8006/api2/json/version | jq .
``` ```
The cold-start contract defaults to the bundled PVE **9.2.3** snapshot in Docker The cold-start contract defaults to the bundled PVE **9.2.3** snapshot in Docker
@@ -104,7 +114,7 @@ Compose. Switch majors 6–9 from the Web UI or
## 5. Authenticate ## 5. Authenticate
```bash ```bash
curl -s -X POST \ curl -sS -X POST \
-d 'username=root@pam&password=secret' \ -d 'username=root@pam&password=secret' \
http://localhost:8006/api2/json/access/ticket | jq . http://localhost:8006/api2/json/access/ticket | jq .
``` ```
@@ -120,10 +130,10 @@ Details: [Authentication](authentication.md).
```bash ```bash
# replace TICKET / CSRF from the previous response # replace TICKET / CSRF from the previous response
curl -s -H "Cookie: PVEAuthCookie=$TICKET" \ curl -sS -H "Cookie: PVEAuthCookie=$TICKET" \
http://localhost:8006/api2/json/nodes/pve01/qemu | jq . http://localhost:8006/api2/json/nodes/pve01/qemu | jq .
curl -s -X POST \ curl -sS -X POST \
-H "Cookie: PVEAuthCookie=$TICKET" \ -H "Cookie: PVEAuthCookie=$TICKET" \
-H "CSRFPreventionToken: $CSRF" \ -H "CSRFPreventionToken: $CSRF" \
http://localhost:8006/api2/json/nodes/pve01/qemu/100/status/start | jq . http://localhost:8006/api2/json/nodes/pve01/qemu/100/status/start | jq .
@@ -132,7 +142,7 @@ curl -s -X POST \
Async operations return a UPID string. Poll until the task finishes: Async operations return a UPID string. Poll until the task finishes:
```bash ```bash
curl -s -H "Cookie: PVEAuthCookie=$TICKET" \ curl -sS -H "Cookie: PVEAuthCookie=$TICKET" \
"http://localhost:8006/api2/json/nodes/pve01/tasks/${UPID}/status" | jq . "http://localhost:8006/api2/json/nodes/pve01/tasks/${UPID}/status" | jq .
``` ```
@@ -140,8 +150,8 @@ curl -s -H "Cookie: PVEAuthCookie=$TICKET" \
Visit [http://localhost:8006/](http://localhost:8006/) for the interactive Visit [http://localhost:8006/](http://localhost:8006/) for the interactive
console, contract catalog (PVE 6–9), compatibility view, runtime contract apply, console, contract catalog (PVE 6–9), compatibility view, runtime contract apply,
and demo-cluster controls. See [Web UI](web-ui.md) for light/dark theme and demo-cluster controls. See [Web UI](web-ui.md) for screenshots and the
screenshots and the full feature list. full feature list.
## 8. Try a client library ## 8. Try a client library
Binary file not shown.

After

Width:  |  Height:  |  Size: 140 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 53 KiB

After

Width:  |  Height:  |  Size: 175 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 135 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 222 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 171 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 86 KiB

Some files were not shown because too many files have changed in this diff Show More