Proxmox VE

FreeSDN supports Proxmox through direct hypervisor endpoints and staged adapter endpoints. Use the staged path when changes require review, durable dispatch history and provider-task observation; direct VM/CT power, cloning, same-cluster migration (including bulk power/migration), manual backup, snapshot create/delete/rollback, disk resize, configuration updates and template conversion now create and apply durable records immediately, while other direct writes do not automatically enter that queue. The adapter implements credential redaction, guest-agent write-path gating, VNC ticket protection, typed-confirm gating, and multi-tenant task isolation; the apply path is covered by a regression test suite.

Implemented coverage and limits

Domain	Operations
VMs	Create, start, stop, reboot, suspend, console, migrate, delete
LXC containers	Create, start, stop, reboot, console, migrate, delete
Snapshots	Create, restore, delete per VM/CT
Storage	Browse storage/content; reviewed ISO/template upload and artifact deletion. Storage-definition CRUD is not provided by these content operations.
Backups	Reviewed schedule changes, manual backup, archive browsing and staged restore/prune paths. A saved schedule is not proof of a successful backup or restore.
Cluster	Membership/quorum reads, reviewed node power and HA resource/group changes; no cluster join/remove workflow.
Nodes	Status, resource usage, active/archived tasks and reviewed task termination.
Replication	Replication job list and status
SDN	Read-only configuration and bounded dependency observations, including subnets, IPAM/DNS, fabrics, routing catalogues and parsed policy entries; guest/host attachment observations and per-node dry-run summaries. FreeSDN SDN writes are unavailable pending qualification of the reviewed native-lock workflow.
Ceph	Status, OSD list, pool list (read-only)
Firewall	Reviewed cluster/node/guest rule create, update, delete and move, plus constrained options. UI supports create/delete/toggle/move-to-first and enable/disable. Effective-policy and hardware qualification remain open.
Guest agent	File read/write/exec (gated to site_admin+)

These rows describe implemented surfaces, not qualification of every option or version. Ceph and replication writes are intentionally unimplemented. The PBS view accesses PBS-backed PVE storage; it is not a standalone PBS management integration.

PMG (Proxmox Mail Gateway) is not covered. Live cluster reconfiguration (add/remove nodes) requires shell access; it is not exposed through the API.

Dashboard observations and capacity

The cluster, fleet and kiosk dashboards separate observed zero usage from missing statistics. CPU, memory and node filesystem totals/percentages are null when a complete metric cannot be established for the known nodes. Per-metric coverage reports contributing/expected nodes and missing, invalid, offline or inconsistent inventory. Partial-node percentages are not presented as full-cluster utilization. Numeric fields reject boolean, negative, non-finite, impossible usage/capacity and other malformed values. A malformed or over-budget essential read fails rather than producing an empty healthy dashboard.

The existing total_storage_bytes, used_storage_bytes and storage_usage_percent fields describe node filesystems, not datastore capacity. Shared storage, overlapping pools and safe placement capacity are not inferred from these totals. Guest counts describe the inventory visible to the controller credential; they do not establish that hidden guests do not exist. Unknown guest runtime status also leaves the running count unknown.

Responses carry the controller identity, collection start/completion, serving time, observation age and coverage. The short backend cache preserves original collection times and expires ten seconds after collection starts. Cache hits return isolated copies. Changed controller connection configuration invalidates the cache; an expired/changed entry cannot survive a failed refresh. Vault contents or native ACLs changed separately are not revalidated during the remaining cache lifetime. Route authorization still applies before service/cache access.

The UI hides values after read errors, mismatched controller responses, invalid observation timing or 90-second expiry. It shows collection age while data remains usable. This interval is a UI observation policy, not a guarantee of native sample freshness: Proxmox resource samples may be older, and the multiple reads are not an atomic snapshot. A substantial client/server clock difference leaves freshness unknown. The kiosk uses node status from the same dashboard response and does not render default zero gauges before its first observation.

Fleet totals are by registered controller. Physical clusters are not deduplicated; registering multiple nodes from one cluster can repeat that infrastructure. Clusters are never merged merely because names match. A failed controller leaves aggregate counts and metrics unknown while successful per-controller observations remain available. Node coverage counts exclude failed controllers whose inventory is unknown; failed-controller counts are shown separately. Aggregate age includes waiting for other controllers. Task statistics have a separate collection lifecycle and are not qualified by the dashboard’s observation metadata.

The dashboard reports the number of visible HA service entries. Quorum, master, LRM or fencing status entries alone do not establish active HA services. The legacy ha_active field is deprecated and remains null; service presence does not prove HA health, fencing readiness or successful failover. Native quorum/online flags are parsed strictly, including numeric string "0" as false.

Limits: at most 1,024 node identities, 20,000 resource rows and 16 MiB of serialized decoded resource data. These are retention/validation limits, not a streaming HTTP body cap. Three native dashboard reads run concurrently with a 15-second shared read deadline after connection. Fleet observations use at most five shared slots and a 20-second deadline per registration that includes slot waiting and connection setup. Timeouts cancel pending reads and are reported as failures. No new native write capability is introduced.

API compatibility: formerly numeric dashboard fields can now be null. Consumers must handle unknown/partial values and the new observation metadata. Private/public fixture tests do not qualify an installed Proxmox version, deployed browser journey, sustained large fleet, upgrade or hardware failure/recovery path.

Adapter tier

DG	CB	RR	TS	SSRF	RG
✓	✓	✓	✓	✓	✓

Prerequisites

• Proxmox VE 7.x or 8.x
• A dedicated API token for FreeSDN (preferred over username/password)
• The token user must have the PVEAdmin or a custom role with the permissions described below

Creating an API token in Proxmox

# On the Proxmox node, create a dedicated user and token
pveum user add freesdn@pve --comment "FreeSDN management"
pveum role add FreesdnRole \
  --privs "VM.Allocate,VM.Config.CDROM,VM.Config.CPU,VM.Config.Disk,VM.Config.HWType,VM.Config.Memory,VM.Config.Network,VM.Config.Options,VM.Console,VM.Migrate,VM.Monitor,VM.PowerMgmt,VM.Snapshot,VM.Snapshot.Rollback,VM.Audit,VM.Backup,Datastore.Allocate,Datastore.AllocateSpace,Datastore.AllocateTemplate,Datastore.Audit,Pool.Allocate,SDN.Allocate,SDN.Audit,Sys.Audit,Sys.Modify,Sys.PowerMgmt"
pveum acl modify / --users freesdn@pve --roles FreesdnRole
pveum user token add freesdn@pve freesdn --privsep 1
pveum acl modify / --tokens 'freesdn@pve!freesdn' --roles FreesdnRole

Copy the token secret - it is shown only once.

Caution

This example grants broad cluster management access. For monitoring, use the native PVEAuditor role and restrict paths to the intended inventory. For management, restrict the user’s and token’s ACLs to the operations and resources you need. Keep privilege separation enabled: a token’s effective permissions are limited by both its own ACLs and those of its user. Do not disable separation or grant full administrator access just to work around a failed permission check.

Effective permissions and operation review

Proxmox’s /access/permissions?path=... response maps each privilege to an inheritance flag. A present privilege with 0 grants access at the requested path; 1 additionally allows inheritance. An absent privilege is not granted. FreeSDN checks the provider’s effective permissions for the configured credential, including token/user restrictions and pool membership, rather than calculating ACL inheritance itself.

The reviewed workflows below retain these explicit preflight requirements:

Workflow	Required effective grants
Cluster or node firewall change	Sys.Audit and Sys.Modify at / or the exact /nodes/{node}
Guest firewall change	VM.Audit and VM.Config.Network at /vms/{vmid}
Task termination	Sys.Audit and Sys.Modify at the exact /nodes/{node}
Scheduled backup create/update	Sys.Audit and Sys.Modify at /; Sys.Audit at the selected node; Datastore.Audit and Datastore.Allocate at the selected storage; VM.Audit and VM.Backup at every explicitly selected guest
Scheduled backup delete	Sys.Audit and Sys.Modify at /
Artifact upload/delete	Datastore.Audit and, respectively, Datastore.AllocateTemplate or Datastore.Allocate at /storage/{storage}
ISO deletion reference check	Additionally, propagating VM.Audit at /
Node/HA impact review	Propagating VM.Audit at /vms for the guest inventory

These are FreeSDN’s reviewed-workflow checks, not an exhaustive list of native permissions for every action. Proxmox still authorizes each read and write. Exact-path grants do not need propagation. Fleet visibility checks deliberately do; even then, child ACL overrides and concurrent changes can hide resources, so these checks do not certify complete visibility. Empty or malformed permission responses cannot authorize a write, and grants are checked again before dispatch. Task termination remains an administrative flow; native ownership of a task alone does not satisfy FreeSDN’s two-privilege requirement.

Adding a Proxmox cluster

Proxmox authentication is embedded directly in the controller record. There is no separate credential store step for Proxmox - pass token_id and token_secret inside the config dict:

# Add controller (API token auth  -  recommended)
curl -X POST https://freesdn.example.com/api/v1/controllers \
  -H "Cookie: freesdn_access=<token>" \
  -H "X-CSRF-Token: <csrf>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "PVE Cluster",
    "controller_type": "proxmox",
    "site_id": "<site-uuid>",
    "host": "pve.example.com",
    "port": 8006,
    "config": {
      "token_id": "freesdn@pve!freesdn",
      "token_secret": "<token-uuid-secret>"
    }
  }'

# Trigger discovery
curl -X POST https://freesdn.example.com/api/v1/discovery/controllers/<controller-uuid> \
  -H "Cookie: freesdn_access=<token>" \
  -H "X-CSRF-Token: <csrf>"

If you prefer username/password auth instead of an API token, omit config and pass username and password at the top level of the controller payload.

FreeSDN enumerates the nodes, VMs and containers visible to the configured credential. A successful, empty response does not establish that the cluster has no other guests.

Staged writes

Gateway-proxmox mutations are staged first. Applies require both gates open:

1. ADAPTER_READ_ONLY=false in the FreeSDN environment. The shipped Docker Compose stack already sets this, so FreeSDN manages your cluster out of the box. Set ADAPTER_READ_ONLY=true in .env for a monitor-only deployment.
2. force: true on the apply request. This is the gate that prevents accidental VM power operations or config changes: a staged change never reaches the hypervisor until you explicitly apply it. See Adapter Contract and Pending Changes for the apply flow.

Example: stage a VM stop, then apply it:

# Stage a VM power-off
curl -X POST 'https://freesdn.example.com/api/v1/gateway-proxmox-vm/<controller-id>/changes/proxmox.vm.stop?operation=create' \
  -H "Cookie: freesdn_access=<token>" \
  -H "X-CSRF-Token: <csrf>" \
  -H "Content-Type: application/json" \
  -d '{"payload": {"node": "pve1", "vmid": 100}, "target_id": "100"}'

# Apply (on Pending Changes page, or via API)
curl -X POST https://freesdn.example.com/api/v1/gateway-vpn/changes/<change-uuid>/apply \
  -H "Cookie: freesdn_access=<token>" \
  -H "X-CSRF-Token: <csrf>" \
  -H "Content-Type: application/json" \
  -d '{"force": true}'

Tracking completion

An accepted Proxmox task is not yet a completed operation. The Pending Changes drawer shows its provider state separately and pauses bulk apply while the outcome is unresolved. See Task completion and interrupted operations for migration, worker, recovery and permission requirements. This durable tracking applies to shared staged changes and immediate VM/CT power, clone and same-cluster migration actions (including bulk power/migration), manual backups, snapshot create/delete/rollback and reviewed disk/configuration/template changes. It does not cover every direct hypervisor action. See Immediate power requests and safe transport retries for the required request IDs, API upgrade contract, and replay-protection limits. See Manual backups, snapshots and reviewed restore for snapshot confirmation, restore staging replay protection and PBS volume IDs. Restore remains a separate review-and-apply operation; task success alone does not prove recoverability.

Reviewed disk, configuration and template changes

The direct resize and config PUT endpoints and template POST endpoint below /api/v1/hypervisor/controllers/{id}/nodes/{node}/{qemu|lxc}/{vmid} now return HTTP 202 with a durable operation receipt. This changes the API contract: clients must send a stable request_id, force: true, and the expected_digest from the guest configuration they reviewed. Resize also requires current_size_bytes; template conversion requires the separate boolean confirmed: true. Unknown request fields are rejected. Use the same body and ID on a transport retry, and a new ID only for a newly reviewed operator intent.

A relative resize such as +10G is recorded as an absolute target derived from the reviewed disk size. FreeSDN checks the disk and configuration again, rejects shrinking and provider locks, and passes the digest to the Proxmox resize/config write. VM disks use scsiN, virtioN, ideN or sataN; containers use rootfs or mpN. Expanding a virtual disk does not prove that partitions or the guest filesystem have expanded.

The UI distinguishes dispatch from verified configuration. A completed provider task still needs a readback; saved settings that differ from current=1 remain pending with a restart message. FreeSDN polls without repeating the write. Template conversion has a pre-write digest check, but Proxmox’s template endpoint has no digest parameter: its native guest-state and configuration locks remain necessary. This is not an atomic external-edit guarantee for template conversion.

These reviewed writes serialize their claims per registered controller and guest. An already active or unknown operation blocks a new reviewed change. This does not establish a shared lock across other operation families or duplicate controller registrations. A terminal provider failure or warning requires a new review before another intent; replaying its old ID never dispatches again. For an unknown observation on these reviewed changes, Pending Changes offers Mark as resolved after you have inspected the guest on Proxmox. This is an acknowledgement, not an undo, cancellation or proof of success; the accepted receipt and timestamps remain in history. Pending/running tasks cannot be cleared this way. If non-running observation remains unresolved after seven days, automatic polling stops and the record changes to unknown/manual verification; this also gives an inactive saved configuration a manual resolution path.

The hardened review contract applies to these direct hypervisor endpoints. Legacy generic staged configuration/resize payloads retain their older contract; they do not acquire a reviewed digest simply by being staged. Do not treat all hypervisor mutation paths as having equivalent safety guarantees.

Guest agent

The QEMU guest agent provides file read/write and command execution inside running VMs. Because this is a high-privilege operation, it is gated to site_admin or higher:

# Read a file from a VM (site_admin required)
curl -X POST 'https://freesdn.example.com/api/v1/hypervisor/controllers/<controller-id>/nodes/pve1/qemu/100/agent/file-read' \
  -H "Cookie: freesdn_access=<token>" \
  -H "X-CSRF-Token: <csrf>" \
  -H "Content-Type: application/json" \
  -d '{"file": "/etc/hostname"}'

Caution

Guest agent exec is force-gated at the adapter layer (CAN-048). The direct HTTP endpoint (/api/v1/hypervisor/controllers/{id}/nodes/{node}/qemu/{vmid}/agent/exec) exists but always returns an error - the site_admin role alone is not sufficient. The only valid path is the staged-write flow: stage via POST /api/v1/gateway-proxmox-vm/{controller_id}/changes/proxmox.vm.guest_agent_exec?operation=create with payload {"command": "...", "node": "...", "vmid": N}, then apply via POST /api/v1/gateway-vpn/changes/{change_id}/apply with body {"force": true}. The staging row is the audit trail for every invocation. Because it executes arbitrary commands inside the VM, prefer SSH or a configuration management tool for routine operations.

SDN management

FreeSDN separates configured (desired), running (committed), and native pending observations for zones, VNets and SDN controllers. Read outcomes distinguish denied, unsupported or missing, unavailable and invalid provider responses. Native inventories are permission-filtered: an empty list is not proof that no objects exist. These sequential reads are not an atomic review and do not establish cluster-wide visibility, node reload completion or connectivity.

# Inspect the three visible configuration views (no writes)
curl https://freesdn.example.com/api/v1/hypervisor/controllers/<controller-id>/sdn/inspection \
  -H "Cookie: freesdn_access=<token>"

# Inspect bounded dependency coverage (no writes or external IPAM/DNS connections)
curl https://freesdn.example.com/api/v1/hypervisor/controllers/<controller-id>/sdn/dependencies \
  -H "Cookie: freesdn_access=<token>"

# Inspect a bounded sample of guest NICs and host interfaces (no writes)
curl https://freesdn.example.com/api/v1/hypervisor/controllers/<controller-id>/sdn/attachments \
  -H "Cookie: freesdn_access=<token>"

# Read a node's native dry-run summary (no apply or lock acquisition)
curl https://freesdn.example.com/api/v1/hypervisor/controllers/<controller-id>/sdn/nodes/pve1/dry-run \
  -H "Cookie: freesdn_access=<token>"

Dry-run reports line counts and whether differences exist. Raw interface and FRR diff content is withheld because generated configuration can contain credentials. Missing capabilities and denied reads are shown explicitly, without a write fallback. The permission fields distinguish a privilege granted at /sdn from whether it propagates; neither establishes access to all child objects.

Compatibility change: the former direct zone/VNet create/delete and SDN apply routes have been removed. New proxmox.sdn.* staging and replay of saved legacy plans are rejected with HTTP 409 before native dispatch. Historical records remain inspectable. This restriction also applies when adapter writes are enabled; SDN apply is no longer an advertised adapter capability. Use native Proxmox management for SDN changes with independent recovery access and a non-management test network.

The separate Inspect dependencies action reads subnet inventories beneath visible VNets, configured IPAM/DNS providers, fabric and fabric-node observations, and prefix-list/route-map catalogues, plus separate prefix-list rules and route-map entries. It also re-reads zones, VNets and controllers for visible object references. It requires FreeSDN hypervisor:read permission and current controller/site access. Native credentials still determine which Proxmox objects are visible. IPAM/DNS have configured-only catalogues; route maps do not expose a pending catalogue through this endpoint. Unexposed views appear as not observed, never successful empty inventories. Routing catalogue identities do not establish the contents of nested policy entries.

The Prefix-list rules and Route-map entries families expose allowlisted configured, running and pending policy observations. Prefix rules include their numeric sequence, action, normalized IP prefix and optional prefix-length bounds. Route-map entries include their map identity, numeric order, action, supported parsed match/set clauses, call target and exit action. The reader parses native property strings, including quoted values and native quote/backslash/newline escapes; quoted commas never introduce extra fields. It never returns raw FRR commands or executes policies. New pending objects can carry their identity only in the pending overlay. Those identities are labelled identity_source: pending, while policy fields stay in the overlay. Pending deletion markers are preserved; overlays are not merged into an effective policy.

Unknown fields/clauses and unconstrained peer text other than valid IP addresses or local are withheld and counted. Malformed known fields invalidate the collection. Catalogue and entry reads can have different native permission filters. References to route-map parents, calls, prefix-list match targets and controller/fabric route maps use the corresponding configured or running catalogue; they do not validate rule ordering, recursion, policy evaluation or FRR enforcement. Native API/version support and live policy behavior need separate qualification.

The Fabrics and Fabric nodes families also expose allowlisted protocol-specific interface and peer configuration. A node can belong to multiple fabrics. Each fabric-node row now has a composite node_key (<fabric_id>_<node_id>), and fabric-node reference IDs use that key instead of the bare node name. API consumers must not treat node_id as globally unique. New pending fabric and node identities can come from the overlay and are labelled identity_source: pending; network fields remain in their original view or overlay.

Supported interface observations include safe names and selected OpenFabric, OSPF, BGP and WireGuard scalar settings. WireGuard peers expose their type, node/interface references and optional route-generation flag. Private/public keys, endpoints, addresses and other unsupported configuration fields are withheld and counted. Unknown protocols retain their identities and an explicit unsupported-details notice; malformed known fields invalidate that collection.

Configured/running peer-node references and interface-name matches stay within the same fabric and configuration view. Internal remote-interface matches also require an observed WireGuard node; other protocols are not substituted. Missing node observations, absent name visibility and external peers have distinct states. These are names in returned fabric configuration, not proof that a physical host interface exists, an active routing session or WireGuard handshake is established, or connectivity works. No peer is contacted and no private-key API is read. Pending peer/interface fields are displayed without graph resolution. Fabric and fabric-node collections fail independently when one returned collection is malformed.

Each inspection allows at most three concurrent collection reads and 121 logical GET calls within a shared 20-second read budget. The shared Proxmox client may retry a GET internally within that budget; requests_made counts started collection reads, not wire attempts or authentication requests. Subnet enumeration selects up to 32 VNets from the union of visible configured, running and pending inventories. The report shows any additional visible parents omitted by that limit; hidden or unavailable parents may also exist. Collections over 256 objects are rejected rather than silently truncated. A separate aggregate budget retains at most 2,048 allowlisted objects, including nested rules, clauses, interfaces, peers and overlays, and 1 MiB of their serialized data before reference annotations; this is not a streaming transport limit or a limit on the entire HTTP response. Budget exhaustion, malformed data, denial and unavailable parent inventories remain explicit. Refresh hides the previous view while fetching and after failure. Each prefix list is limited to 256 rules and each route-map match/set list to 64 clauses. Each fabric-node interface/peer list is limited to 64 items. The UI discloses its own 50-row/100-reference display limits and shows at most 50 items in each nested rules, clauses, interfaces or peers list, including pending lists.

Provider secrets, URLs and arbitrary plugin fields are withheld, and the reader never connects to an external IPAM/DNS service. A reference marked not visible may be absent, hidden by permissions, or changed between reads. A visible target does not prove configuration validity. Running references to IPAM/DNS use their configured-only catalogues. Pending-field overlays are not resolved into a dependency graph. Equal visible catalogues do not prove a clean pending state: the dependency report returns changes detected or unknown.

The separate Inspect guest and host attachments action reads a bounded sample of visible QEMU/LXC guests and hosts. It selects the first 32 guests by numeric VMID and the first eight nodes by name from independently permission-filtered native inventories. Selection and omission counts are explicit; they do not count hidden objects. Guest configurations are read twice: current=1 and the default view with pending edits/removals merged. Both are configuration observations, not guest-network probes. The host read uses type=include_sdn; unsupported provider responses remain unavailable rather than triggering a write or a guessed fallback. Host reports may contain staged settings as well as runtime flags. Result-envelope metadata, host pending diffs and management paths are not inspected.

Only allowlisted NIC bridge/VLAN/flag fields and host interface identities/member lists are returned. Raw guest configs, passwords, SSH keys, addresses, host options and native errors are withheld. Bridge names are matched independently against configured/running VNet catalogues and interfaces on the guest’s observed node. Name visible does not establish validity or connectivity; not visible may mean absent, hidden or changed between reads. A normal local bridge can be visible on a host without appearing in an SDN VNet catalogue. Unavailable targets, unselected hosts and NICs without explicit bridges have separate states.

This action has its own three-concurrent-read limit, 76-logical-read ceiling and shared 20-second deadline. Transport retries and authentication calls are not counted by the logical read counter. Native inventories over 1,024 entries, VNet catalogues over 512 entries, host reports over 256 interfaces, guest configs over 64 NICs and individual serialized native observations over 256 KiB are rejected. The aggregate retention budget is 2,048 parsed entries and 1 MiB before link annotations; neither budget limits streamed HTTP bytes or the complete response size. Failed/oversized observations are never presented as empty. The UI shows one selected guest/view and one host at a time, with a disclosed first-50 interface display limit. Refresh hides old observations while fetching and after failure.

These panels are separate observations, not a merged transaction. Hidden/unselected guests, snapshot and passthrough dependencies, withheld policy fields and policy enforcement, fabric runtime and withheld configuration, installed provider-version/plugin support and live connectivity remain unverified. This inspection is neither atomic nor a complete dependency graph. Restoring FreeSDN writes requires a complete pending-change review, version-qualified native lock ownership with secure durable token handling, and independent node reload/connectivity observations. A successful parent Proxmox task alone is insufficient. No automatic rollback is provided.

Snapshots

# Stage a snapshot create (writes are staged, not applied directly)
curl -X POST 'https://freesdn.example.com/api/v1/gateway-proxmox-snapshot/<controller-id>/changes/proxmox.snapshot.create?operation=create' \
  -H "Cookie: freesdn_access=<token>" \
  -H "X-CSRF-Token: <csrf>" \
  -H "Content-Type: application/json" \
  -d '{"payload": {"node": "pve1", "vmid": 100, "vm_type": "qemu", "snapname": "pre-upgrade", "description": "Before kernel upgrade"}}'

# List snapshots (live read)
curl 'https://freesdn.example.com/api/v1/gateway-proxmox-snapshot/<controller-id>/nodes/pve1/qemu/100/snapshots' \
  -H "Cookie: freesdn_access=<token>"

Backend route prefix

12 sub-routers, one per feature domain:

• /api/v1/gateway-proxmox-vm/ - VM lifecycle
• /api/v1/gateway-proxmox-container/ - LXC lifecycle
• /api/v1/gateway-proxmox-node/ - node info and tasks
• /api/v1/gateway-proxmox-storage/ - storage CRUD + ISO
• /api/v1/gateway-proxmox-backup/ - backup schedules + restore
• /api/v1/gateway-proxmox-cluster/ - cluster status
• /api/v1/gateway-proxmox-ha/ - HA group management
• /api/v1/gateway-proxmox-snapshot/ - VM/CT snapshots
• /api/v1/gateway-proxmox-replication/ - replication jobs
• /api/v1/gateway-proxmox-sdn/ - SDN zone/VNet reads and historical changes
• /api/v1/gateway-proxmox-ceph/ - Ceph status + OSDs (read-only; there is no OSD write path)
• /api/v1/gateway-proxmox-firewall/ - per-VM + node firewall

Frontend pages

• HypervisorPage - cluster overview, node cards, VM/CT grid
• Child components in frontend/src/pages/hypervisor/components/ - detail panels for VMs, storage, HA status

Gotchas

• Self-signed TLS - Proxmox nodes use self-signed certificates by default. FreeSDN accepts them. If you deploy with a trusted CA, set the CA bundle in EXTRA_CA_CERTS.
• API token format - Proxmox API tokens are <user>@<realm>!<tokenid>=<secret>. The username field should be freesdn@pve!freesdn; the api_key field should be the UUID secret from pveum.
• Port 8006 - Proxmox defaults to port 8006. Include it in the host URL.
• HA group names - HA group names are cluster-scoped; collisions across organizations are possible if multiple FreeSDN tenants share a cluster. Use a naming convention like freesdn-<org-slug>-<group>.

For clone/migration intent IDs, receipt handling, and explicit LXC restart consent, see Cloning and same-cluster migration. Running LXC containers cannot migrate live; restart migration involves downtime.

Reviewed guest deletion

Single-guest deletion now returns HTTP 202 with a durable operation receipt. Supply a JSON body with request_id (UUID), expected_digest (the reviewed configuration’s SHA-1 digest), force: true and confirmed: true. The old confirmation-only query is insufficient. Bulk action: delete requires a distinct request_id and expected_digest for every target, force: true on the batch, and confirmed=true on the request query. Bulk results describe acceptance per target, not completion. Reuse the same IDs and exact reviewed payload after response loss; inspect Pending Changes before creating new intents.

The UI reads the selected guests’ names and disks, freezes the review, and requires typed DELETE confirmation. Configuration changes, moved guests, running guests, locks and protection block dispatch. No provider force-stop, skip-lock, purge, or unreferenced-disk option is sent. Legacy staged destroy payloads must be discarded and reviewed again with deletion_version: 1, node, vmid, vm_type and expected_digest; confirmation belongs to the apply request, not the stored payload.

Proxmox DELETE has no digest compare-and-swap parameter. FreeSDN checks the digest immediately before dispatch, but cannot atomically prevent an external operator from replacing or changing a guest between that read and deletion. A digest represents configuration content, not an immutable guest identity; an identical replacement can also evade this check. Avoid concurrent external changes and guest-ID reuse during the operation. The local conflict guard is per registered controller and shared with reviewed resize/configuration/template changes; it is not a universal cluster lock covering every operation or duplicate registrations.

Success requires the matching qmdestroy or vzdestroy task to report OK, followed by GET /cluster/nextid?vmid=... confirming that ID is unused. The permission-filtered guest list alone is not proof of absence. A reused ID, unavailable observation or mismatched task stays uncertain, never triggers another delete, and requires investigation. Unknown observations can be explicitly resolved in Pending Changes without changing their retained receipt. This verifies guest removal at observation time, not secure storage erasure or backup recoverability.

Reviewed VM and container creation

VM/CT creation now uses durable Pending Changes receipts. In the create dialog, choose an explicit VMID, node and resources, then review before submitting. The optional next-ID suggestion is a lookup, not a reservation. The review freezes the controller, guest ID, options and request UUID; after a lost HTTP response, resubmitting the same intent does not dispatch creation again.

The create APIs (POST /api/v1/hypervisor/controllers/{controller_id}/vms and /containers) now require request_id, an explicit vmid, and force: true. They return HTTP 202 with a durable creation receipt (change_id, status, and provider_task), not a completed guest. Clients relying on omitted VMIDs or the old response must migrate. Example VM body:

{
  "request_id": "7667dc35-7a94-47aa-a751-1a19c5e460c0",
  "force": true,
  "vmid": 120,
  "node": "pve1",
  "name": "app-server",
  "cores": 2,
  "memory": 2048,
  "storage": "local-lvm",
  "disk_size": "32G"
}

Container creation requires ostemplate (a storage:vztmpl/filename volume) and accepts a numeric GiB rootfs_size. Its optional root password is encrypted before staging with FreeSDN’s existing credential key and bound to the operation record; it is not returned in receipts. Back up and protect the credential key together with the database. This is application encryption, not an external KMS. Old generic create plans must be discarded and recreated through the reviewed endpoint; generic staging refuses plaintext creation passwords before saving them. Historical rows written by older versions are not automatically scrubbed by this change.

Before dispatch, FreeSDN checks the cluster allocation index. It never sends native Proxmox overwrite/restore/force flags, and Proxmox’s own create lock arbitrates a race with an external allocator. A conflict or an uncertain write does not trigger a new VMID, retry, cleanup, or deletion. Inspect Pending Changes and the cluster before issuing another intent.

Creation completion requires a matching node/type/VMID task followed by readback of the guest, an operation marker in its description, its name (when supplied), cores, memory and allocated disk. Requested auto-start must also be observed running. The marker is editable metadata, not immutable identity. This verifies observed creation and basic resources, not every option, password usability, OS installation, network reachability or application health. Missing permissions, mismatched readback or failed observation remain unknown. An administrator can explicitly resolve an unknown receipt after checking the cluster; that resolution preserves the receipt and does not replay or undo the write.

Reviewed node power and HA changes

Node shutdown/reboot and HA resource/group create/delete now require an impact review and return HTTP 202 with durable operation receipts. Start with GET /api/v1/hypervisor/controllers/{controller_id}/cluster-operations/review, using action (node.reboot, node.shutdown, ha.resource.create, ha.resource.delete, ha.group.create, or ha.group.delete) and target (node name, vm:101/ct:102, or group name). When creating a resource in an existing HA group, include that group’s name as group in the review query.

The UI freezes the action, target, options and request UUID. It displays cluster membership, visible guests and HA configuration and requires the exact target to be typed before submission. Adding an HA resource defaults to state: started and can start a workload; it is not just an inventory registration.

Send a JSON body to the existing node power or HA write endpoint containing request_id, the review’s SHA-256 expected_digest, force: true, and confirmed: true, plus the HA creation fields when applicable. DELETE endpoints also require this body. Old query-only confirmation and unreviewed staged payloads are insufficient. Reuse the same request ID and exact parameters after response loss; inspect Pending Changes before creating another intent.

Before dispatch, the current cluster membership, guest impact and HA configuration must still match the reviewed digest. Non-quorate clusters, offline target nodes, malformed or denied inventory, duplicate identities, missing guests and HA groups still referenced by resources are refused. Read access requires Sys.Audit for the cluster information and propagating VM.Audit at /vms as a minimum. That permission check does not prove complete visibility under every child or pool ACL. Node count and current quorum do not prove quorum will survive shutdown. FreeSDN does not automatically drain guests, enter HA maintenance, validate Ceph or replication health, prevent fencing, or prove backup recovery. Arrange those conditions and an independent management/recovery path before confirming.

The native node power API returns no task ID. Acceptance and a lost connection are not proof of power-off or reboot, so these receipts remain unknown until an administrator verifies the host independently and explicitly resolves the receipt. FreeSDN does not invent a successful task or resend the power command. HA writes likewise return no task ID; a read-only observer verifies the requested configuration before reporting success. This proves configuration at observation time, not workload health, placement convergence or successful failover.

HA resource deletion explicitly sends purge=0 to preserve HA rule references. If the installed Proxmox API rejects this parameter, removal fails safely; there is no fallback that silently enables rule purging. Legacy HA group endpoints are unavailable after migration to HA rules on newer clusters. Such errors block the review instead of being treated as an empty group list. This workflow does not create or migrate the newer HA affinity rules.

A controller-level claim check prevents these node/HA operations from overlapping other unresolved staged Proxmox operations. An unresolved node/HA receipt also blocks new staged writes on that registered controller until investigated and resolved. This is deliberately conservative; it is not a distributed lock against native Proxmox users, duplicate controller registrations, or remaining legacy direct-write paths. The provider offers no atomic review-digest precondition for these six operations, so an external edit can still race the final read. Avoid concurrent external changes during maintenance.

Reviewed certificates and recovery

Custom certificate upload, removal and ACME renewal now use durable operation receipts. Review with POST /api/v1/hypervisor/controllers/{controller_id}/nodes/{node}/certificates/review. The JSON body contains action (upload, delete, or renew). Upload also requires certificates (PEM chain, leaf first) and key (unencrypted PEM private key), with optional overwrite and restart booleans. Removal accepts restart. Renewal accepts acme_force for renewal before the normal expiry window; renewal always reloads the proxy, so it does not accept a separate restart option.

The review only reads Proxmox. It validates the candidate’s RSA/EC key pair, validity, server usage and supplied chain links, and displays the public fingerprint, subject, SANs, expiry, current inventory and ACME domains. It does not establish a complete RFC 5280 trust path, browser trust, hostname coverage, revocation status or compatibility with every supported Proxmox/OpenSSL version. The review needs readable certificate PEMs, cluster node status and, for renewal, node configuration (Sys.Audit at /). The provider write needs Sys.Modify on the node. Unreadable, incomplete or malformed review data blocks submission.

Submit the same fields to the existing custom-certificate POST/DELETE or ACME renewal POST endpoint, adding a unique request_id, the review’s SHA-256 expected_digest, force: true and confirmed: true. The response is HTTP 202 with a change ID and separate provider observation status. force opens the FreeSDN apply gate; it does not mean overwrite or early renewal. The global read-only gate must also permit writes. Old query-only calls and generic plaintext certificate staging are refused. Keep the exact request ID and body when retrying after a lost response; first inspect Pending Changes.

Private keys are encrypted in the durable intent and bound to its change ID. They are redacted from operation responses and events; raw provider upload responses are not saved. Encrypted keys remain in the operation record under the application’s existing retention policy. Protect the database and FreeSDN encryption material. This code change does not retroactively sanitize historical plaintext certificate intents or their backups. Review existing records securely before rollout and rotate keys if exposure is suspected; discarding an intent is not secure erasure. The dialog keeps the key only in memory while needed for the request/retry, removes it from the review display, and does not put it in persistent browser storage or query keys. This is not a memory-zeroization claim.

Before dispatch, FreeSDN re-reads the reviewed state and refuses a changed digest. A claim on the registered controller prevents certificate operations from interleaving with other unresolved staged Proxmox writes. Native Proxmox users, duplicate controller registrations and remaining legacy direct writes are outside that claim. Proxmox offers no atomic certificate digest precondition, so avoid concurrent external maintenance.

Upload acknowledgement only means accepted. The observer independently compares the stored leaf fingerprint and the certificate served by a fresh connection to the configured controller endpoint. Removal requires absence from the readable custom inventory plus the reviewed fallback fingerprint at that endpoint; an omitted inventory entry alone is insufficient. This verifies active fallback, not secure erasure of the old private-key file. ACME renewal requires a matching acmerenew task, a changed leaf fingerprint, expected SANs, validity and endpoint readback. Even a failed renewal task can have installed files, so it stays unresolved for investigation. Task warnings remain visible.

A controller connected through a different node, a TLS-terminating proxy, an unavailable TLS peer, changed trust or missing permissions may require independent verification. Readback cannot verify every node, alternative URL, full installed chain or all clients. FreeSDN preserves the configured TLS verification policy; it never silently switches to unverified TLS or changes trust stores to recover. Stored but inactive changes remain pending or unknown. After seven days, unresolved automatic observation stops and requires manual verification.

Before confirming, verify independent console/SSH recovery access and keep an approved certificate/key backup outside FreeSDN. Proxy reload/restart can interrupt API and console sessions. ACME may revoke the previous certificate, so do not assume rollback is possible. If access is lost, inspect and repair the certificate, key, chain, permissions and proxy from that independent path using the installed Proxmox version’s procedures; do not repeatedly submit another write. Reconcile unknown results in Pending Changes only after verifying the outcome. Resolution preserves the receipt and neither replays the command nor rolls it back.

Reviewed storage uploads and artifact deletion

Use the storage dialog to review an ISO/template upload or artifact removal. Guest disks and root filesystems cannot be deleted here: use the reviewed guest lifecycle instead. Removing an artifact is irreversible; confirm you have another usable copy before deleting a backup. FreeSDN does not establish backup retention compliance or automatically restore a removed file.

POST /hypervisor/controllers/{id}/nodes/{node}/storage/{storage}/review accepts an action (upload or delete). Upload review requires filename, content (iso or vztmpl) and the exact byte size. Deletion review requires the full volume ID. Review returns a snapshot and expected_digest. It checks the storage configuration, availability, permissions and inventory. Deletion refuses protected volumes. ISO removal also checks visible guests’ current, pending and snapshot configurations. Unavailable or malformed data blocks dispatch.

This workflow needs Datastore.Allocate to read the storage configuration, Datastore.Audit to inspect inventory, and the provider’s operation privileges (Datastore.AllocateTemplate for uploads; deletion may also require VM.Backup for an owned backup). ISO reference review requires global VM.Audit and access to guest configurations. It is bounded to 256 visible guests and 64 snapshots per guest. Credential permission overrides can hide guests; the reference check is not proof that an arbitrarily restricted account sees the entire cluster. Qualified deployment credentials and independent review remain necessary.

Submit deletion to the existing content DELETE route with the same reviewed body plus request_id, expected_digest, force: true and confirmed: true. Submit upload as multipart to the existing upload route: file, content, size, request_id, expected_digest, force and confirmed are required for execution. The global read-only gate must permit writes. Old upload/query-only deletion clients and generic server-path staging are refused. These routes now return HTTP 202 operation receipts; inspect Pending Changes for dispatch and provider completion separately. Retry a lost response with the exact request ID, metadata and bytes; never generate another request merely because the response was lost.

Uploads are limited to 4 GiB and conservative ASCII filenames ending in .iso, .tar.gz, .tar.xz or .tar.zst, with no paths, spaces or consecutive dots. Guest image/import uploads and backup-archive uploads are not supported by this workflow. The explicit content category is preserved instead of guessed from MIME. FreeSDN refuses an existing destination filename and repeats review before dispatch. Proxmox has no atomic no-overwrite/digest precondition on this upload API: another native operator can create the filename after the final check. Avoid concurrent external changes. Duplicate controller registrations and legacy direct writes also remain outside the registered-controller claim.

Linux API/worker instances must share the persistent PROXMOX_UPLOAD_DIR volume (the shipped Compose mounts /data/proxmox-uploads). Reviewed sources live in its private reviewed-v1 subdirectory, owned by the service account with mode 0700; files have mode 0600. The source is streamed with a size bound, SHA-256, fsync and an operation-bound filename. Before sending, FreeSDN verifies and retains the same open descriptor, refuses symlink/hardlink substitution, and sends SHA-256 to Proxmox for its import check. The operator’s server filesystem path is never accepted. Pending sources survive API restart. Source cleanup follows a committed dispatch/discard receipt and does not change an acknowledged provider outcome.

A 64 MiB free-space reserve limits normal spool writes. This is not an aggregate quota or ingress capacity guarantee: multipart parsing can create an additional temporary copy before the handler, and concurrent traffic needs deployment-level body/concurrency limits and separate capacity monitoring. A crash before the intent commits can leave incoming-* or unreferenced .blob files; a cleanup failure may also retain a committed source. They are not silently aged out. During a maintenance window with uploads paused, reconcile operation IDs against Pending Changes before removing confirmed orphans. Never delete pending/applying sources based only on age; protect this volume in backup and recovery procedures.

Upload task IDs identify the connected Proxmox endpoint’s imgcopy worker, which may copy to a different destination node. Deletion requires the destination’s imgdel worker and the reviewed storage/owner identity. A task must complete and the observer must read back the same storage configuration plus the expected volume/size (upload) or absence (deletion). Failed tasks may have partial effects and stay unknown. Warnings remain visible. Missing history, changed storage, incomplete inventory or mismatched task identity cannot become success.

These checks verify provider task and inventory evidence. They do not prove secure erasure, an independent checksum of bytes at rest, every storage-plugin identity, or absence of external races. Unsupported inventory/volume formats fail closed. Qualify the exact Proxmox versions, storage plugins, node routing, proxy limits, ACLs, large-file behavior and recovery path before production rollout.

Reviewed scheduled backup jobs

Create, update and delete requests for scheduled backups now require a read-only review, request_id, expected_digest, force: true and confirmed: true. The module routes return HTTP 202 and a durable Pending Changes receipt. A receipt means that FreeSDN recorded the request. backup_job_configuration_verified means that an independent configuration read matched the requested result. Neither proves that a backup ran, that every disk was included, or that a restore will work.

1. POST /api/v1/hypervisor/controllers/{controller_id}/backup/jobs/review with action, job_id and settings (null for deletion).
2. Inspect the prior and desired configuration, selected guests, retention, and inherited node defaults. Type the exact job ID in the UI.
3. Submit the same intent plus the review digest and controls to the existing POST /backup/jobs, PUT /backup/jobs/{job_id}, or DELETE /backup/jobs/{job_id} route. Deletion now requires a JSON body; old confirmed query flags alone are insufficient.
4. Inspect Pending Changes. Do not create a new request after a timeout without first reconciling the saved receipt and the native configuration. Reusing the original request ID and identical intent retrieves its receipt without redispatching it.

Create and update support 1 to 64 explicit guest IDs on one online node, one backup storage, daily HH:MM or comma-separated weekday schedules such as mon,wed 02:00, mode snapshot/suspend/stop, compression zstd/lzo/gzip/0, and an explicit boolean enabled. Times follow the node’s local timezone. Empty guest selection does not mean “all”. Native pool/all/exclusion selectors and advanced execution options are outside this editor and are refused rather than silently rewritten. New IDs are supplied by the client (the UI generates one), limited to 50 characters; safe legacy digest:counter IDs remain supported for update/deletion. Legacy generic staged job payloads must be reviewed again through this workflow.

New jobs default to disabled and explicitly use prune-backups=keep-all=1. This avoids inheriting a pruning policy on creation, but consumes capacity over time. There is no storage quota or backup-size admission estimate in this editor. Updates preserve existing retention and notification settings, including inherited settings; enabling a job can therefore cause pruning when its next run executes. Retention and notification editing remain native operations. Storage reassignment can change inherited retention; the review includes the target storage configuration and node defaults. Stop/suspend mode can interrupt guests. An enabled schedule can start before configuration readback finishes. Deleting a schedule neither deletes its archives nor stops an already running backup.

The review requires an unscoped FreeSDN site administrator (or higher), current site access and an active controller. Provider credentials need Sys.Audit/Sys.Modify on /, Sys.Audit on the selected node, Datastore.Audit/Datastore.Allocate on the storage and VM.Audit/VM.Backup for each selected guest. Explicit node audit prevents hidden hook settings from being mistaken for absent settings. Inherited hooks and fleecing configurations are refused in this workflow. Review reads have a 45-second bound and job inventory is capped at 2,000 records. Unavailable, duplicate or malformed state is not an empty schedule list.

FreeSDN rechecks the reviewed configuration before dispatch and serializes it against unresolved staged Proxmox operations for the same registered controller. Native job update/delete have no atomic review-digest precondition: another native operator or a duplicate controller registration can still race the final check. Selection is a snapshot, not an immutable identity or future authorization guarantee; migration, ID reuse, ACLs, node defaults, retention and storage can change before a later scheduled run. Node-level inherited settings are reviewed, but future execution and scheduler timezone/DST behavior still require supported-version hardware qualification. Pending Changes records job configuration outcomes only; task history, archive integrity, completeness, capacity and restore drills must be checked separately. No automatic retry, rollback, archive pruning, notification delivery or production schedule change is performed by validation tests.

Reviewed task termination

Task termination now uses an exact native UPID, node-bound review and a durable request ID. In the Tasks view, choose Stop, review the running task, then type the node name to confirm. The result opens Pending Changes; it does not announce that the original operation completed successfully.

API clients first call POST /api/v1/hypervisor/controllers/{controller_id}/nodes/{node}/tasks/{upid}/stop-review. The read-only review returns expected_digest, the exact task snapshot and warnings. Then call DELETE on the same task URL (without /stop-review) with a JSON body:

{
  "request_id": "a0393d98-05b6-4893-bf3d-46caa87c191d",
  "expected_digest": "<64-character digest from review>",
  "force": true,
  "confirmed": true
}

Generate one UUID per intended operation and reuse it if the transport response is lost. The endpoint returns HTTP 202 with a durable change ID and provider observation. The legacy body-less DELETE and generic task-stop staged payloads are refused. Old clients must upgrade; do not bypass review by inventing a digest.

The backend checks active controller/site ownership, operator scope, exact task identity, running state and node Sys.Audit/Sys.Modify privileges before dispatch. It rechecks the review at apply time. This admin workflow conservatively requires both privileges even though native Proxmox can allow a task owner to stop their own task with fewer privileges. The global read-only and explicit apply gates still apply.

Native Proxmox acknowledges the stop request synchronously. FreeSDN observes the worker separately: still running remains running, a missing or unverifiable observation remains unknown, and a verified stopped worker gets task_no_longer_running_effects_unverified. The original worker’s success/error code is not used as the stop request’s outcome. A task can also finish naturally between review and dispatch. A stopped worker does not prove rollback, backup integrity, guest health or safe migration recovery. Inspect the original operation, logs and resulting infrastructure before taking further action.

A stop can bypass the conflict caused by its exact acknowledged original worker; unrelated or uncertain claims and another active stop request still block it. This does not serialize external/native operators or duplicate controller registrations. Lost write responses retain an unresolved receipt and are never automatically dispatched again. Explicitly resolving an unknown receipt preserves the unknown observation and does not undo the provider operation.

Task lists request both active and archived workers. Task-list or log failures no longer appear as successful empty results. These paths have controlled regression tests; this source change does not constitute fresh live-cluster qualification.

Reviewed firewall operations

The hypervisor firewall panel keeps cluster, node and guest scopes separate. Changes use POST /api/v1/hypervisor/controllers/{id}/firewall/operations/review followed by POST /api/v1/hypervisor/controllers/{id}/firewall/operations. Submission requires a stable UUID request_id, the returned expected_digest, and strict booleans force, confirmed and recovery_confirmed set to true. Global adapter read-only mode must also be disabled. The latter confirmation means the operator has verified independent console or out-of-band recovery access; FreeSDN cannot prove that access on the operator’s behalf.

The old direct rule POST/DELETE and options PUT routes no longer accept writes. Old generic firewall staging records must be reviewed through the new endpoint; they cannot be replayed. Clients must handle HTTP 202 as a durable receipt, inspect status and provider_task, and never treat transport acceptance as proof of a successful policy change.

Actions are create, update, delete, move and options. The target specifies scope (cluster, node or guest); node scope also needs node, and guest scope adds vm_type (qemu or lxc) and vmid. Update/delete/move require a reviewed pos; move inserts before the original moveto row, matching the native API. Creation inserts at position zero and defaults disabled. Updates preserve omitted rule fields; this interface does not clear optional fields. Options are deliberately constrained: enable at every scope, input/output policy at cluster/guest scope, and selected guest DHCP/IP/MAC/IPv6 flags. Advanced node/cluster settings, aliases, IP-set administration and security-group definition editing remain native-management tasks. Existing security-group references can be represented by the API; the basic UI edits ordinary inbound/outbound rules.

Review reads the rule set, options, guest configuration digest where applicable, and explicit native audit/modify privileges. Apply rechecks the review and passes Proxmox’s native digest for positional operations and options. Native rule creation has no atomic digest guard. These guards do not lock an entire inherited policy graph, external operators, or duplicate controller registrations. An existing native rule error or unsupported flag representation blocks review and needs native inspection.

An acknowledgement is followed by an independent read of the complete expected rules and options. Only matching configuration is recorded as verified. That result does not prove firewall compilation, nftables/legacy-engine behavior, inherited group/IP-set contents, management reachability, packet filtering or workload health. A lost write acknowledgement or mismatched/unavailable observation remains uncertain and must not be automatically retried. There is no automatic rollback. Use Pending Changes, native logs and independent console access to investigate before explicitly resolving uncertainty.

Before a hardware rollout, export the native configuration securely, verify console access, use an isolated non-management test network/guest, and test allow/deny traffic independently. Exercise both deliberate rule reordering and loss of management reachability, then recover through the console using the saved native configuration. Qualify each supported PVE version, firewall engine and token privilege set. Controlled HTTP tests do not replace that drill.