Upgrading

FreeSDN uses CalVer versioning in the format YY.MM.PATCH (e.g. 26.06.1). Releases are tagged in the repository with a leading v (e.g. v26.06.1). There is no automatic update mechanism - upgrades are a deliberate operator action.

Prepared candidate, not a production release

These procedures describe the prepared 26.10.0 candidate. A version in source is not a release announcement. Use the runbooks from the exact reviewed release checkout; do not combine an older backend with new recovery settings. Hosted CI, published-artifact installation and site-specific acceptance remain separate gates.

Before every upgrade: take a backup

Never upgrade without a fresh encrypted, completed pair of primary and LogDB backups and a successful isolated restore drill. A successful dump alone does not establish recoverability.

The configured pg-backup service starts a backup cycle when restarted. Confirm its encryption recipient and public key are configured, restart only that service, then wait for a completed cycle and inspect its health/logs. Do not proceed while backups are disabled, incomplete or stale. Retain the old images, configuration, encryption/signing keys and named volumes. Never use docker compose down -v during an upgrade.

Database runtime transition: read before rebuilding

Applies from 26.10.0. PostgreSQL and LogDB stay on Debian (glibc), the same runtime as 26.09.0, so an existing 26.09.0 database volume upgrades in place. Pull, rebuild, restart. There is no logical migration step, no new volume and no locale change.

On first start against a cluster created before this release, the entrypoint records a runtime marker and continues. It reports what it is doing:

FreeSDN: adopting an unmarked PostgreSQL 18 cluster.
FreeSDN: recording runtime marker postgresql-18-debian-trixie-v1. No data is
read, rewritten or converted.

The guard still refuses a cluster it cannot safely serve, with exit code 78:

• a cluster whose recorded runtime does not match the image, which is how an Alpine/musl cluster is caught
• a different PostgreSQL major version
• an older data layout present alongside the current one

If you are coming from an Alpine-based cluster, that refusal is correct and you do need the offline path: use docker/postgres/UPGRADING and scripts/migrate_database_runtime.py, which performs a logical restore into new named volumes with the same bootstrap role, matching extension versions, stopped writers, role, permission and content verification, and index checks. It never switches the deployment or deletes the old data, so the original volume remains your rollback.

Fresh clusters are created with data checksums and the Debian default locale provider, matching 26.09.0, so text ordering and case behaviour do not change. PostgreSQL’s builtin C.UTF-8 provider is available if you want collation that does not depend on the host libc: set POSTGRES_INITDB_ARGS explicitly before initialising a new cluster. Do not apply it to an existing one.

TimescaleDB is built from checksum-verified source at 2.30.2, the version 26.09.0 resolves, so the extension does not move during the upgrade.

Coordinated application cutover

Do not use an unrestricted docker compose up -d as a migration plan. A new API or worker can accept old credentials, run scheduled work, or contact devices before recovery checks finish. Use a maintenance window and a separate, isolated target deployment. This sequence applies only after the database runtime transition has been completed, or when no database runtime change is involved.

1. Record the exact source release, image identities, enabled profiles, database and application volume names, environment and external dependencies. Preserve those inputs and verified encrypted backups. Select the intended existing, reviewed release tag in a separate checkout; do not fetch a branch and assume it is a production release. Source builds remain the canonical distribution. Build and qualify the target images before the maintenance window, without replacing the source deployment’s only images or attaching its data volumes.
2. Fence ingress and outgoing device/SMTP traffic. Stop all old APIs, schedulers, workers, collectors, plugin workers and external consumers, including HA replicas. Drain or account for active, reserved and scheduled tasks. Preserve broker and Sentinel state. An empty target broker is a deliberate recovery choice, not evidence that customer work was migrated.
3. Keep SECRET_KEY and ENCRYPTION_SALT unchanged for database-level recovery; they must match the existing encrypted vault. Apply the candidate’s normal migrations once against isolated target databases with general workers still stopped. Follow docker/TOKEN-REVOCATION-UPGRADING, docker/redis/QUEUE-BACKUP-RECOVERY and docker/SLA-REPORT-DELIVERY from that checkout. Do not stamp a restored database directly to head to skip migrations.
4. If restoring or rolling back a snapshot, provision a fresh independent JWT_SIGNING_KEY on every target backend process before starting it. Preserve it outside the snapshot. Incrementing a restored token version once is not a complete fence: a later-issued token can already have that value. Restore report-mail state only with a new coordinated SLA_REPORT_DELIVERY_EPOCH, review of old notices and explicit schedule reapproval. Do not copy old epochs or approvals to make delivery resume.
5. Start only the target services needed for isolated acceptance. Recreate services to apply changed environment values; restart alone does not reload them. Verify both database schemas, representative data and vault decryption, new password login, rejection of pre-recovery access/refresh/reset/MFA tokens, report archives and newly generated reports. Review non-JWT credentials, customer jobs and device state separately before allowing any outgoing work.
6. Start reviewed workers under the same current keys and epoch. Verify their queue subscriptions and harmless tasks, then repeat checks after container recreation. Reopen ingress and device/SMTP traffic only after the site’s acceptance criteria pass. Preserve original volumes and images through the agreed retention window.

For read-only checks after the target API is running in isolation:

docker compose --env-file .env.pro exec api alembic current
docker compose --env-file .env.pro exec api python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/api/v1/health/ready', timeout=5); print('ready')"

Select the target project’s Compose files and environment deliberately. A green readiness response is one observation; it does not certify the recovered data, authentication, mail or hardware state. The Lite upgrade rehearsal used current edge and Valkey images in both phases and an empty target broker. It does not qualify a historical UI/broker migration, queued customer work or safe rollback to old authentication keys.

How migrations work

The API entrypoint runs scripts/migrate.py before starting the application. The prepared 26.10.0 startup guard distinguishes a dedicated empty database from one with recognized migration history. Being marked at head is not a complete schema or restored-data integrity check.

What migrate.py detects

DB state	Action taken
No migration table and no non-extension user relations	Creates the current schema and stamps its revision in one transaction. Pre-created empty schemas are allowed.
Existing tables, views, materialized views, sequences or foreign tables without migration history	Refuses before schema changes. This includes unrelated application objects; use a dedicated empty database for installation.
Exactly one current revision	Reports that it is at head and performs no migration. Validate restored schema and data separately.
Exactly one recognized older revision	Runs the incremental migration chain from that revision.
Empty, blank or multiple revision rows	Refuses rather than selecting or guessing a revision.
Revision outside the current migration chain	Refuses. Preserve the database and use its matching release or a separately reviewed migration plan.

Fresh initialization uses the same database transaction for schema creation and revision stamping. An interruption before commit rolls both back, allowing a normal startup retry. If the process loses the commit acknowledgement, the complete schema and revision may already be committed; a normal retry recognizes the current revision. This atomic behavior applies to fresh initialization, not to the entire historical incremental migration chain. If a previous release left a populated database without revision history, preserve it and follow the recovery guide; do not delete tables or force a stamp to make startup proceed.

Compatibility change: older startup code guessed that every unknown revision was pre-consolidation, created missing model tables, re-stamped to 001_initial and replayed migrations. The prepared candidate removes that guess. A future, divergent or incompletely restored schema must not be silently treated as an old baseline. Never delete revision metadata or stamp to head to bypass this guard. See docker/MIGRATION-STARTUP-SAFETY in the matching source checkout for recovery steps and the limits of its cooperating-process advisory lock. The guard does not prevent a database administrator from changing catalogs or metadata.

To check the current migration state without upgrading:

docker compose --env-file .env.pro exec api alembic current
docker compose --env-file .env.pro exec api alembic history

Rollback and PostgreSQL updates

A downgrade is not a routine recovery command. Review each migration’s actual downgrade() implementation and test the complete application on a copy. Some guards deliberately refuse to discard nonempty history, including report delivery records. Never delete that history or stamp around a refusal. An old backend can bypass new authentication or delivery guards even without a schema downgrade.

Before admitting new writes, retained source data can support an operator-reviewed rollback. Keep both deployments fenced while selecting one authority. After new writes, returning to an old snapshot loses or duplicates work unless it is reconciled. JWT, non-JWT credentials, report notices and hardware effects require the recovery review described above. Prefer repairing the current recovery-aware version over starting an older backend against the restored database.

A PostgreSQL minor version alone does not establish image compatibility. Changes to the operating system, locale provider, extension build, UID or entrypoint can require logical migration even within PostgreSQL 18. Use the FreeSDN runtime guide for both primary and TimescaleDB; do not substitute a generic upstream database image or manufacture the FREESDN_RUNTIME marker. Major-version changes and custom extensions need a separately rehearsed migration plan.

Testing adapter changes safely

When upgrading a vendor adapter (e.g. pulling a new Omada adapter version), always test with ADAPTER_READ_ONLY=true first. The shipped Docker Compose stack runs read-WRITE (ADAPTER_READ_ONLY=false), so set ADAPTER_READ_ONLY=true in the selected environment file (for example .env.pro) for the test, then verify the running value:

docker compose --env-file .env.pro exec api \
  python -c "from app.core.config import settings; print('ADAPTER_READ_ONLY=', settings.ADAPTER_READ_ONLY)"

ADAPTER_READ_ONLY is checked by supported adapter paths, including staged apply. It is not a network boundary or a guarantee that every plugin, external worker or operation family is side-effect free. Keep egress restricted to reviewed test targets and retain the operation-specific capability checks. Never infer permission to apply a change from a successful read or from a green health check.

1. Deploy the isolated target with ADAPTER_READ_ONLY=true and verify the running setting on every relevant process.
2. Run reviewed read/sync operations against an authorized disposable target.
3. Check errors, circuit breakers, tenant/site scope and returned data against the device’s actual firmware/API version. A short observation is not soak evidence.
4. Authorize device writes separately, with a rollback/restore plan and operation acceptance for that target. Do not automatically enable writes after 30 minutes.

The 3-day rule for dependency updates

Never adopt a Python or Node package release on the day it ships. Wait 72 hours. This window catches withdrawn releases, missing distribution files, and compromised maintainer incidents that are common on PyPI and npm the day a new version appears. Security CVE patches override this rule - apply them as soon as the patch is signed by upstream.

Checking for migration conflicts after a merge

If two branches each added a migration and both end up in alembic/versions/, alembic upgrade head will error with “Multiple head revisions are present”. Linearize the chain by editing the down_revision of the newer migration to point at the older migration’s revision ID, then re-run alembic heads to confirm a single head before applying.

Never use alembic merge in production without first testing the merged revision on a copy of the production database - the auto-generated merge revision is a no-op that masks the conflict rather than resolving it.