Release environment certification

Release qualification may only run on an environment that has first been certified. The certification phase proves that the PVE and cloud substrate is healthy enough to be used as a test environment. It is not a routerd product test.

When certification or qualification requires a release rollback, follow Release rollback procedure. In particular, a binary rollback does not authorize an implicit configuration rollback.

Phase boundary

Release validation is split into two phases:

1. Environment certification: repair and certify PVE, bridges, QGA, config disks, cloud bootstrap, credentials, quotas, images, and basic reachability.
2. Release qualification: run routerd readiness, connectivity matrix, and provider action checks on a certified environment.

All substrate repair belongs to the certification phase. Once release qualification starts, operators must not fix PVE VMs, bridges, QGA, config disks, cloud-init, NoCloud media, provider bootstrap, identity, routes, security groups, or equivalent lab substrate in place. Stop the qualification run, mark it blocked by an infra failure, repair in certification, and issue a new certification manifest.

Certification status

A certification manifest has one of these statuses:

Status	Meaning
pass	The environment is certified until expiresAt. Release qualification may start.
fail	The environment is not usable. This is an infra failure, not a routerd product failure.
blocked	Required substrate state could not be inspected or repaired. This is an infra failure.
expired	A previous pass is older than its validity window. Qualification must not start.

Certification is valid only for the environment, topology, tool revisions, and commit range recorded in the manifest. Any material change to PVE hosts, VM templates, bridge layout, cloud bootstrap images, cloud account wiring, provider networking, or routerd-labs certification scripts requires a new certification.

Default validity is 24 hours unless the manifest states a shorter window. Release qualification must compare current UTC time with expiresAt; stale certification is equivalent to no certification.

Reproducible environment contract

Each release qualification environment must be reproducible from a recorded contract, not from operator memory. The certification manifest or attached evidence must identify:

• the run ID used to tag cloud resources, PVE VMs, config media, logs, and evidence paths;
• the exact routerd commit or release artifact and the routerd-labs script commit used to provision and qualify the environment;
• provider account, subscription, tenancy, profile, region, and CLI/auth mode, using logical profile names and never relying on an implicit default profile;
• OpenTofu working directory, state path, variables file, provider lock file, and the output JSON consumed by deploy or qualification scripts;
• PVE node, datastore, boot source, underlay bridge, capture bridge, client mode, reusable client VMIDs when retained, and disposable VMIDs when created;
• PVE management address source. DHCP/QGA-discovered addresses are allowed; hard coded management IP assumptions are not release evidence unless explicitly certified for that run;
• whether routerd state databases, provider-side secondary IPs, PVE guests, config media, and guest /tmp/routerd-* artifacts are fresh, retained with matching state, or repaired during certification.
• the rollback scope, release artifact SHA-256, configuration candidate provenance, and resulting canonical configuration SHA-256 when rollback was used to prepare the environment.

The accepted starting modes are:

• fresh fabric, fresh routerd state: all disposable cloud/PVE resources are created for the run ID and routerd starts without previous state;
• retained fabric, retained matching routerd state: reusable resources remain in place and the corresponding routerd state is part of the certified input;
• retained fabric after certified repair: retained resources are inspected, cleaned or patched during certification, and every retained artifact is listed in the manifest with post-repair proof.

Do not start qualification from a retained or dirty fabric with a fresh routerd database unless the certification manifest records the retained artifacts and the repair that makes them compatible. That setup otherwise produces ambiguous provider ownership and action-journal evidence.

Before qualification starts, certification must run the reproducibility checks that match the selected mode:

• refresh OpenTofu state and verify that planned changes are expected for the run mode;
• verify provider inventory is scoped by run ID or certified reusable resource identifiers;
• verify PVE QGA, hostname, config media, and management addresses match the generated topology output;
• remove or explicitly retain guest temporary routerd artifacts such as /tmp/routerd-*;
• run bridge and route reachability audits for the PVE/cloud fabric;
• write the certification manifest and evidence paths before product qualification scripts are invoked.

Required checks

Certification must cover both local and cloud substrate.

PVE substrate

The PVE certification script is expected to verify:

• PVE API access and expected node inventory.
• Required bridges and VLAN-aware settings exist.
• VM templates or base images exist and match the expected identifiers.
• QGA is enabled and responsive for certification test VMs.
• Config disk or NoCloud media is attached, readable, and regenerated when intentionally repaired during certification.
• VM boot reaches SSH and the expected hostname/user-data state.
• Required firewall, forwarding, and L2/L3 reachability checks pass between PVE test endpoints.

Repairs are allowed here. The manifest must record each repair action, including what was changed and how it was verified afterwards.

Cloud substrate

The cloud certification script is expected to verify:

• Provider credentials and selected account/subscription/tenancy identity.
• Required regions, quotas, images, instance shapes, SSH keys, and bootstrap templates.
• Provider networks, route tables, security groups, public IP behavior, and metadata service access needed by the release topology.
• Cloud-init or equivalent bootstrap reaches SSH and expected hostname/user-data state.
• Provider inventory and teardown filters are constrained by release run IDs or test tags.
• Basic east/west and cloud-to-PVE reachability checks pass for certification endpoints.

Repairs are allowed here. The manifest must record each repair action and the post-repair check that made the environment certifiable.

Certification script contract

The generic script implementations live in this repository under scripts/. Environment-specific PVE and provider operations remain in the private routerd-labs driver. This keeps the executable interface, schema validation, preflight, watchdog, and cleanup invariants under public CI without exposing credentials or private infrastructure identifiers.

Both certifiers consume one run contract. The contract fixes the run ID, exact routerd artifact and checksum, routerd-labs revision, logical provider profiles/regions, OpenTofu paths, PVE topology inputs, and lifecycle limits. Drivers receive the contract and must return structured checks and repairs; free-form successful output is not accepted as certification evidence.

certify-pve-substrate.sh

Purpose: inspect and, when requested, repair the PVE substrate before release qualification.

Required interface:

certify-pve-substrate.sh \
  --environment <name> \
  --topology <name> \
  --contract <run-contract.json> \
  --driver <routerd-labs-pve-driver> \
  [--cloud-certification <cloud-manifest.json>] \
  --out <manifest.json> \
  [--repair] \
  [--valid-for 24h]

Behavior:

• Without --repair, perform read-only checks and fail if repair is required.
• With --repair, perform PVE-only substrate repair before producing the final certification result.
• Exit 0 only when the PVE portion is certified.
• Exit non-zero for infra failure or blocked inspection.
• Verify the exact release artifact path and SHA-256 from the run contract.
• When cloud is certified first, merge its passing manifest with the PVE result; reject a different run ID, environment, or topology.
• Write a manifest that conforms to docs/releases/manifests/release-environment-certification.schema.json.

certify-cloud-substrate.sh

Purpose: inspect and, when requested, repair cloud substrate before release qualification.

Required interface:

certify-cloud-substrate.sh \
  --environment <name> \
  --topology <name> \
  --providers aws,azure,oci \
  --contract <run-contract.json> \
  --driver <routerd-labs-cloud-driver> \
  [--pve-certification <pve-manifest.json>] \
  --out <manifest.json> \
  [--repair] \
  [--valid-for 24h]

Behavior:

• Without --repair, perform read-only provider checks and fail if repair is required.
• With --repair, perform cloud-only substrate repair before producing the final certification result.
• Exit 0 only when every requested provider is certified.
• Exit non-zero for infra failure or blocked inspection.
• When PVE is certified first, merge its passing manifest with the cloud result; reject a different run ID, environment, or topology.
• A final merged manifest from either valid phase order conforms to the certification schema and contains both certifier results.

release-environment-preflight.sh

Purpose: verify that release qualification is allowed to start.

Required interface:

release-environment-preflight.sh \
  --certification <manifest.json> \
  --environment <name> \
  --topology <name> \
  --providers aws,azure,oci,pve \
  [--release <version-or-commit>]

Behavior:

• Validate the manifest schema.
• Fail if status is not pass.
• Fail if expiresAt is in the past.
• Fail if environment, topology, or provider set does not match the requested release qualification run.
• Fail if required PVE/cloud check groups are missing.
• Recompute the release artifact SHA-256 and fail on artifact drift.
• When --release is supplied, fail unless it matches the certified version or commit.
• Never repair substrate.

release-qualification-smoke.sh

Purpose: run product qualification on a certified environment.

Required interface:

release-qualification-smoke.sh \
  --certification <manifest.json> \
  --release <version-or-commit> \
  --out <qualification-result.json> \
  --qualification-command <routerd-labs-qualification-driver> \
  --cleanup-command <run-id-scoped-cleanup-driver> \
  --inventory-command <post-cleanup-inventory-driver> \
  --evidence-dir <directory> \
  [--ttl 75m] \
  [--heartbeat-stale 5m] \
  [--cleanup-timeout 15m]

Behavior:

• Run release-environment-preflight.sh first and abort if it fails.
• Run routerd readiness, connectivity matrix, and provider action smoke checks.
• Never repair PVE, bridge, QGA, config disk, cloud bootstrap, provider network, identity, or quota issues.
• Classify failures from routerd readiness, matrix, or provider action checks as routerd product failures only when preflight passed.
• Classify any discovered substrate defect as infra failure and stop the run.
• Start an independent watchdog before the qualification driver. A stale heartbeat or expired TTL terminates the driver, invokes run-ID-scoped cleanup, and then invokes the inventory driver.
• Terminate a hung cleanup or inventory process group after --cleanup-timeout and record the timeout as an infra failure.
• Invoke cleanup and post-cleanup inventory on success and failure. Exit 0 only when product checks pass and both cleanup commands return success.
• Write a result conforming to docs/releases/manifests/release-qualification-result.schema.json.

Evidence and retention

Certification manifests are release evidence. Store them under docs/releases/manifests/ or attach them to the release PR using the same schema. Do not commit secrets, private IPs that identify private infrastructure unless they are already public in the test topology, provider resource IDs, or raw CLI debug logs. Record logical names, check results, timestamps, tool versions, and repair summaries instead.