CloudEdge Autonomous Lab (cloudedge-labctl)

Experimental (CloudEdge). A single-command harness that lets an agent run cloud-edge Selective Address Mobility (SAM) failover labs without a human reading runbooks. It fixes the interface and implements all non-cloud logic (run-id/tag convention, TTL + teardown cost guard, fault primitives, connectivity matrix, evidence assembly). Real per-provider provisioning either wraps the existing examples/cloudedge-mobility-demo/ package or is marked TODO(lab-operator) for Terraform/CLI wiring.

The harness is scripts/cloudedge-labctl.sh, with two helpers:

• scripts/cloudedge-connectivity-matrix.sh — directed ping+ssh matrix + assertions.
• scripts/cloudedge-evidence-schema.json — JSON schema for the run evidence.

--help, dry paths, and down --expired need no cloud credentials.

Observe convergence without repeating the whole test

SAM waits for provider Ready before judging traffic. A validation invocation shares the existing 900-second provider convergence budget with its traffic observations. Only unsuccessful or missing required flow components are reacquired, at most three observations per component; successful components and identical cloud-ingress flows share evidence within that invocation and unchanged fault state. Successful flows do not run traceroute. Missing management-SSH completion is OBSERVATION_INCONCLUSIVE (exit 3), not an observed product ping failure. Raw attempts, including failures before recovery, remain under matrix/<label>/flows.*/.

L2 collection errors or missing metrics are inconclusive and retain stdout/stderr; zero is valid only after successful collection. Gateway unchanged compares the route immediately before and after each flow on its source host, not a deployment-wide baseline. Failover actions must match the current fault start, address and standby. Recovery time includes coordinator/API/SSH/poll overhead; exceeding that upper bound alone does not establish a product regression. The 60-second goal remains unchanged. Failed netns tests and cleanup retain their working directory and cleanup log. FreeBSD same-stack VXLAN pings do not establish forwarding through WireGuard: the smoke reports configuration, handshake, ownership and restart evidence, with forwarding unverified. IPC reads have deadlines within the existing wait budget; unavailable reads use exit 3. The bus slow-subscriber test uses a deadlock watchdog, not a throughput threshold.

Diagnostic collection is separate from acceptance. The convergence loop does not run doctor sam; diagnostics collect it once. Each diagnostic snapshot reads status, applied BGP paths, and the global RIB once and derives summaries from those saved responses. Diagnostic collection failures do not change the traffic verdict.

For retained ARP checks, the 80 ms monitoring HTTP timeout, 100 ms counter cadence, 400 ms continuous coverage and every-interval controller progress are diagnostic constraints, not acceptance conditions. Diagnostic failures must not stop capture or sender collection. Acceptance still requires complete self-ARP capture, original target-command evidence matched to packets, fresh request/TTL evidence, completed communication and bound process identity; missing required evidence is inconclusive. Existing timestamps, targets, command sequences and provider resource IDs remain in raw observations for correlation.

Cleanup inventory distinguishes lifecycle changes from identity mismatches. Matching AWS/OCI identities with disagreeing lifecycle observations may requery only that provider twice within the enclosing inventory invocation. Every raw attempt is retained. Wrong identity/tag/account, malformed responses and transport failures never prove absence. Other provider inventories still run; incomplete scopes retain a null count and cannot pass the zero-resource gate. Existing resource recovery and deletion safeguards are unchanged.

Before another paid test, replay the affected flow and inventory fixtures locally: test_sam_observation.py, test_inventory_resources.py, and test_inventory_driver.py under tools/release-qa-labs/tests/. These do not create resources. Run make cloudedge-representative-redundancy-offline-test for the actual matrix/profile callers. Passing fixtures does not claim a new cloud deployment was tested.

Lifecycle

scripts/cloudedge-labctl.sh up        --profile full --provider aws,oci,azure,onprem --provider-order cost-optimized --ttl 4h
scripts/cloudedge-labctl.sh deploy    --commit HEAD          # or --build <dist path>
CE_MATRIX_PARALLELISM=8 scripts/cloudedge-labctl.sh smoke --matrix d3 --out /tmp/matrix.json
scripts/cloudedge-labctl.sh failover  --provider aws --fault stop-active
CE_MATRIX_PARALLELISM=8 scripts/cloudedge-labctl.sh smoke --matrix d3 --out /tmp/matrix-after.json
scripts/cloudedge-labctl.sh evidence  collect --out evidence/<run-id> --matrix-json /tmp/matrix-after.json
scripts/cloudedge-labctl.sh down      --run-id <run-id> --force

up prints the run-id on stdout; capture it and pass it to later commands. Cloud mutations are DRY by default (CE_DRY_RUN=1); set CE_DRY_RUN=0 to act for real once credentials and budget are approved.

Profiles

Profile	Sites	Use
minimal	on-prem + 1 cloud	cheapest smoke; interface/CI shake-out
provider	one provider A/B routers + client	provider parity (AWS/OCI/Azure seize)
full	on-prem + AWS + OCI + Azure	the 4-site /24 12-flow demo
soak	a full run held open for its whole TTL	long-running reconvergence checks

soak is operationally a full run with a long --ttl left up (do not run down until TTL); use it to exercise BFD/BGP reconvergence and reconverge.

TTL and cost policy

Every cloud resource MUST carry these tags (helpers cloudedge_tags() emit them, up stamps them):

routerd.cloudedge.run_id          <UTCdate>T<time>-cloudedge-<scenario>
routerd.cloudedge.owner
routerd.cloudedge.ttl_expires_at  absolute UTC RFC3339
routerd.cloudedge.provider
routerd.cloudedge.purpose

Cost guard rules:

• up --ttl <dur> stamps ttl_expires_at. Pick the shortest TTL that fits the run.
• down --run-id <id> tears down one run; down --expired tears down any run past its TTL (safe no-op when there is no lab — exit 0).
• up validates --ttl up-front and hard-fails (non-zero) on an invalid duration, so a lab is never started with a broken/already-expired cost guard.
• An EXIT trap in the harness tears down the run if up is interrupted or a provider fails mid bring-up (armed only during the in-progress window, then disarmed on clean success; a successful up leaves the lab alive until explicit down or TTL). up --keep opts out and leaves the partial state for inspection.
• Always run down (or down --expired from a janitor) after every run, even on failure. Past-TTL labs are cleanable without knowing the run-id.

For release qualification, prefer --provider-order cost-optimized. The harness starts non-cloud/on-prem resources first and then starts cloud providers as OCI -> AWS -> Azure, so cheaper OCI capacity is online first and AWS does not sit idle behind slower PVE or Azure work. Use --provider-order declared to preserve an explicit --provider order, or --provider-order fast-first when shortest wall-clock time matters more than idle cloud cost.

Fault primitives (failover --fault)

Fault	Meaning	First-pass wiring
stop-active	stop the active router VM/instance	provider stop CLI (see reset-lab.sh)
drain	MobilityPool maintenance.drain=true on active	reuse run-demo.sh *-drain.yaml
routerd-bgp-stop	stop routerd-bgp (BGP session drops)	ssh systemctl stop routerd-bgp
executor-fail	provider action executor denied (identity scope-down)	identity policy
stale-replay	replay a stale-pathSig action; must be fenced	probe_stale_gate_on_aws_b

Inject a fault, then re-run smoke and evidence collect to prove recovery.

Evidence schema

evidence collect --out <dir> writes <dir>/result.json validating against scripts/cloudedge-evidence-schema.json, plus summary.md and (if provided) the connectivity matrix JSON. Shape:

{
  "runId": "20260601T031500Z-cloudedge-aws-failover",
  "commit": "<sha>",
  "scenario": "aws-active-stop-seize",
  "result": "pass",
  "providers": {
    "aws":    {"dataplane": "pass", "providerState": "pass"},
    "oci":    {"dataplane": "pass", "providerState": "pass"},
    "azure":  {"dataplane": "pass", "providerState": "pass"},
    "onprem": {"dataplane": "pass", "providerState": "pass"}
  },
  "assertions": [
    {"name": "ownership_epoch_bumped", "result": "pass"},
    {"name": "allow_reassignment_maintained_until_success", "result": "pass"},
    {"name": "source_ip_preserved", "result": "pass"},
    {"name": "default_gateway_unchanged", "result": "pass"},
    {"name": "old_holder_residue_absent", "result": "pass"},
    {"name": "stale_action_fenced", "result": "pass"}
  ],
  "qualificationTimings": {
    "infraProvisionSeconds": null,
    "deployRouterdReadySeconds": null,
    "providerConvergenceSeconds": null,
    "normalMatrix": {
      "status": "pass",
      "phase": "normal",
      "elapsedSeconds": 120,
      "parallelism": 8,
      "total": 56,
      "passed": 56,
      "failed": 0
    },
    "cloudIngressMatrix": {
      "status": "pass",
      "phase": "cloud-ingress",
      "elapsedSeconds": 45,
      "parallelism": 8,
      "total": 42,
      "passed": 42,
      "failed": 0
    },
    "dataplaneProbeSeconds": 165
  },
  "costGuard": {"ttlHours": 4, "teardown": "completed"}
}

Data-plane checks and source_ip_preserved / default_gateway_unchanged are derived automatically from the connectivity matrix. The seize/fencing assertions (ownership_epoch_bumped, allow_reassignment_maintained_until_success, old_holder_residue_absent, stale_action_fenced) and providerState start as na and are folded in by the lab operator from provider inventory, BGP mobility paths, provider trap action plans, and the action journal (see collect-evidence.sh). A run is only PASS when result == pass and every required assertion passes.

The connectivity matrix

cloudedge-connectivity-matrix.sh runs every directed src -> dst flow over the shared /24 and asserts, per flow:

• source-IP-preserved — the destination sees the real source client IP (no NAT).
• default-gw-unchanged — the source client's default gateway is unchanged.
• no-NAT — ping reaches the destination and the SSH peer IP equals the source IP.

Execution goes through a MATRIX_RUNNER indirection so the matrix is unit-runnable offline (set MATRIX_RUNNER to a stub); with a real lab the default runner uses ssh/ping against the demo env. Output is per-flow JSON consumable by evidence collect --matrix-json.

Set CE_MATRIX_PARALLELISM or pass --parallel <n> to run directed flows with bounded parallelism. The JSON flows[] order remains deterministic, and summary.phase, summary.parallelism, and summary.elapsedSeconds record the probe phase and harness runtime. When cloud-ingress is measured separately, pass it to evidence collection with --cloud-ingress-matrix-json; result evidence then separates qualificationTimings.normalMatrix, qualificationTimings.cloudIngressMatrix, and qualificationTimings.dataplaneProbeSeconds from VM provisioning and provider convergence.

Autonomy charter (summary)

The agent owns the full loop — lab up -> deploy -> fault inject -> data-plane verify -> evidence -> teardown -> issue/PR update — and runs it without a human reading a runbook. Cloud actions are dry by default and gated behind explicit credential/budget grants. The agent must always leave the lab torn down or within its TTL cost guard, and must attach a schema-valid evidence bundle to any PASS.

Human gates

Only these require a human; everything else is automated:

1. Budget — approving spend / raising the TTL or budget ceiling.
2. Credentials/permissions — granting cloud credentials and the least-privilege identity/roles the executor uses (no secrets are committed or passed to plugins).
3. Merge — final approval to merge a PR.
4. Production — any production rollout (never performed by the lab harness).

Caveats

• This is a lab harness, not a production turnkey.
• First pass: real per-provider allocation/teardown/node-push are TODO(lab-operator) stubs or thin wrappers over the demo package — wire Terraform/OpenTofu or provider CLIs filtered by the run-id tag.
• Never commit real account IDs / subscription IDs / OCIDs / ENI/VNIC IDs / secrets / private keys. Use placeholder logical addresses as in env.example.

Guest preparation evidence

Provider CLI installation is split between OpenTofu cloud-init and SAM preparation. AWS retains its preparation fallback; Azure/OCI use their existing cloud-init installers. Their command lists stop on failure. Preparation saves cloud-init status/output and verifies the required CLI in the root service PATH before deploying routerd. Operator CLI availability or a boot completion marker alone does not prove the guest dependency.

Existing convergence/warm-up attempts preserve stdout/stderr, exit and UTC times. Diagnostics use sudo journal reads and public WireGuard fields; missing GoBGP is recorded as unavailable rather than parsed as JSON. The map of prerequisites for existing test families is tools/release-qa-labs/test-prerequisites.md.

The shared cloud-init CLI preparation retains every attempt. Downloads use curl transient retries; known network/package-lock installer failures retry locally up to three times. Permanent errors and exhausted attempts preserve the failure and stop preparation. Client APT preparation retains its existing four attempts, limited to known transient failures; it does not repeat the scenario.

Performance retries retain raw output, exit codes and timestamps for each server/client attempt. The existing three attempts and canonical metric filenames remain unchanged; a later successful attempt does not erase the failed attempt.