# Staged Upgrade — Use Cases & Requirements

This document captures the staged-upgrade feature use case by use case, for both Edge Routers (existing, in the current PR) and the Network Controller (gap identified in review). For each use case:

- **a. Existing** — what the current PR already supports.
- **b. Expected (new)** — the feature requested via review comments.
- **c. Status after** — the resulting state of the resource(s) once the use case completes.

---

## What Exists Today — Stage & Upgrade-Staged

### Stage (download binaries, no activation)

- **`core`:**
  - `NetworkStageController` — `POST /v2|v3/networks/{id}/stage` — stages many Edge Routers (IDs from request body), optional `target` version.
  - `EdgeRouterStageController` — `POST /v2|v3/edge-routers/{id}/stage` — stages a single Edge Router (ID from path), optional `target` version.
  - Both call the same `StageEdgeRoutersProcess.stageToVersion`.
- **SDS:** `EdgeRouterDeploymentController` — `POST /deployments/edge-routers/stage` → `EdgeRouterService.stageEdgeRouter` → salt orchestration `router_stage_files.sls`, which:
  - Stages the **Network Controller** first (`ziti.stage.controller.binaries`) — downloads/extracts the target binaries onto the controller, purely so it can serve as the file source for routers.
  - Then stages the **Edge Router(s)** (`ziti_upgrade.stage.routers_stage_files`) — copies binaries + writes a `requested_version` file on each router.
  - Nothing is activated; everything keeps running its current version.
- **No equivalent exists for the Network Controller alone** — it can only be staged as a side effect of this router-targeted call.

### Upgrade-Staged (activate previously staged binaries)

- **`core`:** `NetworkUpdateStagedController` — `POST /v2|v3/networks/{id}/update-staged` → `UpgradeStagedEdgeRoutersProcess.upgradeStaged`. Does not accept a target version — it just activates whatever is currently staged on disk.
- **SDS:** `EdgeRouterDeploymentController` — `POST /deployments/edge-routers/upgrade-staged` → `EdgeRouterService.upgradeStagedEdgeRouter` → salt orchestration `router_upgrade_staged.sls`, which enables the `ziti-router-upgrade.timer` on the target **Edge Routers only** — this timer performs the actual binary swap + restart.
- **The Network Controller is never touched by this call** — no step in `router_upgrade_staged.sls` references it, and there is no controller equivalent of this endpoint at all.
- Cleanup: `NetworkUpdateStagedController`'s process also calls SDS `upgradeStagedNetworkFinish` → `upgrade_finish.sls`, which stops/disables the timer on the routers once they're confirmed online at the new version.

#### Detailed request flow — `POST /core/v3/networks/{id}/update-staged`

1. **`core` — `NetworkUpdateStagedController.startNetworkUpdateStaged`**
   - `@ApiVersionRequired` resolves the API version (v2/v3).
   - `@AuthAction(action = "update-version", on = Network.class)` authorizes the caller against the network.
   - `networkService.getProvisionedNetwork(id)` — network must exist and be `PROVISIONED`, else `400`.
   - Builds `UpgradeStagedContext` (`networkId`, `ignoreOffline`, `edgeRouterIds`, `ownerIdentityId`); `upgradeMode` is hardcoded to `EDGE_ROUTERS_ONLY` — the controller is never in scope for this call.
   - Calls `UpgradeStagedEdgeRoutersProcess.upgradeStaged(context)`.
2. **`UpgradeStagedEdgeRoutersProcess` (multi-step `Process`)**
   - *Check Network version* — re-checks `PROVISIONED`/`UPDATING`; **asserts `edgeRouterIds` is non-empty** (`"At least 1 edge router must be included in the request"` if not); asserts every requested ID belongs to the network.
   - *Check EdgeRouter(s) availability* — resolves routers, checks hosts online via `upgradeService.getOnlineHosts`; cancels if any are offline and `ignoreOffline=false`.
   - *Update Staged Edge Routers status to UPDATING* — sets each targeted router's `status` to `UPDATING`.
   - *Pre Upgrade-Staged Hooks* — currently a no-op.
   - *Upgrade Staged Edge Router(s)* — resolves the network's primary controller purely as the salt-master routing target, then calls `sdsService.upgradeStagedEdgeRouters(...)`.
3. **SDS — `EdgeRouterService.upgradeStagedEdgeRouter`** → salt orchestration `ziti_upgrade.orch.router_upgrade_staged`: pings target routers, then runs `ziti_upgrade.stage.routers_enable_timer` on them — enables the `ziti-router-upgrade.timer` systemd unit. **Asynchronous from here** — the timer periodically fires a oneshot service that swaps the binary and restarts `ziti-router`.
4. **Back in `core` — polling for completion**
   - *Upgrade Staged Edge Router* (per router, parallel) — `UpgradeStagedEdgeRouterWaitSubprocess` polls until that router is back online.
   - *Check connections to all upgraded Edge Routers* — `checkOnlineEdgeRouters` retries with backoff until all target routers report online.
   - *Finalize Edge Router Staged Upgrades* — SDS `upgradeStagedNetworkFinish` → `upgrade_finish.sls` (stops/disables the timer).
   - *Post Staged Upgrade Hooks* → `upgradeService.applyStagedUpgradePostHooks`.
   - *Finish Staged Upgrade* — `checkVersionsAndUpdateStatus` sets router status to `PROVISIONED` (version matches target) or `ERROR` (mismatch); publishes success/failure event.
5. **Response** — `202 Accepted` with the `Network` HAL resource; the actual router swap/restart completes asynchronously, tracked internally by step 4 until success, failure, or retry exhaustion.

### Net result

| | Stage exists? | Upgrade-Staged (activate) exists? |
|---|---|---|
| Edge Router | Yes — dedicated, standalone | Yes — dedicated, standalone |
| Network Controller | Only as a side effect of staging routers | **No — does not exist** |

---

## Part 1 — Edge Routers

### 1. Support user wants to stage a router/routers' binaries

- **a. Existing:** Supported today via `POST /networks/{id}/stage` (many routers) or `POST /edge-routers/{id}/stage` (one router). Caller can pick a target version (`target` field), or it defaults to the network's current version. Downloads/extracts the target binaries onto the router(s) and, as a side effect, onto the Network Controller (used as the file-distribution point). No cutover happens.
- **b. Expected (new):** Record that staging occurred — target version + timestamp — per Edge Router, in Redis with a short TTL (~14 days). This is purely a tracking addition; the staging mechanism itself is unchanged.
- **c. Status after:** Edge Router status returns to `PROVISIONED` (unchanged, as today); `productVersion` unchanged (still running old version). **New:** a staged-state record now exists for the router, expiring automatically after the TTL if unused.

### 2. Support user wants to know what version is currently staged on a router

- **a. Existing:** Not possible. No persisted or queryable staged-version state exists anywhere — `EdgeRouter.productVersion` reflects only the currently *running* version, and the actual staged files live only on the host's disk, invisible to any API.
- **b. Expected (new):** Expose the staged-state record from Use Case 1 through a read path, so a support engineer can confirm what's staged (and when it was staged) before deciding to activate or re-stage.
- **c. Status after:** No change to the router itself — this is a read-only visibility feature.

### 3. Support user wants to activate (upgrade) previously staged router binaries

- **a. Existing:** Supported today via `POST /networks/{id}/update-staged`. Enables the upgrade timer on the target router(s), which swaps binaries and restarts, without re-downloading anything. Does not accept or validate a target version — it just activates whatever is currently on disk.
- **b. Expected (new):** No functional change requested to this call itself; it remains the explicit "I know what's staged, activate it now" entry point. (Optionally, it could cross-check against the Use Case 1 record and warn/fail if no valid staged record exists — open question, see below.)
- **c. Status after:** Router status becomes `PROVISIONED` at the new version on success, or `ERROR` if the reported version doesn't match the network's target (existing `checkVersionsAndUpdateStatus` safety net). The staged-state record for that router should be cleared/consumed so it isn't reused for a future, unrelated upgrade.

### 4. Support user runs a normal `/upgrade` and wants it to automatically reuse previously staged binaries

- **a. Existing:** Not supported. `router_upgrade.sls` (behind the regular `/upgrade`) always redoes the full stage step from scratch — it explicitly deletes any previously staged files first — before activating. A prior `/stage` call provides no benefit today if the operator later calls plain `/upgrade`.
- **b. Expected (new):** `UpgradeNetworkProcess` checks the Use Case 1 staged-state record for each Edge Router in scope, before calling SDS:
  - Valid record + version matches the requested upgrade → route through the lightweight `upgradeStagedEdgeRouters` SDS call (activation only).
  - No record, expired record, or version mismatch → fall back to the existing full `upgradeEdgeRouters` SDS call (stage + activate), exactly as today.
- **c. Status after:** All requested routers end up upgraded correctly regardless of path taken; routers with valid staged state upgrade faster (no re-download). Consumed staged-state records are cleared.

### 5. Support user upgrades a mixed fleet (some routers staged, some not) in a single `/upgrade` call

- **a. Existing:** Not applicable — `/upgrade` has no concept of staged state at all, so every router in the request is treated identically (full stage + activate).
- **b. Expected (new):** `/upgrade` partitions the requested routers into "has valid staged state" and "does not," and issues the appropriate SDS call for each subset within the same request — the caller doesn't need to split requests themselves.
- **c. Status after:** All requested routers reach `PROVISIONED` at the target version (or `ERROR` on mismatch); some via the fast path, some via the full path, transparently to the caller.

### 6. Staged state must never cause an incorrect upgrade

- **a. Existing:** N/A today (no staged-state tracking exists yet), but the underlying risk already exists in a different form: `update-staged` blindly activates whatever happens to be on disk, with correctness only verified *after* the fact via `checkVersionsAndUpdateStatus` marking the router `ERROR` on a version mismatch.
- **b. Expected (new):** The staged-state record must be treated as a pure performance optimization, never a correctness dependency. `/upgrade` only takes the fast path when the recorded target version exactly matches the requested version; any mismatch, missing record, or expired TTL always falls back safely to the full stage-then-activate path.
- **c. Status after:** Upgrade correctness is unaffected by whether the cache is present, stale, or absent — worst case is a slower (but correct) upgrade.

---

## Part 2 — Network Controller (gap identified in review; not in current PR)

### 7. Support user wants to stage the Network Controller's binaries

- **a. Existing:** No standalone capability. The controller's binaries only get staged as an incidental side effect of staging Edge Routers (`ziti.stage.controller.binaries`, bundled inside `router_stage_files.sls` / `router_upgrade.sls`) — there is no way to request it independently of targeting routers, and no dedicated SDS endpoint (`NetworkControllerDeploymentController` has no `/stage`).
- **b. Expected (new):** A controller-scoped stage capability, and/or tracking of the incidental staging that already happens today — recording target version + timestamp for the controller the same way as Use Case 1, whenever `ziti.stage.controller.binaries` runs (whether triggered standalone or as a router-stage side effect).
- **c. Status after:** Controller has the target version's binaries extracted on disk (as today); **new:** a staged-state record exists for the controller.

### 8. Support user wants to know what version is currently staged on the Network Controller

- **a. Existing:** Not possible — same gap as Use Case 2, applied to the controller.
- **b. Expected (new):** Same read-path visibility as Use Case 2, for the controller's staged-state record.
- **c. Status after:** Read-only — no change to the controller.

### 9. Support user wants to activate (upgrade) previously staged Network Controller binaries

- **a. Existing:** No such capability exists. The only controller upgrade path is `/deployments/network-controllers/upgrade` (`controller_upgrade.sls`), which always runs preflight → re-stage → stop → backup → install → restart synchronously in one call. There is no "activate only" equivalent to the router's `update-staged`. (The re-stage step is cheap if binaries are already present, thanks to an incidental idempotency guard in `install.sls`, but the full disruptive sequence — stop/backup/restart — still always runs as part of the same call; there's no way to decouple "download now" from "cut over later" for the controller.)
- **b. Expected (new):** A new decoupled activation path, mirroring the router's `update-staged`:
  - New salt orchestration `controller_upgrade_staged.sls` — same as `controller_upgrade.sls` minus the `ziti.stage.controller.binaries` step.
  - New SDS endpoint `POST /deployments/network-controllers/upgrade-staged` + corresponding salt runner.
  - `core`-side controller/process wiring to invoke it.
- **c. Status after:** Controller reaches the target version (stop → swap → restart) without a redundant re-download; staged-state record consumed/cleared.

### 10. Support user runs a normal controller `/upgrade` and wants it to automatically reuse previously staged binaries

- **a. Existing:** Partially, but only by accident: `install.sls` skips its own download/extract if the target version's directory already exists on disk. However, the orchestration still always executes the full stop/backup/install/restart sequence synchronously — there's no explicit, deliberate "detect staged, use it" decision, just an implicit no-op inside one sub-step.
- **b. Expected (new):** Extend `UpgradeNetworkProcess`'s `CONTROLLERS_ONLY`/`FULL` handling to check the Use Case 7 staged-state record for the controller and explicitly route to the new Use Case 9 activation-only path when it's valid, instead of relying on incidental salt-level idempotency.
- **c. Status after:** Same outcome as calling `/upgrade` today, but explicit and deterministic rather than incidental; staged-state record consumed/cleared.

---

## Cross-Cutting Requirements

- **Storage:** Redis, via the existing `CacheManager<K,T>` / `RedissonCacheManager<K,T>` pattern already used in `ServerConfiguration` (e.g. `createNetworkPayloadCache`) — no new infrastructure.
- **TTL:** ~14 days per entry, using `CacheManager#put(key, value, expireAt)`. Loss of a cache entry before the TTL (e.g. Redis restart) is acceptable — it only forces a fallback to the full stage+activate path, never an incorrect upgrade.
- **Consumption:** A staged-state record should be cleared once used by an activation (`update-staged` or the new controller equivalent) or by a fast-path `/upgrade`, so it can't be mistakenly reused for a later, different-version request.

## Open Questions

1. Should `update-staged` (router) and the new controller activation endpoint validate against the staged-state record and fail/warn if no valid entry exists, or continue to trust whatever is physically on disk as they do today?
2. Should the staged-state record be keyed per-controller, or per-network (since there is currently one primary controller per network)?
3. Should the read-path visibility (Use Cases 2 & 8) be a new field on the existing Edge Router / Network Controller resources, or a separate lookup endpoint?
4. Is standalone controller staging (Use Case 7, decoupled from any router stage request) required, or is tracking the existing incidental side-effect staging sufficient for the initial delivery?