Software Deployment Service

Deployment Reference:
NC, ER, Certificates & Upgrades

Repository: sds (Spring Boot 3.4.5, Java 21)
Certs service: certs (Spring Boot 3.5.5)
Salt version: 3006.14
1

Architecture Overview

SDS (Software Deployment Service) orchestrates the lifecycle of OpenZiti Network Controllers (NC) and Edge Routers (ER) on target host machines. It uses two distinct orchestration channels: Ansible for initial installation, and Salt for ongoing state management and upgrades.

System Overview
SDS Spring Boot Ansible ProcessBuilder subprocess Salt Master HTTP API :8001 Target Hosts NC / ER machines Certs Service PKI / BouncyCastle install SSH Salt API (upgrade/state) ZMQ cert callbacks → SDS → certs ― ― Upgrade / State path ― ― Certificate path (install-time only)

Key Components

ComponentRoleKey classes
NetworkControllerAnsibleWrapperRuns NC install playbook via ProcessBuilderAbstractAnsibleWrapper
EdgeRouterAnsibleWrapperRuns ER install playbook via ProcessBuilderAbstractAnsibleWrapper
SaltServicePublic API for all Salt operationsSaltRestClient, SaltClientFactory
JobPollingServiceTwo-phase polling for Salt job resultsRetryTemplate
CertificateServiceProxies cert requests to the certs microserviceInterServiceTokenProvider, WebClient
NetworkControllerServiceNC upgrade orchestration entry pointNetworkControllerUpgradeSaltRunner
EdgeRouterServiceER upgrade orchestration entry pointEdgeRouterUpgradeSaltRunner
2

Network Controller Installation

The NC install is triggered by a REST API call into SDS. SDS builds an Ansible inventory, launches ansible-playbook as a blocking subprocess, and waits for completion (timeout configurable, typically 90 min). The playbook runs entirely over SSH on the target machine.

ℹ
Two connectivity modes: NetFoundry-hosted targets use direct SSH (IP-based inventory). Customer-hosted targets use ansible_connection=community.openziti.paramiko with a Ziti identity file — no public IP required.

Installation Sequence

NC Installation — Full Sequence
REST Client SDS Ansible NC Host Certs Svc POST /networks/{id}/controllers runPlaybook(ziti_controller.yml) fetch SSH key (SSM) write temp inventory + extraVars files ProcessBuilder → ansible-playbook SSH connect (root) FIPS setup, UFW/firewalld rules download ziti binary (or FIPS build) openssl genkey (EC prime256v1) + CSR GET /certificates/root (callback to SDS) GET /ca/{networkId} + JWT auth root CA PEM POST /certificates/intermediate + CSR POST /intermediate {networkId, csr} signed intermediate cert PEM build cert chain, render config ziti controller edge init (init DB) systemd enable/start + verify (18×10s) install Salt Master + Minion (3006.14) monitoring, zitiProxC, frontdoor agents exit code 0 201 Created Certificate callbacks Ansible remote tasks SDS API

Ansible Roles — ziti_controller.yml

RoleTagWhat it does
publicCertV7public_certProvisions public TLS certificate (optional, for HTTPS endpoints)
zitiControllerV7ncCore NC installation: FIPS, binary download, cert chain, config render, edge init, systemd service
ncRouterV7nc_routerInstalls the embedded router that runs alongside the NC
saltMasterV7saltInstalls and configures Salt Master (HTTP API on :8001)
saltMinionV7salt_minionInstalls Salt Minion; minion ID = host UUID; pins to 3006.14
monitoringV7—Installs telemetry/monitoring agent
ddosV7—DDoS mitigation configuration
zitiProxCV7—Installs zitiProxC for control-plane connectivity
frontdoorAgentV7—Installs frontdoor agent for management connectivity
⚠
NC is both Salt Master and Salt Minion. The NC host runs salt-master (for managing ER minions) AND salt-minion (with its own host UUID as minion ID, so SDS can target it for NC upgrades via its own master).

Key extra vars passed to the NC playbook (via getCommonVars() in AbstractAnsibleWrapper):

VariableSource
fips_enabledNetwork/host attribute; affects binary selection and kernel mode
ha_enabledNetwork attribute; enables HA-specific config
cloud_zititrue when platform version ≥ 8
network_idUUID of the network; used for cert callbacks
sds_urlBase URL of this SDS instance; Ansible callbacks use this
target_versionThe Ziti version to install
3

Edge Router Installation

ER installation follows the same Ansible subprocess pattern but uses a different playbook (ziti_edge_router.yml) and a fundamentally different mechanism for certificate provisioning: instead of a PKI callback chain, the ER enrolls with the NC using a pre-issued JWT token.

✓
ER certificates come from JWT enrollment, not the certs microservice. SDS passes the enrollment JWT to Ansible; the ER runs ziti router enroll which contacts the NC's Ziti API and exchanges the JWT for a full mTLS identity (cert + key).

Installation Sequence

ER Installation — Full Sequence
REST Client SDS Ansible ER Host NC / Ziti API POST /networks/{id}/routers runPlaybook(ziti_edge_router.yml, JWT) fetch SSH key (SSM), write inventory ProcessBuilder → ansible-playbook SSH connect (root) FIPS setup, firewall rules download ziti-router binary render router config.yml, write JWT file JWT Enrollment ziti router enroll config.yml --jwt enroll.jwt mTLS identity (cert + key) touch .is_registered marker systemd enable/start ziti-router install Salt Minion (minion_id=host_uuid) TCP tuning (16 MB socket buffers) exit code 0 201 Created

ER Roles & Key Differences from NC

RoleNC playbookER playbook
Core Ziti componentzitiControllerV7zitiEdgeRouterV7
PKI generationopenssl genkey + CSR + callback to certs svcNone — JWT enrollment gives identity
Salt roleSalt Master + MinionSalt Minion only
TCP tuningStandard16 MB socket buffers (high-throughput)
ncRouterYes (ncRouterV7)No
zitiProxC / frontdoorYesNo
4

Certificate Architecture

NetFoundry uses a two-tier PKI. The Root CA is created and persisted by the certs microservice. The Intermediate CA is generated on the NC host itself (EC prime256v1) and signed by the root. All leaf certificates are signed by this intermediate and live only on the NC host.

PKI Hierarchy

Certificate Chain
Root CA
RSA-4096 · 50-year validity
Stored in AWS SSM Parameter Store
/certificates/CA_{uuid}
Intermediate CA
EC prime256v1 · 10-year validity · CA:TRUE
Generated on NC host via openssl
Signed by Root · NOT stored persistently
Server Cert
NC HTTPS endpoint
On NC host
Client Cert
mTLS / admin identity
On NC host
ER Identity
JWT enrollment
Independent of PKI chain

Certificate Generation Flow

ℹ
Callback pattern: Ansible running on the NC host cannot call the certs microservice directly — it lacks a service token. Instead, Ansible calls back into SDS, which uses InterServiceTokenProvider to acquire a JWT and proxy the request to the certs service.
Certificate Flow Detail
NC Host (Ansible) SDS Certs Service AWS SSM openssl genpkey (EC prime256v1) openssl req -new (CSR) GET /{networkId}/certificates/root InterServiceTokenProvider → JWT GET /ca/{networkId} + Bearer JWT GetParameter(/certificates/CA_{id}) Root CA PEM Root CA PEM Root CA PEM POST /certificates/intermediate (CSR) POST /intermediate {networkId, csr} GetParameter(CA_{id}_key) → sign CSR Intermediate cert PEM Intermediate cert PEM build chain: intermediate → root CA

Certs Microservice (/Users/prashantsavadi/git/certs)

EndpointWhat it does
GET /ca/{networkId}Returns the Root CA PEM for the network; reads from SSM /certificates/CA_{networkId}
POST /intermediateSigns the submitted CSR with the Root CA key (fetched from SSM /certificates/CA_{networkId}_key); returns signed intermediate cert PEM; does NOT store it

Root CA properties: RSA-4096, 50-year validity, CA:TRUE, stored as SecureString in AWS SSM Parameter Store. Created once per network on first request.

Intermediate CA properties: The CSR is generated on the NC host using EC prime256v1; the signed certificate has CA:TRUE and 10-year validity. It is returned to Ansible and placed on the NC host but is never persisted by the certs service.

Authentication: SDS uses InterServiceTokenProvider to acquire a JWT with audience io.netfoundry.network before calling the certs service. Ansible running on the NC host has no access to this token — hence the callback pattern through SDS.

5

Network Controller Upgrade via Salt

NC upgrades use Salt orchestration: SDS triggers a state.orchestrate runner on the Salt Master (which lives on the NC host itself), then polls for completion. Maximum wait: 10 minutes (60 retries × 10 s).

NC Upgrade — Orchestration Sequence
SDS Salt Master API (on NC host :8001) Salt Orchestrate controller_upgrade.sls NC Minion (same host as master) JobPollingService SyncAllSaltRunner (fileserver.update) jid state.orchestrate → controller_upgrade jid (async) jobPollingService.await(60 retries, 10s) controller_upgrade.sls ① test.ping (NC minion) ② stage.controller (download binary) ③ fileserver.update (runner) ④ ziti.controller.upgrade state orchestration result Phase 1: jobs.lookup_jid (hasJobResult) true (result available) Phase 2: getOrchestrationResult OrchestrationStateResult upgrade complete

NC Upgrade Orchestration Steps

  • 1SyncAllSaltRunner: Calls fileserver.update runner — syncs Salt fileserver state before upgrade begins.
  • 2NetworkControllerUpgradeSaltRunner: Calls state.orchestrate with mods=ziti_upgrade.orch.controller_upgrade, pillar: {targetVersion, targetMasters: [minionId]}.
  • 3Orchestration step ①: test.ping on controller minion — preflight connectivity check.
  • 4Orchestration step ②: ziti_upgrade.stage.controller — downloads binary bundle to /opt/netfoundry/ziti/{version}/, stages upgrade files (init.sls, localconfig.yaml, service template).
  • 5Orchestration step ③: fileserver.update runner — ensures Salt fileserver sees the staged files.
  • 6Orchestration step ④: ziti.controller.upgrade state — executes the actual upgrade on the NC host (stops service, swaps binary, restarts, verifies).
  • 7Polling: JobPollingService.await(60, 10s) polls using two phases: Phase 1 checks jobs.lookup_jid until result appears; Phase 2 fetches full OrchestrationStateResult.
6

Edge Router Upgrade via Salt

ER upgrades use a "dark upgrade" pattern: Salt stages upgrade files and installs a systemd timer on each ER, then the ER upgrades itself autonomously without further coordination from SDS. The polling window is much longer: ~45 minutes (275 retries × 10 s).

⚠
SDS polls the NC orchestration result, not individual ER minion results. The orchestration completes when staging is done. Each ER's actual upgrade happens independently via salt-call --local (masterless mode) — SDS does not observe individual ER upgrade completion.

Dark Upgrade Pattern — Sequence

ER Upgrade — Dark Upgrade via Systemd Timer
SDS Salt Master (NC host) Salt Orchestrate router_upgrade.sls ER Minions (multiple hosts) Systemd Timer (on each ER) JobPoll SyncAllSaltRunner state.orchestrate → router_upgrade jid (async) jobPollingService.await(275 retries, 10s) ≈ 45 min router_upgrade.sls ① test.ping (masters + minions, parallel) ② stage.controller (NC: download bundle) ③ fileserver.update runner ④ stage.routers (push files to ERs) writes requested_version file installs ziti-router-upgrade.service (oneshot) installs ziti-router-upgrade.timer systemctl enable --now timer orchestration result (staging done) Autonomous (dark) upgrade Timer fires → service runs salt-call --local (masterless) Phase 1: hasJobResult polls NC master result = true (staging complete) upgrade staged — ERs self-upgrade via timer

Dark Upgrade — Step-by-step

  • 1SDS calls EdgeRouterUpgradeSaltRunner: state.orchestrate, mods=ziti_upgrade.orch.router_upgrade, pillar: {targetVersion, targetMasters, targetMinions}.
  • 2Preflight: test.ping runs on both Salt Master minions AND ER minions in parallel — confirms all hosts are reachable.
  • 3Stage on NC: ziti_upgrade.stage.controller downloads the binary bundle to /opt/netfoundry/ziti/{version}/ on the NC host (fileserver). Files include: init.sls, localconfig.yaml, ziti-router.service.j2, defaults.yaml.
  • 4Sync fileserver: fileserver.update runner ensures Salt Master's fileserver reflects the newly staged files.
  • 5Stage on ERs: ziti_upgrade.stage.routers (timeout: 600 s) copies staged files from Salt fileserver to each ER at ziti_upgrade_dir, writes requested_version file, creates and enables ziti-router-upgrade.service (oneshot) and ziti-router-upgrade.timer.
  • 6Dark upgrade fires: The systemd timer on each ER fires and executes the oneshot service. The service runs salt-call --local (masterless mode) to perform the actual binary swap, config update, and service restart — no Salt Master involvement.
  • 7SDS polling: Polls the NC orchestration result (not individual ERs) for up to 275 × 10 s ≈ 45 min. Polling ends when the NC orchestration (staging) reports complete. ER self-upgrades continue independently.
ℹ
Why masterless mode? During the upgrade, the Salt Minion on the ER may be restarted. Using salt-call --local ensures the upgrade can complete even if the minion connection to the master is interrupted during the process.
7

Salt Architecture

Salt provides the post-installation state management and upgrade orchestration channel. Each NC host runs a Salt Master; all associated ER hosts register as Salt Minions of that master. SDS communicates with the master via its HTTP API on port 8001.

Topology

Salt Topology per Network
SDS NC Host Salt Master (:8001) Salt Minion (self) ER Host 1 (Minion) ER Host 2 (Minion) ER Host N (Minion) HTTP :8001 ZMQ (minion connects to master)

Salt Client Factory — Connection Mode Selection

SaltClientFactory picks between two connection strategies based on whether the NC has a managementZitiIdentity:

ConditionClient usedURL formatTLS behavior
controller.managementZitiIdentity != nullcontrolPlaneRestClienthttps://<zitiIdentityName>:8001Ziti overlay; hostname verification disabled
Traditional (no Ziti identity)traditionalRestClienthttps://<hostIpAddress>:8001TrustAllStrategy (self-signed certs)

The controlPlaneRestClient bean uses zitiClientHttpRequestFactory and loads the Ziti identity from AWS Secrets Manager. The SaltRestClient performs a login/logout per API call using an AutoCloseable WithAuth wrapper.

Job Polling — Two-Phase Strategy

JobPollingService.await() uses a two-phase approach to handle Salt's asynchronous job model:

// Phase 1: has any result arrived for this JID?
RetryTemplate(maxRetries, delay)
  .execute(() -> saltService.hasJobResult(jid))  // polls jobs.lookup_jid
  // SaltTerminalFailureException → non-retriable (retcode=1)

// Phase 2: fetch the actual result (up to 60×1s)
supplier.get()  // calls getOrchestrationResult()
Upgrade typePhase 1 retriesPhase 1 delayMax wait
NC upgrade6010 s~10 min
ER upgrade27510 s~45 min

Terminal failures: If Salt returns retcode=1, SaltTerminalFailureException is thrown — this is not retried and propagates immediately to the caller.

Salt operations used by SDS:

MethodSalt functionclientPurpose
applyState()state.applylocal_asyncApply a Salt state to a minion
executeRunner()state.orchestraterunner_asyncRun an orchestration
executeHook()varieslocalSynchronous hook execution
executeFunction()varieslocalSynchronous function call
hasJobResult()jobs.lookup_jidrunnerCheck if job has completed
getOrchestrationResult()runner result—Fetch full orchestration output
listMinions()key.list_acceptedwheelList accepted minion keys
removeMinions()key.deletewheelRemove minion keys (deregister)