# Active and passive share hosts

A share is hosted by one or more agents. Each host is **active** or **passive**. Actives carry the traffic and
share it. Passives carry traffic only when no active is bound. Ziti does the failover. This service decides which
agents host a share and in which role, and reports what Ziti actually has.

Phases 1 to 4 are built, with the role filters on both listings. Phase 5 is still a plan. It applies to
`TcpShare`, `IpsecShare` and `HttpShare`.

## How it works

`frontdoor-agent` (branch `v3`) is a Ziti Go tunneler in host-only mode. It hosts every service its bind policies
give it and creates no service of its own. So a host is added by naming its agent on the share's bind policy, and
the agent needs no change.

The role is carried by **terminator precedence**. Smart routing uses the highest band that holds a terminator (`required` > `default` > `failed`) and balances inside that band by
cost and latency.

| Role    | Precedence | Where it comes from                                                           |
|---------|------------|-------------------------------------------------------------------------------|
| Active  | `required` | The share's own `host.v1`, which carries `listenOptions.precedence: required` |
| Passive | `default`  | A per-identity override: the passive sees the share's `standby` `host.v1`     |

A `host.v1` health check that fails drops that terminator to `failed`, and a passing one returns it to its baseline.
So a failing active drops below the passives, and traffic moves to them.

### Health checks

`HostZitiProjection.hostConfigData` puts one `portChecks` entry in every `host.v1`, active and standby. The agent
runs it (`ziti/v2@v2.0.5/tunnel/intercept/hosting.go:288`) as a TCP connect to `targetAddress:targetPort`, resolved
on its own host. An IPv6 literal is sent in brackets.

The timings come from `io.netfoundry.frontdoor.share.health-check` in `application.yaml`, bound to
`HostHealthCheckProperties`. They have no default. A missing value, a duration under one second, or `failures`
under 1 fails startup. The test `application.yaml` has its own copy of them. Read the values in the YAML, not here.

| Property   | Effect                                                  |
|------------|---------------------------------------------------------|
| `interval` | Time between two checks                                 |
| `timeout`  | How long one check waits for the target                 |
| `failures` | `mark unhealthy` after this many failed checks in a row |

A pass runs `mark healthy` on the first passing check, with no damping.

- Failover takes about `interval` x `failures`, plus the time the controller needs to update the terminator.
- A single host that fails stays reachable. Ziti still routes to a `failed` terminator when no better one exists.
- The passive is checked too, so Ziti does not fail over to a passive that cannot reach the target.
- The check is always on, with the same timings for every share. There is no per-share setting and no HTTP check.
- The check opens a TCP connection and closes it with no handshake. Some targets count this as an error. MySQL's
  `max_connect_errors` is one example. Verify this on a target before you use it.
- A share gets a new or changed check on its next update of any kind. `patchConfig` and `beforeBind` compare the
  stored name and body of each config with the desired state. There is no backfill.

### Why an override and not `serviceHostingPrecedences`

The tunneler reads an identity's `serviceHostingPrecedences` only when it binds, on `ServiceAdded` or
`ServiceChanged` (`ziti/v2@v2.0.5/tunnel/intercept/svcpoll.go:143-170`). The SDK emits `ServiceChanged` only when the
service's id, name, permissions or configs change (`sdk-golang@v1.9.0/ziti/ziti.go:817-834`), so a change on the
identity is never seen by a bound host. It also caches the identity for 30 seconds.

An override changes the configs the identity sees, because the controller merges it into that identity's service
detail (`controller/model/edge_service_manager.go:167`). The SDK then emits `ServiceChanged`, and the tunneler
rebinds with the new precedence.

A service cannot carry both configs itself: the controller allows one config of each type per service (`controller/model/edge_service_model.go:82`).

### Why Ziti writes need no coordination

Each share writes only the (identity, service) pairs for its own service. The controller applies each assign or
remove as one command (`controller/model/identity_manager.go:467-500`). No state is shared between shares, so two
writers cannot lose each other's update.

## Behaviour to know

| Shape                    | Actives | Passives | Behaviour                                   |
|--------------------------|---------|----------|---------------------------------------------|
| Single host              | 1       | 0        | One terminator at `required`                |
| Active/passive           | 1       | 1+       | Passives serve only when no active is bound |
| Active/active            | 2+      | 0        | Each new circuit can land on any active     |
| Active/active + passives | 2+      | 1+       | Both of the above                           |

- **Passives have no order.** When every active is down, all passives share the traffic.
- **Active/active has no affinity.** One TCP connection stays on one host. The next connection can reach another.
- **The target is per share.** Every host forwards to the same `targetAddress:targetPort` and resolves it on its own
  host. A per-host target is out of scope.
- **Circuits die on failover.** Only new circuits reach a passive. When an active comes back, new circuits go to it.
- **A role change rebinds that host.** The tunneler closes its terminator and opens a new one. Whether open circuits
  on that host survive is not verified.
- **The target sees a different source host** after a failover, which matters for an allow-list on the target.
- **A passive holds no idle connection.** The tunneler dials the target only when a circuit arrives.

## Dependency: `ziti-management-client` 0.4.0

`ziti-management-client` 0.4.0 adds the override calls to `IdentityApi`, and `pom.xml` is on `0.4.0-SNAPSHOT`:

```java
ApiEnvelope<List<ServiceConfigOverride>> listServiceConfigs(String identityId);

void assignServiceConfigs(String identityId, Collection<ServiceConfigAssign> serviceConfigs);

void removeServiceConfigs(String identityId, Collection<ServiceConfigAssign> serviceConfigs);
```

What the controller does with overrides, pinned by `IdentityServiceConfigApiTest` in that repo:

- **One override per (service, config type).** A second `host.v1` override for one service replaces the first.
- **A remove works by type**, but the config it names must still exist.
- **A config that an override names cannot be deleted.** The controller answers **500** `UNHANDLED`, "config is in
  use by identity", and leaves the override in place. So a passive's override goes before the `standby` config.
- **Deleting a service removes every override for it.** A share delete therefore needs no override removal.
- **The list has no paging**, and it leaves out an entry whose service or config is gone.
- **A null or empty list is refused** with `IllegalArgumentException`, because an empty remove removes nothing.
- **Both writes are idempotent** and get no TLS-fault poll, so a retry is safe.

**The 500 is an ordering fault, not a transient one.** The circuit breaker records it like any other failure, and a
retry cannot fix it. The write order in phase 3 prevents it; do not add a handler that retries it.

**Pin a release before merge.** `0.4.0-SNAPSHOT` is fine on the branch. The merge to `develop` waits for `0.4.0`.

## Phase 1 — Model

One `Host` table holds the hosts of every share type. Share ids are UUIDs, so they never collide across share
tables. `shareId` has no foreign key, because it points into whichever share table owns the share. Shares are soft-deleted,
so that key would only guard inserts, and every insert runs in the same transaction as its share.

| Column        | Notes                                          |
|---------------|------------------------------------------------|
| `id`          | `BINARY(16)`                                   |
| `shareId`     | The share, of any type. No FK                  |
| `agentId`     | FK to the agent                                |
| `role`        | `ACTIVE` or `PASSIVE`                          |
| audit columns | `createdBy/At`, `updatedBy/At`, `deletedBy/At` |

Unique index `(shareId, agentId, deletedAt)`, so one agent holds one role on one share.

The `Host` entity and the `HostRepository` bean live in `domain/host/`. Each share repository takes the bean and calls it inside its own
transactions. `HostRepository.sharesHostedBy` gives the finders their subselect. A share delete leaves the host rows alone, so a deleted share still reads back
with the hosts it had. Every share type implements `HostedShare`, and every share update implements `HostsUpdate`. So
`HostRepository.withHosts` fills in the hosts of any share type, and `Hosts.apply` merges an update into the hosts.

Flyway `V1.0.14`. Earlier migrations stay as they are.

1. Create the `Host` table.
2. Insert one `ACTIVE` row for every share, from its `agentId`, so seeded shares keep a host.
3. Drop the `agentId` foreign key, index and column from `TcpShare` and `IpsecShare`.

The share entity carries its hosts as two sets, `activeAgentIds` and `passiveAgentIds`, filled by the repository.
`FindTcpShare` and `FindIpsecShare` filter through `Host`, on live host rows only. `agentId` matches a host in either
role; `activeAgentId` and `passiveAgentId` match one role. The host sets are not columns, so `@ValidSort` excludes them.

## Phase 2 — API

`agentId` is gone from the API. `Create*` and `Update*` carry:

```json
{
  "activeAgentIds": [
    "…"
  ],
  "passiveAgentIds": [
    "…"
  ]
}
```

- `activeAgentIds` is required on create and on `PUT`, with at least one entry.
- `passiveAgentIds` is optional and defaults to empty.
- On `PATCH`, a list replaces the whole list. An absent list leaves it alone. `null` on `passiveAgentIds` clears it.
- The view returns both lists, and `hosting` from phase 5 on a single read.
- HAL: one `agents` rel, with one link per host in both roles. The role is in the body.

`@ValidHosts(TcpShare.class)` and `@ValidHosts(IpsecShare.class)` judge the merged share, like
`ValidTcpClientCertificate`, so the `nf-validate` dry run checks the hosts too. The services hold no agent check of their
own. It checks, in this order, and stops at the first rule that fails:

- at least one active;
- the two lists do not overlap;
- the total is at most `io.netfoundry.frontdoor.share.max-hosts`, default 8;
- every agent is live and in the path's account, with one message for every failure, so the endpoint is not an
  oracle for agent ids elsewhere. This rule reports every such agent, on the list that names it.

On a create the two lists carry `@Singular`, so an omitted list is empty, and an empty active list is reported by the
first rule. On a `PUT`, `activeAgentIds` is `@NotNull(groups = PutUpdate.class)`. `tools/seed.sh` sends
`activeAgentIds`. `TcpShareControllerValidationTest` pins each message.

## Phase 3 — Ziti projection

`HostZitiProjection` in `domain/host/` holds all of it. Each share Ziti repository calls it at fixed points: `project`
after the service, `beforeBind` and `afterBind` around the bind policy patch. The three share Ziti repositories extend
`ShareZitiRepository`, which holds the lookups by share tag, the tags, the `host.v1` bodies and the policy create.

1. **The share's `host.v1`** carries `listenOptions.precedence: "required"`.
2. **The `standby` config.** A second `host.v1`, named `<name> | standby | <frontdoorId> | <shareId>` by
   `ZitiUtil`, with the same target and `listenOptions.precedence: "default"`. It is tagged like the share's other
   objects. It exists only while the share has a passive, like the IPsec `intercept` config exists only while a
   tunnel reaches the share. `standby` is shorter than `intercept`, so `MAX_SHARE_NAME_LENGTH` stays at 164.
3. **Overrides.** Each passive's identity is assigned (share's service → `standby` config). Every update checks every
   live passive and assigns the pair when the identity does not hold it. So an override lost with a recreated
   identity comes back on the share's next update, at the cost of one `GET` per passive. Removal compares the old and
   new passive sets, and removes only the pairs of agents that left.
4. **Bind policy.** `bindIdentityRoles` returns `#agent | <id>` for every host, active and passive, patched by
   difference with the whole role set. **A passive goes on the policy only once its override exists.** Without the
   override it binds at `required`, beside the actives, and takes their traffic. A passive left off the policy cannot
   bind, so it cannot take over.

The `agent-id` tag comes off every share object. A tag holds one value, and the hosts are on the rows.

Order:

| Operation        | Order                                                                                    |
|------------------|------------------------------------------------------------------------------------------|
| Create or add    | `standby` config → overrides → bind policy. A passive sees `default` from its first bind |
| Remove a passive | bind policy → override → `standby` config if no passive remains                          |
| Passive → active | remove override. The host rebinds at `required`                                          |
| Active → passive | assign override. The host rebinds at `default`                                           |
| Target change    | patch both `host.v1` configs                                                             |
| Delete the share | policies → service → configs. The service delete removes the overrides                   |

Rules:

- The compensation needs nothing new. `update(previous, current)` is symmetric, so the service's existing rollback,
  `update(current, previous)`, reverts the standby side too, and `deProject` removes it by tag.
- `NetworkAvailability.shouldCleanupZiti` gates the share delete, as before. The share update paths do not call it, so
  neither does the standby side; a network that is gone fails the update against Ziti.
- The `standby` config carries `frontdoor-object: standby-host` beside the share's tags. It shares the `host.v1` type
  with the share's own config, so the tag is what tells them apart, never the name. Each repository leaves it out when
  it patches the share's own `host.v1`.
- A passive's identity is found by its `agent-id` tag. On assign, a **live** agent with no identity fails the create
  or update, because it would stay on the bind policy with no override and bind at `required`. A **deleted** agent is
  skipped: its delete cascade removes it from the share. The service finds deleted passives through
  `HostLiveness` and passes them to `update`. Without the skip, two host agents of one share deleted close
  together would fail each other's cascade forever. On remove, a missing identity is expected, because deleting an
  agent deletes its identity and every override on it.
- `beforeBind` resolves the service and the identities before it creates the `standby` config, so a failed lookup
  leaves nothing in Ziti.

**A share written by an older build is healed on its next update.** Each update compares the share's stored `host.v1`
with the body this build writes, name and data, and patches it when they differ, even when neither the name nor the
target changed. So a share created before the precedence existed gains `required` on the first update that changes it,
in any field, and its hosts rebind once. A `PATCH` that changes nothing writes nothing, because the service returns
before it reaches Ziti, so it does not heal a share. The heal runs before the standby side and the bind policy, so an
update that adds the first passive to an older share puts the actives at `required` before the passive may bind.
`healsAHostConfigWrittenWithoutAPrecedenceOnTheNextUpdate` pins this against a real controller.

## Phase 4 — Cascades

`TcpShareRabbitListener` and `IpsecShareRabbitListener` bind `DELETED` + `agent` and call each service's
`removeDeletedAgent`, one transaction for every share the agent hosts. `Hosts.without(agentId)` decides each share:

1. The agent leaves the share's hosts, on its host row and on the bind policy.
2. If no active remains and passives do, **every passive is promoted to active**: their overrides and the `standby`
   config go. Ziti already routes to all of them at that point, so the model then matches the traffic.
3. If no host remains, the share is deleted, as before.

**Every path that changes a share locks its row first.** The cascade, the account cascade, the API update and the API
delete read the share through `*ShareRepository.lock`, which runs `HostRepository.lock`, a `SELECT … FOR UPDATE`. They
decide from that row and its hosts. They lock in id order, so two transactions never deadlock on each other. Without the
lock, two cascades removing the last two hosts of one share each read both hosts, each takes only its own host off, and
the share stays live with none.

These transactions run at **READ COMMITTED**. MySQL defaults to REPEATABLE READ, where the host reads after the lock
would still come from the snapshot the transaction's first read fixed, before the other writer committed. `lock` refuses
any other isolation level.

The TCP listener tears the ingress down before the transaction, from an unlocked read, so a concurrent cascade can make
this agent the last host after the listener chose. The cascade then refuses to delete a share whose ingress is still up.
The retry runs the whole handler again, and this time the listener tears that ingress down first.

A share that keeps hosts goes through the same repository update and Ziti patch an API update uses, so the rollback is
the same call reversed, and it publishes one `UPDATED`. A deleted share publishes one `DELETED`. The deleted agent's own
override needs no removal: its identity is gone, and the overrides went with it, so the identity lookup skips it.

The TCP listener tears down the AWS ingress first, outside the transaction, but only for the shares the agent was the
only host of (`findLiveHostedOnlyBy`), because only those are deleted. The handler finds live host rows only, so a
redelivery finds nothing to do. The `DELETED` + `frontdoor` cascade does not change.

## Phase 5 — Status

A single read carries a `hosting` block, derived from `GET /terminators?filter=service="<zitiServiceId>"`.

```json
"hosting": {
  "state": "HEALTHY",
  "hosts": [
    {
      "agentId": "…",
      "role": "ACTIVE",
      "bound": true,
      "precedence": "required"
    },
    {
      "agentId": "…",
      "role": "PASSIVE",
      "bound": true,
      "precedence": "default"
    }
  ]
}
```

Match a terminator to a host by `hostId`, which is the agent's Ziti identity id.

| Terminator precedence | Active host                 | Passive host                |
|-----------------------|-----------------------------|-----------------------------|
| `required`            | serving                     | misconfigured               |
| `default`             | misconfigured               | serving                     |
| `failed`              | bound, health check failing | bound, health check failing |
| none                  | not bound                   | not bound                   |

| State           | Meaning                                                                                    |
|-----------------|--------------------------------------------------------------------------------------------|
| `MISCONFIGURED` | A host is bound at a precedence its role does not ask for. This state wins over the others |
| `HEALTHY`       | Every host is serving                                                                      |
| `DEGRADED`      | An active is serving, and some host is not                                                 |
| `FAILED_OVER`   | No active is serving, and a passive is                                                     |
| `DOWN`          | No host is serving                                                                         |

- A listing does not ask Ziti, and its views carry no `hosting`. Absent means "not known", like `AgentService.find`
  when the controller is down.
- There is no scheduled job. The agents follow the policy, so nothing needs repair.

## Tests

- **Migration:** every share has one `ACTIVE` host row, and `agentId` is gone from both share tables.
- **Repository:** hosts round-trip in both roles; `agentIds` on a find matches either role.
- **Validator:** no active; overlap; an agent in another account; a deleted agent; over the cap. Every agent failure
  answers the same message. The dry run reports the same violations.
- **Controller:** create with one active; with actives and passives; `PATCH` replacing one list and leaving the
  other; `PATCH` with `passiveAgentIds: null`; the `agents` links. One test class per controller.
- **Ziti, mocked:** the share's `host.v1` carries `required`; both configs carry a port check of the target, which
  follows a target change; the `standby` config carries `default` and exists only
  with a passive; the bind policy names every host as literal `#agent | <id>`; the write order in the table above;
  a role change touches only the identities that changed; no share object carries `agent-id`.
- **Ziti, container:** the controller stores both `host.v1` bodies, health check included, exactly as sent; `getPolicyAdvice` says every host can bind and an agent outside the
  lists cannot; each passive
  has exactly one override for the service; removing the last passive removes its override and then the `standby`
  config, with no 500; deleting the share leaves no override on any agent identity.
- **Cascade:** a passive deleted; an active deleted with actives left; the last active deleted with passives left (all promoted); the last host deleted (share deleted); a
  redelivery does nothing.
- **Status:** each state, and each cell of the precedence table, from stubbed terminators.
- **Live, manual:** two agents in sandbox on one share. Confirm the passive's terminator shows `default`, the
  active's shows `required`, a role change moves both, and stopping the active moves new circuits to the passive.
  `ZitiTestConfiguration` starts no tunneler, so only this check proves the rebind. Then stop the target on the
  active's host only. Its terminator goes to `failed` after about `interval` x `failures`, and back to `required` on
  the first check after the target starts again.

## Sequencing

All five phases go on one branch, `feature/share-hosts`, built in order 1 → 5. Nothing waits on another repo.

1. Phases 1 and 2 together, since the migration drops a field the API returns.
2. Phase 3, built. It must land in the same change as 1 and 2: dropping `agentId` breaks the bind policy, and a
   passive on the bind policy without its override binds at `required` and takes traffic.
3. Phase 4, for the same reason: dropping `agentId` breaks the agent cascade.
4. Phase 5.
5. The live check, then the `0.4.0` release pin, then the merge.