# Testing harness for the IPsec ingress flow

Notes on the test topology the `ip-sec-tf-prototype` prototype used to
validate the full chain — customer IPsec peer → TGW → NLB → Fargate
(`frontdoor-mtls`) → zrok fabric → tcpShare backend. Document this here
so when `zrok-orchestration` stands up new Endpoint Sites and Shares we
can reproduce the same tests against the new stacks without re-inventing
the wheel.

Companion to [`HANDOFF-ipsec-tiering.md`](./HANDOFF-ipsec-tiering.md),
which owns the architecture side.

## Test topology

Three hosts, three roles. Everything below assumes `zrok v2.0.1`-class
semantics (the controller is at `api-v2.frontdoor.staging.netfoundry.io`).

```
           (customer tier)                  (NetFoundry AWS)                  (Service Site / backend)
┌───────────────────────────────┐   ┌──────────────────────────────┐   ┌──────────────────────────────┐
│ host inside 172.31.0.0/16     │   │ Pinned NLB @ 10.1.5.10       │   │ mg-ipsec-testing-zrok        │
│ (e.g. 172.31.12.137)          │   │   listener 9001 → task → sh. │   │ (100.55.0.139)               │
│                               │   │   listener 9002 → task → sh. │   │                              │
│ nc 10.1.5.10 9001              │──►│   listener 9003 → task → sh. │──►│ podman echo-a  :9101         │
│                               │   │ Fargate task runs            │   │ podman echo-b  :9102         │
│                               │   │   frontdoor-mtls             │   │ podman echo-c  :9103         │
│                               │   │   → zrok access private <tk>│   │ zrok2 share private  ×3      │
│                               │   │   → ziti fabric              │   │                              │
│                               │   │                              │   │ zrok2 access private  ×3     │
│ IPsec tunnel terminates at    │   │ NLB ENI ingress via subnet   │   │   (loopback validators       │
│ customer gateway → AWS VPN    │   │ 10.1.5.0/24 + TGW attach     │   │    bound on 127.0.0.1:92xx)  │
│ connection → TGW → VPC        │   │                              │   │                              │
└───────────────────────────────┘   └──────────────────────────────┘   └──────────────────────────────┘
```

Three tiers of test exist on this topology, ordered by scope:

| # | Where you run it | What it proves |
|---|---|---|
| Test 0 | On the backend host (`100.55.0.139`), against the local zrok-loopback validators at `127.0.0.1:9201/9202/9203` | zrok share → fabric → share-backend → echo container works, end-to-end, with **zero AWS involvement**. Fastest path to isolate "zrok side is fine" vs "AWS side is broken". |
| Test 1 | From any in-VPC host (ideally `vpn-testrouter` at `10.1.5.69` since it's in the same subnet & route table that customer traffic uses) | The full AWS-side path, minus IPsec: VPC client → NLB → Fargate → zrok → backend. Localizes to before vs after the NLB. |
| Test 2 | From the customer side (e.g. `172.31.12.137`) | The real thing: IPsec → TGW → NLB → Fargate → zrok → backend. If this works, ship it. |

## Backend host: `mg-ipsec-testing-zrok` / `100.55.0.139`

### Access

```bash
ssh -i /opt/sandbox.key ubuntu@100.55.0.139
```

The box is a dev EC2 in a NetFoundry-CGNAT subnet (`100.55.0.0/16`)
reachable from the operator's workstation via an existing NF overlay.
For production, spin up an equivalent (EC2 / on-prem box / anything) per
tenant that wants to publish backends; this one exists solely for the
prototype.

### Prerequisites on the box

Already installed + enrolled in the prototype box; listing here so you can
reproduce on a fresh host:

- `zrok2` binary at `/usr/local/bin/zrok2` (v2.0.1+).
- `~/.zrok2/` populated by a prior `zrok2 enable <accountToken>` against
  `https://api-v2.frontdoor.staging.netfoundry.io`. Confirm with
  `zrok2 status --secrets` (shows `Account Token` and `EnvZId` both
  `<<SET>>`).
- `podman` (`apt install -y podman`; version 4.x is fine).
- Inbound SSH allowed from wherever you're running tests from.
- Outbound TCP/443 to `api-v2.frontdoor.staging.netfoundry.io` and the
  Ziti edge routers (`*.production.netfoundry.io:443`).

### Echo servers (the "TCP application" stand-in)

Use `istio/tcp-echo-server` with `--network=host` so ports bind directly
on the host — rootless podman's slirp4netns port-forwarder rots after
the SSH session that birthed it dies, so port mappings are not reliable
for multi-day uptime. Direct host-port binding sidesteps this entirely.

```bash
for L in a b c; do
  case $L in a) P=9101 B="APP-A:";; b) P=9102 B="APP-B:";; c) P=9103 B="APP-C:";; esac
  podman run -d --name echo-$L --network=host --restart=always \
    docker.io/istio/tcp-echo-server:1.3 $P "$B"
done

# Smoke test — expect `APP-A: …` etc. on each port
for P in 9101 9102 9103; do echo "direct-$P" | nc -w2 127.0.0.1 $P; done
```

Banners (`APP-A:` / `APP-B:` / `APP-C:`) are what let a human tell which
path the packet took end-to-end. Keep this pattern whenever you stand up
new test backends — use something with a distinguishable response so
mis-wiring is obvious at the `nc` output.

### Zrok share backends (publish the echo servers into the fabric)

Ephemeral-token form (prototype, simplest):

```bash
mkdir -p /tmp/zrok-shares
for L in a b c; do
  case $L in a) P=9101;; b) P=9102;; c) P=9103;; esac
  setsid bash -c "nohup zrok2 share private --headless -b tcpTunnel 127.0.0.1:$P > /tmp/zrok-shares/$L.log 2>&1 &" </dev/null >/dev/null 2>&1
done
sleep 5

# Capture the tokens the controller assigned
for L in a b c; do
  grep -oE 'zrok2 access private [a-z0-9]+' /tmp/zrok-shares/$L.log | awk -v l=$L '{print l": "$NF}' | head -1
done
```

**Gotchas (burned in prototype):**

- The target form is bare `host:port` (e.g. `127.0.0.1:9101`). Do **not**
  write `tcp://127.0.0.1:9101` — zrok v2 tcpTunnel backend fails every
  subsequent dial with `too many colons in address`. Share starts up fine
  but silently drops all traffic.
- Ephemeral tokens regenerate on every `zrok share private` restart.
  Dying SSH sessions kill nohup-detached processes less often than you'd
  think but still happen; when they do, all shares and their tokens
  vanish. Production should use **reserved named tokens** (`zrok2 create
  share -s <name>`) + systemd units. See
  `../ip-sec-tf-prototype/PLAN-reserved-share-tokens.md`.
- `setsid` detaches from the SSH session's process group so `pkill` from
  the SSH session doesn't cascade.

### Zrok access loopback validators (the self-test harness)

Three `zrok2 access private` processes bound to `127.0.0.1:9201/9202/9203`
that connect back through the fabric to each share's backend. This is
Test 0 — it exercises zrok + backend without any AWS in the path.

```bash
mkdir -p /tmp/zrok-access
# Replace TOKENS with whatever the share-backends reported
TOK_A=... TOK_B=... TOK_C=...
for L in a b c; do
  case $L in a) T=$TOK_A PORT=9201;; b) T=$TOK_B PORT=9202;; c) T=$TOK_C PORT=9203;; esac
  setsid bash -c "nohup zrok2 access private $T --bind 127.0.0.1:$PORT --headless > /tmp/zrok-access/$L.log 2>&1 &" </dev/null >/dev/null 2>&1
done
sleep 6

# Test 0 — run on the same box
for P in 9201 9202 9203; do echo "lb-$P" | nc -w5 127.0.0.1 $P; done
```

Expected output:

```
APP-A: lb-9201
APP-B: lb-9202
APP-C: lb-9203
```

If this fails — silence, empty bytes, or a connection close with no
banner — the fault is strictly on the **backend side** (share, echo
container, or the box's networking). Everything AWS-side is irrelevant
until Test 0 is green. That's its whole job.

### Backend health spot-checks

```bash
# On 100.55.0.139
podman ps --format "table {{.Names}}\t{{.Status}}\t{{.Command}}"
ss -tlnp 2>/dev/null | grep -E ':910[123]|:920[123]'
pgrep -af 'zrok2 share private'
pgrep -af 'zrok2 access private'

# Watch share backends dial the echo container (this is what fires on every Fargate hit)
tail -f /tmp/zrok-shares/a.log   # expect no errors; "connection refused" means the echo container isn't listening
tail -f /tmp/zrok-access/a.log   # expect no errors; accepts on local port, dials fabric
```

## AWS-side: ingress validation

### Preflight — Fargate tasks booted clean

```bash
source creds      # whatever you use for AWS sts credentials

# All three services should be desired=running=1
aws ecs describe-services --cluster ip-sec-proto-cluster \
  --services ip-sec-proto-frontend-a ip-sec-proto-frontend-b ip-sec-proto-frontend-c \
  --region us-east-1 \
  --query 'services[].{n:serviceName,d:desiredCount,r:runningCount,p:pendingCount}' \
  --output table

# Tail logs — expect "zrok environment was successfully enabled" and "access the zrok share at '0.0.0.0:9191'"
for k in a b c; do
  echo "=== frontend-$k ==="
  aws logs tail /ecs/ip-sec-proto/frontend-$k --region us-east-1 --since 10m | head -40
done
```

### Target-group health

```bash
terraform output -json listener_map | jq -r '.[] | "\(.service) \(.target_group_arn)"' | \
  while read svc arn; do
    echo "=== $svc ==="
    aws elbv2 describe-target-health --target-group-arn "$arn" --region us-east-1 \
      --query 'TargetHealthDescriptions[*].{Target:Target.Id,State:TargetHealth.State,Reason:TargetHealth.Reason}' \
      --output table
  done
```

All three targets `healthy` within ~90s of task launch (60s grace period
+ 2 × 30s healthy threshold from target-group config).

### Test 1 — in-VPC TCP reachability

From any host inside `vpc-0ddf3f3271533cbb4` with `nc`. For a test that
exercises the same route-table/path bits the real customer traffic uses,
run from **`vpn-testrouter` (10.1.5.69)** — same subnet + RT as the NLB.

```bash
NLB=10.1.5.10
for p in 9001 9002 9003; do echo "probe-$p" | nc -w5 $NLB $p; done
```

Expected: `APP-A: probe-9001` / `APP-B: probe-9002` / `APP-C: probe-9003`.

If Test 0 was green but Test 1 fails, narrow to NLB / SG / Fargate
wiring (not zrok).

### Test 2 — from the customer site (over IPsec via TGW VPN)

From any host with a `172.31.x.x` address (the VPN-connected side):

```bash
NLB=10.1.5.10
for p in 9001 9002 9003; do echo "tunnel-$p" | nc -w5 $NLB $p; done
```

Expected: same banners as Test 1. If only Test 2 fails, the fault is
the IPsec tunnel, TGW routing, or the `nlb_from_ipsec` SG rule — check:

```bash
aws ec2 describe-vpn-connections --vpn-connection-ids vpn-0982939c03412d0d2 \
  --region us-east-1 --query 'VpnConnections[0].VgwTelemetry'
```

Both tunnels should be `UP`.

### tcpdump from the customer side (when all else fails)

The prototype burned an afternoon on this. If `nc` hangs, capture on the
customer host for the full TCP state machine:

```bash
sudo tcpdump -n -v -i any host 10.1.5.10
```

Diagnostic patterns the prototype hit:

- SYN sent, SYN-ACK received, client ACKs, client PSH with data, server
  ACKs the data, server FINs **with no data back** → share backend isn't
  relaying bytes back. Check share log for `too many colons` or
  `connection refused`.
- SYN sent, SYN-ACK received, client ACK, immediate server RST → NLB
  target is unhealthy (Fargate task not listening yet, or SG denies
  the probed port).
- No SYN-ACK at all → IPsec tunnel down, TGW routing broken, or NLB SG
  denies the source CIDR.

## Operational commands (cheat sheet)

```bash
# Restart all backend shares after killing them
ssh -i /opt/sandbox.key ubuntu@100.55.0.139
pkill -f 'zrok2 share private'
# … then re-run the for-loop from the "Zrok share backends" section above.
# Capture new tokens and propagate them into whatever stack config the
# Share-tier Pulumi stack is reading from (today: the prototype's
# terraform.tfvars; tomorrow: Pulumi config on the Share stack).

# Rotate access-side validators after killing shares
pkill -f 'zrok2 access private'
# … then re-run the validators section with the new tokens.

# Stop everything on the backend box (cleanup)
pkill -f 'zrok2 share private'
pkill -f 'zrok2 access private'
podman rm -f echo-a echo-b echo-c

# Check if the backend host is still reachable at all
ping -c 3 100.55.0.139
ssh -i /opt/sandbox.key -o ConnectTimeout=5 ubuntu@100.55.0.139 'hostname; uptime'
```

## Translating this harness to a new Endpoint Site + Share

When `zrok-orchestration` stands up a new Endpoint Site + Share pair,
validate in this order:

1. **Spin up the backend side** (echo containers + zrok2 share processes)
   on whatever host is playing "Service Site" for this test.
2. **Run Test 0 locally on that host** against `zrok2 access private`
   loopback validators with the new share tokens. Green = zrok side is
   healthy.
3. **Configure the Share stack** with the new share tokens; let Pulumi
   deploy. Wait for Fargate tasks to `RUNNING` and target groups to
   `healthy`.
4. **Run Test 1 from any VPC host** (or SSM exec into one of the
   Fargate tasks via ECS Exec if enabled). Green = AWS-side ingress is
   healthy.
5. **Run Test 2 from the Endpoint Site** — the real thing. Green = done.

If anything fails, re-run Test 0 first. Its point is to eliminate half
the search space in one command.

## What to carry into `zrok-orchestration` proper

When implementing in this repo:

- Replicate the **zrok loopback validator pattern** as a first-class
  per-Share test harness. Either a sidecar in the same task (check
  overhead) or a separate "validator" task def that each Share stack
  optionally deploys — same share token, different bind port.
  Controller can expose `/frontdoor-tcp-share/{id}/test` that runs
  `nc` against that bind port internally and reports a pass/fail.
- Encode the **ephemeral-token dance** as a tech-debt ticket in the
  repo's issue tracker, and wire the Share stack to only accept
  **reserved tokens** going forward (`zrok2 create share -s <name>`).
  Ephemeral tokens are the prototype's shortcut; the orchestration
  layer should not embrace them.
- Adopt the `--network=host` convention on any container backend this
  repo eventually spawns. Rootless-podman port-mapping is a known
  stale-port-forwarder source.
- Keep the `APP-A/B/C:` banner pattern (or equivalent distinguishable
  responses) in any integration test so mis-wired listeners are obvious
  at first probe.

## Cross-references

- [`HANDOFF-ipsec-tiering.md`](./HANDOFF-ipsec-tiering.md) — architecture
  side of the handoff.
- `../ip-sec-tf-prototype/TESTING.md` — the prototype's test plan,
  operator-facing.
- `../ip-sec-tf-prototype/PLAN-reserved-share-tokens.md` — future work on
  replacing the ephemeral-token prototype with stable reserved tokens.
- `../ip-sec-tf-prototype/ARCHITECTURE.md` — canonical tier model (the
  "living triumvirate" with this doc's companion Confluence page).