# Audit

**Status: all tasks built.** The tasks are at the bottom.

The audit is an immutable record of every entity change in this service. Each `ResourceChangeMessage` becomes one
Elasticsearch document. A caller lists the documents at `GET /{scopeId}/audit-events`. Process runs are not
audited here. They stay in `domain/execution/`.

```
*Service ──(same DB tx)──▶ ResourceChangeOutbox ──relay──▶ Platform-Private ──▶ frontdoor.audit.ResourceChangeListener ──▶ AuditListener ──▶ ES audit-event
                                                                   └──────────▶ cascade queues (unchanged)
```

## Decisions

| Topic | Decision |
|---|---|
| Scope | Entity changes only, no process events. The audit is a library, `io.netfoundry.audit`, and each service runs it for itself |
| System | `@EnableAudit(system = "frontdoor")`, required. Every message carries it in `x-audit-system`; each system stores and reads only its own events |
| Delivery | Zero loss, through a transactional outbox. At-least-once, with duplicates removed by `_id` |
| Content | Full `fromVersion` / `toVersion`, redacted at publish time |
| Immutability | Application-level: every write is `op_type=create`, and no update or delete path exists |
| Retention | No deletion. One index shared by every system, no rollover |
| ES outage | Messages wait in Rabbit. The listener retries without a limit and logs at `ERROR` |
| Poison message | Dead-letter queue. Replay is a manual shovel |
| Read access | Auth type `frontdoor-audit-event`, child of `frontdoor`, with the `read` action only, in the Admin and Read-Only roles. The code names the system, because a code is unique per domain and the next system in `io.netfoundry.app` needs its own |
| API | The library owns the controller as `AbstractAuditEventController`. A service subclasses it for its path and parent link, and declares an `AuditReadAuthorizer` for the grant and the scope |
| Paging | `search_after` cursor, as in `../alerting` |
| Client | Spring Data Elasticsearch, as in `../alerting`. Same cluster, new index. One cluster per environment |
| CRUD audit in MySQL | Removed with its code. `V1.0.16` deletes its rows; no data migration |
| Message building | A public `ResourceChangeMessageFactory`, extracted in `messaging-framework` |
| Rotated secrets | Not shown. The update still records the actor and the time |
| Resource kinds | A generic `resourceKind` field (`tcp`, `ipsec`) tells two classes with one type code apart |

## 1. Transactional outbox

**Built.** The package `io.netfoundry.audit.outbox` replaces the framework's publisher for **every** consumer, not only
the audit. `@EnableAudit(resourceChangeOutbox = true)`, the default, turns it on. All audit features go in
`io.netfoundry.audit`, which moves to its own repository later, so it imports nothing from this service. [messaging.md](messaging.md)
describes how it works: *The outbox*.

- The publisher writes one `ResourceChangeOutbox` row in the caller's transaction (`MANDATORY`). There is no Rabbit call
  on the request thread.
- The relay deletes a row only after a confirm with no return. Any other result keeps the row, so a message may arrive
  twice and is never lost. A failed row waits for a doubling back-off, so it cannot block newer rows.
- The library's `AuditQueueConfiguration` declares the audit queue, bound on every resource type. So every message is routable
  from the first deploy, and changes made before the listener exists wait in Rabbit.

## 2. Redaction and resource kind at publish time

**Built**, in `io.netfoundry.audit.serialization`. `@EnableAudit` always imports it, with or without the outbox.

- Every resource-change message is built by one `ResourceChangeMessageFactory`. Its mapper is a copy of the service's
  `ObjectMapper` with `AuditAnnotationIntrospector` ahead of the mapper's own introspector. The service's mapper is
  not changed, so the process tracker, the archive and the API serialize as before.
- `@Redacted` on a field or record component writes a non-null value as `"[REDACTED]"` and a null as null. The same
  value on both sides of an update means that `changedPaths` does not show that a secret changed.
- `@ResourceKind("tcp")` on a class adds `"resourceKind"` to its payload. `TcpShare` is `tcp` and `IpsecShare` is
  `ipsec`, because they share the type code `share`. No field in the payload could tell them apart, because
  `ipsecTunnelId` can be `null`.
- **No frontdoor field needs `@Redacted` today.** `Agent.registrationCode` and `CertificateRequestToken.token` have
  `@JsonIgnore`, so no mapper writes them. The enrollment JWT is not an entity field. `ResourceChangeSerializationTest`
  checks both. A new secret field gets `@JsonIgnore` or `@Redacted`.

## 3. Index `audit-event`

**Built**, in `io.netfoundry.audit.index`. One index for every system that uses the library, defined in
`io/netfoundry/audit/index/audit-event.json`, with `"dynamic": "strict"`. It holds no field that belongs to one service.

- `AuditIndexInitializer` (a `SmartLifecycle`, phase 0) creates the index or applies an additive mapping change
  through `ElasticIndexManager`, copied from `alerting`.
- **An unreachable cluster does not stop startup.** The initializer logs at `ERROR`, retries every
  `io.netfoundry.audit.index.retry-interval` (30s), and `isReady()` stays false. The listener (task 5) consumes only
  once it is ready, so messages wait in the broker.
- **A rejected mapping fails startup**, because waiting cannot fix a code error. If the rejection only arrives after
  an outage, the index stays not ready and the initializer stops trying.
- `ElasticFailures.isTransient` draws the line: an I/O failure, a 5xx, a 429, a 401 or a 403 is an outage; any other
  status, including `400` and `409`, is not. A 401 or 403 is about the service's credentials, not the message, so the
  listener waits for a fix and does not dead-letter the queue. The listener and the initializer use the same test.
- **`spring.elasticsearch.uris` must be set explicitly** while the index is enabled, because Boot's own default
  (`localhost:9200`) would hide a missing setting. `io.netfoundry.audit.index.enabled=false` turns the index off, as
  frontdoor's test profile does.
- `fromVersion` and `toVersion` have `ignore_above: 8191`. A longer leaf stays in `_source` and is not indexed, so one
  huge value cannot make Elasticsearch reject the whole event.
- **Frontdoor's wiring.** Terraform sets `SPRING_ELASTICSEARCH_URIS` from the workspace's `elasticsearch_host`, or
  the shared `mop_core` cluster when that is empty. The credentials come from the hand-seeded
  `elasticsearch_frontdoor_credentials` secret. `management.health.elasticsearch.enabled` is `false`.

| Field | Type | Source |
|---|---|---|
| `id` | keyword | `message.id`, also `_id` |
| `system` | keyword | the `x-audit-system` header. Every query the library runs filters on it |
| `resourcePath` | keyword | message |
| `resourcePathElements` | flattened | message, for example `{frontdoor: <id>, share: <id>}`. A service scopes on its own parent type |
| `resourceType`, `resourceId`, `changeType` | keyword | message |
| `resourceKind` | keyword | `toVersion.resourceKind`, or `fromVersion` on a delete. Absent when the class has no `@ResourceKind` |
| `correlationId`, `traceId`, `spanId` | keyword | message |
| `occurredAt` | date | message |
| `recordedAt` | date | listener clock |
| `actor.id`, `actor.name` | keyword | message, plus the service's `AuditActorResolver` at write time |
| `changedPaths` | keyword[] | JSON pointers that differ between the from and to versions, on the redacted values |
| `fromVersion`, `toVersion` | flattened | message |
| `searchText` | match_only_text | the distinct scalar values of both versions, without `[REDACTED]`, at most 32 KB |

- `flattened` keeps the full entity without a mapping explosion, and it supports exact and prefix filters on a leaf.
  Its values are case-sensitive keywords, and a range compares them as strings.
- `searchText` gives `q` case-insensitive, single-word matching. It stores no positions or scores.
- A caller cannot choose `system`. The library adds its own `AuditSystem` to every query, as the listing adds the
  path's parent id.
- One index with no rollover. A rollover alias would break idempotency, because `op_type=create` checks
  only the current write index.

## 4. Listener

**Built**, in `io.netfoundry.audit.listener`.

- **The queue belongs to the library.** `AuditQueueConfiguration` declares `<system>.audit.ResourceChangeListener`
  (for frontdoor, `frontdoor.audit.ResourceChangeListener`): durable, quorum, bound with `ResourceChange.v1.*.<type>`
  for each `@AuthResourceType` code under the service's base package. That is Boot's `AutoConfigurationPackages`, or
  else the package of the class with `@EnableAudit`. It is declared even when the listener is off, so messages wait.
  Frontdoor's `AuditQueueConfigurationTest` asserts its keys as literals.
- **Dead letters go to `<queue>.dlq`**, through the platform dead-letter exchange with key `<queue>.deadLetter`. The
  queue carries the dead-letter arguments, so a rejected delivery lands there with no republish code. Replay is a
  manual shovel.
- **A routing key does not name its publisher.** The listener acks and drops a message whose `x-audit-system` header
  is not its own system, or is absent. `resourceDomain` cannot do this, because several services may share an auth
  domain.
- `AuditListener` runs its own `SimpleMessageListenerContainer`: one consumer, manual ack, prefetch 10. It is not a
  `@RabbitListener`, so the shared retry factory and `RabbitListenerIdentityAspect` do not apply. It starts only once
  `AuditIndexInitializer.isReady()`, and until then messages stay in the queue.
- `AuditMessageHandler` decides each delivery:

  | Delivery | Outcome |
  |---|---|
  | Another system's, or no header | ack, dropped |
  | Body cannot be read or mapped | reject into the DLQ |
  | Stored | ack |
  | `409`: already stored | ack; it is a redelivery |
  | Any other rejection (400…) | reject into the DLQ |
  | Outage (I/O, 5xx, 429, 401, 403) | wait on the consumer thread and try again, unacked. Backoff 1s doubling to 1m, one `ERROR` per minute |
  | Stopping during an outage | nack with requeue |

  Waiting on the consumer thread keeps the order and loses nothing. A stopping listener leaves a wait within 200 ms.
- The write is `op_type=create` under the message id, with refresh `NONE`.
- `AuditEventFactory` builds the document; `DefaultAuditEventFactory` maps the message field by field. A service may
  declare its own.
- **Metrics:** `audit.listener.messages{outcome=stored|duplicate|other-system|dead-lettered}` and
  `audit.listener.retrying`, which is 1 while an outage holds the listener.
- Settings: `io.netfoundry.audit.listener.{enabled,prefetch,initial-backoff,max-backoff,error-interval}`. The
  listener is also off when the index is off.
- **Derived fields** (task 6), in `DefaultAuditEventFactory` through `VersionDiff`:
  - `changedPaths`: objects are compared key by key, and an array or a scalar is one value. A create or a hard delete
    is compared with an empty object, so it lists every leaf. An object that appears or goes is listed leaf by leaf.
  - `searchText`: the distinct scalar values of both versions, the to version first.
- **Actor names** (`io.netfoundry.audit.actor`). `AuditActorNames` wraps the service's `AuditActorResolver` in a
  Caffeine cache (`io.netfoundry.audit.actor.cache-ttl`, 10m). An empty answer is cached. A lookup that throws stores
  `null`, is not cached, and never blocks the write. A resolver answers empty for a permanent "no name" and throws only
  for a failure that may pass. The library's default resolver names nobody. Frontdoor's (`domain/audit/AuditConfiguration`)
  reads `GET /identities/{id}` as the service identity, under the `V2.179.0` grants on `user-identity` and
  `api-account-identity`. It answers `name`, or `email` when there is no name. A 403, which a service identity gets,
  or a 404 answers empty, so the cache holds it. The API reads names through the same cache (task 7). A page names each distinct actor
  once, with at most 8 lookups at a time, within `io.netfoundry.audit.actor.read-budget` (2s). An actor not named in
  time has no `actor.name` on that read; its lookup still finishes and fills the cache.

## 5. API

**Built**, in `io.netfoundry.audit.api`. The library owns the routes, the view, the paging and the failure statuses. Frontdoor
declares the path, the parent link and the grant.

| Route | Behaviour |
|---|---|
| `GET /{scopeId}/audit-events` | Filters: `resourceType`, `resourceKind`, `resourceId`, `changeType`, `actorId`, `correlationId`, `traceId`, `changedPath`, each with up to 25 values, repeated or comma-separated; plus `from` (inclusive), `to` (exclusive), `q`. Paging: `cursor`, `size` (default 25, 1–100). A HAL `CollectionModel` with a `next` link on every page but the last |
| `GET /{scopeId}/audit-events/{id}` | One event under the scope. An event under another scope answers 404 |

- **What a service declares.** A subclass of `AbstractAuditEventController` with `@RestController`, `@Scope`, a
  `@RequestMapping` that holds `{scopeId}`, and `parentLink(scopeId)`. Also one `AuditReadAuthorizer` bean.
  `AuditApiConfiguration` makes the `AuditEventReader` only when that bean exists, so the library has no default grant.
  Frontdoor's are `controllers/AuditEventController` and `domain/audit/AuditConfiguration`.
- **The authorizer gives the grant and the scope together.** `AuditReadAuthorizer.actionInParent(resolver, type,
  action)` checks `action` on `type` at its parent's path, and scopes on the parent's code. Frontdoor passes
  `FrontdoorAuditEvent` and `read`, so the scope is `frontdoor`. One bean holds both, so a listing cannot be scoped on
  one resource and checked on another.
- **The library scopes every query.** `AuditEventSearch` adds `system = <@EnableAudit system>`. `AuditEventReader`
  replaces whatever scope a query carries with `resourcePathElements.<scope> = <path id>`.
- **Order and paging.** Newest first, by `occurredAt` and then `id`, so the order is total. The cursor is opaque
  (base64url of the last event's millis and id) and feeds `search_after`. One extra row is fetched, so the last page
  is known to be the last.
- **Filters.** The values of one filter are alternatives, and separate filters must all match. Each multi-value filter
  is one `terms` query. More than `AuditQuery.MAX_VALUES` (25) values in one filter answers 400. `system` is not a
  filter, because a system reads only its own events.
- **`q`** matches `searchText` with every word required. It is case-insensitive.
- **Authorization.** An Elasticsearch query cannot take an `@AuthRead` where clause, so the check is explicit. A listing
  without the grant answers `200` with an empty list and does not search. A single read answers 403.
- **The view** is the library's `AuditEventView`. It holds `scopeId` and drops `system`, `searchText` and the raw
  `resourcePath`. `actor.name` is the name now, through the `AuditActorNames` cache, and `actor.nameAtEvent` is what
  was stored.
- **Failures.** A bad cursor or size answers 400. An index that is off or not ready yet, or an Elasticsearch outage,
  answers 503. The reader checks `AuditIndexInitializer.isReady()`, because a missing index answers 404, which is no
  outage to `ElasticFailures`.
  `ElasticFailures` counts a JSON parse error as a client error, not an outage. Each failure is a
  `ResponseStatusException`, answered by the base controller's own `@ExceptionHandler`, because a service's catch-all
  advice would make it 500. A service overrides `problem(..)` for its own error body; frontdoor uses
  `CommonExceptionHandler.buildErrorResponse`.
- **OpenAPI.** The base methods document every status, because springdoc reads a class-level `@ApiDefaultResponses`
  from the class that declares the method, not from the subclass. There is no OpenAPI link to the parent operation,
  because the library cannot name it.
- `FrontdoorAuditEvent` is registered in `PostConfiguration` and `PostTestConfiguration`. Its lookup is
  `AuditEventReader.scopeIdOf`, which is unscoped, answers empty when the index is off or down, and never fails the
  request.
- No create, update or delete route, and no dry-run validation, because there is nothing to write.
  `ApiDocumentationTest` pins that, with the filters and the cursor.

## 6. Library packaging

`io.netfoundry.audit` becomes its own artifact. A service adds that one dependency and `@EnableAudit(system = "...")`,
and declares nothing else for the audit.

- **Every dependency is inherited.** The library's POM declares, in `compile` scope, everything its features need:
  `messaging-framework`, `spring-boot-starter-amqp`, `spring-data-elasticsearch`, `spring-boot-autoconfigure`,
  the `authorization` client (the queue reads `@AuthResourceType`, the API checks grants), Caffeine (actor names),
  Jackson, Micrometer, and for the API `spring-webmvc`, `spring-hateoas` and `swagger-annotations`. A service does not repeat them, and it takes their versions from the library. The library
  pins them to the same Boot and messaging-framework line as the platform parent.
- **The one exception is the database.** `spring-jdbc` is `optional` in the library's POM, so it is not inherited.
  Only the outbox uses it. A service that keeps the outbox on already has a `DataSource` and Flyway, and it adds the
  outbox table to its own migrations. A service with `@EnableAudit(resourceChangeOutbox = false)` gets no JDBC
  class from the library. If it did, Boot's `DataSourceAutoConfiguration` would switch on and fail startup for a
  service that has no database.
- **Code rule that follows.** Only `io.netfoundry.audit.outbox` may import `org.springframework.jdbc` or
  `javax.sql`, and `AuditImportSelector` must not load an outbox class when the outbox is off. The import is by
  class name, so a JDBC class never loads. A test runs a context with the outbox off and no JDBC on the classpath,
  and the boundary test checks the import rule.
- **Test dependencies are not inherited.** The library's own tests use MySQL, RabbitMQ and Elasticsearch through
  `AuditContainers`: containers locally, the pipeline's services under `pipeline-test`. The outbox table is in the
  service's own database, created by the service's Flyway migrations. A service's tests set `io.netfoundry.audit.index.enabled=false` and
  `io.netfoundry.audit.outbox.relay.enabled=false`, or start their own containers.
- **Resources ship in the jar:** the outbox DDL (`io/netfoundry/audit/outbox/mysql/`) and the index definition
  (`io/netfoundry/audit/index/audit-event.json`).
- **Service configuration that the library cannot set:** `spring.elasticsearch.uris` and its credentials, and
  `management.health.elasticsearch.enabled: false`, so an Elasticsearch outage does not turn `/actuator/health`
  `DOWN`.

## Open

1. **Deleted accounts.** The documents stay. After the `frontdoor` row is gone, the `@AuthRead` path cannot be built,
   so nobody can read them. Deferred.

## Tasks

0. `messaging-framework`: extract `ResourceChangeMessageFactory`, make `ResourceChangeMessageService` use it, release.
1. Outbox: table, publisher, relay, confirms, metrics. Test support.
2. The publisher's mapper: `@Redacted` and `@ResourceKind`, in the library. Annotate both share entities.
3. auth-service migration `V2.179.12` for `frontdoor-audit-event` (`read`). Done in `../authorization`. The
   `PostConfiguration` and `PostTestConfiguration` registration comes with task 7, with the class that carries
   `@AuthResourceType`. That class also adds a binding to the audit queue, and to `AuditQueueConfigurationTest`'s literal list.
4. Spring Data Elasticsearch config, `ElasticIndexManager`, index definition, Testcontainers. Done, with the
   Terraform wiring.
5. The listener: queue and DLQ in the library, the system filter, the outage wait. Done. `DefaultAuditEventFactory`
   maps the message directly; task 6 adds `changedPaths`, `searchText` and the actor name to it.
6. Document builder: `changedPaths`, `searchText`, actor enrichment. Done.
7. The audit API: filters, cursor, read-time enrichment. Done. The controller is the library's `AbstractAuditEventController`.
8. Remove `ExecutionService.created/updated/deleted` and the calls to them. Done: the four services that called it lose the calls and
   the `try`/`catch` that recorded a failure, and `V1.0.16` deletes the rows without a process.