diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 16b6145..2445904 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -120,7 +120,7 @@ A Rust supervisor that runs Vector as a child process so a Vector pipeline behav | `src/deployment.rs`, `src/vector-layer.dockerfile` | `contract()`, the source the Dockerfile and the released chart are generated from, and the two-binary image override spliced onto scalo's generated Dockerfile | | `templates/bus.yaml`, `templates/direct.yaml` | Whole runnable Vector topologies, one per transport, commented per field | | `pipelines/filebeat/` | The bundled filebeat pipeline. The image does not ship it | -| `docs/` | `architecture.md` for why the service is built this way and the invariants, `internals.md` for field-by-field depth, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` | +| `docs/` | `architecture.md` for the parts and why, a page per part (`internals.md` the config engine, `vector.md`, `direct-transport.md`), `reference/configuration.md`, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` | | `tests/` | `integration`, `e2e`, `smoke`, and `TESTING.md` for how the broker and Vector binary are resolved | | `scripts/fetch-vector.sh`, `scripts/pgo-workload.sh` | Downloads the pinned Vector for tests, and drives the PGO/BOLT workload | @@ -142,6 +142,10 @@ What a passing run does NOT mean: - `e2e::filebeat_kafka` runs by default and needs a dfe-transform-vrl checkout, for its cases and the bundled pipeline: beside this repo, or at `DFE_TRANSFORM_VRL_DIR`. Without one it skips locally and fails under CI, which checks the repo out beside this one. - A docs-only push runs no CI at all. `paths-ignore` for `docs/**` and `**.md` is on the `push` trigger but not on `pull_request`, so the PR is the only place a docs change is checked. A silent push is not a pass. +To upgrade Vector, change `VECTOR_VERSION` and `VECTOR_SHA256_AMD64` and `VECTOR_SHA256_ARM64` in `src/deployment.rs` from Vector's published `SHA256SUMS`, then regenerate the Dockerfile. The Dockerfile, the default `vector.version`, the contract's `default_config` and `scripts/fetch-vector.sh` all read the constant. Move `services.vector` in dfe-infra's `versions.yaml` to the same release, tagged `-alpine`, because dfe-fetcher's container Vector extractor runs that image. The full test tier runs the `vector_validate` cases against the new Vector. + +The image has two binaries, and scalo's contract builds only the supervisor. `src/deployment.rs::emit_dockerfile()` splices `src/vector-layer.dockerfile` in before the generated `USER` line, with the version and checksums substituted. The fragment is a Dockerfile, not a Rust string, so hadolint and shellcheck read it. + ### What tends to bite | Don't | Do | Why | @@ -156,6 +160,7 @@ What a passing run does NOT mean: | Assume a config key you added is wired because it parses and validates | Test that it changes behaviour, and keep `every_committed_config_file_loads_and_validates` passing over each committed config file | A key can parse, validate and reach nothing, so a deployment that moves a port on it keeps the old port and loses every probe | | Set `enable.auto.commit: false` here, as the shared DFE consumer baseline does | Leave auto-commit on | Vector's kafka source only stores offsets and leaves librdkafka's commit timer to flush them, and that timer is armed only when auto-commit is on, so with it off the stored offsets reach the broker only on a rebalance or a clean shutdown | | Read `/readyz` as "carrying traffic" | Read it as "Vector survived 500 ms after spawn, and the sink is not stalled" | A pod with an unreachable broker and nothing held stays Ready with no record moving | +| Replace the Vector licence files in the image with a separate download | Keep shipping `LICENSE`, `NOTICE` and the `licenses` tree from the release archive | Vector is redistributed unmodified under MPL-2.0, which needs the licence text and the source location shipped. Taking them from the archive keeps the attribution matched to the exact binary | | Swap the reload poller for inotify | Keep polling | S3-backed mounts -- s3fs, goofys, Mountpoint for S3 -- generate no filesystem notification events, so inotify works on every laptop and stops reloading in production with nothing logged | | Add or rename a credential field without a schema marker | Schema it as `scalo::SensitiveString` with `#[schemars(with = ...)]`, as `SaslConfig::password` is | That puts `x-scalo-secret` and `writeOnly` on the field in the emitted config-schema, which is what tells a console to mask it. Without it only dfe-engine's name-based fallback can mask it, so a name that fallback does not read as a credential returns an operator's Kafka password in the clear | diff --git a/README.md b/README.md index 9dc2659..857b68e 100644 --- a/README.md +++ b/README.md @@ -64,33 +64,55 @@ config: --- flowchart TB accTitle: L3 Components -- dfe-transform-vector - accDescr: The previous stage sends records into the pod and the config and transform files configure it. Inside, the config engine writes the Vector config that the process manager starts Vector on, and on the direct transport the bridge intake hands records to Vector and the bridge return takes them back. Vector's metrics are merged into the supervisor's and served on the ops port. Records leave for the next stage and metrics go to the collector. + accDescr: The previous stage sends records into the pod and the config and transform files configure it. Inside, the config engine writes the Vector config that the process manager starts Vector on. On the direct transport the bridge intake hands records to Vector and the bridge return takes them back. Vector's metrics are merged into the supervisor's. Records leave for the next stage and metrics go to the collector. - prev["Previous stage"] - files[("Config files")] + subgraph PREV["Previous stage"] + direction LR + prev("Records in
*_land topic or Push") + end - subgraph pod["dfe-transform-vector"] + subgraph CFGIN["Config"] direction LR - cfg[["Config engine"]] - proc["Process manager"] - intake[["Bridge intake"]] - vec[["Vector"]] - ret[["Bridge return"]] - merge["Metrics merge"] - cfg --> proc - proc --> vec - intake --> vec - vec --> ret - vec --> merge + files[("Config files
config and transforms")] end - next["Next stage"] - collector["Metrics collector"] + subgraph POD["dfe-transform-vector"] + direction TB + subgraph SUP["Supervisor"] + direction LR + cfg("Config engine
writes Vector's config") + proc("Process manager
spawns, restarts") + merge("Metrics merge
vector_* into ours") + cfg --> proc + end + subgraph DATA["Records"] + direction LR + intake("Bridge intake
direct only") + vec("Vector
the transform") + ret("Bridge return
direct only") + intake -.-> vec -.-> ret + end + SUP -->|starts, scrapes| DATA + end - prev -->|records| pod - files -->|config| pod - pod -->|records| next - pod -->|metrics| collector + subgraph NEXT["Next stage"] + direction LR + next("Records out
*_load topic or Push") + end + + subgraph COL["Telemetry"] + direction LR + collector("Metrics collector
scrape or OTLP") + end + + PREV -->|records| POD + CFGIN -->|config| POD + POD -->|records| NEXT + POD -->|metrics| COL + class prev,next bus + class collector ext + class files store + class cfg,proc,intake,vec,ret,merge part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -102,13 +124,16 @@ flowchart TB classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class prev,next,collector ext - class files store - class cfg,proc,intake,vec,ret,merge part - style pod fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style PREV fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style CFGIN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style POD fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style SUP fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style DATA fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style NEXT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style COL fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L3 Components -- dfe-transform-vector. Dashed boxes are external.* Up: top of dfe-transform-vector. Down: [L4 Detail -- Config assembly](docs/architecture.md#config-assembly), [L4 Detail -- Direct transport bridge](docs/architecture.md#where-the-transports-differ), [L4 Detail -- Vector lifecycle](docs/internals.md#the-vector-subprocess). +*Figure 1. L3 Components -- dfe-transform-vector. Dotted arrows are the direct transport. On the bus Vector reads and writes Kafka itself.* Up: top of dfe-transform-vector. Down: [dfe-transform-vector architecture](docs/architecture.md). On `bus` Vector reads and writes Kafka itself and the bridge does nothing. On `direct` the bridge takes records from the previous stage's Push client and hands Vector's output to the next stage's Push listener. @@ -136,12 +161,15 @@ One port, 9090 by default, serves the ops surface: `/livez`, `/readyz` and `/met The config file (`--config`, else `config.yaml` or `config.yml` in the working directory) sits over the defaults, and `DFE_TRANSFORM_VECTOR_*` environment variables sit over the file. The working directory's `.env` is read into the environment first, and a variable set before it wins. The flat names take a single underscore and a comma list (`DFE_TRANSFORM_VECTOR_SOURCE_BROKERS=a:9092,b:9092`, `_SINK_TOPIC`, `_TRANSFORMS_DIR`, the SASL pairs) and beat the double-underscore form. That form reaches any key and takes a list in brackets (`DFE_TRANSFORM_VECTOR_SOURCE__BROKERS='[a:9092,b:9092]'`), and a bare comma list there fails to load. `--log-level`, `--log-format` and `--metrics-addr`, or `LOG_LEVEL`, `LOG_FORMAT` and `METRICS_ADDR`, outrank `logging.*` and `metrics.address`. [docs/config-schema.yaml](docs/config-schema.yaml) lists the app's keys with their types and defaults. -With reload on, the default, a transform file change applies once it validates, with no restart. Any other change to the app's keys, `dfe_source`, `pipeline.name` and `scaling.pressure_threshold` included, is logged as needing a pod restart, and no later transform change applies until the pod restarts or the edit is reverted. scalo's sections in the file, such as `version_check`, `scaling.enabled` and `metrics.namespace`, are read once at startup, so an edit there needs a restart, and the watcher logs a successful reload without applying it. In a chart deployment a config or file set change rolls the pods anyway, because the chart hashes both into the pod template. The reload path, the readiness rules, the metrics merge and the Kafka settings are in [docs/internals.md](docs/internals.md) and [docs/librdkafka.md](docs/librdkafka.md). +With reload on, the default, a transform file change applies once it validates, with no restart. Any other change to the app's keys, `dfe_source`, `pipeline.name` and `scaling.pressure_threshold` included, is logged as needing a pod restart, and no later transform change applies until the pod restarts or the edit is reverted. scalo's sections in the file, such as `version_check`, `scaling.enabled` and `metrics.namespace`, are read once at startup, so an edit there needs a restart, and the watcher logs a successful reload without applying it. In a chart deployment a config or file set change rolls the pods anyway, because the chart hashes both into the pod template. The reload path is in [docs/internals.md](docs/internals.md), the readiness rules and the metrics merge in [docs/vector.md](docs/vector.md), and the Kafka settings in [docs/librdkafka.md](docs/librdkafka.md). ## Documentation -- [docs/architecture.md](docs/architecture.md): the problem, the ownership boundary and the invariants. -- [docs/internals.md](docs/internals.md): the config engine, hot reload, the Vector subprocess, health and metrics, dead letters, the Vector version pin, the chart. +- [docs/architecture.md](docs/architecture.md): the parts, how data flows and why it is built this way. +- [docs/internals.md](docs/internals.md): the config engine and hot reload. +- [docs/vector.md](docs/vector.md): the Vector subprocess, health and metrics, and dead letters. +- [docs/direct-transport.md](docs/direct-transport.md): the bridge on the direct transport. +- [docs/reference/configuration.md](docs/reference/configuration.md): the config layers and the chart. - [docs/migration.md](docs/migration.md): moving a pipeline off the official Vector chart. - [docs/librdkafka.md](docs/librdkafka.md): the librdkafka settings Vector's Kafka source and sink are given. - [tests/TESTING.md](tests/TESTING.md): the test tiers, and the e2e cases that need Docker and a Vector binary. diff --git a/docs/README.md b/docs/README.md index df21a65..f7fa864 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,7 +3,10 @@ How dfe-transform-vector is built and configured, and how to move a pipeline onto it from Vector. - [Architecture](architecture.md) -- [Internals](internals.md) +- [Config engine](internals.md) +- [Vector](vector.md) +- [Direct transport](direct-transport.md) +- [Configuration](reference/configuration.md) - [librdkafka settings](librdkafka.md) - [Moving off the Vector chart](migration.md) - [config-schema.yaml](config-schema.yaml) -- the configuration as a JSON Schema (YAML) diff --git a/docs/architecture.md b/docs/architecture.md index b28cdf6..0f5d4f6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,41 +1,6 @@ -# Architecture +# dfe-transform-vector architecture -Why this repo exists, what owns what, and the invariants you cannot read off a single file. Field-level detail is in [internals.md](internals.md). - -## The problem - -DFE needs a transform stage between its sources and its loader, and Vector does the per-record transform work well. What it lacks is the rest of a DFE app. Vector has its own config format, metrics endpoint, process lifecycle and idea of what "started" means. Every DFE app presents one control surface instead: a big-dial YAML, one ops port carrying `/livez`, `/readyz` and `/metrics`, an OTLP push, KEDA scaling and a chart assembled from its deployment contract. - -So Vector is kept, and wrapped. The Rust supervisor is PID 1 and Vector is its child. The supervisor supplies the production half -- config, credentials, health, metrics, restarts -- and Vector does the transform. - -## Why a subprocess - -Vector is not designed as a library and has no stable public API for embedding, so the image downloads the upstream release binary at a pinned version and the supervisor spawns it with `--config-dir`. A Vector upgrade is then a version bump and an image rebuild, where a compiled-in Vector would mean rebuilding against every Vector release. - -This DFE Rust app alone ships two binaries in its runtime image: the supervisor, which scalo's deployment contract builds, and `vector`, which has no slot in that contract. The Dockerfile is therefore generated by scalo and then spliced: `src/deployment.rs::emit_dockerfile()` puts `src/vector-layer.dockerfile` in before the generated `USER` line, with the version and the two archive checksums substituted. - -The fragment is a Dockerfile rather than a Rust string literal so hadolint and shellcheck can read it. - -## Who owns what - -Most confusion here is a question asked of the wrong layer. - -| Layer | Owns | -|---|---| -| Vector (the child) | Per-record transform work, the Kafka client on the `bus` transport, its internal metrics | -| This supervisor | Config assembly, process lifecycle and crash recovery, health reporting, the metrics merge, the direct-transport bridge, the deployment artefacts | -| dfe-engine | What the app IS -- it reads dfe-infra's `apps.yaml`, compiles the app's routing into its chart values and holds the config in its registry | -| dfe-infra | The image pin and the GitOps deployment | - -Nothing about how this app is deployed is coded into dfe-engine. dfe-infra's `apps.yaml` declares it `per_config` with `scale_deployed: true` -- one deployment per source config, not a singleton, and KEDA drives the replica count. Its `default_in` list names `docker-single` alone, so any other deploy composition gets it when a deployment asks for it. - -## The pieces - -The pod's parts are in [Figure 1 of the README](../README.md). The config engine turns a handful of big dials into a Vector config directory: a global file, a source file, the user's transform files, a sink file and an observability file. The user authors only the `transforms:` and `enrichment_tables:` blocks. `wiring` parses the component labels and `inputs` references in those files and joins `dfe_source` to `dfe_sink` through them, so a transform author does not hand-wire the topology. - -### Config assembly - -The supervisor loads and validates the config, writes the secret files and the source, then wires the transforms, writes the sink and runs `vector validate` before Vector reads the directory. +dfe-transform-vector runs a Vector pipeline as a DFE transform stage, for the transforms beyond VRL's `remap`. A Rust supervisor is PID 1 and gives Vector what every DFE app has: one config, one ops port, health, metrics and restarts. ```mermaid --- @@ -84,18 +49,56 @@ config: .ext .label-icon{color:#64748B} --- flowchart TB - accTitle: L4 Detail -- Config assembly - accDescr: The config is loaded through its cascade and validated, the secret files are written and the source is generated. The transform files are read and wired into a checked graph, the sink is generated, and vector validate gates the directory before Vector is started or sent a SIGHUP. + accTitle: L3 Components -- dfe-transform-vector + accDescr: The previous stage sends records into the pod and the config and transform files configure it. Inside, the config engine writes the Vector config that the process manager starts Vector on. On the direct transport the bridge intake hands records to Vector and the bridge return takes them back. Vector's metrics are merged into the supervisor's. Records leave for the next stage and metrics go to the collector. - subgraph s1["1 Prepare"] + subgraph PREV["Previous stage"] direction LR - load["Load cascade"] --> check["Validate config"] --> secrets["Write secret files"] --> source["Generate source"] + prev("Records in
*_land topic or Push") end - subgraph s2["2 Wire and gate"] + + subgraph CFGIN["Config"] + direction LR + files[("Config files
config and transforms")] + end + + subgraph POD["dfe-transform-vector"] + direction TB + subgraph SUP["Supervisor"] + direction LR + cfg("Config engine
writes Vector's config") + proc("Process manager
spawns, restarts") + merge("Metrics merge
vector_* into ours") + cfg --> proc + end + subgraph DATA["Records"] + direction LR + intake("Bridge intake
direct only") + vec("Vector
the transform") + ret("Bridge return
direct only") + intake -.-> vec -.-> ret + end + SUP -->|starts, scrapes| DATA + end + + subgraph NEXT["Next stage"] direction LR - read["Read transforms"] --> wire["Wire and check"] --> sink["Generate sink"] --> gate["vector validate"] + next("Records out
*_load topic or Push") end - s1 --> s2 + + subgraph COL["Telemetry"] + direction LR + collector("Metrics collector
scrape or OTLP") + end + + PREV -->|records| POD + CFGIN -->|config| POD + POD -->|records| NEXT + POD -->|metrics| COL + class prev,next bus + class collector ext + class files store + class cfg,proc,intake,vec,ret,merge part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -107,135 +110,89 @@ flowchart TB classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class load,check,secrets,source,read,wire,sink,gate part - style s1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 - style s2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style PREV fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style CFGIN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style POD fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style SUP fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style DATA fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style NEXT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style COL fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 2. L4 Detail -- Config assembly. Read each stage left to right, then down.* Up: [L3 Components -- dfe-transform-vector](../README.md). +*Figure 1. L3 Components -- dfe-transform-vector. Dotted arrows are the direct transport. On the bus Vector reads and writes Kafka itself.* Up: [L2 Services -- DFE deployment](https://github.com/hyperi-io/dfe-infra/blob/main/docs/architecture.md). Down: [L4 Detail -- Config engine](internals.md), [L4 Detail -- Vector](vector.md), [L4 Detail -- Direct transport bridge](direct-transport.md#how-it-works). -`run` assembles into `vector.config_dir`, and a [reload](internals.md#hot-reload) into `.next`. The wiring check rejects an undefined input, a transform `dfe_source` cannot reach and a cycle. +## How data flows -The process manager spawns Vector, forwards SIGTERM at shutdown, and restarts it after a crash with exponential backoff from one to sixty seconds. +1. The **Config engine** turns the big-dial config and your transform files into a Vector config directory, wires the graph and runs `vector validate`. +2. The **Process manager** spawns **Vector** on that directory, and restarts it after a crash. +3. On the bus, Vector's `kafka` source reads `_land`. On the direct transport, the **Bridge intake** takes Push calls on port 6000 and hands records to Vector over loopback. +4. Vector runs the transforms. +5. On the bus, Vector's `kafka` sink writes `_load`. On direct, the **Bridge return** takes Vector's output and pushes it to the next stage. +6. The **Metrics merge** scrapes Vector's exporter and serves its `vector_*` series beside the supervisor's, on one port and one OTLP push. -## Runtime invariants +## Components -**Reload polls, and must.** The watcher polls rather than using inotify because S3-backed mounts -- s3fs, goofys, Mountpoint for S3 -- generate no filesystem notification events. Inotify would work on every laptop and stop reloading in production with nothing logged. +| Part | Role | Page | +|---|---|---| +| Config engine | Builds and validates Vector's config, and reloads transforms | [internals.md](internals.md) | +| Process manager | Spawns Vector, forwards SIGTERM, restarts it with backoff | [vector.md](vector.md#how-it-works) | +| Metrics merge | Serves Vector's metrics with the supervisor's | [vector.md](vector.md#interfaces) | +| Bridge intake | Translates scalo `Transport/Push` into Vector's `PushEvents` | [direct-transport.md](direct-transport.md) | +| Vector | The upstream binary, at the version `VECTOR_VERSION` pins | [vector.md](vector.md) | +| Bridge return | Takes Vector's output and pushes it onward | [direct-transport.md](direct-transport.md) | -**Only transforms hot-reload.** A transform change is validated, SIGHUPed into Vector, and reported from Vector's reload counters, since the signal succeeds even when Vector rejects it. Everything else is set at startup -- the consumer, the Push listener, the bridge legs, the group id, the ops listener -- and needs a pod restart. A reload compares the app's config with the running one, so a restart-only change in the file blocks every later transform change until a restart or a revert. An edit to scalo's sections in the file is not compared: it logs a successful reload and waits for a restart. +You write the `transforms:` and `enrichment_tables:` blocks. The supervisor generates the source, the sink and the observability config. -**A push on the direct path is answered only once the next hop has it.** Both bridge listeners are built armed, so a push is held until the pipeline releases its records. A hop that rejects is retried until the hold runs out, and then the push is answered `Unavailable` and its sender retries. The holds nest, 18 s at the intake, 16.5 s on the leg back from Vector and 15 s per send onward, so an inner hop times out first and the outer one can still answer its sender. A record no resend could deliver is not resent forever: one over the next stage's message-size ceiling goes to the DLQ, or is dropped and counted with the DLQ off, and one Vector refused for good is released dropped and counted. +## External dependencies -**Only the intake sheds under pressure.** The leg back from Vector drains everything the intake holds, so shedding it would stall the stage until the holds expired into duplicates. Both listeners lease held bytes on the runtime's memory guard. +| Dependency | Needed for | Swappable | +|---|---|---| +| Kafka | The bus transport, read and written by Vector | Any broker `source.brokers` and `sink.brokers` reach. Direct needs none | +| A DFE service over gRPC | The direct sink, `http://dfe-loader:6000` by default | Any stage running a Push listener | +| Dead-letter queue | Records over the next hop's size ceiling | `dfe_transform_dlq` on the bus, a file spool on direct | +| dfe-engine | Writing a deployment per source config | dfe-infra's `apps.yaml` declares the app, so the engine has no code for it | -**A failed bridge leg stops the process.** Its senders would otherwise retry for ever against a live pod, so the service exits non-zero. +## Deployment variants -**Transforms cannot reach the Kafka credentials.** Vector runs whatever a transform file declares, with read access to the credentials. So a transform file declares only `transforms` (never `lua`) and `enrichment_tables`, each table resolves inside `transforms.data_dir` after `..` and symlinks and outside every credential path (the config directory, a mounted `sasl.secret_dir`, a TLS key), and a `SECRET[...]` naming an assembler backend is refused. Vector runs without the `DFE_TRANSFORM_VECTOR_*` variables. +| Variant | What runs | +|---|---| +| Docker `single` | On by default. One instance per source that binds a Vector transform | +| Kubernetes, every profile | When a deployment asks for it. One deployment per source config, KEDA sets replicas | -## Health and metrics invariants +One Vector runs per pod, and replicas scale out. The released chart scales on CPU. dfe-infra's values add a pressure trigger in `single`, `scale` and `mesh`. Lag isn't a trigger, because it rises when a downstream stage breaks and more pods can't fix that. -**Port 9598 is a loopback debug surface, not a scrape target.** Vector's `prometheus_exporter` binds it. The supervisor GETs it on scalo's metrics interval and registers every sample on the registry scalo serves and pushes over OTLP, so one endpoint serves both. The merge is what makes the app's throughput counters non-zero, since Vector owns the Kafka client and the supervisor never sees a record on the bus transport. +The chart mounts the config and transform files, and an edit to either rolls the pods. Outside a chart, a transform change hot-reloads. [reference/configuration.md](reference/configuration.md) has the config layers and what the chart fixes. -**`/readyz` checks that Vector runs and its sink moves.** The gate is that the child was still alive 500 ms after spawn, which catches bad argv, an unreadable config and a pod in crash-recovery backoff, and that the sink has not held or refused records for `metrics.sink_stall_secs` without delivering any. A large program can take Vector longer than 500 ms to build, so the pod can report ready before Vector has finished starting. A pod with an unreachable broker and nothing held stays Ready. So does a bus config with no topics, which idles with no Vector running. +## Design considerations -## Build, chart and licence invariants +**Security.** Vector runs whatever a transform file declares, with read access to the Kafka credentials. So a transform file may declare `transforms` (never `lua`) and `enrichment_tables` alone, and a table must resolve inside `transforms.data_dir` and outside every credential path. Credentials reach Vector as `SECRET[...]` file references, never as values in its config. Vector runs without the `DFE_TRANSFORM_VECTOR_*` variables. -**The Vector version has one source.** The constant `VECTOR_VERSION` in `src/deployment.rs` is substituted into the Dockerfile's `ARG VECTOR_VERSION`, and both `VectorConfig::default()` and the contract's `default_config` read it for `vector.version`. `vector.version_check: strict` stops a pod starting when the config's version and the binary's disagree, so a deployment that sets `vector.version` itself has to move it with the image. dfe-infra's `versions.yaml` records the same release as `services.vector`, and dfe-fetcher's container Vector extractor runs that image. +**Reliability.** Delivery is at least once on both transports. A crashed Vector restarts on an exponential backoff capped at a minute, with no crash limit. A failed bridge leg stops the process, because its senders would otherwise retry forever against a live pod. -**`password` is a `String` on purpose, not a `SensitiveString`.** The config goes through a figment serialize-merge-deserialize round trip in `apply_figment_env()`, and `SensitiveString` serialises as `***REDACTED***`, which destroys the value in transit. The protection is elsewhere: the env var is masked in logs, `Debug` redacts the field, and no credential is written into the Vector config. The generated components name it `SECRET[.]` and Vector's `directory` secret backend reads it from files: a mounted secret at `sasl.secret_dir`, or owner-only files the assembler writes under `/.secrets` from a credential given as text. The released chart takes the second path: its Kafka Secret arrives in `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_*`, and `` sits on the `run` writable path. +**Operations.** `/readyz` means Vector survived 500 ms after spawn and its sink isn't stalled. A pod with an unreachable broker and nothing held stays Ready. Vector's exporter on 9598 is a loopback debug surface, and 9090 is the scrape target. -**Vector expands no `${VAR}`, and `vector validate` resolves no `SECRET[...]`.** Since 0.57 interpolation needs `--dangerously-allow-env-var-interpolation` or `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION`. The supervisor passes neither, but Vector inherits the pod's environment, so keep that variable unset. `validate()` rejects a `${...}` in a SASL credential. Validate would hand the broker the literal `SECRET[...]` reference, so the supervisor runs it with `--no-environment`, and a missing secret file fails when Vector starts, not at validate -- which is why the assembler checks `secret_dir` itself. +**Performance.** Vector's buffers have no memory cap of their own, so the pod's limit has to cover Vector's worst case. For VRL alone, [dfe-transform-vrl](https://github.com/hyperi-io/dfe-transform-vrl) runs in-process with bounded buffers. -Vector is redistributed unmodified under MPL-2.0, which has two obligations -- ship the licence text and tell recipients where the source is. Both are discharged inside the image from the release archive's `LICENSE`, `NOTICE` and `licenses` tree, so the attribution always matches the exact binary shipped. Do not replace that with a separate download. +## Why it's built this way -**No chart is committed.** At release, hyperi-ci emits the contract with `generate-artefacts` and assembles a thin chart from it on the scalo-service library chart at `release.helm.library`. [internals.md](internals.md#the-chart) lists what the contract fixes. +**A subprocess, not a library.** Vector has no stable API for embedding. Running the upstream release binary makes a Vector upgrade a version bump and an image rebuild, not a rebuild against every Vector release. -**The ScaledObject scales on CPU alone.** `contract()` sets `KafkaLagTrigger::disabled()`: consumer-group lag rises when a downstream stage breaks, and the library's lag trigger reads `config.kafka.*`, a block this app does not have, so the library will not render it. A deployment adds a trigger with the library's `keda.extraTriggers` value. +**One pin.** `VECTOR_VERSION` in `src/deployment.rs` feeds the Dockerfile, the default `vector.version` and the contract. `vector.version_check: strict` stops a pod whose config and binary disagree. -## Where the transports differ +**Polling for reload.** S3-backed mounts (s3fs, goofys, Mountpoint for S3) raise no filesystem events, so inotify would work on a laptop and silently stop reloading in production. -`source.transport` and `sink.transport` each pick `bus` or `direct`, and the transform you author is identical on both. What changes is who hands the record over: on `bus` it is a Kafka topic and Vector uses its `kafka` source and sink, on `direct` there is no broker, the rest of DFE speaks scalo's `Transport/Push` and Vector speaks `vector.Vector/PushEvents`, so the supervisor's bridge translates in both directions over loopback. `templates/bus.yaml` and `templates/direct.yaml` are the whole topology for each, and `vector validate` passes on both as they stand. In a DFE deployment the supervisor generates the `sources:` and `sinks:` blocks, so a whole template mounted as a transform file is refused. +**Wiring from labels.** The config engine reads each transform's `inputs` and joins `dfe_source` to `dfe_sink` through them, so a transform author never hand-wires the topology. -On direct a record enters at the intake listener, crosses Vector over loopback and leaves from the return listener, which sends a record over the next stage's size ceiling to the DLQ. +## Diagram map -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Direct transport bridge - accDescr: The previous stage pushes records to the intake listener, which hands them to Vector over loopback. Vector's output comes back to the return listener, which pushes it to the next stage, or sends a record over the size ceiling to the DLQ. - - prev["Previous
stage
"] - intake["Intake
listener
"] - vec["Vector"] - ret["Return
listener
"] - next["Next
stage
"] - dlq[("DLQ")] - - prev -->|Push| intake - intake --> vec - vec --> ret - ret -->|Push| next - ret -.->|too large| dlq - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class prev,next ext - class intake,vec,ret part - class dlq bus -``` +- [Figure 1. L3 Components -- dfe-transform-vector](architecture.md) + - [L4 Detail -- Config engine](internals.md) + - [L4 Detail -- Vector lifecycle](vector.md) + - [L4 Detail -- Direct transport bridge](direct-transport.md) -*Figure 3. L4 Detail -- Direct transport bridge. The dotted arrow is the dead letter path.* Up: [L3 Components -- dfe-transform-vector](../README.md). +## What's next -The intake listens on `source.listen`, Vector's `vector` source on `bridge.to_vector`, the return listener on `bridge.from_vector`, and the onward client dials `sink.endpoint`. A stage with one end on `direct` runs that half alone. +- [config.example.yaml](../config.example.yaml), the annotated config, and [config-schema.yaml](config-schema.yaml), every key +- [reference/configuration.md](reference/configuration.md), the config layers and the chart +- [librdkafka.md](librdkafka.md), the Kafka client settings, and [migration.md](migration.md), moving off the Vector chart +- [CONTRIBUTING.md Context](../CONTRIBUTING.md#context), where things live and the commands that test a change diff --git a/docs/direct-transport.md b/docs/direct-transport.md new file mode 100644 index 0000000..e00ce77 --- /dev/null +++ b/docs/direct-transport.md @@ -0,0 +1,144 @@ +# Direct transport + +The direct transport runs [dfe-transform-vector](architecture.md) with no broker. The supervisor's bridge translates between scalo's `Transport/Push`, which the rest of DFE speaks, and Vector's `vector.Vector/PushEvents`. + +| | | +|---|---| +| Repo | [hyperi-io/dfe-transform-vector](https://github.com/hyperi-io/dfe-transform-vector), `src/bridge.rs` | +| Image and chart | `ghcr.io/hyperi-io/dfe-transform-vector`. The chart opens the Push port when `config.source.transport` is `direct` or `grpc` | +| Config reference | `source.transport`, `source.listen`, `sink.transport`, `sink.endpoint` and `bridge` in [config.example.yaml](../config.example.yaml). [templates/direct.yaml](../templates/direct.yaml) is the whole topology | +| Endpoints | Push listener on `source.listen` (`0.0.0.0:6000`). Loopback legs on `bridge.to_vector` (`127.0.0.1:6100`) and `bridge.from_vector` (`127.0.0.1:6101`) | +| Metrics | `pipeline_dead_letters_dropped_total{reason}` on `:9090/metrics` | + +`source.transport` and `sink.transport` each pick `bus` or `direct`, so a stage can take Kafka in and push out, or the reverse. A stage with one end on `direct` runs that half of the bridge alone, and the transform you write is the same either way. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart TB + accTitle: L4 Detail -- Direct transport bridge + accDescr: The previous stage pushes records to the intake listener, which hands them to Vector over loopback. Vector's output comes back to the return listener, which pushes it to the next stage, or sends a record over the size ceiling to the DLQ. + + subgraph IN["Previous stage"] + direction LR + prev("Push client
the stage in front") + end + + subgraph BR["dfe-transform-vector"] + direction LR + intake("Intake listener
source.listen") + vec("Vector
vector source and sink") + ret("Return listener
bridge.from_vector") + intake --> vec --> ret + end + + subgraph OUT["Next stage"] + direction LR + next("Push listener
sink.endpoint") + end + + subgraph DL["Dead letters"] + direction LR + dlq[("DLQ
file spool")] + end + + IN -->|Push| BR + BR -->|Push| OUT + BR -.->|too large| DL + class prev,next ext + class intake,vec,ret part + class dlq bus + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style IN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style BR fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style OUT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style DL fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 1. L4 Detail -- Direct transport bridge. The dotted arrow is the dead-letter path.* Up: [L3 Components -- dfe-transform-vector](architecture.md). + +## How it works + +1. The previous stage pushes a batch to the intake listener. +2. The intake hands the records to Vector's `vector` source on `bridge.to_vector`, over loopback. +3. Vector runs the transforms, and its `vector` sink dials the return listener on `bridge.from_vector`. +4. The return listener pushes the records to `sink.endpoint`, or sends one over the next stage's size ceiling to the DLQ. +5. The intake answers its sender once the next hop has the records. + +## Interfaces + +In a DFE deployment the supervisor generates the `sources:` and `sinks:` blocks, so a whole template mounted as a transform file is refused. `templates/bus.yaml` and `templates/direct.yaml` both pass `vector validate` as they stand. Copy a template's `transforms:` block, not the file. + +## Scaling and failure + +Both listeners are built armed, so a push is held until the pipeline releases its records. A hop that rejects is retried until the hold runs out, then the push is answered `Unavailable` and its sender retries. The holds nest: 18 s at the intake, 16.5 s on the leg back from Vector and 15 s per send onward. An inner hop times out first, so the outer one can still answer its sender. + +Only the intake sheds under memory pressure. The return leg drains everything the intake holds, so shedding it would stall the stage until the holds expired into duplicates. Both listeners lease held bytes on the runtime's memory guard. + +A record over the next stage's size ceiling goes to the DLQ, or is dropped and counted with the DLQ off. A record Vector refused for good is released dropped and counted. A failed bridge leg stops the process with a non-zero exit, because its senders would otherwise retry forever against a live pod. + +## Key metrics + +| Metric | Healthy when | +|---|---| +| `pipeline_dead_letters_dropped_total{reason="too_large"}` | Flat. A rise is records over the next stage's ceiling with no DLQ | +| `pipeline_dead_letters_dropped_total{reason="rejected"}` | Flat. A rise is Vector refusing records for good | +| `records_delivered_total` | Tracks `records_received_total` | + +## Related + +- [Vector](vector.md), the process between the two legs +- [Config engine](internals.md), which writes the `vector` source and sink diff --git a/docs/internals.md b/docs/internals.md index aa526c5..8f82101 100644 --- a/docs/internals.md +++ b/docs/internals.md @@ -1,97 +1,16 @@ -# dfe-transform-vector internals - -Field-level depth under [architecture.md](architecture.md): how a config becomes the directory Vector reads, how reloads, crashes and shutdown run, what the ops port reports, and what the chart fixes. - -## The config - -The app's config layers in this order, each over the one before: - -1. The defaults, which [config-schema.yaml](config-schema.yaml) lists. -2. The config file: `--config`, else `config.yaml` or `config.yml` in the working directory. -3. `DFE_TRANSFORM_VECTOR_
__`, which reaches any key. A list takes brackets: `DFE_TRANSFORM_VECTOR_SOURCE__BROKERS='[a:9092,b:9092]'`. -4. The flat names, such as `DFE_TRANSFORM_VECTOR_SOURCE_BROKERS=a:9092,b:9092`, `_SINK_TOPIC`, `_TRANSFORMS_DIR`, `_VECTOR_BINARY` and the SASL pairs. - -The working directory's `.env` is read into the environment before the env layers, and a variable set before it wins. `--log-level`, `--log-format` and `--metrics-addr`, or `LOG_LEVEL`, `LOG_FORMAT` and `METRICS_ADDR`, outrank `logging.*` and `metrics.address`. Nothing reads `DFE_TRANSFORM_VECTOR_CONFIG`: name the file with `--config`, before any subcommand. - -Two flat names cover both endpoints. A `DFE_TRANSFORM_VECTOR_KAFKA_SECURITY_PROTOCOL` naming SSL, such as `SASL_SSL`, turns on `source.tls.enabled` and `sink.tls.enabled`, and any other value leaves them as they were. `DFE_TRANSFORM_VECTOR_KAFKA_SASL_MECHANISM` sets both SASL mechanisms, and the per-endpoint `_SOURCE_SASL_MECHANISM` and `_SINK_SASL_MECHANISM` beat it. `source.transport` and `sink.transport` take `kafka` for `bus` and `grpc` for `direct`, the names the other DFE stages use. - -The same file seeds scalo's cascade as well, so scalo's sections in it apply: `version_check`, `scaling.enabled`, `scaling.memory_gate_threshold` and `metrics.namespace`. `dfe_source` fills in `_land`, `_load` and the consumer group, `dfe-transform-vector-` or `dfe-transform-vector-` without a pipeline name, wherever the config leaves them at their defaults. - -```yaml -pipeline: - name: syslog-enrichment -source: - brokers: ["kafka:9092"] # or DFE_TRANSFORM_VECTOR_SOURCE_BROKERS - topics: - - raw_syslog_land - group_id: dfe-transform-vector-syslog-enrichment - sasl: - mechanism: scram_sha_512 - # A mounted Secret holding `username` and `password` files. Vector reads - # them through its directory secret backend, and expands no ${VAR}. - secret_dir: /var/run/secrets/dfe-kafka - tls: - enabled: true -sink: - brokers: ["kafka:9092"] # or DFE_TRANSFORM_VECTOR_SINK_BROKERS - topic: enriched_syslog_land - key_field: ".org_id" - compression: zstd - sasl: # each endpoint has its own SASL and TLS - mechanism: scram_sha_512 - secret_dir: /var/run/secrets/dfe-kafka - tls: - enabled: true -transforms: - dir: /etc/dfe-transform-vector-transforms # a file set's mountPath, see The chart -``` - -## The assembled directory +# Config engine -`dfe-transform-vector --config assemble ` writes what `run` hands Vector, without starting it. +The config engine is the part of [dfe-transform-vector](architecture.md) that turns one small YAML and your transform files into the directory Vector runs on. -| File | What it holds | +| | | |---|---| -| `00_global.yaml` | `timezone: UTC`, `data_dir`, the API switch, one `directory` secret backend per credential source | -| `00_source.yaml` | `dfe_source`: a `kafka` source on bus, carrying the librdkafka layers in [librdkafka.md](librdkafka.md), or a `vector` source on `bridge.to_vector` on direct | -| `50_NNN_` | Each transform file, in filename order, re-serialised with its wired `inputs` | -| `90_sink.yaml` | `dfe_sink`: on bus a `kafka` sink behind the `dfe_size_cap` route, plus `dfe_size_cap_dlq` and `dfe_dlq_sink` for the records it holds back, on direct a `vector` sink dialling `bridge.from_vector` | -| `99_observability.yaml` | `internal_metrics` into a `prometheus_exporter` on `metrics.vector_metrics_address` | -| `.secrets/` | Owner-only files for a SASL credential given as text | - -Credentials appear only as references: `SECRET[dfe_source_sasl.username]` for a `secret_dir`, `SECRET[dfe_credentials.source_sasl_username]` for a credential given as text. The assembler rejects a `secret_dir` that lacks its `username` or `password` file. - -`dfe_size_cap` is a `route` that sends a record whose encoded length is over the producer's `message.max.bytes` less 128 bytes for the key, headers and framing out of its `_unmatched` output, and the rest out of `within_cap` to the sink. `message.max.bytes` is scalo's pipeline-wide record ceiling of 16 MiB unless the central profile or `sink.librdkafka_options` sets it. librdkafka's default of 1,000,000 bytes would refuse records the rest of the chain accepts. - -While `dlq.enabled` is true, `dfe_size_cap_dlq` remaps each record from `_unmatched` into scalo's DLQ entry and `dfe_dlq_sink` writes it to `sink.dlq_topic` (`dfe_transform_dlq`, the shared topic dfe-loader loads). The entry is `truncated`: `original_bytes` is the record's length, `reason` names it and the ceiling, `destination` is `sink.topic`, and `_raw` is the first bytes as base64 with `_raw_encoding: base64`. The `_org_id`, `_source`, `_timestamp_received` and `_tags` values that fit are copied beside it, and any that do not are named in `header_dropped`. The head and the copied values share `dlq.max_payload_bytes`, 512 KiB by default, held to half of what `message.max.bytes` leaves. `dfe_dlq_sink` uses the main sink's brokers, credentials and librdkafka options, writes JSON with no key, and holds in a 500-event memory buffer that blocks when full, so an unreachable broker back-pressures the source rather than dropping the entry. With `dlq.enabled: false` there is no remap or second sink, and `dfe_size_cap` is a `filter` that drops the record. - -The wiring follows these rules: - -- The source is always `dfe_source` and the sink always `dfe_sink`. -- A transform with no or empty `inputs` reads `dfe_source`, and an input written `source` becomes `dfe_source` in the file Vector reads. -- An input names a component (`parse`) or one of its named outputs (`split.accepted`). A `route` or `exclusive_route` has named outputs only, so a bare reference to one is refused. -- Every output nothing reads feeds the sink, through `dfe_size_cap` on bus, each unread route and its `_unmatched` included. -- A transform file declares only `transforms` and `enrichment_tables`. A `sources:` key, a `lua` transform, or a table outside `transforms.data_dir` or on a credential path is refused. -- Every `inputs` reference must resolve, every transform must be reachable from `dfe_source`, and the graph must have no cycle. The error names the component: `component 'enrich' references undefined input 'parser'`. -- Once every enrichment table file proves readable, `vector validate --no-environment` over the directory is the last gate. - -## Hot reload - -The watcher polls the config file and the `.yaml` and `.yml` files of `transforms.dir`, or the listed `transforms.files`, every `reload.poll_interval_secs` (30 by default). A SIGHUP to the supervisor starts the same reload. Enrichment table files are not watched, and with `reload.enabled: false` nothing is, and a SIGHUP is logged and ignored. - -1. The config is re-read and validated. -2. A change to the app's keys outside the `transforms` block is refused. The watcher logs `non-transform config changed -- requires pod restart`, counts `rejected`, and keeps the running config. The change stays in the file, so every later transform change is refused until the pod restarts or the edit is reverted, which lets the next transform change reload. scalo's sections in the file are read once at startup and not compared: an edit to one needs a restart, and the watcher reloads the transforms and counts `success` without applying it. -3. A transform change is assembled into `.next`, its table files checked and `vector validate` run there. A failure is logged and counted `error`, and the directory Vector reads is untouched. -4. The two directories swap in one `renameat2(RENAME_EXCHANGE)`, so a Vector restarted mid-reload reads the old directory or the staged one. A filesystem that cannot swap directories gets the files copied in place, which is not atomic. -5. Vector gets a SIGHUP, and its exporter is read for the outcome: `vector_reloaded_total` counts `success`, a refusal in `vector_component_errors_total` counts `error` and swaps the old directory back, and no answer within 60 s counts `unconfirmed`. - -A failed attempt is retried on the same files after one poll interval, doubling to five minutes. `transform_vector_config_reloads_total{result}` counts `success`, `error`, `rejected` and `unconfirmed`. scalo's `config_reloads_total{result}` counts `success` and `error`, a rejection included. - -In a chart deployment an edit to the config or a file set rolls the pods anyway, because the chart hashes both into the pod template. +| Repo | [hyperi-io/dfe-transform-vector](https://github.com/hyperi-io/dfe-transform-vector), `src/config/` | +| Image and chart | `ghcr.io/hyperi-io/dfe-transform-vector`, and the thin chart each release assembles from `src/deployment.rs` | +| Config reference | [config.example.yaml](../config.example.yaml), [reference/configuration.md](reference/configuration.md) and [librdkafka.md](librdkafka.md) | +| Endpoints | None. It writes `vector.config_dir`, and stages a reload in `.next` | +| Metrics | `transform_vector_config_reloads_total{result}`, `config_reloads_total{result}`, `config_validation_errors_total` on `:9090/metrics` | -## The Vector subprocess - -Vector is `Running` once it is alive 500 ms after spawn, and an exit nobody asked for is a crash, followed by a restart after the backoff. +It runs at startup and again on every transform reload. Vector never reads a directory that failed `vector validate`. ```mermaid --- @@ -99,110 +18,136 @@ config: theme: base look: neo fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder themeVariables: fontSize: 15px lineColor: "#64748B" primaryTextColor: "#0F172A" clusterBorder: "#0072B2" edgeLabelBackground: "#FFFFFF" - primaryColor: "#FFFFFF" - primaryBorderColor: "#94A3B8" - tertiaryColor: "#F4F6F8" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} --- -stateDiagram-v2 - direction LR - accTitle: L4 Detail -- Vector lifecycle - accDescr: The supervisor initialises and validates, then starts Vector. Vector is running once it is alive 500 ms after spawn, and a reload returns to running. A crash from starting or running waits out the backoff and starts again. Shutdown ends the lifecycle. - state "Shutting down" as ShuttingDown - [*] --> Initialising - Initialising --> Validating - Validating --> Starting - Starting --> Running: settled - Running --> Reloading: change - Reloading --> Running: done - Starting --> Crashed: exit - Running --> Crashed: exit - Crashed --> Starting: backoff - Running --> ShuttingDown: SIGTERM - ShuttingDown --> [*] - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - class Crashed error +flowchart LR + accTitle: L4 Detail -- Config engine + accDescr: The config is loaded through its cascade and validated, and the secret files are written. The source is generated, the transform files are read and wired into a checked graph. The sink is generated and vector validate gates the directory before Vector is started or sent a SIGHUP. + + subgraph s1["1 Load"] + direction TB + load("Load cascade") --> check("Validate config") --> secrets("Write secret files") + end + subgraph s2["2 Wire"] + direction TB + source("Generate source") --> read("Read transforms") --> wire("Wire and check") + end + subgraph s3["3 Gate"] + direction TB + sink("Generate sink") --> gate("vector validate") + end + s1 --> s2 --> s3 + class load,check,secrets,source,read,wire,sink,gate part + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style s1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style s2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style s3 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 4. L4 Detail -- Vector lifecycle. Orange is the crash state.* Up: [L3 Components -- dfe-transform-vector](../README.md). - -Of these states `/readyz` can answer 200 in `Running` and `Reloading` alone, and the sink and idle rules under Health and metrics apply on top. - -Startup runs in this order: the config loads and validates, the runtime binds the ops listener, `/readyz` starts reporting the lifecycle, `vector --version` is checked against `vector.version`, the directory is assembled, `vector validate` passes, the bridge binds its listeners on direct, and Vector is spawned. A failure before the spawn stops the process with its error. - -Any exit Vector was not asked for is a crash, exit 0 included. `crashes_total` counts it, the lifecycle reads `Crashed`, and Vector is respawned after a backoff that starts at 1 s, doubles to 60 s and resets once a run has lasted five minutes. There is no crash limit, and `restarts_total` counts each respawn. One Vector runs per pod, and replicas scale out. - -On SIGTERM or SIGINT the intake stops, Vector gets a SIGTERM and 55 s to exit before a SIGKILL, the bridge drains for up to 15 s, and the process exits 0. A failed bridge leg runs the same shutdown and exits non-zero. +*Figure 1. L4 Detail -- Config engine. Read each stage top to bottom, then right.* Up: [L3 Components -- dfe-transform-vector](architecture.md). -On a version mismatch `strict` and `warn` emit a security data-quality event, and no metric records it. +## How it works -| `vector.version_check` | On a mismatch | -|---|---| -| `strict` (the default) | Logs it and exits | -| `warn` | Logs a warning and continues | -| `disabled` | Runs no check, not even `vector --version` | - -## Health and metrics - -| Path | Answers | -|---|---| -| `/livez` | 200 whenever the ops listener answers | -| `/readyz` | 200 or 503, below | -| `/metrics` | The supervisor's metrics and every merged `vector_*` | -| `/scaling/pressure` | scalo's scaling pressure | +`dfe-transform-vector --config assemble ` writes what `run` hands Vector, without starting it. -All of them are on `metrics.address`, 9090 by default. `/readyz` answers 503 `{"status":"starting"}` until the runtime hands the service over. A bus config with no topics then idles: 200, `pipeline_idle` 1, no Vector, until the config file names a topic. Once running, it answers 503 unless the lifecycle is `Running` or `Reloading`, and when the sink has held or refused records for `metrics.sink_stall_secs` (60 by default) without delivering any. On direct the bridge's gRPC transports report there as well. `Reloading` is entered only from `Running`, so a change landing mid crash-backoff cannot turn it to 200. +The wiring follows these rules: -Metric names are bare unless `metrics.namespace` is set, and `dfe-transform-vector metrics-manifest` prints the catalogue. The supervisor's metrics: +- The source is always `dfe_source` and the sink always `dfe_sink`. +- A transform with no or empty `inputs` reads `dfe_source`, and an input written `source` becomes `dfe_source` in the file Vector reads. +- An input names a component (`parse`) or one of its named outputs (`split.accepted`). A `route` or `exclusive_route` has named outputs only, so a bare reference to one is refused. +- Every output nothing reads feeds the sink, through `dfe_size_cap` on bus, each unread route and its `_unmatched` included. +- A transform file declares only `transforms` and `enrichment_tables`. A `sources:` key, a `lua` transform, or a table outside `transforms.data_dir` or on a credential path is refused. +- Every `inputs` reference must resolve, every transform must be reachable from `dfe_source`, and the graph must have no cycle. The error names the component: `component 'enrich' references undefined input 'parser'`. +- Once every enrichment table file proves readable, `vector validate --no-environment` over the directory is the last gate. -- `transform_vector_lifecycle_state{state}`, 1 on the current state, and `pipeline_ready`. -- `crashes_total`, `restarts_total` and `uptime_seconds`, since the service started. -- `transform_vector_config_reloads_total{result}`, `config_reloads_total{result}` and `config_validation_errors_total`. -- `transform_vector_scrape_failures_total` and `transform_vector_sink_errors_total`. -- The record counters below, derived from Vector's component counters, since Vector and not the supervisor owns the Kafka client. +## Interfaces -| Supervisor metric | Vector counter it reads | +| File | What it holds | |---|---| -| `records_received_total` | `vector_component_received_events_total{component_id="dfe_source"}` | -| `records_processed_total`, `records_delivered_total` | `vector_component_sent_events_total{component_id="dfe_sink"}` | -| `records_error_total` | `vector_component_discarded_events_total` on `dfe_sink`, `dfe_size_cap` and `dfe_dlq_sink` | -| `transform_vector_sink_errors_total` | `vector_component_errors_total{component_id="dfe_sink"}` | -| `pipeline_dead_letters_dropped_total{reason="too_large"}` | the growth of `vector_component_discarded_events_total` on `dfe_size_cap` and on `dfe_dlq_sink` | -| Dead letters held by the DLQ topic, with no supervisor counter of their own | `vector_component_sent_events_total{component_id="dfe_dlq_sink"}`, merged with the other `vector_*` series | -| `pipeline_dead_letters_dropped_total{reason="rejected"}` | the growth of `vector_component_discarded_events_total` on `dfe_sink` | - -The scrape reads Vector's exporter on scalo's metrics interval and re-registers every sample. A failed scrape is not fatal: it counts `transform_vector_scrape_failures_total`, warns at most once every five minutes, and leaves readiness as it was. A response over 4 MiB is a failed scrape, not a partial merge. A counter is set with `absolute` and carried across Vector restarts: a total that falls is a restarted Vector, and the old count becomes a base. A histogram arrives pre-aggregated, so each `_bucket` registers as a counter with its `le` label. A gauge Vector stops reporting is zeroed after `metrics.vector_metrics_expiry_ticks` scrapes, a counter is forgotten after 240, and at most 50,000 series merge. - -dfe-infra's values set a CPU target and, on its single, scale and mesh profiles, add a pressure trigger through `keda.extraTriggers`. The `scaling_pressure` this app publishes stays 0 when Vector runs, since nothing feeds it a component or a memory reading, and `spawn_circuit_gate_task` pins it to 0 when Vector is down. +| `00_global.yaml` | `timezone: UTC`, `data_dir`, the API switch, one `directory` secret backend per credential source | +| `00_source.yaml` | `dfe_source`: a `kafka` source on bus, carrying the librdkafka layers in [librdkafka.md](librdkafka.md), or a `vector` source on `bridge.to_vector` on direct | +| `50_NNN_` | Each transform file, in filename order, re-serialised with its wired `inputs` | +| `90_sink.yaml` | `dfe_sink`: on bus a `kafka` sink behind the `dfe_size_cap` route, plus `dfe_size_cap_dlq` and `dfe_dlq_sink` for the records it holds back, on direct a `vector` sink dialling `bridge.from_vector` | +| `99_observability.yaml` | `internal_metrics` into a `prometheus_exporter` on `metrics.vector_metrics_address` | +| `.secrets/` | Owner-only files for a SASL credential given as text | -## Delivery and dead letters +Credentials appear only as references: `SECRET[dfe_source_sasl.username]` for a `secret_dir`, `SECRET[dfe_credentials.source_sasl_username]` for a credential given as text. The assembler rejects a `secret_dir` that lacks its `username` or `password` file. -With `source.acknowledgements.enabled`, the default, delivery is at-least-once on both transports. On bus the sink's acknowledgements hold Vector's source offsets until delivery, so a crash redelivers from the last committed offset. On direct a push is answered once the next hop has it. With acknowledgements off, both answer at receipt. +`dfe_size_cap` is a `route` that sends a record whose encoded length is over the producer's `message.max.bytes` less 128 bytes for the key, headers and framing out of its `_unmatched` output, and the rest out of `within_cap` to the sink. `message.max.bytes` is scalo's pipeline-wide record ceiling of 16 MiB unless the central profile or `sink.librdkafka_options` sets it. librdkafka's default of 1,000,000 bytes would refuse records the rest of the chain accepts. -On direct a record over the next stage's message-size ceiling goes to scalo's file DLQ, NDJSON under `/var/spool/scalo/dlq` on the contract's `dlq` writable path. With `dlq.enabled: false` it is dropped and counted in `pipeline_dead_letters_dropped_total{reason="too_large"}`. A record Vector rejects for good is dropped and counted under `reason="rejected"`, whatever `dlq` says. On bus a record over the producer's ceiling goes to `sink.dlq_topic` (see [The assembled directory](#the-assembled-directory)), counted in `vector_component_sent_events_total{component_id="dfe_dlq_sink"}` and in no drop counter. It counts under `too_large` only when the DLQ sink rejects it for good, or with `dlq.enabled: false`, and the Kafka sink's rejections count under `rejected`. When no enabled DLQ backend can start, such as a file DLQ with no writable directory off the image, a direct run logs `DLQ enabled but no backend started` and leaves dead letters unacknowledged at their source. Point `dlq.file.path` at a directory you can write. +While `dlq.enabled` is true, `dfe_size_cap_dlq` remaps each record from `_unmatched` into scalo's DLQ entry and `dfe_dlq_sink` writes it to `sink.dlq_topic` (`dfe_transform_dlq`, the shared topic dfe-loader loads). The entry is `truncated`: `original_bytes` is the record's length, `reason` names it and the ceiling, `destination` is `sink.topic`, and `_raw` is the first bytes as base64 with `_raw_encoding: base64`. The `_org_id`, `_source`, `_timestamp_received` and `_tags` values that fit are copied beside it, and any that do not are named in `header_dropped`. The head and the copied values share `dlq.max_payload_bytes`, 512 KiB by default, held to half of what `message.max.bytes` leaves. `dfe_dlq_sink` uses the main sink's brokers, credentials and librdkafka options, writes JSON with no key, and holds in a 500-event memory buffer that blocks when full, so an unreachable broker back-pressures the source rather than dropping the entry. With `dlq.enabled: false` there is no remap or second sink, and `dfe_size_cap` is a `filter` that drops the record. -On bus Vector holds the records, bounded by its `buffer` settings and the container's memory limit. +## Scaling and failure -`vector.data_dir` sits on the contract's `data` writable path, an emptyDir unless `writablePaths.data.persistence.enabled` gives it a claim, so a disk buffer outlives a Vector restart but not a pod. The supervisor logs `DISK BUFFER HAS NO DURABLE STORAGE` at startup when a disk buffer sits on ephemeral storage. +The watcher polls the config file and the `.yaml` and `.yml` files of `transforms.dir`, or the listed `transforms.files`, every `reload.poll_interval_secs` (30 by default). A SIGHUP to the supervisor starts the same reload. Enrichment table files are not watched, and with `reload.enabled: false` nothing is, and a SIGHUP is logged and ignored. -## Upgrading Vector +1. The config is re-read and validated. +2. A change to the app's keys outside the `transforms` block is refused. The watcher logs `non-transform config changed -- requires pod restart`, counts `rejected`, and keeps the running config. The change stays in the file, so every later transform change is refused until the pod restarts or the edit is reverted, which lets the next transform change reload. scalo's sections in the file are read once at startup and not compared: an edit to one needs a restart, and the watcher reloads the transforms and counts `success` without applying it. +3. A transform change is assembled into `.next`, its table files checked and `vector validate` run there. A failure is logged and counted `error`, and the directory Vector reads is untouched. +4. The two directories swap in one `renameat2(RENAME_EXCHANGE)`, so a Vector restarted mid-reload reads the old directory or the staged one. A filesystem that cannot swap directories gets the files copied in place, which is not atomic. +5. Vector gets a SIGHUP, and its exporter is read for the outcome: `vector_reloaded_total` counts `success`, a refusal in `vector_component_errors_total` counts `error` and swaps the old directory back, and no answer within 60 s counts `unconfirmed`. -`VECTOR_VERSION` and the two archive checksums in `src/deployment.rs` are the only pin. The Dockerfile, the default `vector.version`, the contract's `default_config` and `scripts/fetch-vector.sh` all read the constant. +A failed attempt is retried on the same files after one poll interval, doubling to five minutes. `transform_vector_config_reloads_total{result}` counts `success`, `error`, `rejected` and `unconfirmed`. scalo's `config_reloads_total{result}` counts `success` and `error`, a rejection included. -1. Change `VECTOR_VERSION`, and `VECTOR_SHA256_AMD64` and `VECTOR_SHA256_ARM64` from Vector's published `SHA256SUMS` for that release. -2. Regenerate the Dockerfile: `cargo run --bin dfe-transform-vector -- emit-dockerfile > Dockerfile`. `checked_in_dockerfile_matches_emit_dockerfile` fails when it drifts. -3. Move `services.vector` in dfe-infra's `versions.yaml` to the same release, tagged `-alpine`. dfe-fetcher's container Vector extractor runs that image. -4. Release this repo. The full test tier, nightly and before a release, runs the `vector_validate` cases against the bumped Vector. +In a chart deployment an edit to the config or a file set rolls the pods anyway, because the chart hashes both into the pod template. -## The chart +## Key metrics -No chart is committed. At release hyperi-ci runs `generate-artefacts` for the deployment contract and assembles a thin chart on the scalo-service library chart, at the version `release.helm.library` names in `.hyperi-ci.yaml`. The library renders the Deployment, Service, ConfigMap, ServiceAccount and KEDA ScaledObject, and its README documents the values. To see the chart a release would ship, build the binary and run `hyperi-ci chart assemble --binary target/debug/dfe-transform-vector --image ghcr.io/hyperi-io/dfe-transform-vector:@sha256:`, which prints the directory it wrote. +| Metric | Healthy when | +|---|---| +| `transform_vector_config_reloads_total{result}` | `success` after each transform change. `rejected` means a restart-only edit is waiting | +| `config_validation_errors_total` | Flat | -`dfe-transform-vector emit-contract` prints what the contract fixes: the probes on the metrics port and their startup budget, the Push port when `config.source.transport` is `direct` or `grpc`, the Kafka Secret in `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_{USERNAME,PASSWORD}` and `DFE_TRANSFORM_VECTOR_KAFKA_SASL_MECHANISM`, the writable paths `data`, `run` and `dlq` beside the library's `/tmp` under a read-only root, the requests and limits, the grace period, and KEDA on CPU with no Kafka lag trigger. +## Related -The mounted config is the chart's `config` with `configOverrides` merged over it. The contract's `default_config` is never merged in, so the app's defaults apply to anything a deployment leaves unset, `vector.version_check: strict` included. The ConfigMap mounts over the config file's whole directory unless `configMount.subPath` is set, which hides the image's `/etc/dfe-transform-vector/transforms`: deliver transform files with a `fileSets` entry and point `transforms.dir` at its `mountPath`. +- [Vector](vector.md), the process the directory is handed to +- [reference/configuration.md](reference/configuration.md), the config layers and the chart +- [templates/bus.yaml](../templates/bus.yaml) and [templates/direct.yaml](../templates/direct.yaml), whole runnable topologies diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md new file mode 100644 index 0000000..a8bfdd4 --- /dev/null +++ b/docs/reference/configuration.md @@ -0,0 +1,55 @@ +# Configuration + +How dfe-transform-vector reads its config, and what the released chart fixes. Every key is in [config-schema.yaml](../config-schema.yaml). + +## Config layers + +The app's config layers in this order, each over the one before: + +1. The defaults, which [config-schema.yaml](../config-schema.yaml) lists. +2. The config file: `--config`, else `config.yaml` or `config.yml` in the working directory. +3. `DFE_TRANSFORM_VECTOR_
__`, which reaches any key. A list takes brackets: `DFE_TRANSFORM_VECTOR_SOURCE__BROKERS='[a:9092,b:9092]'`. +4. The flat names, such as `DFE_TRANSFORM_VECTOR_SOURCE_BROKERS=a:9092,b:9092`, `_SINK_TOPIC`, `_TRANSFORMS_DIR`, `_VECTOR_BINARY` and the SASL pairs. + +The working directory's `.env` is read into the environment before the env layers, and a variable set before it wins. `--log-level`, `--log-format` and `--metrics-addr`, or `LOG_LEVEL`, `LOG_FORMAT` and `METRICS_ADDR`, outrank `logging.*` and `metrics.address`. Nothing reads `DFE_TRANSFORM_VECTOR_CONFIG`: name the file with `--config`, before any subcommand. + +Two flat names cover both endpoints. A `DFE_TRANSFORM_VECTOR_KAFKA_SECURITY_PROTOCOL` naming SSL, such as `SASL_SSL`, turns on `source.tls.enabled` and `sink.tls.enabled`, and any other value leaves them as they were. `DFE_TRANSFORM_VECTOR_KAFKA_SASL_MECHANISM` sets both SASL mechanisms, and the per-endpoint `_SOURCE_SASL_MECHANISM` and `_SINK_SASL_MECHANISM` beat it. `source.transport` and `sink.transport` take `kafka` for `bus` and `grpc` for `direct`, the names the other DFE stages use. + +The same file seeds scalo's cascade as well, so scalo's sections in it apply: `version_check`, `scaling.enabled`, `scaling.memory_gate_threshold` and `metrics.namespace`. `dfe_source` fills in `_land`, `_load` and the consumer group, `dfe-transform-vector-` or `dfe-transform-vector-` without a pipeline name, wherever the config leaves them at their defaults. + +```yaml +pipeline: + name: syslog-enrichment +source: + brokers: ["kafka:9092"] # or DFE_TRANSFORM_VECTOR_SOURCE_BROKERS + topics: + - raw_syslog_land + group_id: dfe-transform-vector-syslog-enrichment + sasl: + mechanism: scram_sha_512 + # A mounted Secret holding `username` and `password` files. Vector reads + # them through its directory secret backend, and expands no ${VAR}. + secret_dir: /var/run/secrets/dfe-kafka + tls: + enabled: true +sink: + brokers: ["kafka:9092"] # or DFE_TRANSFORM_VECTOR_SINK_BROKERS + topic: enriched_syslog_land + key_field: ".org_id" + compression: zstd + sasl: # each endpoint has its own SASL and TLS + mechanism: scram_sha_512 + secret_dir: /var/run/secrets/dfe-kafka + tls: + enabled: true +transforms: + dir: /etc/dfe-transform-vector-transforms # a file set's mountPath, see The chart +``` + +## The chart + +No chart is committed. At release hyperi-ci runs `generate-artefacts` for the deployment contract and assembles a thin chart on the scalo-service library chart, at the version `release.helm.library` names in `.hyperi-ci.yaml`. The library renders the Deployment, Service, ConfigMap, ServiceAccount and KEDA ScaledObject, and its README documents the values. To see the chart a release would ship, build the binary and run `hyperi-ci chart assemble --binary target/debug/dfe-transform-vector --image ghcr.io/hyperi-io/dfe-transform-vector:@sha256:`, which prints the directory it wrote. + +`dfe-transform-vector emit-contract` prints what the contract fixes: the probes on the metrics port and their startup budget, the Push port when `config.source.transport` is `direct` or `grpc`, the Kafka Secret in `DFE_TRANSFORM_VECTOR_{SOURCE,SINK}_SASL_{USERNAME,PASSWORD}` and `DFE_TRANSFORM_VECTOR_KAFKA_SASL_MECHANISM`, the writable paths `data`, `run` and `dlq` beside the library's `/tmp` under a read-only root, the requests and limits, the grace period, and KEDA on CPU with no Kafka lag trigger. + +The mounted config is the chart's `config` with `configOverrides` merged over it. The contract's `default_config` is never merged in, so the app's defaults apply to anything a deployment leaves unset, `vector.version_check: strict` included. The ConfigMap mounts over the config file's whole directory unless `configMount.subPath` is set, which hides the image's `/etc/dfe-transform-vector/transforms`: deliver transform files with a `fileSets` entry and point `transforms.dir` at its `mountPath`. diff --git a/docs/vector.md b/docs/vector.md new file mode 100644 index 0000000..e6f4d79 --- /dev/null +++ b/docs/vector.md @@ -0,0 +1,120 @@ +# Vector + +Vector is the child process in [dfe-transform-vector](architecture.md) that does the transform work. The supervisor spawns it, restarts it and merges its metrics. + +| | | +|---|---| +| Repo | [hyperi-io/dfe-transform-vector](https://github.com/hyperi-io/dfe-transform-vector), `src/vector/`, `src/health.rs` and `src/metrics/` | +| Image and chart | `ghcr.io/hyperi-io/dfe-transform-vector`, which bakes the upstream Vector release `VECTOR_VERSION` pins in `src/deployment.rs` | +| Config reference | `vector`, `metrics` and `source.acknowledgements` in [config.example.yaml](../config.example.yaml) | +| Endpoints | `/livez`, `/readyz`, `/metrics` and `/scaling/pressure` on 9090. Vector's exporter on `127.0.0.1:9598`, loopback only | +| Metrics | `transform_vector_lifecycle_state`, `crashes_total`, `restarts_total` and every merged `vector_*` on `:9090/metrics` | + +Vector is `Running` once it is alive 500 ms after spawn. An exit nobody asked for is a crash, followed by a restart after the backoff. + +```mermaid +--- +config: + theme: base + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + actorBkg: "#FFFFFF" + actorBorder: "#94A3B8" + actorTextColor: "#0F172A" + actorLineColor: "#64748B" + signalColor: "#64748B" + signalTextColor: "#0F172A" + themeCSS: >- + .messageText{paint-order:stroke;stroke:#FFFFFF;stroke-width:3px;stroke-linejoin:round} +--- +stateDiagram-v2 + direction LR + accTitle: L4 Detail -- Vector lifecycle + accDescr: The supervisor initialises and validates, then starts Vector. Vector is running once it is alive 500 ms after spawn, and a reload returns to running. A crash from starting or running waits out the backoff and starts again. Shutdown ends the lifecycle. + state "Shutting down" as ShuttingDown + [*] --> Initialising + Initialising --> Validating + Validating --> Starting + Starting --> Running: settled + Running --> Reloading: change + Reloading --> Running: done + Starting --> Crashed: exit + Running --> Crashed: exit + Crashed --> Starting: backoff + Running --> ShuttingDown: SIGTERM + ShuttingDown --> [*] +``` + +*Figure 1. L4 Detail -- Vector lifecycle.* Up: [L3 Components -- dfe-transform-vector](architecture.md). + +## How it works + +Of these states `/readyz` can answer 200 in `Running` and `Reloading` alone, and the sink and idle rules under Interfaces apply on top. + +Startup runs in this order: the config loads and validates, the runtime binds the ops listener, `/readyz` starts reporting the lifecycle, `vector --version` is checked against `vector.version`, the directory is assembled, `vector validate` passes, the bridge binds its listeners on direct, and Vector is spawned. A failure before the spawn stops the process with its error. + +Any exit Vector was not asked for is a crash, exit 0 included. `crashes_total` counts it, the lifecycle reads `Crashed`, and Vector is respawned after a backoff that starts at 1 s, doubles to 60 s and resets once a run has lasted five minutes. There is no crash limit, and `restarts_total` counts each respawn. One Vector runs per pod, and replicas scale out. + +On SIGTERM or SIGINT the intake stops, Vector gets a SIGTERM and 55 s to exit before a SIGKILL, the bridge drains for up to 15 s, and the process exits 0. A failed bridge leg runs the same shutdown and exits non-zero. + +On a version mismatch `strict` and `warn` emit a security data-quality event, and no metric records it. + +| `vector.version_check` | On a mismatch | +|---|---| +| `strict` (the default) | Logs it and exits | +| `warn` | Logs a warning and continues | +| `disabled` | Runs no check, not even `vector --version` | + +## Interfaces + +| Path | Answers | +|---|---| +| `/livez` | 200 whenever the ops listener answers | +| `/readyz` | 200 or 503, below | +| `/metrics` | The supervisor's metrics and every merged `vector_*` | +| `/scaling/pressure` | scalo's scaling pressure | + +All of them are on `metrics.address`, 9090 by default. `/readyz` answers 503 `{"status":"starting"}` until the runtime hands the service over. A bus config with no topics then idles: 200, `pipeline_idle` 1, no Vector, until the config file names a topic. Once running, it answers 503 unless the lifecycle is `Running` or `Reloading`, and when the sink has held or refused records for `metrics.sink_stall_secs` (60 by default) without delivering any. On direct the bridge's gRPC transports report there as well. `Reloading` is entered only from `Running`, so a change landing mid crash-backoff cannot turn it to 200. + +## Scaling and failure + +With `source.acknowledgements.enabled`, the default, delivery is at-least-once on both transports. On bus the sink's acknowledgements hold Vector's source offsets until delivery, so a crash redelivers from the last committed offset. On direct a push is answered once the next hop has it. With acknowledgements off, both answer at receipt. + +On direct a record over the next stage's message-size ceiling goes to scalo's file DLQ, NDJSON under `/var/spool/scalo/dlq` on the contract's `dlq` writable path. With `dlq.enabled: false` it is dropped and counted in `pipeline_dead_letters_dropped_total{reason="too_large"}`. A record Vector rejects for good is dropped and counted under `reason="rejected"`, whatever `dlq` says. On bus a record over the producer's ceiling goes to `sink.dlq_topic` (see [Config engine](internals.md#interfaces)), counted in `vector_component_sent_events_total{component_id="dfe_dlq_sink"}` and in no drop counter. It counts under `too_large` only when the DLQ sink rejects it for good, or with `dlq.enabled: false`, and the Kafka sink's rejections count under `rejected`. When no enabled DLQ backend can start, such as a file DLQ with no writable directory off the image, a direct run logs `DLQ enabled but no backend started` and leaves dead letters unacknowledged at their source. Point `dlq.file.path` at a directory you can write. + +On bus Vector holds the records, bounded by its `buffer` settings and the container's memory limit. + +`vector.data_dir` sits on the contract's `data` writable path, an emptyDir unless `writablePaths.data.persistence.enabled` gives it a claim, so a disk buffer outlives a Vector restart but not a pod. The supervisor logs `DISK BUFFER HAS NO DURABLE STORAGE` at startup when a disk buffer sits on ephemeral storage. + +## Key metrics + +Metric names are bare unless `metrics.namespace` is set, and `dfe-transform-vector metrics-manifest` prints the catalogue. The supervisor's metrics: + +- `transform_vector_lifecycle_state{state}`, 1 on the current state, and `pipeline_ready`. +- `crashes_total`, `restarts_total` and `uptime_seconds`, since the service started. +- `transform_vector_config_reloads_total{result}`, `config_reloads_total{result}` and `config_validation_errors_total`. +- `transform_vector_scrape_failures_total` and `transform_vector_sink_errors_total`. +- The record counters below, derived from Vector's component counters, since Vector and not the supervisor owns the Kafka client. + +| Supervisor metric | Vector counter it reads | +|---|---| +| `records_received_total` | `vector_component_received_events_total{component_id="dfe_source"}` | +| `records_processed_total`, `records_delivered_total` | `vector_component_sent_events_total{component_id="dfe_sink"}` | +| `records_error_total` | `vector_component_discarded_events_total` on `dfe_sink`, `dfe_size_cap` and `dfe_dlq_sink` | +| `transform_vector_sink_errors_total` | `vector_component_errors_total{component_id="dfe_sink"}` | +| `pipeline_dead_letters_dropped_total{reason="too_large"}` | the growth of `vector_component_discarded_events_total` on `dfe_size_cap` and on `dfe_dlq_sink` | +| Dead letters held by the DLQ topic, with no supervisor counter of their own | `vector_component_sent_events_total{component_id="dfe_dlq_sink"}`, merged with the other `vector_*` series | +| `pipeline_dead_letters_dropped_total{reason="rejected"}` | the growth of `vector_component_discarded_events_total` on `dfe_sink` | + +The scrape reads Vector's exporter on scalo's metrics interval and re-registers every sample. A failed scrape is not fatal: it counts `transform_vector_scrape_failures_total`, warns at most once every five minutes, and leaves readiness as it was. A response over 4 MiB is a failed scrape, not a partial merge. A counter is set with `absolute` and carried across Vector restarts: a total that falls is a restarted Vector, and the old count becomes a base. A histogram arrives pre-aggregated, so each `_bucket` registers as a counter with its `le` label. A gauge Vector stops reporting is zeroed after `metrics.vector_metrics_expiry_ticks` scrapes, a counter is forgotten after 240, and at most 50,000 series merge. + +dfe-infra's values set a CPU target and, on its single, scale and mesh profiles, add a pressure trigger through `keda.extraTriggers`. The `scaling_pressure` this app publishes stays 0 when Vector runs, since nothing feeds it a component or a memory reading, and `spawn_circuit_gate_task` pins it to 0 when Vector is down. + +## Related + +- [Config engine](internals.md), what Vector is started on +- [Direct transport](direct-transport.md), the bridge on the direct path +- [librdkafka.md](librdkafka.md), the Kafka client Vector runs on the bus