Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ A Rust supervisor that runs Vector as a child process so a Vector pipeline behav
| `src/deployment.rs`, `src/vector-layer.dockerfile` | `contract()`, the source the Dockerfile and the released chart are generated from, and the two-binary image override spliced onto scalo's generated Dockerfile |
| `templates/bus.yaml`, `templates/direct.yaml` | Whole runnable Vector topologies, one per transport, commented per field |
| `pipelines/filebeat/` | The bundled filebeat pipeline. The image does not ship it |
| `docs/` | `architecture.md` for why the service is built this way and the invariants, `internals.md` for field-by-field depth, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` |
| `docs/` | `architecture.md` for the parts and why, a page per part (`internals.md` the config engine, `vector.md`, `direct-transport.md`), `reference/configuration.md`, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` |
| `tests/` | `integration`, `e2e`, `smoke`, and `TESTING.md` for how the broker and Vector binary are resolved |
| `scripts/fetch-vector.sh`, `scripts/pgo-workload.sh` | Downloads the pinned Vector for tests, and drives the PGO/BOLT workload |

Expand All @@ -142,6 +142,10 @@ What a passing run does NOT mean:
- `e2e::filebeat_kafka` runs by default and needs a dfe-transform-vrl checkout, for its cases and the bundled pipeline: beside this repo, or at `DFE_TRANSFORM_VRL_DIR`. Without one it skips locally and fails under CI, which checks the repo out beside this one.
- A docs-only push runs no CI at all. `paths-ignore` for `docs/**` and `**.md` is on the `push` trigger but not on `pull_request`, so the PR is the only place a docs change is checked. A silent push is not a pass.

To upgrade Vector, change `VECTOR_VERSION` and `VECTOR_SHA256_AMD64` and `VECTOR_SHA256_ARM64` in `src/deployment.rs` from Vector's published `SHA256SUMS`, then regenerate the Dockerfile. The Dockerfile, the default `vector.version`, the contract's `default_config` and `scripts/fetch-vector.sh` all read the constant. Move `services.vector` in dfe-infra's `versions.yaml` to the same release, tagged `<version>-alpine`, because dfe-fetcher's container Vector extractor runs that image. The full test tier runs the `vector_validate` cases against the new Vector.

The image has two binaries, and scalo's contract builds only the supervisor. `src/deployment.rs::emit_dockerfile()` splices `src/vector-layer.dockerfile` in before the generated `USER` line, with the version and checksums substituted. The fragment is a Dockerfile, not a Rust string, so hadolint and shellcheck read it.

### What tends to bite

| Don't | Do | Why |
Expand All @@ -156,6 +160,7 @@ What a passing run does NOT mean:
| Assume a config key you added is wired because it parses and validates | Test that it changes behaviour, and keep `every_committed_config_file_loads_and_validates` passing over each committed config file | A key can parse, validate and reach nothing, so a deployment that moves a port on it keeps the old port and loses every probe |
| Set `enable.auto.commit: false` here, as the shared DFE consumer baseline does | Leave auto-commit on | Vector's kafka source only stores offsets and leaves librdkafka's commit timer to flush them, and that timer is armed only when auto-commit is on, so with it off the stored offsets reach the broker only on a rebalance or a clean shutdown |
| Read `/readyz` as "carrying traffic" | Read it as "Vector survived 500 ms after spawn, and the sink is not stalled" | A pod with an unreachable broker and nothing held stays Ready with no record moving |
| Replace the Vector licence files in the image with a separate download | Keep shipping `LICENSE`, `NOTICE` and the `licenses` tree from the release archive | Vector is redistributed unmodified under MPL-2.0, which needs the licence text and the source location shipped. Taking them from the archive keeps the attribution matched to the exact binary |
| Swap the reload poller for inotify | Keep polling | S3-backed mounts -- s3fs, goofys, Mountpoint for S3 -- generate no filesystem notification events, so inotify works on every laptop and stops reloading in production with nothing logged |
| Add or rename a credential field without a schema marker | Schema it as `scalo::SensitiveString` with `#[schemars(with = ...)]`, as `SaslConfig::password` is | That puts `x-scalo-secret` and `writeOnly` on the field in the emitted config-schema, which is what tells a console to mask it. Without it only dfe-engine's name-based fallback can mask it, so a name that fallback does not read as a credential returns an operator's Kafka password in the clear |

Expand Down
86 changes: 57 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

The config engine and the process manager run Vector inside the pod, the bridge moves records on the direct transport, and the metrics merge serves Vector's metrics beside the supervisor's.

```mermaid

Check warning on line 19 in README.md

View workflow job for this annotation

GitHub Actions / ci / Quality

hyperi-ci diagram-check: R9

frontmatter configuration
---
config:
theme: base
Expand Down Expand Up @@ -64,33 +64,55 @@
---
flowchart TB
accTitle: L3 Components -- dfe-transform-vector
accDescr: The previous stage sends records into the pod and the config and transform files configure it. Inside, the config engine writes the Vector config that the process manager starts Vector on, and on the direct transport the bridge intake hands records to Vector and the bridge return takes them back. Vector's metrics are merged into the supervisor's and served on the ops port. Records leave for the next stage and metrics go to the collector.
accDescr: The previous stage sends records into the pod and the config and transform files configure it. Inside, the config engine writes the Vector config that the process manager starts Vector on. On the direct transport the bridge intake hands records to Vector and the bridge return takes them back. Vector's metrics are merged into the supervisor's. Records leave for the next stage and metrics go to the collector.

prev["<b>Previous stage</b>"]
files[("<b>Config files</b>")]
subgraph PREV["Previous stage"]
direction LR
prev("<b>Records in</b><br/><span>*_land topic or Push</span>")
end

subgraph pod["dfe-transform-vector"]
subgraph CFGIN["Config"]
direction LR
cfg[["<b>Config engine</b>"]]
proc["<b>Process manager</b>"]
intake[["<b>Bridge intake</b>"]]
vec[["<b>Vector</b>"]]
ret[["<b>Bridge return</b>"]]
merge["<b>Metrics merge</b>"]
cfg --> proc
proc --> vec
intake --> vec
vec --> ret
vec --> merge
files[("<b>Config files</b><br/><span>config and transforms</span>")]
end

next["<b>Next stage</b>"]
collector["<b>Metrics collector</b>"]
subgraph POD["dfe-transform-vector"]
direction TB
subgraph SUP["Supervisor"]
direction LR
cfg("<b>Config engine</b><br/><span>writes Vector's config</span>")
proc("<b>Process manager</b><br/><span>spawns, restarts</span>")
merge("<b>Metrics merge</b><br/><span>vector_* into ours</span>")
cfg --> proc
end
subgraph DATA["Records"]
direction LR
intake("<b>Bridge intake</b><br/><span>direct only</span>")
vec("<b>Vector</b><br/><span>the transform</span>")
ret("<b>Bridge return</b><br/><span>direct only</span>")
intake -.-> vec -.-> ret
end
SUP -->|starts, scrapes| DATA
end

prev -->|records| pod
files -->|config| pod
pod -->|records| next
pod -->|metrics| collector
subgraph NEXT["Next stage"]
direction LR
next("<b>Records out</b><br/><span>*_load topic or Push</span>")
end

subgraph COL["Telemetry"]
direction LR
collector("<b>Metrics collector</b><br/><span>scrape or OTLP</span>")
end

PREV -->|records| POD
CFGIN -->|config| POD
POD -->|records| NEXT
POD -->|metrics| COL
class prev,next bus
class collector ext
class files store
class cfg,proc,intake,vec,ret,merge part
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
Expand All @@ -102,13 +124,16 @@
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
class prev,next,collector ext
class files store
class cfg,proc,intake,vec,ret,merge part
style pod fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style PREV fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style CFGIN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style POD fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style SUP fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style DATA fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style NEXT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style COL fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
```

*Figure 1. L3 Components -- dfe-transform-vector. Dashed boxes are external.* Up: top of dfe-transform-vector. Down: [L4 Detail -- Config assembly](docs/architecture.md#config-assembly), [L4 Detail -- Direct transport bridge](docs/architecture.md#where-the-transports-differ), [L4 Detail -- Vector lifecycle](docs/internals.md#the-vector-subprocess).
*Figure 1. L3 Components -- dfe-transform-vector. Dotted arrows are the direct transport. On the bus Vector reads and writes Kafka itself.* Up: top of dfe-transform-vector. Down: [dfe-transform-vector architecture](docs/architecture.md).

On `bus` Vector reads and writes Kafka itself and the bridge does nothing. On `direct` the bridge takes records from the previous stage's Push client and hands Vector's output to the next stage's Push listener.

Expand Down Expand Up @@ -136,12 +161,15 @@

The config file (`--config`, else `config.yaml` or `config.yml` in the working directory) sits over the defaults, and `DFE_TRANSFORM_VECTOR_*` environment variables sit over the file. The working directory's `.env` is read into the environment first, and a variable set before it wins. The flat names take a single underscore and a comma list (`DFE_TRANSFORM_VECTOR_SOURCE_BROKERS=a:9092,b:9092`, `_SINK_TOPIC`, `_TRANSFORMS_DIR`, the SASL pairs) and beat the double-underscore form. That form reaches any key and takes a list in brackets (`DFE_TRANSFORM_VECTOR_SOURCE__BROKERS='[a:9092,b:9092]'`), and a bare comma list there fails to load. `--log-level`, `--log-format` and `--metrics-addr`, or `LOG_LEVEL`, `LOG_FORMAT` and `METRICS_ADDR`, outrank `logging.*` and `metrics.address`. [docs/config-schema.yaml](docs/config-schema.yaml) lists the app's keys with their types and defaults.

With reload on, the default, a transform file change applies once it validates, with no restart. Any other change to the app's keys, `dfe_source`, `pipeline.name` and `scaling.pressure_threshold` included, is logged as needing a pod restart, and no later transform change applies until the pod restarts or the edit is reverted. scalo's sections in the file, such as `version_check`, `scaling.enabled` and `metrics.namespace`, are read once at startup, so an edit there needs a restart, and the watcher logs a successful reload without applying it. In a chart deployment a config or file set change rolls the pods anyway, because the chart hashes both into the pod template. The reload path, the readiness rules, the metrics merge and the Kafka settings are in [docs/internals.md](docs/internals.md) and [docs/librdkafka.md](docs/librdkafka.md).
With reload on, the default, a transform file change applies once it validates, with no restart. Any other change to the app's keys, `dfe_source`, `pipeline.name` and `scaling.pressure_threshold` included, is logged as needing a pod restart, and no later transform change applies until the pod restarts or the edit is reverted. scalo's sections in the file, such as `version_check`, `scaling.enabled` and `metrics.namespace`, are read once at startup, so an edit there needs a restart, and the watcher logs a successful reload without applying it. In a chart deployment a config or file set change rolls the pods anyway, because the chart hashes both into the pod template. The reload path is in [docs/internals.md](docs/internals.md), the readiness rules and the metrics merge in [docs/vector.md](docs/vector.md), and the Kafka settings in [docs/librdkafka.md](docs/librdkafka.md).

## Documentation

- [docs/architecture.md](docs/architecture.md): the problem, the ownership boundary and the invariants.
- [docs/internals.md](docs/internals.md): the config engine, hot reload, the Vector subprocess, health and metrics, dead letters, the Vector version pin, the chart.
- [docs/architecture.md](docs/architecture.md): the parts, how data flows and why it is built this way.
- [docs/internals.md](docs/internals.md): the config engine and hot reload.
- [docs/vector.md](docs/vector.md): the Vector subprocess, health and metrics, and dead letters.
- [docs/direct-transport.md](docs/direct-transport.md): the bridge on the direct transport.
- [docs/reference/configuration.md](docs/reference/configuration.md): the config layers and the chart.
- [docs/migration.md](docs/migration.md): moving a pipeline off the official Vector chart.
- [docs/librdkafka.md](docs/librdkafka.md): the librdkafka settings Vector's Kafka source and sink are given.
- [tests/TESTING.md](tests/TESTING.md): the test tiers, and the e2e cases that need Docker and a Vector binary.
Expand Down
5 changes: 4 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,10 @@
How dfe-transform-vector is built and configured, and how to move a pipeline onto it from Vector.

- [Architecture](architecture.md)
- [Internals](internals.md)
- [Config engine](internals.md)
- [Vector](vector.md)
- [Direct transport](direct-transport.md)
- [Configuration](reference/configuration.md)
- [librdkafka settings](librdkafka.md)
- [Moving off the Vector chart](migration.md)
- [config-schema.yaml](config-schema.yaml) -- the configuration as a JSON Schema (YAML)
Expand Down
Loading
Loading