diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0144b67..f578ca3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -192,6 +192,100 @@ Two ways a green run says less than it looks: | Set `DFE_UPDATE_WIPE_STATE=1` on a box that also sets `DFE_DATA_ROOT` | Leave `DFE_DATA_ROOT` unset if you rely on the wipe | `make clean` does not chain the storage overlay, so it removes the volume objects while the bind directories keep their contents and the wipe silently becomes a no-op (`ops/daemon-update/README.md`) | | Take a local, gitignored note as current | Read the tracked file you are asking about | A local note is nobody's to maintain, and the ones seen so far named repos and directories that no longer exist | +### Profile resolution + +Compose profiles select infrastructure. `service_profiles.yaml` selects DFE services. `scripts/resolve_profile.py` reads the second and derives the first, and writes `.tmp/profile.mk` for the Makefile to include. Each service entry names the config file it mounts, which is how one service gets a Kafka config and another a gRPC one without a second compose file. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart TB + accTitle: L4 Detail -- Profile resolution + accDescr: resolve_profile.py reads the active profile from service_profiles.yaml and the DFE_PROFILE and footprint flags from the environment. It writes profile.mk, which the Makefile includes to pass the Compose profiles and DFE services to the targets that start or check a stack. + subgraph inputs["Inputs"] + direction LR + sp("service_profiles.yaml
the profile") + env(".env flags
overrides") + sp ~~~ env + end + subgraph resolver["Resolve"] + direction LR + resolve("resolve_profile.py") + mk(".tmp/profile.mk") + make("make targets") + resolve -->|writes| mk --> make + end + inputs --> resolver + class sp,env,mk store + class resolve,make part + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style inputs fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style resolver fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 1. L4 Detail -- Profile resolution. The flags are `DFE_PROFILE` and the footprint toggles such as `DFE_CORE_ENABLED`.* Up: [L2 Services -- DFE Compose stack](docs/architecture.md). + +### Start order + +`compose/one-shots.yml`, included by `docker-compose.yml`, holds every container that runs once and exits: dlq-init, bearer-tokens-init, the `contract-` emitters, dfe-dashboards, hyperdx-documentdb-tls, and a `config-check-` for dfe-archiver, dfe-loader and dfe-receiver. A config check runs its app's `config-check` with the image, environment and config file the app is about to start with, once the engine has rendered that file. A key the app doesn't know still passes, because the apps ignore unknown keys. + +The apps wait on `depends_on: dfe-engine: service_healthy`, because healthy means their tables and topics exist. A loader started first holds messages pending-schema and dead-letters them, which reads as data loss and is start ordering. + +Each limit tier extends `base` in `compose/base.yml`, and its `dns_search: ["."]` clears the search domain the host passes into every container. A bare name with no alias (`kafka` on a stack without a broker) then fails to resolve, rather than reaching a host on that domain. + ### Where this sits The suite graph in `dfe-infra/suite.yaml` records exactly one edge on this repo and no outbound edges at all. Read it with `dfe-stack suite --consumer dfe-docker` and `--producer dfe-docker`. diff --git a/Makefile b/Makefile index d79b3ee..719746b 100644 --- a/Makefile +++ b/Makefile @@ -515,8 +515,8 @@ up: ## Start the stack from the pinned registry images (PROFILE=slim-grpc and VE @$(MAKE) --no-print-directory ci @python3 scripts/creds.py --write -# The command dfe-engine's restart and recreate hints name: docs/architecture.md, -# "The engine renders each app's config". DEV=1 recreates from the local images. +# The command dfe-engine's restart and recreate hints name: docs/per-source-apps.md. +# DEV=1 recreates from the local images. ifneq ($(strip $(DEV)),) APPLY_FLAGS = $(DEV_FLAGS) else diff --git a/docs/README.md b/docs/README.md index d4c141c..a0a0540 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,6 +15,7 @@ Running DFE on a single host with Docker Compose: a first look, deploying and up - [Console edits without a rebuild](live-console.md) - [How the stack reports on itself](observability.md) - [Operating the DFE Compose stack](operating.md) +- [Per-source apps and make apply](per-source-apps.md) - [Profiles](profiles.md) - [Proving the pipeline with the e2e suite](testing.md) - [Secrets and storage](secrets-and-storage.md) diff --git a/docs/architecture.md b/docs/architecture.md index 56ef781..6f1d55b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -8,21 +8,9 @@ License: BUSL-1.1 Copyright: (c) 2026 HYPERI PTY LIMITED --> -# How the Compose stack fits together +# dfe-docker architecture -This repo is **packaging, not product**. It ships a `docker-compose.yml`, the -config files the components mount, and Python helpers that resolve a profile and -pin image versions. It builds no DFE component image. Each release publishes the -dfe-docker image, which sets up a folder that runs the stack, and the compose -files `docker compose -f oci://` runs (`.github/workflows/image.yml`). - -When something in the data path is wrong, the fix is almost always in a component -repo, not here. - -## Events arrive two ways and leave one - -Pushed to `dfe-receiver` or pulled by `dfe-fetcher`, and out through `dfe-loader` -into ClickHouse. Everything between is optional and profile-selected. On a Kafka profile, the receiver and fetcher produce to Kafka, and the loader reads the topics into ClickHouse. +dfe-docker runs a whole DFE stack on one host under Docker Compose, from the same images and pins as the Kubernetes path. It is packaging, not product: it builds no DFE component, so a fault in the data path is almost always fixed in a component repo. ```mermaid --- @@ -70,42 +58,60 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L2 Services -- DFE Compose stack - accDescr: Push clients send events to dfe-receiver and dfe-fetcher pulls them from APIs, databases and files. Both produce to Kafka topics. dfe-loader reads the topics and writes ClickHouse, dfe-archiver writes each source's raw events to its archive, and a transform reads a source's topic and writes the transformed topic the loader then reads. - + accDescr: Push clients send events to dfe-receiver and dfe-fetcher pulls them from APIs, databases and files. On a Kafka profile both produce to a source topic, an optional transform per source reads it and writes a transformed topic the loader reads instead, dfe-loader writes ClickHouse and dfe-archiver writes each source's raw events to its archive. On a gRPC profile the receiver sends straight to the loader. The DFE console and dfe-engine run beside the data path. push("Push clients") pull("APIs, databases, files") - subgraph stack["DFE Compose stack"] + direction TB subgraph ingestgrp["Ingest"] + direction LR receiver("dfe-receiver
terminates every protocol") fetcher("dfe-fetcher
pulls on a schedule") end - topics[("Kafka topics")] - loader("dfe-loader
schema-free insert") - transforms("Transforms
one per source") - archiver("dfe-archiver
raw events to the archive") - ch[("ClickHouse")] - archive[("Archive")] + subgraph busgrp["Kafka profiles"] + direction LR + topics[("Source topic
*_land")] + end + subgraph xfgrp["Optional transform"] + direction LR + transforms("Transforms
one per source") + loadtopic[("Transformed topic
*_load")] + transforms --> loadtopic + end + subgraph storegrp["Store"] + direction LR + loader("dfe-loader
schema-free insert") + ch[("ClickHouse")] + archiver("dfe-archiver
raw events, Kafka only") + archive[("Archive")] + loader --> ch + archiver --> archive + end + subgraph ctlgrp["Console and control"] + direction LR + console("DFE console
dfe-ui and Search") + engine("dfe-engine
config, schema, topics") + console --> engine + end + ingestgrp --> busgrp + busgrp --> storegrp + busgrp -.-> xfgrp + xfgrp -.-> storegrp + ingestgrp -.->|gRPC profiles, no Kafka| storegrp + storegrp --> ctlgrp end - - push -->|events| receiver - pull -->|events| fetcher - ingestgrp --> topics - topics --> loader - topics --> transforms - topics --> archiver - loader --> ch - archiver --> archive - transforms -.-> topics - + push -->|events| stack + pull -->|events| stack class push,pull ext class receiver,fetcher ingest - class topics bus + class topics,loadtopic bus class loader,archiver load class transforms xform class ch,archive store + class console ui + class engine control classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -119,118 +125,60 @@ flowchart LR classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A style stack fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style ingestgrp fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style busgrp fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style xfgrp fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style storegrp fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style ctlgrp fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L2 Services -- DFE Compose stack. The dotted arrow is a transform writing `_load` back to the broker.* Up: top of dfe-docker. Down: [L4 Detail -- Two listeners on port 6000](#component-repos-and-their-grpc-listeners), [L4 Detail -- Loader topic choice](known-issues.md#a-_load-topic-changes-what-the-loader-reads), [L4 Detail -- Loader and schema checks](troubleshooting.md#events-are-accepted-but-nothing-lands), [L4 Detail -- Topic and dead-letter checks](troubleshooting.md#events-are-accepted-but-nothing-lands). +*Figure 1. L2 Services -- DFE Compose stack. Dotted paths are optional.* Up: top of dfe-docker. Down: [L4 Detail -- Loader topic choice](known-issues.md#a-_load-topic-changes-what-the-loader-reads), [L4 Detail -- Loader and schema checks](troubleshooting.md#events-are-accepted-but-nothing-lands), [L4 Detail -- One console origin](operating.md#console-origin-and-the-port-6000-listeners), [L4 Detail -- Two listeners on port 6000](operating.md#console-origin-and-the-port-6000-listeners). -Push clients are HTTP, Vector, Beats, OTLP, HEC, syslog and flow senders. dfe-engine, the console and Search run beside this path, and Figure 3 shows the console's side. + -Transport is a per-profile choice, not a global one. On a `*-kafka` profile the -ingest components produce to a `*_land` topic derived from the event's `_source`. -`config/loader/kafka.yaml` names no topics, so the loader auto-discovers every -topic ending in one of its `routing.topic_suffixes`, `_land` and `_load` by -default. On a `*-grpc` profile there is no broker: the receiver dials -`dfe-loader:6000` directly. +## How data flows -A transform reads `_land` and writes `_load`, and runs on a Kafka profile only. `dfe-archiver` archives each source from its landing topic, and no gRPC profile runs it, so archiving needs a Kafka profile. Once a transform writes `_load`, the loader reads it instead of `_land`: scalo drops `_land` from its subscription once `_load` exists. +1. Push clients send to dfe-receiver over HTTP, Vector, Beats, OTLP, HEC, syslog or flow. dfe-fetcher pulls from REST APIs, databases and files, and idles until a fetcher source exists. +2. On a `*-kafka` profile both produce to a `*_land` topic named after the event's `_source`. The name carries through character for character, so `filebeat-vector` gives `filebeat-vector_land` and a table that needs backticks in SQL. +3. A source with a transform gets its own instance. It reads `_land` and writes `_load`, and from then on the loader reads `_load` instead. Transforms run on Kafka profiles alone. +4. dfe-loader auto-discovers every topic ending in `_land` or `_load` and inserts into ClickHouse. dfe-archiver writes each source's landing topic to its archive, so archiving needs a Kafka profile. +5. On a `*-grpc` profile there is no broker, and the receiver dials `dfe-loader:6000` directly. +6. The DFE console sends every change through dfe-engine. The engine creates each table and topic, and renders each app's config. -dfe-engine creates a transform's table and nothing here provisions it. Create the source before sending it data: a loader that meets rows for a table it has no schema for buffers them, then dead-letters them. +Create a source before you send it data. A loader that meets rows for a table it has no schema for buffers them, then dead-letters them. -The source NAME propagates character for character. The engine needs a Kubernetes DNS-1123 label (hyphens, never underscores, because a source-bound app deploys one instance called after its source), the receiver substitutes it into `<_source>_land`, and the loader strips `_load` back off for the table. So `filebeat-vector` gives `filebeat-vector_land`, `filebeat-vector_load` and a table name that needs backticks in SQL. +## Components -`slim-*` is the core data path alone: receiver, loader, engine, console and dfe-hyperdx. `single-kafka` adds the per-source apps its keys enable ([profiles.md](profiles.md#per-source-apps)) and one archiver. The `-kafka` profiles add the broker ([broker.md](broker.md)). dfe-fetcher runs idle until a fetcher source exists, and no transform starts until a source created through the engine binds one. +| Part | Role | Its page | +|---|---|---| +| dfe-receiver | Push ingest, every protocol, HTTP on `:8080` | [dfe-receiver](https://github.com/hyperi-io/dfe-receiver/blob/main/docs/architecture.md) | +| dfe-fetcher | Pull ingest from REST APIs, databases and files | [dfe-fetcher](https://github.com/hyperi-io/dfe-fetcher/blob/main/docs/architecture.md) | +| Source topic | The `*_land` topic each source writes | [Broker](broker.md) | +| Transforms | VRL, Vector or compiled Elastic pipelines, one instance per source | [vrl](https://github.com/hyperi-io/dfe-transform-vrl/blob/main/docs/architecture.md), [vector](https://github.com/hyperi-io/dfe-transform-vector/blob/main/docs/architecture.md), [elastic](https://github.com/hyperi-io/dfe-transform-elastic/blob/main/docs/architecture.md) | +| Transformed topic | The `*_load` topic a transform writes | [Known issues](known-issues.md#a-_load-topic-changes-what-the-loader-reads) | +| dfe-loader | Writes ClickHouse, and hosts the gRPC Push listener | [dfe-loader](https://github.com/hyperi-io/dfe-loader/blob/main/docs/architecture.md) | +| ClickHouse | Every table, plus the stack's self-telemetry | [Secrets and storage](secrets-and-storage.md) | +| dfe-archiver | Writes raw events to a filesystem, S3, MinIO, GCS or Azure Blob | [dfe-archiver](https://github.com/hyperi-io/dfe-archiver/blob/main/docs/architecture.md) | +| Archive | The archiver's files | [Secrets and storage](secrets-and-storage.md) | +| DFE console | dfe-ui and its Search, dfe-hyperdx, behind dfe-proxy | [dfe-ui](https://github.com/hyperi-io/dfe-ui/blob/main/docs/architecture.md) | +| dfe-engine | Config and schema API, the schema and topic authority | [dfe-engine](https://github.com/hyperi-io/dfe-engine/blob/main/docs/architecture.md) | -## The engine renders each app's config, one instance per source +`dfe-transform-splack` and `dfe-transform-wasm` have repos but no service here. -On Kubernetes an app's chart turns its values overlay into a ConfigMap the pod mounts. Compose runs no chart, so dfe-engine does that step: it merges each instance's overlay over the committed `config//.yaml` and writes the app's config file, its programs and its enrichment tables into the `dfe-app-config` volume, one directory per app. Every app mounts that volume read-only at `/etc/dfe/apps`. +## External dependencies -The committed file stays the DEPLOYMENT's (brokers, warehouse credentials, the spool) and the overlay wins wherever the two name the same key, so nothing deployment-specific is written into the deploy repo. `resolve_profile.py` turns this on wherever the core footprint runs. dfe-fetcher and the three transforms have no committed file: the overlay is their whole config, and their broker comes from `DFE_KAFKA_BOOTSTRAP_SERVERS` in `.env`. +| Dependency | Needed | Choices | +|---|---|---| +| Docker with Compose | always | one Linux host | +| Registry | always | the registry the stack pins, at the digests the Kubernetes path runs | +| Broker | on `*-kafka` profiles | Redpanda, or Apache Kafka with `KAFKA_BACKEND=apache` ([broker](broker.md)) | +| Object storage | for an archive past local disk | S3, MinIO, GCS, Azure Blob | +| OIDC provider | with the `auth` profile | any, through oauth2-proxy ([auth profile](auth-profile.md)) | -Nothing restarts a container. A change the running app cannot take in place is reported by the API as `restart required: make apply SERVICES=` and one to its env file as `recreate required:` with the same command. `make apply` creates a per-source container the engine has only just declared, recreates one that exists and removes the container of a source the engine has deleted, which it finds by the `dfe.instance-of` label only generated services have. A delete names no command, so `make apply` with no `SERVICES` does that removal alone and starts nothing. +## Deployment variants -A source with a transform stops sharing `main_land`: the receiver routes it to `_land`, and only a transform reading THAT topic sees it. So a transform is deployed once per source, as the Kubernetes tier deploys one per Source definition, and its input topic, program directory and output topic all have the name. dfe-engine renders the instance's config into `dfe-app-config`, and `make apply` starts a service that extends the committed `dfe-transform-vrl`, `dfe-transform-vector` or `dfe-transform-elastic`, which have the broker env and nothing of a source's own. +A profile is `-`: `slim-grpc`, `slim-kafka`, `single-grpc` or `single-kafka` ([profiles](profiles.md)). `slim` is the core data path: receiver, loader, engine, console and Search. `single-kafka` adds the per-source apps and one archiver. There is no `scale`, because Compose can't scale out. -## Two independent selectors decide what runs - -Compose profiles select INFRASTRUCTURE. `service_profiles.yaml` selects DFE SERVICES. `scripts/resolve_profile.py` reads the second and derives the first, writing `.tmp/profile.mk` for the Makefile. The resolver reads the profile and the environment, and the Makefile includes the `profile.mk` it writes. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Profile resolution - accDescr: resolve_profile.py reads the active profile from service_profiles.yaml and the DFE_PROFILE and footprint flags from the environment. It writes profile.mk, which the Makefile includes to pass the Compose profiles and DFE services to the targets that start or check a stack. - - sp("service_profiles.yaml") - env(".env flags") - resolve("resolve_profile.py") - mk(".tmp/profile.mk") - make("make targets") - - sp -->|profile| resolve - env -->|overrides| resolve - resolve -->|writes| mk - mk --> make - - class sp,env,mk store - class resolve,make part - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A -``` - -*Figure 2. L4 Detail -- Profile resolution. The flags are `DFE_PROFILE` and the footprint toggles such as `DFE_CORE_ENABLED`.* Up: [dfe-docker documentation](README.md). +A profile declares its footprint through the `clickhouse`, `core`, `kafbat`, `hyperdx`, `otel` and `auth` keys, each overridable by its `.env` flag. Each footprint is a Compose profile: | Compose profile | Services | |---|---| @@ -244,230 +192,48 @@ flowchart LR | `otel` | `otel-collector` | | `auth` | the three `oauth2-proxy-*` gates ([auth-profile.md](auth-profile.md)) | -A profile declares its whole footprint through the optional `clickhouse`, `core`, `kafbat`, `hyperdx`, `otel` and `auth` keys, each overridable by the matching `.env` flag. Each service entry names the config file it mounts, which is how one service gets a Kafka config and another a gRPC one without a second compose file. + Four limit tiers set CPU and memory ceilings ([operating](operating.md#resource-limits-four-tiers-ceilings-not-reservations)). -The `core` footprint is the management plane, independent of transport. `dfe-proxy` gives the console and the engine API one origin, because the console's client calls the engine with a relative `/api/v1/...` base URL. dfe-proxy sends `/api/v1` to dfe-engine and every other path to dfe-ui. +Kubernetes is the primary DFE path, and Compose is for small environments. Three things differ on purpose: -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- One console origin - accDescr: A browser reaches dfe-proxy on host port 3000. dfe-proxy sends paths under /api/v1 to dfe-engine and every other path to dfe-ui. dfe-engine reads and writes ClickHouse. - - browser("Browser") - proxy("dfe-proxy") - engine("dfe-engine
:8000") - console("dfe-ui
:3000") - ch[("ClickHouse")] - - browser -->|host :3000| proxy - proxy -->|/api/v1| engine - proxy -->|other paths| console - engine --> ch - - class browser ext - class proxy part - class engine control - class console ui - class ch store - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A -``` - -*Figure 3. L4 Detail -- One console origin. The ports in the boxes are container ports.* Up: [dfe-docker documentation](README.md). - -## dfe-engine owns the schema and the topics, and starts first - -Nothing here provisions a ClickHouse table or a Kafka topic. `dfe-engine` ships `/app/schemas` inside its own image, applies every object and every bootstrap topic from that manifest at startup, and reports healthy only once that pass converged. `dfe-schemas` rides inside the engine image, so there is no separate schema pin here and a schema change is a `dfe-engine` release. The ClickHouse service mounts only server settings from `config/clickhouse/`. A DDL file or a topic-creating step appearing in this repo is wrong by construction, and `scripts/tests/test_engine_only_schema_control.py` fails the build on one. - -`compose/one-shots.yml`, which `docker-compose.yml` includes, holds every container that runs once and exits: dlq-init, bearer-tokens-init, the `contract-` emitters, dfe-dashboards, hyperdx-documentdb-tls and a `config-check-` for dfe-archiver, dfe-loader and dfe-receiver. Each config check runs its app's own `config-check` with the image, environment and config file the app is about to start with, once the engine has rendered that file, and the app starts only after it passes. A broken config therefore stops `make up`, `make dev`, `make ci` and `make apply` with the app's own error instead of a restart loop. A key the app does not know still passes, because the apps ignore unknown keys. - -The apps then wait on `depends_on: dfe-engine: service_healthy`, because healthy means their tables and topics exist. A loader started first holds messages pending-schema and dead-letters them, which reads as data loss and is really start ordering. - -Each limit tier extends `base` in `compose/base.yml`, and its `dns_search: ["."]` clears the search domain the host passes into every container. A bare name with no alias (`kafka` on a stack without a broker) then fails to resolve instead of reaching a host on that domain. - -## Component repos and their gRPC listeners - -Dev builds fetch source into a managed git cache, or from your own checkouts when -`DFE_SRC_ROOT` is set -- see -[developing.md](developing.md#some-from-source-the-rest-pinned). - -| Repo | Language | Role | Profile | -|---|---|---|---| -| `dfe-receiver` | Rust | Push ingest. HTTP `:8080` published, Vector gRPC `:6000`, OTLP, Beats, HEC, syslog and flow published once their port key is set | `dfe` | -| `dfe-fetcher` | Rust | Pull ingest from REST APIs, databases and files | `dfe` | -| `dfe-loader` | Rust | Writes ClickHouse and hosts `scalo.transport.v1.Transport/Push` | `dfe` | -| `dfe-archiver` | Rust | Archive sink (filesystem, S3, MinIO, GCS, Azure Blob) | `dfe` | -| `dfe-transform-elastic` | Rust | Beats and Elastic Agent pipelines compiled to Rust, one transform selected per instance by `source.name` or, in agentless mode, detected per event | `dfe` | -| `dfe-transform-vector` | Rust | Vector.dev subprocess wrapper, Kafka to Kafka | `dfe` | -| `dfe-transform-vrl` | Rust | Embedded VRL transform engine, deployed once per source that has a transform | `dfe` | -| `dfe-engine` | Python | Config and schema API, and the schema authority | `core` | -| `dfe-ui` | TypeScript | The DFE console | `core` | -| `dfe-hyperdx` | TypeScript | The console's Search, a fork of HyperDX. Repo and published image are both `dfe-hyperdx` | `hyperdx` | - -dfe-transform-vector mounts `resources/enrichment/` at `/etc/dfe-transform-vector/data`, where its bundled filebeat pipeline reads `timezones.csv`. dfe-transform-elastic compiles its transforms in, and the engine uploads the enrichment tables a dfe-transform-vrl program reads, so neither mounts them. `make test-source` drives the filebeat, Vector and Elastic cases through every hop. dfe-transform-vector's one port serves everything: the wrapper scrapes Vector's `internal_metrics` off its `prometheus_exporter` on loopback and registers the samples on scalo's own recorder (dfe-transform-vector#70), so `vector_*` reads on its metrics port and rides the OTLP push. The exporter's 9598 is not published. - -Two gRPC listeners share port 6000, and they are not the same thing. A `*-grpc` profile runs both. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Two listeners on port 6000 - accDescr: An external Vector client can push to dfe-receiver's Vector protocol listener on port 6000 once it is enabled. dfe-receiver pushes to dfe-loader's internal Push listener, also on port 6000, over the compose network. +- A new source's apps start when you run `make apply`. Argo does that on Kubernetes. +- Envoy's OIDC filters stay unconfigured, because Docker mode can't assume an issuer exists. The engine API still needs a login per route. +- Container logs reach the collector through Docker's `fluentd` log driver, and the host's CPUs are the one host metric collected ([observability](observability.md)). - client("Vector client") +## Design considerations - subgraph net["Compose network"] - receiver("dfe-receiver") - loader("dfe-loader") - end +**Security.** The console and its Search authenticate. The ingest ports, Kafbat and the metrics ports ask for no login, so put the stack behind a VPN, a tunnel or a firewall, or arm the `auth` profile ([operating](operating.md)). - client -.->|Vector :6000| receiver - receiver -->|Push :6000| loader +**Reliability.** Every app waits for dfe-engine to report healthy, which means its tables and topics exist. Before an app starts, a one-shot runs its `config-check` against the exact config it will use, so a broken config stops `make up` with the app's own error and not a restart loop. - class client ext - class receiver ingest - class loader load - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - style net fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 -``` +**Operations.** Nothing restarts a container on its own. A change an app can't take in place is reported as `restart required: make apply SERVICES=` ([per-source apps](per-source-apps.md)). -*Figure 4. L4 Detail -- Two listeners on port 6000. The dotted arrow is off until `config.grpc.enabled` is set.* Up: [L2 Services -- DFE Compose stack](#events-arrive-two-ways-and-leave-one). +**Performance.** The Compose stack is sized for one host. `DFE_SERVICE_CPUS` has a floor of 2.0 ([operating](operating.md)). -| Container port | Host port | Owner | Protocol | Audience | Binds | -|---|---|---|---|---|---| -| 6000 | none, until `DFE_RECEIVER_GRPC_PORT` is set | dfe-receiver | Vector protocol, an ingest source that listens only once `config.grpc.enabled` is set through the engine | external clients | `DFE_INGRESS_BIND_HOST` (`0.0.0.0`) | -| 6000 | none | dfe-loader | `scalo.transport.v1.Transport/Push`, internal transport | dfe-receiver, on a `*-grpc` profile | the compose network | +## Why it's built this way -Every Push listener in the suite answers on 6000 and dfe-engine compiles each sender's endpoint against that port. The receiver's listener is meant to be reachable from outside the stack. The loader's is not, so it publishes no host port. +**Same images, same registry, same pins.** Compose is safe to deploy because it runs the exact digests Kubernetes runs, pinned from the same stack definition. -`dfe-transform-splack` and `dfe-transform-wasm` have repos but no service here, -so this stack cannot run them. +**dfe-engine renders each app's config.** Compose runs no chart, so the engine merges each instance's overlay over the committed `config//.yaml` and writes the result to the `dfe-app-config` volume. Deployment facts stay in the committed file, and nothing deployment-specific goes into the deploy repo. -## Where this differs from Kubernetes +**dfe-engine alone owns the schema and topics.** The schemas ride inside the engine image, so a schema change is an engine release and there's no schema pin here. `scripts/tests/test_engine_only_schema_control.py` fails the build on any DDL or topic step in this repo. -Kubernetes is the primary DFE path. Compose is supported for small environments -and is not co-equal -- if both would work, choose Kubernetes. What makes Compose -safe to deploy is that it consumes the SAME image from the SAME registry, pinned -from the same stack SSoT, so both run identical digests. +**One transform per source.** A source with a transform stops sharing `main_land`, so only a transform reading that source's topic sees it. That matches Kubernetes, which deploys one per source definition. -Three deliberate asymmetries: +## Diagram map -- An operator applies a new source. Both tiers run a fetcher and a transform per source. On Kubernetes Argo creates the new Deployment. Here the engine declares the instance and reports `make apply`, which starts it. After deleting a source, run `make apply` with no `SERVICES` to stop its transform, because the delete names no command and the container runs until an apply or a restart removes it. `make ps` warns about any it finds. -- No edge authentication. Envoy fronts both tiers, but its OIDC filters stay unconfigured here, because Docker mode can never assume an issuer exists. The engine API still needs a login per route, which the proxy neither adds nor removes. The opt-in `auth` profile gates the infra UIs alone ([auth-profile.md](auth-profile.md)). -- Self-monitoring collects differently. Container stdout reaches the collector through Docker's `fluentd` log driver rather than a DaemonSet filelog receiver, and of the host's own metrics only its CPUs are collected. See [observability.md](observability.md). +- [Figure 1. L2 Services -- DFE Compose stack](#dfe-docker-architecture) + - [L4 Detail -- Loader topic choice](known-issues.md#a-_load-topic-changes-what-the-loader-reads) + - [L4 Detail -- Loader and schema checks](troubleshooting.md#events-are-accepted-but-nothing-lands) + - [L4 Detail -- Topic and dead-letter checks](troubleshooting.md#events-are-accepted-but-nothing-lands) + - [L4 Detail -- One console origin](operating.md#console-origin-and-the-port-6000-listeners) + - [L4 Detail -- Two listeners on port 6000](operating.md#console-origin-and-the-port-6000-listeners) + - [L4 Detail -- Self-telemetry](observability.md) + - [L4 Detail -- Profile resolution](../CONTRIBUTING.md#profile-resolution), for contributors -## Related +## What's next -- [operating.md](operating.md) -- access, exposure, limits -- [secrets-and-storage.md](secrets-and-storage.md) -- secrets, volumes, retention -- [deploying.md](deploying.md) -- the dial, pinning, upgrades -- [observability.md](observability.md) -- health, self-telemetry, the self test -- [developing.md](developing.md) -- dev mode, profiles, proving the pipeline +- Try it: [evaluating](evaluating.md). +- Run it: [operating](operating.md), [deploying](deploying.md) and [secrets and storage](secrets-and-storage.md). +- Every setting: [configuration](configuration.md). +- Change dfe-docker: [developing](developing.md) and the Context section of [CONTRIBUTING.md](../CONTRIBUTING.md#context). diff --git a/docs/developing.md b/docs/developing.md index 7d6fef4..2025767 100644 --- a/docs/developing.md +++ b/docs/developing.md @@ -137,21 +137,23 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Dev image build accDescr: The component's source is staged without target and .git. This repo's Rust builder image compiles it into a binary. The component's own Dockerfile packages that binary into the service's local image. - - src("Component source") - builder("dfe-rust-builder
Dockerfile") - bin("Compiled binary") - own("Component
Dockerfile") - img("service:local
image") - - src --> builder - builder --> bin - bin --> own - own --> img - + subgraph build["Compile"] + direction LR + src("Component source") + builder("dfe-rust-builder
Dockerfile") + bin("Compiled binary") + src --> builder --> bin + end + subgraph pack["Package"] + direction LR + own("Component
Dockerfile") + img("service:local
image") + own --> img + end + build --> pack class src,bin,img store class builder,own part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A @@ -165,6 +167,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style build fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style pack fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Dev image build. Green boxes are files and images, grey boxes are build steps.* Up: [dfe-docker documentation](README.md). diff --git a/docs/observability.md b/docs/observability.md index c38e613..01dcfe7 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -115,26 +115,29 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Self-telemetry accDescr: DFE services push metrics and traces over OTLP to the otel-collector. Docker's log driver forwards the stdout of the DFE containers to it while container log shipping is on. The collector pulls ClickHouse's system tables and the host's CPUs itself. It writes logs, metrics and traces to the dfe.otel_ tables, which dfe-hyperdx queries. - - apps("DFE services") - driver("Docker log driver") + subgraph pushed["Pushed"] + direction LR + apps("DFE services
OTLP :4317") + driver("Docker log driver
fluent :24224") + end subgraph pulled["Pulled"] + direction LR chsys[("ClickHouse
system tables")] host("Host /proc") end - col("otel-collector") - otel[("dfe.otel_* tables")] - hdx("dfe-hyperdx") - - apps -->|OTLP :4317| col - driver -->|fluent :24224| col - pulled -->|scraped| col - col -->|writes| otel - otel -.-> hdx - + subgraph collect["Collect and query"] + direction LR + col("otel-collector") + otel[("dfe.otel_* tables")] + hdx("dfe-hyperdx") + col -->|writes| otel + otel -.-> hdx + end + pushed -->|pushed| collect + pulled -->|scraped| collect class apps,col part class driver,host ext class chsys,otel store @@ -150,7 +153,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style pushed fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style pulled fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style collect fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Self-telemetry. The dotted arrow is a read, and the collector's own `:8888` metrics and dfe-hyperdx's `:8000` are scraped.* Up: [dfe-docker documentation](README.md). diff --git a/docs/operating.md b/docs/operating.md index e2e6c74..df5d6cb 100644 --- a/docs/operating.md +++ b/docs/operating.md @@ -121,7 +121,7 @@ flowchart TB *Figure 1. L4 Detail -- Who reaches which port. Dotted arrows answer only on the box until `DFE_BIND_SCOPE=all` or `DFE_BIND_HOST=0.0.0.0` widens them.* Up: [dfe-docker documentation](README.md). -Behind those ports, dfe-proxy sends `/api/v1/*` to the engine ([architecture.md](architecture.md#two-independent-selectors-decide-what-runs)) and dfe-hyperdx queries ClickHouse for the browser. +Behind those ports, dfe-proxy sends `/api/v1/*` to the engine ([console origin](#console-origin-and-the-port-6000-listeners)) and dfe-hyperdx queries ClickHouse for the browser. ## Two consequences survive the loopback defaults @@ -172,6 +172,180 @@ Setting `DFE_BIND_HOST=0.0.0.0` opens every operator port at once, including a ClickHouse that runs with `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1` behind one password. Do it knowingly. +## Console origin and the port 6000 listeners + +The console's client calls the engine with a relative `/api/v1/...` base URL, so `dfe-proxy` gives both one origin. It sends `/api/v1` to dfe-engine and every other path to dfe-ui. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart TB + accTitle: L4 Detail -- One console origin + accDescr: A browser reaches dfe-proxy on host port 3000. dfe-proxy sends paths under /api/v1 to dfe-engine and every other path to dfe-ui. dfe-engine reads and writes ClickHouse. + subgraph front["Front"] + direction LR + browser("Browser") + proxy("dfe-proxy") + browser -->|host :3000| proxy + end + subgraph back["Behind the proxy"] + direction LR + console("dfe-ui
:3000, other paths") + engine("dfe-engine
:8000, /api/v1") + ch[("ClickHouse")] + engine --> ch + end + front --> back + class browser ext + class proxy part + class engine control + class console ui + class ch store + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style front fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style back fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 2. L4 Detail -- One console origin. The ports in the boxes are container ports.* Up: [L2 Services -- DFE Compose stack](architecture.md). + +Two gRPC listeners share port 6000, and they aren't the same thing. A `*-grpc` profile runs both. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart TB + accTitle: L4 Detail -- Two listeners on port 6000 + accDescr: An external Vector client can push to dfe-receiver's Vector protocol listener on port 6000 once it is enabled. dfe-receiver pushes to dfe-loader's internal Push listener, also on port 6000, over the compose network. + client("Vector client") + subgraph net["Compose network"] + direction LR + receiver("dfe-receiver") + loader("dfe-loader") + receiver -->|Push :6000| loader + end + client -.->|Vector :6000, to dfe-receiver| net + class client ext + class receiver ingest + class loader load + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style net fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 3. L4 Detail -- Two listeners on port 6000. The dotted arrow is off until `config.grpc.enabled` is set.* Up: [L2 Services -- DFE Compose stack](architecture.md). + +| Container port | Host port | Owner | Protocol | Audience | Binds | +|---|---|---|---|---|---| +| 6000 | none, until `DFE_RECEIVER_GRPC_PORT` is set | dfe-receiver | Vector protocol, an ingest source that listens only once `config.grpc.enabled` is set through the engine | external clients | `DFE_INGRESS_BIND_HOST` (`0.0.0.0`) | +| 6000 | none | dfe-loader | `scalo.transport.v1.Transport/Push`, internal transport | dfe-receiver, on a `*-grpc` profile | the compose network | + +Every Push listener in the suite answers on 6000 and dfe-engine compiles each sender's endpoint against that port. The receiver's listener is meant to be reachable from outside the stack. The loader's is not, so it publishes no host port. + ## Web UIs: one scope dial, one kill switch Every web UI publishes by default. `DFE_BIND_SCOPE` says where: diff --git a/docs/per-source-apps.md b/docs/per-source-apps.md new file mode 100644 index 0000000..6a778f9 --- /dev/null +++ b/docs/per-source-apps.md @@ -0,0 +1,23 @@ + + +# Per-source apps and make apply + +On Kubernetes an app's chart turns its values overlay into a ConfigMap the pod mounts. Compose runs no chart, so dfe-engine does that step. It merges each instance's overlay over the committed `config//.yaml` and writes the app's config file, its programs and its enrichment tables into the `dfe-app-config` volume, one directory per app. Every app mounts that volume read-only at `/etc/dfe/apps`. + +The committed file stays the deployment's (brokers, warehouse credentials, the spool), and the overlay wins wherever the two name the same key. So nothing deployment-specific is written into the deploy repo. `resolve_profile.py` turns this on wherever the core footprint runs. dfe-fetcher and the three transforms have no committed file: the overlay is their whole config, and their broker comes from `DFE_KAFKA_BOOTSTRAP_SERVERS` in `.env`. + +Nothing restarts a container. A change the running app can't take in place is reported by the API as `restart required: make apply SERVICES=`, and a change to its env file as `recreate required:` with the same command. + +`make apply` creates a per-source container the engine has just declared, recreates one that exists, and removes the container of a source the engine has deleted. It finds those by the `dfe.instance-of` label, which generated services alone have. A delete names no command, so `make apply` with no `SERVICES` does that removal alone and starts nothing. + +A source with a transform stops sharing `main_land`. The receiver routes it to `_land`, and a transform reading that topic is the one that sees it. So a transform is deployed once per source, as the Kubernetes tier deploys one per Source definition, and its input topic, program directory and output topic all carry the source's name. + +dfe-engine renders the instance's config into `dfe-app-config`. `make apply` then starts a service that extends the committed `dfe-transform-vrl`, `dfe-transform-vector` or `dfe-transform-elastic`, which have the broker env and nothing of a source's own.