diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6a7773fb..f335cbde 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -157,6 +157,13 @@ System design and why it is built this way: | `src/dfe_engine/schema/` | ClickHouse DDL planning and apply | | `src/dfe_engine/source/` | Source definitions. Receiver and loader routing is COMPILED from these | | `src/dfe_engine/gitops/` | every mutation is a git commit, and the deploy repo is the authority | +| `src/dfe_engine/gitcrud/` | one git-native CRUD engine for every YAML resource class, with the commit policy and RBAC bound per class | +| `src/dfe_engine/sigma/` | Sigma rule import, conversion to ClickHouse SQL, propagation to hunts and the `_sigma` views | +| `src/dfe_engine/kafka/` | the Kafka credential contract and topic admin | +| `src/dfe_engine/hyperdx/` | dfe-hyperdx teams, per-source sources and dashboards | +| `src/dfe_engine/governance/ch/` | ClickHouse roles, quota tiers and the per-org row policies, reconciled | +| `src/dfe_engine/hunts/` | hunt and rule models, the compiler to SQL, alert delivery and the `dfe-hunt-runner` loop | +| `src/dfe_engine/keda_shim/` | `dfe-keda-shim`: config-driven ClickHouse queries answered to KEDA's `metrics-api` scaler | ### Commands that prove a change diff --git a/README.md b/README.md index bcc2ba4f..2ccfab55 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ The image ships the dfe-schemas trees at `/app/schemas-seed`, copies them into ` ## Documentation -- [docs/architecture.md](docs/architecture.md): what the engine is, its invariants and why it is built this way. +- [docs/architecture.md](docs/architecture.md): what the engine is, its parts and why it is built this way. - [docs/control-plane/](docs/control-plane/README.md): the API, auth, RBAC and governed gitops. - [docs/data-plane/](docs/data-plane/README.md): sources, schemas, DDL, the query API and hunts. - [docs/deployment/](docs/deployment/README.md): the chart, the image and the deployment contract. diff --git a/docs/README.md b/docs/README.md index 33d7c70c..0c6b4e65 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,7 +2,7 @@ The DFE engine's documentation: the architecture and data evolution, plus one directory each for the control plane, data plane, deployment and integrations, and the API route map. -- [DFE architecture](architecture.md) +- [dfe-engine architecture](architecture.md), with its [API server](api-server.md) and [hunt runner](hunt-runner.md) - [Data evolution](data-evolution.md) and its [acceptance path](data-evolution-acceptance.md) - [The DFE Control-Plane Layer](control-plane/README.md) - [Data plane](data-plane/README.md) diff --git a/docs/api-server.md b/docs/api-server.md new file mode 100644 index 00000000..ebd986ab --- /dev/null +++ b/docs/api-server.md @@ -0,0 +1,317 @@ +# API server + +The API server is the part of [dfe-engine](architecture.md) that every operator change goes through: it keeps the deployment's config, compiles what the data apps read, and applies it. + +| | | +|---|---| +| Repo | [hyperi-io/dfe-engine](https://github.com/hyperi-io/dfe-engine) | +| Image and chart | `ghcr.io/hyperi-io/dfe-engine`, entry point `dfe-engine`. Chart `oci://ghcr.io/hyperi-io/charts/dfe-engine` | +| Config reference | `DFE_ENGINE_` environment variables, `src/dfe_engine/settings.py` | +| Endpoints | `:8000` the API under `/api/v1` ([route map](reference/api-route-map.md)). `:9090` `/livez`, `/readyz` and `/metrics` | +| Metrics | Prometheus `/metrics` and OTLP, names prefixed `dfe_` ([observability standard](deployment/observability-standard.md)) | + +Every change enters through the routes and leaves as a commit or as DDL. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart TB + accTitle: L3 Components -- API server + accDescr: dfe-ui and the dfe CLI call the routes. The routes drive the config registries and the compilers. The GitOps writer commits registry changes and compiled overlays to the deploy repo, and the provisioners apply DDL to ClickHouse and create Kafka topics. + subgraph CALLERS["Callers"] + direction LR + ui("dfe-ui
the DFE console") + cli("dfe CLI
every operation") + end + subgraph API["API server"] + direction TB + routes("Routes and auth
/api/v1, RBAC") + registries("Config registries
sources, hunts, rules") + compilers("Compilers
routing, DDL, Sigma") + writer("GitOps writer
one commit a change") + provisioners("Provisioners
tables, topics, roles") + routes --> registries + routes --> compilers + registries --> writer + compilers --> writer + compilers --> provisioners + end + subgraph TARGETS["Deploy and store"] + direction LR + repo[("Deploy repo
git")] + ch[("ClickHouse")] + kafka[("Kafka
optional")] + end + CALLERS -->|calls| routes + writer -->|commits| repo + provisioners -->|DDL| ch + provisioners -.->|topics| kafka + class ui ui + class cli,repo control + class routes,registries,compilers,writer,provisioners part + class ch store + class kafka bus + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style CALLERS fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style API fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style TARGETS fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 1. L3 Components -- API server. The dotted arrow runs only where the deployment has Kafka.* Up: [L2 Services -- dfe-engine](architecture.md). Down: none. + +## How it works + +1. Routes and auth authenticate the request by trusted OIDC headers, an API key or a bearer token, and check its RBAC scope. +2. The config registries read and write sources, hunts, rules, field maps, governance and app overlays. With GitOps on they live in the deploy repo, otherwise in directories under `DFE_CONFIG_DIR`. +3. The compilers turn a source into the receiver's routing rules, the loader's table map, the table's DDL and its Sigma view, and turn Sigma rules into SQL. +4. The GitOps writer makes one attributed commit per change. +5. The provisioners apply the dfe-schemas manifest and each source's table, create Kafka topics, and keep the dfe-hyperdx sources and the ClickHouse roles and row policies in step. + +A source definition carries its routing rule, schema binding and field maps together, and every stage reads what is compiled from it. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart LR + accTitle: L4 Detail -- One source definition, compiled + accDescr: A source definition goes to the compilers, which produce the receiver's rules, the loader's table map, the source's table and its Sigma view. + src("Source definition") + compilers("Compilers
in the API server") + subgraph OUT["Compiled from it"] + direction TB + rules("Receiver rules") + tablemap("Loader table map") + table[("Source table")] + view[("Sigma view")] + rules ~~~ tablemap ~~~ table ~~~ view + end + src -->|definition| compilers + compilers --> OUT + class table,view store + class src,compilers,rules,tablemap part + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style OUT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 2. L4 Detail -- One source definition, compiled.* Up: [L3 Components -- API server](api-server.md). + +The `_sigma` view projects the source's columns under Sigma field names for detection. Routing that disagrees with the sources is reported as drift and re-synced. + +## Interfaces + +- Inbound: HTTP on port 8000. The dfe CLI and dfe-ui's types are both generated from the OpenAPI specs in `openapi/`. +- Deploy repo: app overlays in `values/`, backing-service overlays in `infra/`, governance in `governance/`, and with GitOps on the sources in `config/sources/` and the hunts and rules in `config/hunts/` and `config/rules/`. GitHub, GitLab or the in-cluster Forgejo. +- ClickHouse: DDL, roles and row policies. The engine is the only thing that creates ClickHouse objects. +- Kafka: topic creation, on tiers that run Kafka. +- dfe-hyperdx: the sources the Observe view queries. + +Adding a service instance is a commit of one values file. dfe-infra's ApplicationSet turns each file into one Argo CD Application. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart LR + accTitle: L4 Detail -- Instance deploy path + accDescr: The GitOps writer commits an instance values file to the deploy repo. dfe-infra's ApplicationSet makes one Argo CD Application per values file, which renders the app's chart into service pods. + subgraph REPO["Commit"] + direction TB + writer("GitOps writer
in the API server") + values[("Instance values file")] + writer -->|commit| values + end + subgraph ARGO["Argo CD sync"] + direction TB + appset("ApplicationSet") + app("Argo Application") + pods("Service pods") + appset --> app -->|helm| pods + end + REPO -->|matched| ARGO + class values store + class writer part + class appset,app control + class pods part + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style REPO fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style ARGO fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 3. L4 Detail -- Instance deploy path.* Up: [L3 Components -- API server](api-server.md). + +The file is `values/--values.yaml`, and dfe-infra's git-files ApplicationSet (`argocd/appsets/layer2-apps.yaml`) matches it. Removing the file removes the app. An instance removed while its source still exists keeps its files in `values/--parked.yaml`, which the glob does not match, and its next deploy restores them. + +## Scaling and failure + +- Nothing on the data path or the hunt path calls it, so an API server that is down delays config changes and nothing else. +- A write that names the revision it read in `If-Match` gets `409 Conflict` with the current document when that revision is stale. A write with no `If-Match` is not checked. +- With no ClickHouse at start it waits up to 180 seconds for the schema bootstrap (`DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS`), then serves with `/readyz` reporting `clickhouse: false`. +- The engine reports what it committed. Argo CD, and any approval gate, decides when it is live. + +## Key metrics + +`/readyz` reports `clickhouse: false` while ClickHouse is unreachable. `api_bearer_tokens_refused_total{reason}` counts refused bearer tokens. The [observability standard](deployment/observability-standard.md) lists the rest. + +## Related + +- [Control plane](control-plane/README.md): governed GitOps, RBAC and OIDC in depth. +- [Source definition](data-plane/source-definition.md) and [schema](data-plane/schema.md). +- [Deployment](deployment/README.md): the tiers and the deploy-repo providers. diff --git a/docs/architecture.md b/docs/architecture.md index f3473599..710d94ec 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,27 +1,6 @@ -# DFE architecture +# dfe-engine architecture -DFE is a deployable product suite: event-data ingest, ClickHouse storage, -scheduled detection, and exploration, all governed through one git-backed -control plane. This page is the map - where each repo sits, the two principles -that define the system, the invariants, and links into the depth docs. - -The depth docs: - -| Area | Start at | Covers | -|------|----------|--------| -| Control-plane surface | [control-plane/README.md](control-plane/README.md) | governed ops, gitops commit standard, RBAC, OIDC, CLI, UI API, type-safety sync | -| Data plane | [data-plane/README.md](data-plane/README.md) | sources, schemas, field mapping, CEL, query API, hunts | -| Deployment | [deployment/README.md](deployment/README.md) | tiers + composition, backing services, KEDA, observability, supply-chain pinning, e2e acceptance | -| Integrations | [integrations/README.md](integrations/README.md) | dfe-ui guide, and where the dfe-hyperdx fork runbook lives | -| Archive | [archive/README.md](archive/README.md) | superseded research + snapshots, kept for the decision trail | - -Edge auth: [auth-envoy-topology.md](auth-envoy-topology.md). - -## Where the engine sits in the suite - -dfe-engine is the control plane: a Python library and FastAPI server that turns -operator changes into git commits Argo CD syncs into the cluster, and applies the -ClickHouse DDL itself. Operators call the engine through dfe-ui and the CLI, and the engine commits to the deploy repo that Argo CD syncs. +dfe-engine is DFE's control plane: it turns an operator's change into a commit in the deployment's git repo, applies the ClickHouse schema and runs the scheduled hunts. Records never pass through it, because the engine only decides how the receiver, loader and the other data apps are configured. ```mermaid --- @@ -70,511 +49,38 @@ config: .ext .label-icon{color:#64748B} --- flowchart LR - accTitle: L2 Services -- DFE - accDescr: Operators call dfe-engine through dfe-ui and the dfe CLI. The engine commits config to the deploy repo and applies DDL to ClickHouse. Argo CD syncs the deploy repo into the runtime services. - - subgraph clients["Clients"] - ui["dfe-ui"] - cli["dfe CLI"] + accTitle: L2 Services -- dfe-engine + accDescr: dfe-ui and the dfe CLI call the engine, and KEDA polls it. The engine's API server, hunt runner and KEDA shim write to the deploy repo, ClickHouse and Kafka. + subgraph CALLERS["Callers"] + direction TB + ui("dfe-ui
the DFE console") + cli("dfe CLI
every operation") + keda("KEDA
autoscaler") + ui ~~~ cli ~~~ keda end - - engine[["dfe-engine"]] - repo[("Deploy repo")] - ch[("ClickHouse")] - argo["Argo CD"] - runtime["Runtime services"] - - clients -->|calls| engine - engine -->|commits| repo - engine -->|DDL| ch - repo --> argo - argo -->|helm| runtime - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class ch store - class ui ui - class cli,engine,repo,argo control - class runtime part - style clients fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 -``` - -*Figure 1. L2 Services -- DFE.* Up: top of dfe-engine. Down: [L3 Components -- dfe-engine](#inside-the-engine). - -Argo CD also reads dfe-infra's charts. The runtime services write to ClickHouse, and dfe-hyperdx, embedded in dfe-ui, reads it. - -| Repo | Role | Relationship to the engine | -|------|------|---------------------------| -| dfe-engine | config control plane + API server | the engine itself | -| dfe-infra | charts, ApplicationSets, bootstrap | consumes engine-authored overlays via the deploy repo | -| dfe-ui | web UI | calls the engine API and nothing else | -| dfe-hyperdx | the Observe view: our extended fork of the upstream HyperDX project, explore UI + telemetry sink | embedded in dfe-ui, reads ClickHouse | -| dfe-schemas | every table, view, role and bootstrap topic, plus the apply manifest | where they are defined, and nowhere else. The engine applies the manifest at boot ([schema bootstrap](data-plane/schema-bootstrap.md)) | -| scalo-py / scalo-rs | shared libraries (neutral-identity OSS) | engine imports scalo-py, Rust services import scalo-rs | -| dfe-receiver/loader/archiver/fetcher/transform-* | Rust runtime services | read YAML config the engine writes | -| dfe-deploy | example deploy/overlay repo | the GitOps hand-off surface | - -An event goes from the receiver through Kafka to the loader and the archiver. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Event path - accDescr: Log shippers send events to dfe-receiver, which routes them to Kafka topics. dfe-loader writes them to ClickHouse and dfe-archiver writes them to object storage. - - shippers["Log shippers"] - receiver["dfe-receiver"] - topics[("Kafka topics")] - - subgraph readers["Topic readers"] - loader["dfe-loader"] - archiver["dfe-archiver"] + subgraph ENGINE["dfe-engine"] + direction TB + api("API server
config and schema") + runner("Hunt runner
scheduled hunts") + shim("KEDA shim
scaling metrics") + api ~~~ runner ~~~ shim end - - subgraph stores["Stores"] + subgraph STATE["Deploy and store"] + direction TB + repo[("Deploy repo
git")] ch[("ClickHouse")] - bucket[("Object storage")] + kafka[("Kafka
optional")] + repo ~~~ ch ~~~ kafka end - - shippers -->|events| receiver - receiver -->|routed| topics - topics --> readers - readers --> stores - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class shippers ext - class receiver ingest - class loader,archiver load - class topics bus - class ch,bucket store - style readers fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 - style stores fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 -``` - -*Figure 6. L4 Detail -- Event path. The dashed grey box is outside DFE.* Up: [L2 Services -- DFE](#where-the-engine-sits-in-the-suite). - -The loader writes to ClickHouse and the archiver to object storage. With no broker, the receiver pushes to them over gRPC instead. The hunt runner, dfe-hyperdx and the query API read ClickHouse. - -Matched rows are written at full fidelity up to `hunts.max_detections_per_run` -per rule, and past it the runner writes one summary row. When a hunt names an alert -destination, the runner sends one message per fire that matched. There is no -alert grouping or cooldown. - -## The two principles that define DFE - -The first is orchestration over infra: helm through Argo, never hands on the -cluster. The engine makes every operational change as a governed git commit of -compiled Helm overlay values, and Argo CD reconciles the cluster to match. The -one thing it applies itself is ClickHouse DDL. The engine never deploys backing -services, never authors Argo Applications, and never claims a commit is live -(an org-specific approval gate may sit between commit and sync). One committed values file becomes one Argo Application and its pods. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Instance deploy path - accDescr: dfe-engine commits an instance values file to the deploy repo. dfe-infra's ApplicationSet makes one Argo Application per values file, which renders the app's chart into service pods. - - engine[["dfe-engine"]] - values[("Instance values file")] - appset["ApplicationSet"] - app["Argo Application"] - pods["Service pods"] - - engine -->|commit| values - values -->|matched| appset - appset --> app - app -->|helm| pods - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class values store - class engine,appset,app control - class pods part -``` - -*Figure 2. L4 Detail -- Instance deploy path.* Up: [L2 Services -- DFE](#where-the-engine-sits-in-the-suite). - -Adding a service instance is a git write of one -`values/--values.yaml` file. dfe-infra's git-files -ApplicationSet (`argocd/appsets/layer2-apps.yaml`) makes one Argo Application per -values file. Removing the file removes the app. The engine declares instances. -The appset deploys them. -An instance removed while its source still exists leaves the files authored into it in `values/--parked.yaml`, which the glob does not match, and its next deploy restores them. -Depth: [deployment/README.md](deployment/README.md) and -[control-plane/governed-ops-design.md](control-plane/governed-ops-design.md). - -The second is close coupling of source, ingest, schema and transform, where -other stacks run separate ETL. Traditional pipelines split ingest, schema management, and transformation -across separate tools with separate config stores that drift apart. DFE keeps -them one artifact: a source definition carries the routing rule, the schema -binding, and the field-map bindings together, and every runtime stage reads -what the engine compiles from it. One definition compiles to the receiver's rules, the loader's table map, the table and its Sigma view. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- One source definition, compiled - accDescr: A source definition goes to dfe-engine, which compiles the receiver's rules, the loader's table map, the source's table and its Sigma view from it. - - src["Source definition"] - engine[["dfe-engine"]] - - subgraph out["Compiled from it"] - direction LR - rules["Receiver rules"] - tablemap["Loader table map"] - table[("Source table")] - view[("Sigma view")] - rules ~~~ tablemap ~~~ table ~~~ view - end - - src -->|definition| engine - engine --> out - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class table,view store - class engine control - class src,rules,tablemap part - style out fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 -``` - -*Figure 3. L4 Detail -- One source definition, compiled.* Up: [L3 Components -- dfe-engine](#inside-the-engine). - -One definition, read by every stage: the receiver routes an event by the -source's match rule, the loader writes it to that source's table, and the -`
_sigma` view projects the same columns under Sigma field names for -detection (`services/source_routing.py`, `schema/schema_ddl.py`). Routing that -disagrees with the sources is reported as drift and re-synced -(`appmgmt/routing.py`), so there is no separate ETL config to fall out of step. -Depth: [data-plane/README.md](data-plane/README.md). - -## The control pyramid - -The control path has four layers, each a narrower window onto the one below. The API is the single -control plane: dfe-ui and the generated CLI commands call it, and the CLI's `local` -and `schema` groups are break-glass paths that work with no engine. The API never -reaches past git to the cluster. Both clients reach the engine through its API, and the API reaches the cluster through git. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Control layers - accDescr: dfe-ui covers the common tasks and the dfe CLI every listed operation, both through the engine API. The API covers every control function and works through the charts and YAML in git. - - subgraph clients["Clients"] - ui["dfe-ui
common tasks"] - cli["dfe CLI
every operation"] - end - - api["Engine API
every control function"] - git[("Charts and YAML in git")] - - clients -->|API calls| api - api -->|commits| git - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + CALLERS --> ENGINE + ENGINE --> STATE class ui ui - class cli,api,git control - style clients fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 -``` - -*Figure 4. L4 Detail -- Control layers.* Up: [L2 Services -- DFE](#where-the-engine-sits-in-the-suite). - -## Inside the engine - -Inside the engine, the API server drives the registries and compilers, which commit through the gitops writer or apply DDL through the provisioners. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L3 Components -- dfe-engine - accDescr: Clients call the API server. It reads and writes the config registries and runs the compilers. The gitops writer commits registry changes and compiled overlays to the deploy repo, and the provisioners apply compiled DDL to ClickHouse. - - clients["Clients"] - - subgraph engine["dfe-engine"] - api["API server"] - registries["Config registries"] - compilers["Compilers"] - writer["GitOps writer"] - provisioners["Provisioners"] - end - - repo[("Deploy repo")] - ch[("ClickHouse")] - - clients -->|calls| api - api --> registries - api --> compilers - registries --> writer - compilers --> writer - compilers --> provisioners - writer -->|commits| repo - provisioners -->|DDL| ch + class keda ext + class cli control + class api,runner,shim part + class repo control + class ch store + class kafka bus classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -586,97 +92,85 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class ch store - class clients ui - class repo control - class api,registries,compilers,writer,provisioners part - style engine fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style CALLERS fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style ENGINE fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style STATE fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 5. L3 Components -- dfe-engine.* Up: [L2 Services -- DFE](#where-the-engine-sits-in-the-suite). +*Figure 1. L2 Services -- dfe-engine. The dashed grey box is outside DFE.* Up: [L2 Services -- DFE](https://github.com/hyperi-io/dfe-infra/blob/main/docs/architecture.md). Down: [L3 Components -- API server](api-server.md), [L3 Components -- Hunt runner](hunt-runner.md). -| Part | Code | What it does | -|---|---|---| -| API server | `api/` | FastAPI routers, the four auth paths and RBAC | -| Config registries | `source/registry.py`, `hunts/`, `fieldmap/registry.py`, `gitcrud/registry.py` | Read and write sources, hunts, rules, field maps, governance and app overlays | -| Compilers | `services/source_routing.py`, `appmgmt/routing.py`, `schema/schema_ddl.py`, `sigma/` | Receiver rules and loader table maps, table DDL and Sigma views, Sigma rules to SQL | -| GitOps writer | `gitcrud/`, `gitops/` | One attributed commit per change, with an `If-Match` revision check | -| Provisioners | `schema/phase.py`, `kafka/topics.py`, `hyperdx/`, `governance/ch/` | The dfe-schemas manifest and source tables, Kafka topics, dfe-hyperdx sources, ClickHouse roles and row policies | +## How data flows -- The config directory (`DFE_CONFIG_DIR`) is the deployment's own config: a - mounted volume, or a checkout of its config repo. The engine keeps its - registries and its auth store (`auth/`) there. -- The deploy repo is the GitOps hand-off. The engine commits app overlays to - `values/`, backing-service overlays to `infra/` and governance to - `governance/` (`gitcrud/registry.py`), and Argo CD reads them. With gitops - enabled it also holds the source definitions (`config/sources/`) and the hunt - definitions and detection rules (`config/hunts/`, `config/rules/`), one - attributed commit per change. A write that names the revision it read - (`If-Match`) answers `409 Conflict` with the current document when that - revision is stale, rather than overwrite a human or CI edit. A write with no - `If-Match` is not checked, so a client that needs the guard sends it. Standalone deployments fall back to - plain directories under the config directory. The hunt runner git-syncs the - hunt and rule directories, so on Kubernetes they are how a hunt reaches the pod - that runs it. Provider-agnostic: GitHub / GitLab external git, or the - in-cluster Forgejo fallback (see [deployment/README.md](deployment/README.md)). -- dfe-infra holds the charts and pinned versions that deploy the stack. The engine never writes to it. +1. dfe-ui and the dfe CLI call the API server on port 8000. The CLI is generated from the API's OpenAPI spec. Its `local` and `schema` groups work with no engine running, as the break-glass path. +2. The API server writes each change as one attributed commit to the deploy repo, and Argo CD syncs the repo into the cluster. A commit is not a deployment: an organisation's approval gate can sit between the commit and the sync. +3. The API server applies ClickHouse DDL itself, the dfe-schemas manifest at every start and a source's table when that source is deployed. On a tier with Kafka it creates the topics too. +4. The hunt runner reads hunts and rules from the deploy repo through a git-sync sidecar. It runs them inside ClickHouse, writes the matches to a detection table and sends an alert for each fire that matched. +5. KEDA polls the KEDA shim, which answers from ClickHouse. Scaling never waits on the API server. -## First principles (invariants) +## Components -1. Generic product, not an internal tool. Ships into a customer cluster - we never see. Nothing site-specific in code. -2. YAML + git is the single source of truth. No config database. Rust - services read config files directly. The engine reads/writes them through - GitCrud on the deploy repo, or a cached YAML directory (DirectoryConfigStore - from scalo) for registries that live off it. Identity - comes from the FILENAME, never a field inside the YAML (the store loads - YAML 1.1, where `off`/`yes`/`no` become booleans). -3. Engine writes config, never installs infrastructure. The hard - engine<->infra boundary: dfe-infra deploys backing services and owns - ApplicationSets. The engine commits config and overlays to the deploy - repo, applies DDL to ClickHouse itself, and turns dials through overlays. - If the engine is about to install ClickHouse or author an Argo Application, - the work belongs in dfe-infra. -4. Assume a bare cluster. Layer 0 detect-or-install brings - everything else: a default StorageClass, cert-manager, External Secrets, - and Argo CD, installed into `argocd` or, where one already runs, adopted. -5. ClickHouse is the only operational store. All non-gitops state - - events, results, watermarks, leases - lives in CH via - `effective_data_database`. The DocumentDB store is dfe-hyperdx's, never - DFE state. -6. A commit is not a deployment. The engine reports what it committed. - Argo (and any org approval gate) decides when it is live. -7. Fully operable from the repos alone. Engine and UI down = running - system unaffected, manageable by editing the repos directly. +| Part | Role | Page | +|---|---|---| +| API server | The registries, compilers, GitOps writer and provisioners behind `/api/v1` | [API server](api-server.md) | +| Hunt runner | Runs scheduled hunts in ClickHouse, coordinated through leases in ClickHouse | [Hunt runner](hunt-runner.md) | +| KEDA shim | Answers KEDA's `metrics-api` scaler from ClickHouse, and holds its last good value when a query fails | none | +| dfe CLI | Every API operation from a terminal, plus the offline `local` and `schema` groups | none | -The deployment is layered. Layer 0 platform baseline (Argo CD + detect-or-install operators) -> -Layer 1 backing services (ClickHouse, optional Kafka, DocumentDB) -> -Layer 2 apps, dfe-hyperdx among them, and config (digest-pinned thin charts -built from each app's deployment contract + engine-authored overlays). Four -tiers select defaults: **dfe-docker** (Compose), **slim** (gRPC, no Kafka), -**single** (+ single-broker Kafka) and **scale** (everything clustered). A -fifth, **mesh** (scale's replicas and ClickHouse cluster over gRPC, no broker), -is in the charts, but dfe-infra's bootstrap refuses to select it. dfe-hyperdx -ships in EVERY tier - it is the explore UI over all DFE data and the -self-monitoring sink, not an add-on. Full detail including the per-tier -composition table, deploy-repo providers, backing-service modes, and the -network model: [deployment/README.md](deployment/README.md). +All four ship in one image, `ghcr.io/hyperi-io/dfe-engine`, as the entry points `dfe-engine`, `dfe-hunt-runner`, `dfe-keda-shim` and `dfe` (`pyproject.toml`, `[project.scripts]`). The engine chart renders the API server. dfe-infra's `dfe-extras` chart renders the hunt runner and the shim as their own pods. -## Technology decisions +## External dependencies -| Decision | Choice | Rationale | -|----------|--------|-----------| -| Config storage | YAML + git (no DB) | human-readable, versioned, Rust reads directly | -| API host | dfe-engine FastAPI | one artifact | -| Auth | local JWT by default, plus OIDC, API key and disabled | OIDC through the engine's own relying party (`/api/v1/auth/oidc/{provider}/login`) | -| Conflict handling | optional `If-Match` revision + 409 | git-native, standard HTTP semantics | -| Deployment | Argo CD + Helm, thin chart + dfe-extras + overlay, multi-source | engine turns dials, appsets author apps | -| Platform baseline | Layer 0 detect-or-install | assume a bare cluster | -| Deploy repo | provider-agnostic (GitHub/GitLab/Forgejo) | no hard git-server dependency | -| Backing services | DFE-owned, mode-driven (single/cluster/external) | never piggyback shared infra | -| Secrets | External Secrets Operator for deployment secrets, `DFE_ENGINE_SECRETS_PROVIDER` for secrets the engine mints | ESO: customer Vault, or in-cluster generators on bare. Engine: file by default, OpenBao/Vault or AWS Secrets Manager | -| Observability | OTel -> ClickHouse -> dfe-hyperdx | single storage backend | -| Ingress / OIDC | Envoy Gateway (k8s) / oauth2-proxy (docker) | native OIDC at the edge | -| Expression language | CEL (Python + Rust), DFE subset | one standard, tiered for performance | -| Field mapping | YAML, two-tier (default + source) | human-editable, git-tracked, UI-editable | -| Stage transport | one per source, `bus` or `direct`, default from the profile | a flow is never mixed. The bus provider (Kafka) is a deployment fact, so a second provider joins without a model change | +| Dependency | Needed | Choices | +|---|---|---| +| ClickHouse | Always | In-cluster single node or cluster, or an external server | +| Git host for the deploy repo | On Kubernetes | GitHub, GitLab, or the in-cluster Forgejo | +| Argo CD | On Kubernetes | Installed by dfe-infra, or an existing one adopted | +| Secrets backend | Always | `file` (default), OpenBao or Vault, or AWS Secrets Manager (`DFE_ENGINE_SECRETS_PROVIDER`) | +| Kafka | single and scale tiers | In-cluster broker or cluster | +| Identity provider | When OIDC sign-in is on | Any OIDC provider. Local accounts and API keys need none | +| KEDA | When autoscaling is on | Installed by dfe-infra | +| Alert destinations | When a hunt names one | Any Apprise URL: Slack, email, PagerDuty, Teams and others | + +## Deployment variants + +| Variant | What changes for the engine | +|---|---| +| Docker (dfe-docker) | No Argo CD and no KEDA shim. The engine commits to a local deploy repo in its config volume | +| slim | No Kafka, so every source uses the direct gRPC transport. Saving a source that needs a fetcher or a transform is refused | +| single | Kafka with one broker. The engine creates each source's topics | +| scale | ClickHouse is a cluster, so the hunt coordination tables are replicated across its nodes | + +Each tier sets defaults, and a deployer overrides any one dial in the deploy repo's overlay. The hunt runner is one steady pod on every Kubernetes tier. [Deployment](deployment/README.md) has the full composition per tier. + +## Design considerations + +- Security. A request is authenticated by trusted OIDC headers, an API key or an ES384 bearer token, then checked against scoped RBAC. ClickHouse roles and row policies, which the engine provisions, keep each organisation to its own rows. A secret the engine mints lives in the secrets backend, and the API masks it on every read. +- Reliability. With the API server down, the data path, the hunt runner and the shim keep running, because each needs only ClickHouse. A runner or shim pod that is starting waits for the engine, which applies the schema at its own start. A write that names the revision it read (`If-Match`) gets `409 Conflict` when the file has changed, so a human or CI edit is never overwritten. +- Operations. The git history is the audit log. `/livez`, `/readyz` and `/metrics` answer on port 9090, apart from API traffic. Compiled routing that disagrees with the sources is reported as drift and re-synced. +- Performance. Hunt load runs in ClickHouse, so one runner pod covers most deployments. Per-fire caps keep a flood of matches to a bounded write. + +## Why it's built this way + +- Config is YAML in git, with no config database. It is reviewable and versioned, the Rust services read the files directly, and an operator can manage the system by editing the repo when the engine and UI are down. +- The engine writes config and never installs infrastructure. dfe-infra deploys the backing services and owns the Argo CD ApplicationSets. The one thing the engine applies itself is ClickHouse DDL, because it controls every ClickHouse object. +- A source is defined once. The receiver's routing rules, the loader's table map, the source's table and its Sigma view are all compiled from that definition, so there is no separate ETL config to drift out of step. +- ClickHouse is the only operational store. Hunt results, watermarks and leases all live there, so the runner works the same on Docker, where ClickHouse is the only store guaranteed. +- The hunt runner and the shim are separate pods on the engine image. Neither puts the API server on a hot path, and neither stops when it does. +- A file's name is its identity, never a field inside it. YAML 1.1 reads `off`, `yes` and `no` as booleans, so an id inside the file is not safe. + +## Diagram map + +- Figure 1. L2 Services -- dfe-engine, on this page + - [L3 Components -- API server](api-server.md) + - [L4 Detail -- One source definition, compiled](api-server.md#how-it-works) + - [L4 Detail -- Instance deploy path](api-server.md#interfaces) + - [L3 Components -- Hunt runner](hunt-runner.md) + +## What's next + +- Configuration: every setting is a `DFE_ENGINE_` environment variable, listed in `src/dfe_engine/settings.py` and the [README](../README.md#configuration). +- [Running dfe-engine locally](local-dev.md). +- [Control plane](control-plane/README.md): the API, auth, RBAC and governed GitOps. +- [Data plane](data-plane/README.md): sources, schemas, the query API and hunts. +- [Deployment](deployment/README.md), including [KEDA scaling](deployment/keda-scaling.md), which covers the shim's queries. +- [CONTRIBUTING.md](../CONTRIBUTING.md#context): where the code lives and what tends to bite. diff --git a/docs/auth-envoy-topology.md b/docs/auth-envoy-topology.md index d8cdd1b0..8198ea31 100644 --- a/docs/auth-envoy-topology.md +++ b/docs/auth-envoy-topology.md @@ -60,7 +60,7 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Token check at the edge accDescr: The browser calls the edge proxy, which routes API paths to dfe-engine, the rest to dfe-ui, and the embed to dfe-hyperdx. The engine publishes its JWKS, and dfe-hyperdx verifies the dfe_token cookie against it. @@ -68,6 +68,7 @@ flowchart LR proxy["Edge proxy"] subgraph apps["Apps"] + direction LR engine["dfe-engine"] ui["dfe-ui"] hyperdx["dfe-hyperdx"] @@ -81,6 +82,11 @@ flowchart LR proxy -->|embed| hyperdx engine -->|publishes| jwks hyperdx -.->|verifies| jwks + class browser ext + class jwks store + class ui,hyperdx ui + class engine control + class proxy part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -92,15 +98,10 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class browser ext - class jwks store - class ui,hyperdx ui - class engine control - class proxy part style apps fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Token check at the edge. The dotted arrow is a read.* Up: [DFE architecture](architecture.md). +*Figure 1. L4 Detail -- Token check at the edge. The dotted arrow is a read.* Up: [L2 Services -- dfe-engine](architecture.md). - Local login: `POST /api/v1/auth/login`, and the engine signs the JWT (`src/dfe_engine/api/v1/auth.py`). - External OIDC: the engine is the relying party. `/api/v1/auth/oidc/{provider}/login` redirects to the IdP, and the callback validates the IdP's token and mints an engine token (`src/dfe_engine/api/v1/oidc_login.py`). The IdP's token stays on the engine-to-IdP leg. diff --git a/docs/control-plane/README.md b/docs/control-plane/README.md index eb854252..2d034320 100644 --- a/docs/control-plane/README.md +++ b/docs/control-plane/README.md @@ -52,12 +52,15 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Control-plane repos accDescr: dfe-ui calls the engine API and dfe-schemas ships the schema trees the engine reads. The engine commits config to the deploy repo, which Argo CD syncs from dfe-infra's ApplicationSets. - ui["dfe-ui"] - schemas["dfe-schemas"] + subgraph inputs["Inputs"] + direction LR + ui["dfe-ui"] + schemas["dfe-schemas"] + end engine["dfe-engine"] @@ -69,6 +72,9 @@ flowchart LR schemas -->|schema trees| engine engine -->|commits| deploy deploy --> infra + class schemas store + class ui ui + class engine,deploy,infra control classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -80,12 +86,10 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class schemas store - class ui ui - class engine,deploy,infra control + style inputs fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Control-plane repos.* Up: [DFE architecture](../architecture.md). +*Figure 1. L4 Detail -- Control-plane repos.* Up: [L2 Services -- dfe-engine](../architecture.md). | Repo | Role in the layer | |---|---| @@ -155,17 +159,23 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Control-plane sources of truth accDescr: The engine reads the dfe-schemas trees and the RBAC roles, and commits overlays and config to the deploy repo. Argo CD syncs the deploy repo together with the charts and dfe-infra values. - schemas[("dfe-schemas")] - rbac[("RBAC roles")] + subgraph engreads["Read by the engine"] + direction LR + schemas[("dfe-schemas")] + rbac[("RBAC roles")] + end engine["dfe-engine"] - deploy[("Deploy repo")] - charts[("Charts and dfe-infra values")] + subgraph argoreads["Read by Argo CD"] + direction LR + deploy[("Deploy repo")] + charts[("Charts and dfe-infra values")] + end argo["Argo CD"] @@ -175,6 +185,8 @@ flowchart LR engine ~~~ charts deploy --> argo charts --> argo + class schemas,rbac store + class engine,deploy,charts,argo control classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -186,11 +198,11 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class schemas,rbac store - class engine,deploy,charts,argo control + style engreads fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style argoreads fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 2. L4 Detail -- Control-plane sources of truth.* Up: [DFE architecture](../architecture.md). +*Figure 2. L4 Detail -- Control-plane sources of truth.* Up: [L2 Services -- dfe-engine](../architecture.md). To change one, go through its writer: edit the routes and regenerate the spec, commit to dfe-schemas, turn the engine's dials, or edit roles through the API. Never hand-edit a generated artefact such as `openapi.json` or a compiled overlay. @@ -244,24 +256,34 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- openapi.json and its consumers accDescr: The FastAPI routes generate openapi.json and feed the dfe CLI and Swagger UI directly. dfe-ui types and the Prism mock read the committed file. routes["FastAPI routes"] - spec[("openapi.json")] - cli["dfe CLI"] - swagger["Swagger UI"] + subgraph fromroutes["From the routes"] + direction LR + spec[("openapi.json")] + cli["dfe CLI"] + swagger["Swagger UI"] + end - ui["dfe-ui types"] - mock["Prism mock"] + subgraph fromspec["From openapi.json"] + direction LR + ui["dfe-ui types"] + mock["Prism mock"] + end routes -->|generate| spec routes -->|app factory| cli routes -->|served| swagger spec -->|sync PR| ui spec -->|mounted| mock + class spec store + class ui ui + class cli control + class routes,swagger,mock part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -273,13 +295,11 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class spec store - class ui ui - class cli control - class routes,swagger,mock part + style fromroutes fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style fromspec fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 3. L4 Detail -- openapi.json and its consumers.* Up: [DFE architecture](../architecture.md). +*Figure 3. L4 Detail -- openapi.json and its consumers.* Up: [L2 Services -- dfe-engine](../architecture.md). - The `dfe` CLI builds its whole command tree at runtime from the installed engine's app factory, so an endpoint becomes a command with no CLI code. An endpoint opts out with `x-scalo.cli.hidden`. - dfe-ui copies both specs into its `@repo/dfe-engine-types` package through a PR the [sync workflow](type-safety-sync.md) opens, and generates its types from them. diff --git a/docs/control-plane/governed-ops-design.md b/docs/control-plane/governed-ops-design.md index 55013034..86a28a6f 100644 --- a/docs/control-plane/governed-ops-design.md +++ b/docs/control-plane/governed-ops-design.md @@ -53,15 +53,18 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Governed write path accDescr: dfe local, the Helm vars router, the backing services router and governance actions all write through GitCrud. GitCrud builds the message under the commit policy and commits through GitopsRepo, and Argo CD reconciles the commit. subgraph callers["Callers"] + direction LR local["dfe local"] helm["Helm vars router"] backing["Backing services router"] actions["Governance actions"] + local ~~~ helm + backing ~~~ actions end gitcrud["GitCrud"] @@ -75,6 +78,8 @@ flowchart LR gitcrud -->|message| policy gitcrud -->|commit| repo repo --> argo + class argo control + class local,helm,backing,actions,gitcrud,policy,repo part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -86,12 +91,10 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class argo control - class local,helm,backing,actions,gitcrud,policy,repo part style callers fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Governed write path. Argo CD is outside the engine.* Up: [L3 Components -- dfe-engine](../architecture.md). +*Figure 1. L4 Detail -- Governed write path. Argo CD is outside the engine.* Up: [L3 Components -- API server](../api-server.md). Tier 1 is raw var CRUD over the overlay files (`/api/v1/helm`), admin-grade because it can set anything. Tier 2 is curated actions (`/api/v1/governance`), each behind a permission of its name, so an operator can run `receiver-surge` without the keys to edit any var. RBAC binds to a resource class, an action or an operation, never to a field. @@ -179,7 +182,7 @@ sequenceDiagram repo-->>argo: New commit ``` -*Figure 2. L4 Detail -- Writing a helm var. Solid arrows are requests, dashed arrows are responses.* Up: [L3 Components -- dfe-engine](../architecture.md). +*Figure 2. L4 Detail -- Writing a helm var. Solid arrows are requests, dashed arrows are responses.* Up: [L3 Components -- API server](../api-server.md). The full path is `PUT /api/v1/helm/files/{name}/vars/{path}`, for example `receiver-default-values` and `keda.maxReplicaCount`. A production `DFE_ENGINE_ENV` in team mode routes the write to a review branch instead of `main`. Validation, the protected-var check and the commit live in the engine, so the CLI and the API take the same path. @@ -254,7 +257,7 @@ sequenceDiagram crud-->>op: Diff and commit SHA ``` -*Figure 3. L4 Detail -- Invoking an action. Solid arrows are requests, dashed arrows are responses.* Up: [L3 Components -- dfe-engine](../architecture.md). +*Figure 3. L4 Detail -- Invoking an action. Solid arrows are requests, dashed arrows are responses.* Up: [L3 Components -- API server](../api-server.md). A `ProtectedPolicy` is a list of locked `cls:name:path` globs that Tier 1 respects too, unless the caller holds the override grant. A set or revert at a parent, or a set below a locked scalar, is rejected like a write to the var itself, and a whole-document write is gated on the leaves it adds, changes or removes. Three policies ship: `baseline` locks image references, `storage-layout` locks the data-layer modes, the storage model, the `objectStore`, `tieredBlock` and `tieredObject` dial blocks and the disk size and class, and `sizing-locks` locks the Kafka sizing inputs, the KRaft controller layout and the cloud. @@ -336,6 +339,8 @@ flowchart LR user -->|tier| tier user -->|tenant| tenant user -->|pinned| setting + class user,tier,tenant,setting store + class group part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -347,11 +352,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class user,tier,tenant,setting store - class group part ``` -*Figure 4. L4 Detail -- ClickHouse identity.* Up: [L3 Components -- dfe-engine](../architecture.md). +*Figure 4. L4 Detail -- ClickHouse identity.* Up: [L3 Components -- API server](../api-server.md). A group binding ties an RBAC group's ClickHouse user to one tier and at most one org. A tier is grants, a settings profile and a quota (`dfe__role`, `_profile`, `_quota`). The tenant axis is one shared role plus a restrictive row policy on each `_org_id` table, reading the caller's `SQL_current_tenant_id`, which an org-tied user holds read-only, so a `SETTINGS` override in a query is rejected. The catalogue of tiers and service roles is data in dfe-schemas (`roles/clickhouse.yaml`), editable through the `ch_tiers` and `ch_service_roles` classes, and `ChRbacReconciler` renders it into idempotent DDL and runs it. The hunt runner connects as the `dfe_hunt_runner` service role, with grants and no quota (`src/dfe_engine/hunts/connect.py`), and the catalogue's `hunt_tier_*` tiers set per-query memory and time limits and an hourly quota for a group bound to one. @@ -407,13 +410,14 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Hunt claim and run accDescr: When a fire is due, runner workers race to claim it in hunt_lease. The winner runs the window query, writes matched rows to the results table and advances hunt_watermark, which sets the next window. due(["Fire due"]) subgraph workers["Runner workers"] + direction LR w1["Worker"] w2["Worker"] end @@ -422,8 +426,11 @@ flowchart LR run["Run the window query"] - results[("Results table")] - watermark[("hunt_watermark")] + subgraph written["Written by the run"] + direction LR + results[("Results table")] + watermark[("hunt_watermark")] + end due --> workers workers -->|claim| lease @@ -431,6 +438,8 @@ flowchart LR run --> results run --> watermark watermark -.->|next window| workers + class lease,results,watermark store + class due,w1,w2,run part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -442,9 +451,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class lease,results,watermark store - class due,w1,w2,run part style workers fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style written fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 5. L4 Detail -- Hunt claim and run. The dotted arrow is the next run reading the watermark.* Up: [The DFE control-plane layer](README.md). @@ -659,6 +667,7 @@ flowchart LR surv --> chsafe surv --> nodouble nodouble --> crash + class surv,rbac,chsafe,nodouble,crash part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -670,7 +679,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class surv,rbac,chsafe,nodouble,crash part ``` *Figure 8. L4 Detail -- What governed ops rests on.* Up: [The DFE control-plane layer](README.md). diff --git a/docs/control-plane/oidc-rbac-architecture.md b/docs/control-plane/oidc-rbac-architecture.md index 986fda77..62421f29 100644 --- a/docs/control-plane/oidc-rbac-architecture.md +++ b/docs/control-plane/oidc-rbac-architecture.md @@ -89,18 +89,27 @@ flowchart LR accTitle: L4 Detail -- OIDC sign-in layers accDescr: The external IdP authenticates the user and hands the engine an id_token. The engine's relying party validates it and re-mints an engine token. The provider adapter turns the raw group references into group ids, the group files map them to roles and org ids, and those feed the control plane and the data plane. - idp["External IdP"] - rp["Relying party"] - adapt["Provider adapter"] - files[("Group files")] - ctrl["Control plane
roles to actions"] - data["Data plane
orgs to rows"] - - idp -->|id_token| rp - rp -->|groups| adapt - adapt -->|ids| files - files --> ctrl - files --> data + subgraph SIGNIN["Sign-in"] + direction TB + idp["External IdP"] + rp["Relying party"] + adapt["Provider adapter"] + idp -->|id_token| rp + rp -->|groups| adapt + end + subgraph RESOLVE["Roles and orgs"] + direction TB + files[("Group files")] + ctrl["Control plane
roles to actions"] + data["Data plane
orgs to rows"] + files --> ctrl + files --> data + end + SIGNIN -->|ids| RESOLVE + class idp ext + class files store + class ctrl control + class rp,adapt,data part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -112,10 +121,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class idp ext - class files store - class ctrl control - class rp,adapt,data part + style SIGNIN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style RESOLVE fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- OIDC sign-in layers. The provider adapter is the one layer that knows the provider.* Up: [rbac.md](rbac.md). diff --git a/docs/control-plane/rbac-groups.md b/docs/control-plane/rbac-groups.md index cc958f93..32066863 100644 --- a/docs/control-plane/rbac-groups.md +++ b/docs/control-plane/rbac-groups.md @@ -120,24 +120,36 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Group sync decision accDescr: A remote group linked to a stored group through this provider updates that group. One with an id linked to another provider is skipped as provider_mismatch. Otherwise a free name creates a linked group, and a name already held is skipped as name_taken. - linked{"Linked here?"} - other{"Id linked elsewhere?"} - update(["Update description"]) - free{"Name free?"} - mismatch(["Skip provider_mismatch"]) - create(["Create, linked"]) - taken(["Skip name_taken"]) - - linked -->|no| other + subgraph q1["Linked here"] + direction LR + linked{"Linked here?"} + update(["Update description"]) + end + subgraph q2["Linked elsewhere"] + direction LR + other{"Id linked elsewhere?"} + mismatch(["Skip provider_mismatch"]) + end + subgraph q3["Name free"] + direction LR + free{"Name free?"} + create(["Create, linked"]) + taken(["Skip name_taken"]) + end + + q1 -->|no| q2 linked -->|yes| update - other -->|no| free + q2 -->|no| q3 other -.->|yes| mismatch free -->|yes| create free -.->|no| taken + class linked,other,free decision + class mismatch,taken error + class update,create part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -149,9 +161,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class linked,other,free decision - class mismatch,taken error - class update,create part + style q1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style q2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style q3 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 2. L4 Detail -- Group sync decision. Dotted arrows end in a skip.* Up: [rbac.md](rbac.md). diff --git a/docs/control-plane/rbac-sign-in.md b/docs/control-plane/rbac-sign-in.md index 3b118553..44a3fd3b 100644 --- a/docs/control-plane/rbac-sign-in.md +++ b/docs/control-plane/rbac-sign-in.md @@ -52,34 +52,37 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Identity resolution chain accDescr: OIDC headers, a password login and a bearer JWT each bind an account, and the account's groups resolve. An API key resolves the groups the key lists. The groups give the roles, grants and orgs, and the authorisation check allows or denies the action. - hdr["OIDC headers"] - login["Password login"] - jwt["Bearer JWT"] - key["API key"] - - acct["Account groups"] - keyg["Key groups"] - - res["Roles, grants and orgs"] - - authz{"Action allowed?"} - - allow(["Allowed"]) - deny(["Denied"]) - - hdr --> acct - login --> acct - jwt --> acct - key --> keyg - acct --> res - keyg --> res - res --> authz - authz -->|yes| allow - authz -.->|no| deny + subgraph CREDS["Who is asking"] + direction LR + hdr["OIDC headers"] + login["Password login"] + jwt["Bearer JWT"] + key["API key"] + acct["Account groups"] + keyg["Key groups"] + hdr --> acct + login --> acct + jwt --> acct + key --> keyg + end + subgraph DECIDE["What they may do"] + direction LR + res["Roles, grants and orgs"] + authz{"Action allowed?"} + allow(["Allowed"]) + deny(["Denied"]) + res --> authz + authz -->|yes| allow + authz -.->|no| deny + end + CREDS --> DECIDE + class authz decision + class deny error + class hdr,login,jwt,key,acct,keyg,res,allow part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -91,9 +94,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class authz decision - class deny error - class hdr,login,jwt,key,acct,keyg,res,allow part + style CREDS fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style DECIDE fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Identity resolution chain. The dotted arrow is the refusal.* Up: [rbac.md](rbac.md). diff --git a/docs/control-plane/rbac.md b/docs/control-plane/rbac.md index bacebb7e..2594fa3c 100644 --- a/docs/control-plane/rbac.md +++ b/docs/control-plane/rbac.md @@ -70,14 +70,17 @@ flowchart LR resolve(["AuthContext
groups to roles"]) - which -->|trusted headers| hdr - which -->|API key| key + which -->|headers| hdr + which -->|key| key which -->|bearer| tok which -->|auth off| root which -.->|none| reject hdr --> resolve key --> resolve tok --> resolve + class which decision + class reject error + class hdr,key,tok,root,resolve part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -89,9 +92,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class which decision - class reject error - class hdr,key,tok,root,resolve part ``` *Figure 1. L4 Detail -- Four authentication paths. The dotted arrow is the refusal, and `auth off` needs a dev `DFE_ENGINE_ENV`.* Up: [Control plane](README.md). @@ -206,6 +206,8 @@ flowchart LR files -->|hashes, roles| api sec -->|keys, passwords| api + class acc,grp,keys,roles,sec store + class api control classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -217,8 +219,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class acc,grp,keys,roles,sec store - class api control style files fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` @@ -373,6 +373,7 @@ flowchart LR end ctrl ~~~ dataplane + class admin,infra_admin,infra_viewer,data_analyst,data_analyst_viewer,dfe_operator,data_viewer,org_viewer part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -384,7 +385,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class admin,infra_admin,infra_viewer,data_analyst,data_analyst_viewer,dfe_operator,data_viewer,org_viewer part style ctrl fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style dataplane fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` @@ -681,7 +681,7 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- OIDC provider sync accDescr: The group sync reads a provider from the provider registry, calls the provider's adapter, which lists groups from Google, Entra ID or Okta, and writes groups to the GroupStore. It records the sync's time and status back on the provider. @@ -691,18 +691,24 @@ flowchart LR adapt["Provider adapter"] gstore[("GroupStore")] + regstat[("Provider status
in the provider registry")] subgraph idps["Directory APIs"] + direction LR google["Google"] entra["Entra ID"] okta["Okta"] + google ~~~ entra ~~~ okta end reg --> sync sync --> adapt sync --> gstore adapt -->|list groups| idps - sync -.->|status| reg + sync -.->|status| regstat + class google,entra,okta ext + class reg,gstore,regstat store + class sync,adapt part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -714,9 +720,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class google,entra,okta ext - class reg,gstore store - class sync,adapt part style idps fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` @@ -859,27 +862,36 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Who reads as whom accDescr: A caller without query execute is refused. A caller with a platform grant reads as the platform reader, which sees every org. Any other caller that resolves to exactly one registered org reads as that org's pinned user, and the rest are refused with no_single_org. - exec{"query:execute?"} - - plat{"Platform grant?"} - forbid(["403 forbidden"]) - - one{"Exactly one org?"} - reader(["Platform reader"]) - - pinned(["Org's pinned user"]) - nosingle(["403 no_single_org"]) + subgraph q1["Execute"] + direction LR + exec{"query:execute?"} + forbid(["403 forbidden"]) + end + subgraph q2["Platform"] + direction LR + plat{"Platform grant?"} + reader(["Platform reader"]) + end + subgraph q3["One org"] + direction LR + one{"Exactly one org?"} + pinned(["Org's pinned user"]) + nosingle(["403 no_single_org"]) + end - exec -->|yes| plat + q1 -->|yes| q2 exec -.->|no| forbid - plat -->|no| one + q2 -->|no| q3 plat -->|yes| reader one -->|yes| pinned one -.->|no| nosingle + class reader,pinned store + class exec,plat,one decision + class forbid,nosingle error classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -891,9 +903,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class reader,pinned store - class exec,plat,one decision - class forbid,nosingle error + style q1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style q2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style q3 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 6. L4 Detail -- Who reads as whom. Green ends are ClickHouse users, orange are refusals.* Up: [Control plane](README.md). @@ -1181,6 +1193,7 @@ flowchart LR end auth ~~~ oidc ~~~ outside + class engine,roles_mod,accounts,groups,oidc_reg,oidc_sync,adapters,deps,chrbac,orgreg part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -1192,7 +1205,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class engine,roles_mod,accounts,groups,oidc_reg,oidc_sync,adapters,deps,chrbac,orgreg part style auth fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style oidc fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style outside fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 diff --git a/docs/control-plane/synthetic-data.md b/docs/control-plane/synthetic-data.md index 38ca9a1e..259d1cdf 100644 --- a/docs/control-plane/synthetic-data.md +++ b/docs/control-plane/synthetic-data.md @@ -52,15 +52,21 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Synthetic data paths accDescr: dfe-schemas meta packs feed the schema event factory, and sampler rows or lines feed the sample event factory, which scrubs identities. Both feed a Poisson-paced stream that posts events to dfe-receiver's ingest endpoint. - packs[("dfe-schemas
meta packs")] - sample[("Sampler rows or lines")] + subgraph inputs["Inputs"] + direction LR + packs[("dfe-schemas
meta packs")] + sample[("Sampler rows or lines")] + end - schemaf["SchemaEventFactory"] - samplef["SampleEventFactory
identity scrub"] + subgraph factories["Factories"] + direction LR + schemaf["SchemaEventFactory"] + samplef["SampleEventFactory
identity scrub"] + end stream["Poisson-paced stream"] @@ -71,6 +77,9 @@ flowchart LR schemaf --> stream samplef --> stream stream --> receiver + class receiver ingest + class packs,sample store + class schemaf,samplef,stream part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -82,9 +91,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class receiver ingest - class packs,sample store - class schemaf,samplef,stream part + style inputs fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style factories fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Synthetic data paths. dfe-receiver is outside the engine.* Up: [The DFE control-plane layer](README.md). diff --git a/docs/control-plane/ui-api-guide.md b/docs/control-plane/ui-api-guide.md index e4c16a76..b712f0ef 100644 --- a/docs/control-plane/ui-api-guide.md +++ b/docs/control-plane/ui-api-guide.md @@ -240,7 +240,7 @@ There is no progress stream: poll the operation, and read `metadata` for progres | POST | `/transforms/compile` | Compile VRL/CEL transform | | POST | `/transforms/test` | Test transform against sample data | | GET | `/system/version` | Stack, engine, ui and every pinned app's version (the console footer) | -| GET | `/system/deployment` | Profile, transports, mesh, whether compiled routing reaches the running apps, and versions -- what a pane must not guess | +| GET | `/system/deployment` | Profile, transports, whether compiled routing reaches the running apps, and versions -- what a pane must not guess | | GET | `/deployment/admin-links` | The admin UIs this deployment runs: name, purpose, URL and `up`/`down`/`unknown`. `admin` and `infra_admin` only (`deployment:admin_links:read`) | | GET | `/sources/{name}/signals` | Records a minute and last arrival for one source | | GET | `/system/settings` | Current settings | diff --git a/docs/data-evolution.md b/docs/data-evolution.md index dfd2ddb9..4c64dbc8 100644 --- a/docs/data-evolution.md +++ b/docs/data-evolution.md @@ -68,7 +68,7 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Data evolution stages accDescr: In stage 0 dfe-receiver or dfe-fetcher lands data in the main table as _json. Promoting fields gives the source a meta schema in stage 1. In stage 2 the source gets a routing condition and its own topic and table, with an optional transform. @@ -79,13 +79,16 @@ flowchart LR meta["Meta schema"] end subgraph s2["2 DFE source"] + direction LR route["Routing condition"] own[("Own topic and table")] + route --> own end - main -.->|promote| meta - meta -.->|own source| route - route --> own + s0 -.->|promote| s1 + s1 -.->|own source| s2 + class main,own store + class meta,route part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -97,14 +100,12 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class main,own store - class meta,route part style s0 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style s1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style s2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Data evolution stages. Dotted arrows are optional steps, and dfe-receiver or dfe-fetcher fills the main table.* Up: [DFE architecture](architecture.md). Down: [L4 Detail -- How a table gets built](#how-a-table-gets-built), [L4 Detail -- What a transform does](#what-a-transform-is-actually-for). +*Figure 1. L4 Detail -- Data evolution stages. Dotted arrows are optional steps, and dfe-receiver or dfe-fetcher fills the main table.* Up: [L2 Services -- dfe-engine](architecture.md). Down: [L4 Detail -- How a table gets built](#how-a-table-gets-built), [L4 Detail -- What a transform does](#what-a-transform-is-actually-for). Nothing forces you down the stages. A source can sit in stage 0 forever, and stage 3 below drops `_json` from a source you know fully. @@ -207,6 +208,8 @@ flowchart LR meta -->|select fields| derived derived --> table header --> table + class table store + class meta,derived,header part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -218,8 +221,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class table store - class meta,derived,header part ``` *Figure 2. L4 Detail -- How a table gets built.* Up: [L4 Detail -- Data evolution stages](#the-path). @@ -341,6 +342,8 @@ flowchart LR raw -->|record| xform xform -->|columns| cols + class xform xform + class raw,cols store classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -352,8 +355,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class xform xform - class raw,cols store ``` *Figure 3. L4 Detail -- What a transform does.* Up: [L4 Detail -- Data evolution stages](#the-path). diff --git a/docs/data-plane/README.md b/docs/data-plane/README.md index 0224ed77..bdfcb994 100644 --- a/docs/data-plane/README.md +++ b/docs/data-plane/README.md @@ -54,7 +54,7 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- What a source definition compiles into accDescr: The engine compiles one source definition into the receiver's routing rules, the loader's table routing and the table with its standard views. The hunt runner and the query API read the table and its views. @@ -62,19 +62,27 @@ flowchart LR engine["dfe-engine
compilers"] - receiver["Receiver rules"] - loader["Loader table routing"] - views[("Table and standard views")] + subgraph compiled["Compiled"] + direction LR + receiver["Receiver rules"] + loader["Loader table routing"] + views[("Table
and standard views")] + end - hunts["Hunt runner"] - query["Query API"] + subgraph readers["Readers"] + direction LR + hunts["Hunt runner"] + query["Query API"] + end source --> engine engine --> receiver engine --> loader engine --> views - views --> hunts - views --> query + views --> readers + class views store + class engine control + class source,receiver,loader,hunts,query part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -86,12 +94,11 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class views store - class engine control - class source,receiver,loader,hunts,query part + style compiled fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style readers fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- What a source definition compiles into.* Up: [L3 Components -- dfe-engine](../architecture.md). +*Figure 1. L4 Detail -- What a source definition compiles into.* Up: [L3 Components -- API server](../api-server.md). ## Documents in this area diff --git a/docs/data-plane/external-pipelines.md b/docs/data-plane/external-pipelines.md index 0fa3b5d0..f6ceb73b 100644 --- a/docs/data-plane/external-pipelines.md +++ b/docs/data-plane/external-pipelines.md @@ -52,21 +52,25 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart TB +flowchart LR accTitle: L4 Detail -- External pipeline plug-in points accDescr: As an origin, your pipeline produces to the source's landing topic and the loader reads it. As a transform, the receiver produces to the landing topic, your pipeline reads it and produces to the transformed topic, and the loader reads that. subgraph origin["External origin"] - direction LR + direction TB pipe1["Your pipeline"] --> land1[("acme-logs_land")] --> loader1["dfe-loader"] end subgraph transform["External transform"] - direction LR + direction TB receiver["dfe-receiver"] --> land2[("acme-logs_land")] --> pipe2["Your pipeline"] --> load2[("acme-logs_load")] --> loader2["dfe-loader"] end origin ~~~ transform + class pipe1,pipe2 ext + class receiver ingest + class loader1,loader2 load + class land1,land2,load2 bus classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -78,10 +82,6 @@ flowchart TB classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class pipe1,pipe2 ext - class receiver ingest - class loader1,loader2 load - class land1,land2,load2 bus style origin fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style transform fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` diff --git a/docs/data-plane/hunt-runner-scaling.md b/docs/data-plane/hunt-runner-scaling.md index f909f2dd..cadb2021 100644 --- a/docs/data-plane/hunt-runner-scaling.md +++ b/docs/data-plane/hunt-runner-scaling.md @@ -10,7 +10,7 @@ The hunt runner is pull-based and multi-pod. The default deployment is ONE steady pod with no ScaledObject, because the runner is a thin orchestrator and all hunt load runs in ClickHouse. One pod covers the large majority of deployments. -Autoscaling is a beta, off by default (`huntRunner.autoscaling.enabled`). Leave it off. It reads the due backlog from the `hunt_schedule` table and nothing in the charts writes that table, so the backlog always reads 0. With autoscaling on, the runner never scales out, and at `minReplicaCount: 0` no hunt ever fires. To try it anyway, run `dfe-hunt-runner materialise` after every hunt-config change. +Autoscaling is a beta, off by default (`huntRunner.autoscaling.enabled`). Leave it off. It reads the due backlog from the `hunt_schedule` table. `hunt_schedule` is written only through `publish_schedule`: by `dfe-hunt-runner materialise` at config-deploy time, and by the phase smoother when smoothing is on. The charts never run `materialise`, so with smoothing off the backlog always reads 0. With autoscaling on, the runner never scales out, and at `minReplicaCount: 0` no hunt ever fires. To try it anyway, run `dfe-hunt-runner materialise` after every hunt-config change. When on, KEDA scales the pod count on the deterministic-due backlog described below, read through the fail-safe `dfe-keda-shim`. KEDA never asks the engine. @@ -61,13 +61,13 @@ Once deployed, the running system needs only ClickHouse: - Workers read hunt definitions from the gitops config, coordinate through the coordination tables and run queries against ClickHouse. - The shim reads `hunt_schedule` and the coordination tables from ClickHouse. It is a separate, distinctly labelled pod on the engine container image with its own entrypoint (`dfe-keda-shim`), not the live engine API. -- `hunt_schedule` is written at config-deploy time by `dfe-hunt-runner materialise`, a one-shot, idempotent command that rewrites it from the hunt config. The charts do not run it. +- `hunt_schedule` is written only through `publish_schedule`: by `dfe-hunt-runner materialise` at config-deploy time, and by the phase smoother when smoothing is on. `materialise` is a one-shot, idempotent command that rewrites the table from the hunt config, and the charts do not run it. If the engine pod is down, running hunt-runner pods keep firing hunts. What you lose is the ability to change configs, which needs the engine anyway. A pod that is starting is the exception: the runner and the shim each have a `wait-for-engine` init container (dfe-infra `helm/library/dfe-common/templates/_waitfor.tpl`) that holds the pod until the engine reports ready, because the engine applies the ClickHouse schema at its own start. A scale-out or a restart while the engine is down waits for it. ClickHouse is DFE's single operational store and the runner holds to that, which is also why the same runner works on a non-k8s single deploy (dfe-docker), where only ClickHouse is guaranteed present. Every worker and the shim must see the same coordination tables. On the scale tier's multi-node cluster behind a round-robin load balancer, a plain `ReplacingMergeTree` lands on one node, and workers connected to other nodes see an empty or absent table (`UNKNOWN_DATABASE`, missing leases). The schema phase therefore creates the coordination tables as `ReplicatedReplacingMergeTree` inside a Replicated database on every deployment, so lease, watermark and schedule state is shared across nodes (). -`schedule.publish_schedule(ch, db, specs)` is the only writer of `hunt_schedule`, and `dfe-hunt-runner materialise` is its only caller. It is idempotent and diff-aware: it tombstones hunts that vanished from config. On Kubernetes the table stays empty until an operator runs the command after each hunt-config change. A hunt added while scaled to zero then still gets a row, so KEDA can wake a worker for it. +`schedule.publish_schedule(ch, db, specs)` is the only function that writes `hunt_schedule`. Its two callers are `dfe-hunt-runner materialise` and the phase smoother (`smoothing.py`). It is idempotent and diff-aware: it tombstones hunts that vanished from config. On Kubernetes with smoothing off, the table stays empty until an operator runs the command after each hunt-config change. A hunt added while scaled to zero then still gets a row, so KEDA can wake a worker for it. ## The ScaledObject @@ -121,20 +121,30 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Hunt runner autoscaling accDescr: The KEDA shim reads the due hunts from ClickHouse's hunt_schedule and coordination tables and serves the due count. KEDA's metrics-api trigger reads it and sets the hunt runner Deployment's replicas. On a failure the shim serves the last good value, or holds cold. subgraph CH["ClickHouse"] - hs[("hunt_schedule +
hunt_lease/watermark/state")] + hs[("hunt_schedule
and hunt_lease, watermark, state")] end - shim["dfe-keda-shim
GET /keda/hunt-backlog"] - ko["KEDA metrics-api trigger
ScaledObject: dfe-hunt-runner-scaler"] - hr["dfe-hunt-runner Deployment
replicas: N"] - hs -->|"due_query(db)"| shim - shim -->|"{value: due_count}"| ko - ko -->|"AverageValue, target = targetBacklog"| hr - shim -.->|"on failure: last-good cache / cold-hold"| ko + subgraph SHIM["Shim"] + direction LR + shim["dfe-keda-shim
GET /keda/hunt-backlog"] + cache("Last good value
or cold_hold") + shim -.->|on failure| cache + end + subgraph KEDA["KEDA"] + direction LR + ko["metrics-api trigger
dfe-hunt-runner-scaler"] + hr["dfe-hunt-runner
replicas: N"] + ko -->|AverageValue| hr + end + CH -->|due_query| SHIM + SHIM -->|due_count| KEDA + class hs store + class ko control + class shim,hr,cache part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -146,12 +156,13 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class hs store - class ko control - class shim,hr part style CH fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style SHIM fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style KEDA fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` +*Figure 1. L4 Detail -- Hunt runner autoscaling. The dotted arrow is the failure path.* Up: [L3 Components -- Hunt runner](../hunt-runner.md). + With `huntRunner.autoscaling.enabled`, the Deployment's `replicas` field is omitted, because the ScaledObject owns the count, and a `dfe-hunt-runner-scaler` ScaledObject is rendered beside it: ```yaml diff --git a/docs/data-plane/query-api.md b/docs/data-plane/query-api.md index 067f1187..fe11c82d 100644 --- a/docs/data-plane/query-api.md +++ b/docs/data-plane/query-api.md @@ -173,7 +173,7 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Query routes accDescr: A client calls the query routes. A view execution goes to the ViewExecutor, which runs the view on ClickHouse. A raw query goes to a datasource adapter, which reads ClickHouse or lists S3, MinIO or a local directory. @@ -181,11 +181,17 @@ flowchart LR api["Query routes"] - executor["ViewExecutor"] - adapters["Datasource adapters"] + subgraph handlers["Handlers"] + direction LR + executor["ViewExecutor"] + adapters["Datasource adapters"] + end - ch[("ClickHouse")] - storage[("S3, MinIO or files")] + subgraph stores["Stores"] + direction LR + ch[("ClickHouse")] + storage[("S3, MinIO or files")] + end client --> api api -->|view| executor @@ -193,6 +199,9 @@ flowchart LR executor --> ch adapters --> ch adapters --> storage + class client ext + class ch,storage store + class api,executor,adapters part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -204,12 +213,11 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class client ext - class ch,storage store - class api,executor,adapters part + style handlers fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style stores fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 2. L4 Detail -- Query routes. The dashed grey box is the caller.* Up: [L3 Components -- dfe-engine](../architecture.md). +*Figure 2. L4 Detail -- Query routes. The dashed grey box is the caller.* Up: [L3 Components -- API server](../api-server.md). The executor looks each view up in the `ViewCatalog`, which discovers the views from ClickHouse and caches them for 60 seconds. Every route authenticates by JWT, OIDC headers or API key first. @@ -397,6 +405,7 @@ flowchart TB end offset ~~~ keyset + class o1,o2,o3,k1,k2,k3 part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -408,7 +417,6 @@ flowchart TB classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class o1,o2,o3,k1,k2,k3 part style offset fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style keyset fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` @@ -639,20 +647,29 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Query route authorisation accDescr: An authenticated request is checked for the route's scope. Without it the answer is 403. With it the executor injects org_id from the auth context and runs the query on ClickHouse. - request(["Authenticated request"]) - scope{"Route scope?"} - inject["Inject org_id"] - deny(["403"]) - run(["Run on ClickHouse"]) + subgraph check["Check"] + direction LR + request(["Authenticated request"]) + scope{"Route scope?"} + deny(["403"]) + end + subgraph runq["Run"] + direction LR + inject["Inject org_id"] + run(["Run on ClickHouse"]) + end request --> scope - scope -->|yes| inject + check -->|yes| runq scope -.->|no| deny inject --> run + class scope decision + class deny error + class request,inject,run part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -664,9 +681,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class scope decision - class deny error - class request,inject,run part + style check fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style runq fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 4. L4 Detail -- Query route authorisation. The dotted arrow is the refusal.* Up: [Data plane](README.md). diff --git a/docs/data-plane/schema-classes.md b/docs/data-plane/schema-classes.md index 2813a385..55eb9470 100644 --- a/docs/data-plane/schema-classes.md +++ b/docs/data-plane/schema-classes.md @@ -72,29 +72,30 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Schema build accDescr: SchemaBuilderV2 loads the header profile's columns and the source's schema columns, composes them, and validates them against the TypeRegistry. DDLGenerator writes the CREATE TABLE and ViewGenerator the standard views, and both go into the SchemaBuildResult. - profile["Header profile"] - schema["Source schema
meta, derived, additional"] - - compose["SchemaLoader
compose"] - - validate["Validate against TypeRegistry"] - - ddl["DDLGenerator
CREATE TABLE"] - views["ViewGenerator
standard views"] - - result["SchemaBuildResult"] - - profile --> compose - schema --> compose - compose --> validate - validate --> ddl - validate --> views - ddl --> result - views --> result + subgraph LOAD["Load and validate"] + direction LR + profile["Header profile"] + schema["Source schema
meta, derived, additional"] + compose["SchemaLoader
compose"] + validate["Validate
against TypeRegistry"] + profile --> compose + schema --> compose + compose --> validate + end + subgraph GEN["Generate"] + direction LR + ddl["DDLGenerator
CREATE TABLE"] + views["ViewGenerator
standard views"] + result["SchemaBuildResult"] + ddl --> result + views --> result + end + LOAD --> GEN + class profile,schema,compose,validate,ddl,views,result part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -106,7 +107,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class profile,schema,compose,validate,ddl,views,result part + style LOAD fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style GEN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Schema build. Validation errors go into the result and do not stop the build.* Up: [Data plane](README.md). diff --git a/docs/data-plane/schema-sync.md b/docs/data-plane/schema-sync.md index c6be0d11..3be690b5 100644 --- a/docs/data-plane/schema-sync.md +++ b/docs/data-plane/schema-sync.md @@ -58,13 +58,15 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Engine and loader schema contract accDescr: The dfe-schemas wheel and the source definition feed the engine. The engine renders the DDL and applies it to the ClickHouse table. dfe-loader reads the table's columns and comments back from system.columns. subgraph defs["Definitions"] + direction LR schemas["dfe-schemas
wheel"] source["Source definition"] + schemas ~~~ source end engine["dfe-engine"] @@ -76,6 +78,10 @@ flowchart LR defs --> engine engine -->|DDL| table table -->|system.columns| loader + class loader load + class schemas,table store + class engine control + class source part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -87,10 +93,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class loader load - class schemas,table store - class engine control - class source part style defs fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` diff --git a/docs/data-plane/schema.md b/docs/data-plane/schema.md index f3960a0b..3e36aa37 100644 --- a/docs/data-plane/schema.md +++ b/docs/data-plane/schema.md @@ -709,18 +709,25 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Elastic template conversion accDescr: An Elastic index template goes to the template converter, which maps Elastic types to primitives and analyzers to use cases. It writes a source meta schema in YAML, which a source definition then names. - template["Elastic index template"] - converter["Template converter"] - schema["Source meta schema YAML"] - source["Source definition"] - - template --> converter - converter --> schema - schema --> source + subgraph CONVERT["Convert"] + direction LR + template["Elastic index template"] + converter["Template converter"] + template --> converter + end + subgraph USE["Use"] + direction LR + schema["Source meta schema
YAML"] + source["Source definition"] + schema --> source + end + CONVERT -->|writes| USE + class template ext + class converter,schema,source part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -732,8 +739,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class template ext - class converter,schema,source part + style CONVERT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style USE fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Elastic template conversion. The dashed box comes from outside DFE.* Up: [Data plane](README.md). diff --git a/docs/data-plane/sigma.md b/docs/data-plane/sigma.md index 24e50df6..0825824f 100644 --- a/docs/data-plane/sigma.md +++ b/docs/data-plane/sigma.md @@ -66,19 +66,36 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Sigma detection pipeline accDescr: Providers pull rules from SigmaHQ git, a local directory and the Valhalla API, and normalise them into the catalogue. The operator selects rules, and propagation generates DFE rules and hunts, which the hunt runner runs against each source's Sigma view. - GIT[("SigmaHQ git")] --> P["providers"] - LOCAL[("local dir")] --> P - VAL[("Valhalla API")] --> P - P -->|normalise + upsert| CAT[("catalogue
id-keyed gitcrud")] - CAT -->|operator selects| SEL["selection"] - SEL -->|propagate| RULES["DFE rules + hunts"] - RULES --> HR["hunt runner"] - SRC["source _json"] --> VIEW["{source}_sigma view"] - HR --> VIEW + subgraph COLLECT["Collect"] + direction LR + GIT[("SigmaHQ git")] --> P["providers"] + LOCAL[("local dir")] --> P + VAL[("Valhalla API")] --> P + end + subgraph CATALOGUE["Catalogue"] + CAT[("catalogue
id-keyed gitcrud")] + end + subgraph SELECT["Select"] + direction LR + SEL["selection"] + RULES["DFE rules + hunts"] + SEL -->|propagate| RULES + end + subgraph RUN["Run"] + direction LR + HR["hunt runner"] --> VIEW["{source}_sigma view"] + SRC["source _json"] --> VIEW + end + COLLECT -->|normalise + upsert| CATALOGUE + CATALOGUE -->|operator selects| SELECT + SELECT --> RUN + class GIT,VAL ext + class LOCAL,CAT,SRC,VIEW store + class P,SEL,RULES,HR part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -90,11 +107,14 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class GIT,VAL ext - class LOCAL,CAT,SRC,VIEW store - class P,SEL,RULES,HR part + style COLLECT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style CATALOGUE fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style SELECT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style RUN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` +*Figure 1. L4 Detail -- Sigma detection pipeline. Dashed grey boxes are outside DFE.* Up: [Data plane](README.md). + ## Providers - pull external feeds A provider fetches Sigma rules from an external feed and normalises each via diff --git a/docs/data-plane/source-definition.md b/docs/data-plane/source-definition.md index 82e52a15..086175eb 100644 --- a/docs/data-plane/source-definition.md +++ b/docs/data-plane/source-definition.md @@ -236,18 +236,20 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Source table columns accDescr: The source table takes the common header profile's columns first. Then it takes the meta schema's columns, narrowed by the derived schema when there is one. Then it takes the additional fields. subgraph cols["Columns, in order"] direction LR - header["Common header"] ~~~ meta["Meta schema, or its derived selection"] ~~~ additional["Additional fields"] + header["Common header"] ~~~ meta["Meta schema
or its derived selection"] ~~~ additional["Additional fields"] end table[("Source table")] cols --> table + class table store + class header,meta,additional part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -259,8 +261,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class table store - class header,meta,additional part style cols fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` diff --git a/docs/data-plane/source-detections.md b/docs/data-plane/source-detections.md index 6702278f..2fd95e48 100644 --- a/docs/data-plane/source-detections.md +++ b/docs/data-plane/source-detections.md @@ -119,18 +119,25 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Rule to detection table accDescr: A hunt groups rules and carries the schedule. The hunt runner runs each rule over the hunt's window and writes every matched row to the detection table. - rules["Rules"] - hunt["Hunt"] - runner["Hunt runner"] - detection[("detection table")] - - rules --> hunt - hunt --> runner - runner -->|matched rows| detection + subgraph DEFINE["Define"] + direction LR + rules["Rules"] + hunt["Hunt"] + rules --> hunt + end + subgraph RUN["Run"] + direction LR + runner["Hunt runner"] + detection[("detection table")] + runner -->|matched rows| detection + end + DEFINE --> RUN + class detection store + class rules,hunt,runner part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -142,8 +149,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class detection store - class rules,hunt,runner part + style DEFINE fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style RUN fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Rule to detection table.* Up: [Data plane](README.md). diff --git a/docs/data-plane/source-flow.md b/docs/data-plane/source-flow.md index 634d91b3..bfad9c73 100644 --- a/docs/data-plane/source-flow.md +++ b/docs/data-plane/source-flow.md @@ -12,7 +12,7 @@ A source names where its records come in, whether they are transformed, and wher ## The flow on each transport -Two transports carry the same three stages. The bus (Kafka on the `single` and `scale` profiles) holds records between stages, so a stage can be down and nothing is lost. Direct gRPC (`slim`, `mesh`) stores nothing between stages, so the receiver's own buffer is the only slack. +Two transports carry the same three stages. The bus (Kafka on the `single` and `scale` profiles) holds records between stages, so a stage can be down and nothing is lost. Direct gRPC (`slim`) stores nothing between stages, so the receiver's own buffer is the only slack. On the bus, a source's records go through its landing topic, its transform and its transformed topic to the loader. @@ -62,19 +62,24 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Source flow on the bus accDescr: The receiver pool and the source's fetcher write the source's landing topic. The source's transform reads it and writes the transformed topic, which the loader pool reads. The archiver reads the landing topic. subgraph input["Input"] + direction LR receiver["Receiver pool"] fetcher["Fetcher
one per source"] + receiver ~~~ fetcher end land[("source_land")] - transform["Transform
one per source"] - archiver["Archiver"] + subgraph landreaders["Read the landing topic"] + direction LR + transform["Transform
one per source"] + archiver["Archiver"] + end load[("source_load")] @@ -85,6 +90,10 @@ flowchart LR land -.-> archiver transform --> load load --> loader + class receiver,fetcher ingest + class transform xform + class archiver,loader load + class land,load bus classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -96,11 +105,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class receiver,fetcher ingest - class transform xform - class archiver,loader load - class land,load bus style input fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style landreaders fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Source flow on the bus. With no transform the loader reads `source_land` itself.* Up: [Data plane](README.md). @@ -169,6 +175,9 @@ flowchart LR receiver -->|matched| loader receiver -.->|archived| archiver loader --> ch + class receiver ingest + class loader,archiver load + class ch store classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -180,14 +189,11 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class receiver ingest - class loader,archiver load - class ch store ``` *Figure 2. L4 Detail -- Source flow on direct gRPC. The loader and the archiver each take the push on a scalo Push listener.* Up: [Data plane](README.md). -Direct runs the receiver to loader path alone (figure 2). A fetcher or a transform is one app per source, and addressing a pool of those over gRPC needs the mesh, which is not GA. On a direct deployment a source that needs one is rejected at save and left out of the compile. `GET /api/v1/system/deployment` reports `transports.per_source_apps: false` there, and each `GET /api/v1/sources/catalogue` entry has `runnable`, `runnable_reason` and `runnable_intakes` from the same resolver as the save. +Direct runs the receiver to loader path alone (figure 2). A fetcher or a transform is one app per source, and direct gRPC has no way to address a pool of those. On a direct deployment a source that needs one is rejected at save and left out of the compile. `GET /api/v1/system/deployment` reports `transports.per_source_apps: false` there, and each `GET /api/v1/sources/catalogue` entry has `runnable`, `runnable_reason` and `runnable_intakes` from the same resolver as the save. The archiver keeps the record as it arrived, before any transform. On the bus it reads the landing topic. On direct the receiver sends the record to it beside the loader. `archive: true` is rejected at save, naming the profile, where apps.yaml does not offer dfe-archiver in the profile, or does not run it by default (`default_in`) and no archiver instance is deployed. A source stored with it before still reads and compiles. @@ -204,7 +210,6 @@ A source declares `transport: bus` or `transport: direct`. Left unset, it takes | slim | direct | gRPC | one replica. The KEDA ceiling of two would pin senders to pods | | single | bus | Kafka topics on one broker | consumer groups | | scale | bus | Kafka topics on a broker cluster | consumer groups | -| mesh (in the charts, refused by dfe-infra's bootstrap) | direct | gRPC | a listener per stage pool, see [../deployment/transports.md](../deployment/transports.md) | The source records bus or direct and nothing more. Which bus (Kafka) and which direct protocol (gRPC) are deployment facts, so a second bus provider joins behind the same `transport: bus` value with no change to the source model. @@ -241,7 +246,7 @@ Every compiled block lands in the deploy repo under `values/`, Argo CD applies i The receiver's destinations are a named set: the loader and the archiver are destinations, and a rule sends a matched record to one by name. The fetcher has the same named set as `output.routes` over its default destination. -A transform is an apps.yaml entry with a `source_binding` and its `transports`, and a chart that exposes the transport dial and the Push Service. No engine release is involved. dfe-transform-vrl, dfe-transform-vector and dfe-transform-elastic run a Push listener beside their bus consumer, but a transform is one deployment per source, so none runs on a direct deployment until the mesh is GA. dfe-transform-elastic's `source.name` selects one of the compiled-in transforms its contract lists. +A transform is an apps.yaml entry with a `source_binding` and its `transports`, and a chart that exposes the transport dial and the Push Service. No engine release is involved. dfe-transform-vrl, dfe-transform-vector and dfe-transform-elastic run a Push listener beside their bus consumer, but a transform is one deployment per source, so none runs on a direct deployment. dfe-transform-elastic's `source.name` selects one of the compiled-in transforms its contract lists. A transform can be a pipeline you run yourself. `engine: external` deploys nothing, and your pipeline reads `_land` in `dfe-external-` and writes `_load` ([external-pipelines.md](external-pipelines.md)). @@ -320,7 +325,7 @@ Every app rolls on every config change, so nothing depends on which app can relo ## What the deployment is, and what a source is doing -`GET /api/v1/system/deployment` answers what this deployment is: the profile the deployer set as `DFE_ENGINE_PROFILE`, the one transport it carries a source on, whether the stage pools sit behind mesh listeners, whether compiled routing reaches the running apps (`applies_routing`), and the pinned versions of the stack, the engine, dfe-ui and each app. The stack comes from the deploy repo's pins, else the published stack whose image digests match `DFE_ENGINE_STACK_IMAGES`, else the `DFE_ENGINE_STACK_VERSION` label, and `source` says which (`deploy-repo`, `registry`, `deployment`, `engine`). `DFE_ENGINE_STACK_RESOLVE=false` skips the registry read on an air-gapped site, and `DFE_ENGINE_STACK_MANIFEST_REPOSITORY` points it at a mirror. `GET /api/v1/system/version` reads the same function, so the footer and the deployment card agree. +`GET /api/v1/system/deployment` answers what this deployment is: the profile the deployer set as `DFE_ENGINE_PROFILE`, the one transport it carries a source on, whether compiled routing reaches the running apps (`applies_routing`), and the pinned versions of the stack, the engine, dfe-ui and each app. The stack comes from the deploy repo's pins, else the published stack whose image digests match `DFE_ENGINE_STACK_IMAGES`, else the `DFE_ENGINE_STACK_VERSION` label, and `source` says which (`deploy-repo`, `registry`, `deployment`, `engine`). `DFE_ENGINE_STACK_RESOLVE=false` skips the registry read on an air-gapped site, and `DFE_ENGINE_STACK_MANIFEST_REPOSITORY` points it at a mirror. `GET /api/v1/system/version` reads the same function, so the footer and the deployment card agree. `GET /api/v1/sources/{name}/signals` answers what one source is doing: `records_per_min` and `last_seen` over five minutes, read from the loader's `loader_messages_by_table_total` counter in the otel database, plus `landed_in_default`. The counter is per table, so `landed_in_default` says the number covers the whole shared landing table, as it does for a fetcher pushing to the main topic. A reading with no series behind it is null and the route still answers 200, because a zero would claim a measurement nobody took. diff --git a/docs/deployment/README.md b/docs/deployment/README.md index c7df027b..9ff50fe0 100644 --- a/docs/deployment/README.md +++ b/docs/deployment/README.md @@ -12,7 +12,7 @@ The pages in this area: | Doc | Covers | |---|---| | [backing-services.md](backing-services.md) | per-backend swap matrix | -| [transports.md](transports.md) | profile x transport, the Kafka provider swap, the mesh tier's Envoy balancer and receiver buffers | +| [transports.md](transports.md) | profile x transport, the Kafka provider swap and receiver buffers | | [keda-scaling.md](keda-scaling.md) | CPU plus ScalingPressure via dfe-keda-shim, on by default for the ingest apps | | [observability-standard.md](observability-standard.md) | one OTel destination, modes, log schema | | [supply-chain-pinning.md](supply-chain-pinning.md) | pick-latest-per-rules then SHA-pin | @@ -68,23 +68,27 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart TB +flowchart LR accTitle: L4 Detail -- Deployment layers accDescr: Layer 0 brings a StorageClass, cert-manager and External Secrets, Argo CD and the cluster operators such as KEDA. Layer 1 deploys ClickHouse, an optional Kafka, the DocumentDB store and the OTel collector. Layer 2 deploys the DFE apps from the engine's overlay values. subgraph l0["Layer 0 platform"] - direction LR + direction TB storage["StorageClass"] --> certs["cert-manager, ESO"] --> argo["Argo CD"] --> ops["Operators"] end subgraph l1["Layer 1 backing services"] - direction LR + direction TB ch[("ClickHouse")] ~~~ kafka[("Kafka")] ~~~ docdb[("DocumentDB")] ~~~ otel["OTel collector"] end subgraph l2["Layer 2 apps"] - direction LR + direction TB overlay[("Overlay values")] --> apps["DFE apps"] end l0 --> l1 --> l2 + class kafka bus + class ch,docdb,overlay store + class argo control + class storage,certs,ops,otel,apps part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -96,16 +100,12 @@ flowchart TB classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class kafka bus - class ch,docdb,overlay store - class argo control - class storage,certs,ops,otel,apps part style l0 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style l1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style l2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Deployment layers. Read each layer left to right, then down.* Up: [DFE architecture](../architecture.md). +*Figure 1. L4 Detail -- Deployment layers. Read each layer top to bottom, then the next layer to the right.* Up: [L2 Services -- dfe-engine](../architecture.md). Layer 0 is the platform baseline. The deployment assumes a bare cluster and nothing more. `bootstrap/bootstrap.sh` (in dfe-infra) lays down the baseline with a **detect-or-install** probe for every @@ -151,10 +151,10 @@ one config each). KEDA scaling is folded into each app chart - see ## Deployment tiers -Four selectable tiers, smallest to largest, and `mesh` beside them. The first is +Four selectable tiers, smallest to largest. The first is Docker, and the rest are Kubernetes profiles selected by the cluster secret's `dfe.hyperi.io/profile`. Transport -and the balancer per tier: [transports.md](transports.md). +per tier: [transports.md](transports.md). | Tier | Platform | Transport | Kafka | Sizing | Fits | |------|----------|-----------|-------|--------|-------| @@ -162,11 +162,8 @@ and the balancer per tier: [transports.md](transports.md). | **slim** | k8s | gRPC (receiver->loader direct) | **no** | minimum-viable + safety | bare-minimum k8s. Smallest | | **single** | k8s | Kafka | **yes** (single broker) | reliability | one node of everything, with Kafka | | **scale** | k8s | Kafka | **yes** (multi-broker cluster) | reliability | everything clustered (ReplicatedMergeTree + dedicated Keeper) | -| **mesh** | k8s | gRPC, Envoy listener per stage pool | **no** | reliability | in the charts but never run live: dfe-infra's `bootstrap.sh` stops on `DFE_PROFILE=mesh` ([dfe-infra#875](https://github.com/hyperi-io/dfe-infra/issues/875)). Scale's replicas and ClickHouse cluster with no broker, see [transports.md](transports.md#buffering-without-a-bus) | -A tier sets defaults, and a deployer overrides any single dial. `mesh` -composes as `scale` below with Kafka and Kafbat off and, like slim, no -fetchers or transforms. +A tier sets defaults, and a deployer overrides any single dial. ## Default composition per k8s tier diff --git a/docs/deployment/backing-services.md b/docs/deployment/backing-services.md index a93e0514..0d0881e3 100644 --- a/docs/deployment/backing-services.md +++ b/docs/deployment/backing-services.md @@ -87,6 +87,8 @@ flowchart LR cfg -->|selects| eng eng -->|talks to| be + class be store + class cfg,eng control classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -98,8 +100,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class be store - class cfg,eng control ``` *Figure 1. L4 Detail -- Backing-service binding.* Up: [Deployment](README.md). @@ -159,19 +159,20 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart TB +flowchart LR accTitle: L4 Detail -- Size and platform axes - accDescr: Size runs from dfe-docker through slim, single and scale to mesh and sets the composition. Platform runs on-prem, AWS, GCP and Azure and sets which product fills each backing service. + accDescr: Size runs from dfe-docker through slim and single to scale and sets the composition. Platform runs on-prem, AWS, GCP and Azure and sets which product fills each backing service. subgraph size["Size: composition"] - direction LR - docker["dfe-docker"] ~~~ slim["slim"] ~~~ single["single"] ~~~ scale["scale"] ~~~ mesh["mesh"] + direction TB + docker["dfe-docker"] ~~~ slim["slim"] ~~~ single["single"] ~~~ scale["scale"] end subgraph platform["Platform: which product"] - direction LR + direction TB onprem["On-prem or Rancher"] ~~~ aws["AWS"] ~~~ gcp["GCP"] ~~~ azure["Azure"] end size ~~~ platform + class docker,slim,single,scale,onprem,aws,gcp,azure part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -183,14 +184,13 @@ flowchart TB classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class docker,slim,single,scale,mesh,onprem,aws,gcp,azure part style size fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style platform fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 2. L4 Detail -- Size and platform axes.* Up: [Deployment](README.md). -Size is the four dfe-infra profiles (`argocd/values/profile-slim|single|scale|mesh.yaml`) +Size is the three dfe-infra profiles (`argocd/values/profile-slim|single|scale.yaml`) plus `dfe-docker`, which is not Kubernetes. It changes replica counts, single node against cluster and which optional apps run, never which product a row binds to. Platform is the swap (`argocd/values/local|rancher|aws|gcp|azure.yaml`). @@ -290,6 +290,10 @@ flowchart LR choice -->|openbao, vault| bao choice -->|aws| sm choice -.->|gcp, azure| refused + class file,bao,sm store + class choice decision + class refused error + class eng control classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -301,10 +305,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class file,bao,sm store - class choice decision - class refused error - class eng control ``` *Figure 3. L4 Detail -- Secrets provider choice. The dotted arrow is the refusal.* Up: [Deployment](README.md). @@ -405,6 +405,10 @@ flowchart LR ch ~~~ core ~~~ observe broker ~~~ persource ~~~ secrets end + class broker bus + class ch,secrets store + class observe ui + class core,persource part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -416,10 +420,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class broker bus - class ch,secrets store - class observe ui - class core,persource part style docker fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` diff --git a/docs/deployment/e2e-acceptance.md b/docs/deployment/e2e-acceptance.md index 62d899b6..fd11fdad 100644 --- a/docs/deployment/e2e-acceptance.md +++ b/docs/deployment/e2e-acceptance.md @@ -59,7 +59,7 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Acceptance scenarios accDescr: A local dev hybrid is not sufficient for acceptance. The full Kubernetes deploy reads its config and credentials from the deployment's private config and secrets backend, and the standalone docker run feeds what it learns back to dfe-docker. Both must pass. @@ -67,6 +67,7 @@ flowchart LR hybrid["Local dev hybrid"] subgraph accept["Acceptance, both must pass"] + direction LR k8s["Clustered k8s
dfe-infra deploy"] docker["Single-host docker"] end @@ -76,6 +77,8 @@ flowchart LR cfg -->|config, creds| k8s hybrid -.->|not sufficient| accept docker -->|feedback| dfedocker + class cfg store + class hybrid,k8s,docker,dfedocker part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -87,8 +90,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class cfg store - class hybrid,k8s,docker,dfedocker part style accept fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` diff --git a/docs/deployment/keda-scaling.md b/docs/deployment/keda-scaling.md index baf0e90f..b6630682 100644 --- a/docs/deployment/keda-scaling.md +++ b/docs/deployment/keda-scaling.md @@ -87,14 +87,20 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- KEDA triggers accDescr: metrics-server feeds the CPU trigger on every app. The scaling_pressure gauge rows in otel_metrics_gauge feed dfe-keda-shim, which serves the pressure on the ingest apps. KEDA's ScaledObject takes the higher of the two and sets the app Deployment's replicas. - ms["metrics-server"] - gauge[("otel_metrics_gauge")] - cpu["CPU trigger
every app"] - shim["dfe-keda-shim"] + subgraph signals["Signals"] + direction LR + ms["metrics-server"] + gauge[("otel_metrics_gauge")] + end + subgraph triggers["Triggers"] + direction LR + cpu["CPU trigger
every app"] + shim["dfe-keda-shim"] + end so["ScaledObject"] dep["App Deployment"] @@ -103,6 +109,9 @@ flowchart LR cpu -->|utilisation| so shim -->|pressure| so so -->|replicas| dep + class gauge store + class so control + class ms,cpu,shim,dep part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -114,9 +123,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class gauge store - class so control - class ms,cpu,shim,dep part + style signals fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style triggers fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- KEDA triggers.* Up: [Deployment](README.md). @@ -156,7 +164,7 @@ The docs site fills the table below from the dfe-infra chart renders at the stac | App | KEDA | Triggers by default | |-----|------|---------------------| | dfe-receiver / dfe-loader | on | CPU ~70% and scalo ScalingPressure (shim, target 70) | -| dfe-archiver, dfe-transform-elastic / vrl / vector | on | CPU ~70%, plus ScalingPressure on the single, scale and mesh profiles | +| dfe-archiver, dfe-transform-elastic / vrl / vector | on | CPU ~70%, plus ScalingPressure on the single and scale profiles | | dfe-ui | on | CPU ~70% | @@ -176,7 +184,6 @@ Replica bounds come from dfe-infra `argocd/values/common.yaml` and the tier prof | slim | 1 | 2 | 75 (unused, slim clears the trigger) | | single | 1 | 3 | 70 | | scale | 2 | 12 | 70 | -| mesh | 2 | 10 | 70 | No tier scales to zero. diff --git a/docs/deployment/observability-standard.md b/docs/deployment/observability-standard.md index 4835e69c..1b23e64e 100644 --- a/docs/deployment/observability-standard.md +++ b/docs/deployment/observability-standard.md @@ -53,7 +53,7 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Telemetry path accDescr: DFE apps push OTLP to the collector gateway, which scrapes their metrics as well, and their stdout reaches it through the collector daemonset. The gateway's ClickHouse exporter writes the otel tables in the DFE database. Observe and the engine's KEDA shim read those tables. @@ -72,6 +72,9 @@ flowchart LR collector -->|exporter| otel otel --> observe otel --> shim + class otel store + class observe ui + class apps,ds,gw,shim part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -83,9 +86,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class otel store - class observe ui - class apps,ds,gw,shim part style collector fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` diff --git a/docs/deployment/transports.md b/docs/deployment/transports.md index 8f287ce7..00501af9 100644 --- a/docs/deployment/transports.md +++ b/docs/deployment/transports.md @@ -1,7 +1,7 @@ @@ -17,9 +17,8 @@ Pick the profile by what you need to survive. | smallest footprint, a stage may drop records when another is down | slim | one pod per stage, direct gRPC, no broker | | records survive a stage outage on one node | single | one broker holds them | | records survive outages and the stack scales out | scale | broker cluster, replicated ClickHouse, KEDA | -| scale out without running a broker | mesh | receiver and loader pools behind per-pool listeners, no fetchers or transforms. In the charts but never run live: dfe-infra's `bootstrap.sh` stops on it ([dfe-infra#875](https://github.com/hyperi-io/dfe-infra/issues/875)) | -slim and mesh run on direct, where the receiver pushes records to the loader over gRPC, and to the archiver as well when a source is archived. A source that needs a fetcher or a transform is rejected at save there, so it needs single or scale. +slim runs on direct, where the receiver pushes records to the loader over gRPC, and to the archiver as well when a source is archived. A source that needs a fetcher or a transform is rejected at save there, so it needs single or scale. ## The bus is Kafka, the broker is a swap @@ -79,18 +78,20 @@ flowchart LR single[("One broker
StatefulSet")] subgraph deployed["Operator-deployed"] - direction LR + direction TB strimzi[("Strimzi
default")] ~~~ redpanda[("Redpanda
licence gate")] end subgraph supplied["Supplied broker"] - direction LR + direction TB msk[("MSK
provisioned")] ~~~ confluent[("Confluent Cloud")] ~~~ rpcloud[("Redpanda Cloud")] end mode -->|single| single mode -->|cluster| deployed mode -->|external| supplied + class single,strimzi,redpanda,msk,confluent,rpcloud bus + class mode decision classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -102,8 +103,6 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class single,strimzi,redpanda,msk,confluent,rpcloud bus - class mode decision style deployed fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style supplied fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` @@ -114,119 +113,11 @@ Values in the deploy repo select the broker. `kafka.provider` chooses which oper ## Buffering without a bus -On the direct transport no stage stores records, and no fetcher runs, so the receiver is the one stage that holds them. What it holds depends on the receiver's acknowledgement mode, and dfe-infra's `apps/dfe-receiver/profile-mesh.yaml` sets the figures for both. +On the direct transport no stage stores records, and no fetcher runs, so the receiver is the one stage that holds them. What it holds depends on the receiver's acknowledgement mode. With acknowledgements on, the default, the receiver builds no buffer or spool. It holds each request until the loader confirms delivery. Once that hold expires the sender gets a 503 and retries from its copy. -With acknowledgements off, the buffer and spool apply. mesh sets `buffer.memory_limit` to 3 GiB of the pod's 4Gi limit, so back-pressure starts at 2.4 GiB, and turns on `buffer.spillover` with 20 GiB of node-local spool per replica. A loader outage shorter than that is invisible to senders, and a longer one back-pressures them (503) until the loader returns. A record held there has had its 2xx, so it is lost if the pod dies, the trade for turning acknowledgements off. - -The knobs are the receiver's (`config.example.yaml`, `buffer:`), so a deployer tunes them in the overlay like any other dial. - -## The mesh tier balances between pools with a listener per pool - -Status: design built, never run against a live cluster. The charts, the `mesh` profile and the engine's pool addressing are on main, and dfe-infra's `bootstrap.sh` stops on `DFE_PROFILE=mesh` ([dfe-infra#875](https://github.com/hyperi-io/dfe-infra/issues/875)). - -Every stage on `mesh` is a pool of replicas. A Kubernetes Service in front of a pool balances per connection, and gRPC holds long connections, so one sender would pin one pod. The profile places a listener in front of each pool. Senders address the listener and every request is balanced. One chart helper renders the same listener for every pool as Gateway API resources (a Gateway listener plus a GRPCRoute), so the balancer is a GatewayClass value: Envoy Gateway by default, because the deployment runs it for its external routes, or a cloud provider's Gateway API class, where the swap is the class name alone. - -On `mesh` one listener balances each pool, receivers in front and loaders behind. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Mesh pool listeners - accDescr: Senders reach the receiver listener, which balances each request across the receiver pool. The receivers push to the loader listener, which balances across the loader pool, and the loaders write the ClickHouse cluster. - - rlisten["Receiver listener"] - - subgraph rpool["Receiver pool"] - r1["Receiver 1"] - r2["Receiver 2"] - end - - llisten["Loader listener"] - - subgraph lpool["Loader pool"] - l1["Loader 1"] - l2["Loader 2"] - end - - ch[("ClickHouse cluster")] - - rlisten --> rpool - rpool --> llisten - llisten --> lpool - lpool --> ch - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class r1,r2 ingest - class l1,l2 load - class ch store - class rlisten,llisten part - style rpool fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 - style lpool fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 -``` - -*Figure 2. L4 Detail -- Mesh pool listeners.* Up: [Deployment](README.md). - -All three transforms have a Push listener, but no transform runs on direct, so none sits in this chain - see [../data-plane/source-flow.md](../data-plane/source-flow.md). - -On the direct transport no stage stores records, and no fetcher runs, so the receiver is the one stage that holds them. What it holds depends on the receiver's acknowledgement mode, and dfe-infra's `argocd/values/apps/dfe-receiver/profile-mesh.yaml` sets the mesh figures below. - -With acknowledgements on, the default, the receiver builds no buffer or spool. It holds each request until the loader confirms delivery. Once that hold expires the sender gets a 503 and retries from the copy it kept. +With acknowledgements off, the receiver's memory buffer and spool apply. A loader outage shorter than they hold is invisible to senders, and a longer one back-pressures them (503) until the loader returns. A record held there has had its 2xx, so it is lost if the pod dies. That is the trade for turning acknowledgements off. -With acknowledgements off, the buffer and spool apply. mesh sets `buffer.memory_limit` to 3 GiB of the pod's 4Gi limit, so back-pressure starts at 2.4 GiB, and turns on `buffer.spillover` with 20 GiB of node-local spool per replica. A loader outage shorter than that is invisible to senders, and a longer one back-pressures them (503) until the loader returns. A record held there has had its 2xx, so it is lost if the pod dies. That is the trade for turning acknowledgements off. +The knobs are the receiver's (dfe-receiver `config.example.yaml`, `buffer:`), so a deployer sizes them in the overlay like any other dial. -The knobs are the receiver's (dfe-receiver `config.example.yaml`, `buffer:`), so a deployer tunes them in the overlay like any other dial. diff --git a/docs/hunt-runner.md b/docs/hunt-runner.md new file mode 100644 index 00000000..601ce9ac --- /dev/null +++ b/docs/hunt-runner.md @@ -0,0 +1,140 @@ +# Hunt runner + +The hunt runner is the part of [dfe-engine](architecture.md) that runs each scheduled hunt inside ClickHouse and tells someone when it matched. + +| | | +|---|---| +| Repo | [hyperi-io/dfe-engine](https://github.com/hyperi-io/dfe-engine) | +| Image and chart | `ghcr.io/hyperi-io/dfe-engine`, command `dfe-hunt-runner run`. dfe-infra chart `helm/charts/dfe-extras`, `templates/engine-hunt-runner.yaml` | +| Config reference | `DFE_ENGINE_HUNTS_` environment variables (`src/dfe_engine/settings.py`), and the chart's `huntRunner` values | +| Endpoints | None. The runner has no HTTP listener | +| Metrics | `dfe_hunt_*`, pushed over OTLP ([observability standard](deployment/observability-standard.md#the-hunt-runner-pushes-its-metrics)) | + +A hunt needs only the deploy repo and ClickHouse to fire. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart LR + accTitle: L3 Components -- Hunt runner + accDescr: A git-sync sidecar brings the hunts and rules from the deploy repo to the hunt runner. Its scheduler claims due fires, the worker runs them, and the notifier sends alerts. The runner writes to ClickHouse and the alert destinations. + repo[("Deploy repo
hunts and rules")] + subgraph RUNNER["Hunt runner"] + direction TB + sched("Scheduler
due fires, leases") + worker("Worker
windowed INSERT") + notifier("Notifier
one message a fire") + sched --> worker --> notifier + end + subgraph OUT["Writes to"] + direction TB + ch[("ClickHouse
detections, state")] + dest("Alert destinations
Apprise URLs") + ch ~~~ dest + end + repo -->|git-sync| RUNNER + RUNNER --> OUT + class repo control + class sched,worker,notifier part + class ch store + class dest ext + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style RUNNER fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style OUT fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 1. L3 Components -- Hunt runner. The dashed grey box is outside DFE.* Up: [L2 Services -- dfe-engine](architecture.md). Down: none. + +## How it works + +1. The runner loads each hunt file from the synced `config/hunts/` directory, and the file's name is the hunt's id. A hunt that names rules gets one `INSERT ... SELECT` per rule, compiled at load, so a hunt made in dfe-ui runs with no hand-written SQL. +2. Every poll (15 seconds, `huntRunner.poll`) the scheduler finds the fires that are due. A hunt fires once per interval at a stable offset inside the first 80% of it (`hunts.phase_spread_fraction`), so hunts do not all start at once. +3. A due fire waits until every ClickHouse replica holds its window, then the scheduler claims a lease on it in ClickHouse. At most 8 fires run at once across all runner pods (`huntRunner.cap`), so ClickHouse is never flooded. +4. The worker runs the fire's statements over its window, writing matches to the hunt's target table, `detection` by default. It advances the watermark only after they commit. +5. The notifier sends one message per fire whose `alerts` trigger holds. A failed send is logged and the fire stands, because its rows are already written. + +## Interfaces + +- In: hunt and rule YAML from the deploy repo's `config/hunts/` and `config/rules/`, through a git-sync sidecar. +- In: alert destination URLs, read from the engine's secrets backend on every fire. +- ClickHouse: the source tables it reads, the detection table it writes, and the coordination tables `hunt_lease`, `hunt_watermark`, `hunt_state`, `hunt_run`, `hunt_runner_heartbeat` and `hunt_schedule`. +- Out: Apprise messages to Slack, email, PagerDuty, Teams and the other services Apprise covers. + +## Scaling and failure + +- One steady pod covers most deployments, since the query load runs in ClickHouse. A second pod competes for the same leases and nothing runs twice. +- A pod that dies mid-fire loses its lease after 300 seconds, and another pod re-runs the window from the last committed watermark. Nothing is lost and nothing is skipped. +- A ClickHouse that is down or full never ends the process. Each tick that hits it is skipped until the next poll. +- A failed fire keeps its watermark, so it stays due. It is retried after 30 seconds, doubling to 600. +- A rule writes at most 1,000 matches a fire (`hunts.max_detections_per_run`), then one summary row with the true count. +- With the API server down, a running pod keeps firing hunts. A pod that is starting waits for it. + +## Key metrics + +| Metric | Healthy | +|---|---| +| `dfe_hunt_runs_total{outcome}` | Rising, mostly `success` | +| `dfe_hunt_backlog` | Near 0 between fires | +| `dfe_hunt_fire_failures_total{reason}` | Flat | +| `dfe_hunt_clickhouse_unavailable_total` | Flat | +| `dfe_hunt_detections_capped_total` | Flat. A rise means a rule matches more than its cap | + +## Related + +- [Hunt-runner scaling](data-plane/hunt-runner-scaling.md): the leases, the schedule table and KEDA. +- [Hunt schedule smoothing](data-plane/hunt-schedule-smoothing.md). +- [How a hunt match reaches a person](data-plane/hunt-match-delivery.md). +- [Source detections](data-plane/source-detections.md) and [Sigma](data-plane/sigma.md).