diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fa1827a..8c42f34 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -143,6 +143,112 @@ NOT a transform stage, NOT a schema manager. Its only outbound topic is the DLQ. | `src/buffer/`, `src/clickhouse/` | Per-table buffers, and the query client, schema cache and inserter | | `tests/` | `smoke.rs`, `integration_tests.rs` (`integration/`, `e2e/`, `unit/`) -- see `tests/TESTING.md` | +The code sits in four layers. Orchestration calls the transform and the insert layers, and both call into the ClickHouse layer, never back up. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart LR + accTitle: L4 Detail -- Loader layers + accDescr: Orchestration holds the Orchestrator and the MessageProcessor. It calls the transform layer, the Router, the HeaderExtractor and the EnrichmentPipeline, and the buffer and insert layer, the BufferManager and the Inserter. Both call the ClickHouse layer, the query client, the schema cache, the dynamic insert and the ChClient. + + subgraph orchestration["1 Orchestration"] + direction TB + orchestrator["Orchestrator"] ~~~ processor["MessageProcessor"] + end + subgraph transform["2 Transform"] + direction TB + router["Router"] ~~~ extractor["HeaderExtractor"] ~~~ enrichment["EnrichmentPipeline"] + end + subgraph insertlayer["3 Buffer and insert"] + direction TB + buffers["BufferManager"] ~~~ inserter["Inserter"] + end + subgraph chlayer["4 ClickHouse layer"] + direction TB + queryclient["ClickHouseQueryClient"] ~~~ schemacache["SchemaCache"] ~~~ dynamic["dfe_clickhouse
dynamic"] ~~~ chclient["ChClient"] + end + orchestration --> transform + orchestration --> insertlayer + transform --> chlayer + insertlayer --> chlayer + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + class orchestrator,processor,router,extractor,enrichment,buffers,inserter,queryclient,schemacache,dynamic,chclient part + style orchestration fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style transform fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style insertlayer fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style chlayer fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 1. L4 Detail -- Loader layers. Each layer calls only the layers to its right.* Up: [L3 Components -- dfe-loader](docs/architecture.md). + +| Component | What it does | +|---|---| +| `Orchestrator` | Wiring, shutdown, hot-reload, the flush cycle and the offset commit | +| `MessageProcessor` | The per-message pass, run on the worker pool | +| `BufferManager` | Per-table rows, raw payloads and offsets | +| `Inserter` | Format dispatch, retry and salvage | +| `ClickHouseQueryClient` | Schema, comment and health queries | +| `ChClient` | The official `clickhouse::Client` over HTTP, or `dfe_clickhouse::TcpClient` over native | + +A higher layer depends on lower layers, never the reverse. The insert path's +external dependencies are the official `clickhouse` crate's HTTP `Client`, and +`dfe-clickhouse` for the native-protocol `TcpClient` and for reading the +server's exception code out of an HTTP rejection so the inserter can classify it. + ### Commands that prove a change ```bash diff --git a/docs/README.md b/docs/README.md index d35f84d..40170d2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,7 +12,7 @@ Loads records from Kafka or a gRPC push into ClickHouse. Point it at a topic, give it a routing rule, and it parses, routes, promotes schema columns, enriches, buffers per table, and inserts -- RowBinary by default so ClickHouse parses no row text, with at-least-once Kafka delivery. -[architecture.md](architecture.md) covers the loader's layers, [clickhouse/](clickhouse/) the insert layer and [pipeline/](pipeline/) the hot path. +[architecture.md](architecture.md) covers the loader's parts, [clickhouse/](clickhouse/) the insert layer and [pipeline/](pipeline/) the hot path. | You give it | It handles | You don't write | |-------------|------------|-----------------| @@ -24,94 +24,9 @@ Loads records from Kafka or a gRPC push into ClickHouse. Point it at a topic, gi | Enrichment toggles (geoip/reputation/risk) | Flat enriched columns on the promoted row, never in `_json` | Lookup, caching, CIDR matching | | Nothing extra | Per-table buffering, batch salvage, held-batch retry, schema-cache recovery, one offset commit per flush cycle | The resilience layer | -## 10,000-foot view +## How it fits together -Every record takes the same six steps whichever transport brought it, and normally leaves as a row in a batched ClickHouse insert or as a dead letter. - -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart TB - accTitle: L3 Components -- dfe-loader - accDescr: Records arrive from Kafka topics or gRPC senders. Inside the loader each record is parsed, routed to a table, has its columns extracted, is enriched and is buffered per table, and the buffers are inserted into ClickHouse in batches. Rows ClickHouse refuses for good and records that fail processing go to the DLQ. - - kafka[("Kafka topics")] - grpc["gRPC senders"] - - subgraph loader["dfe-loader"] - direction LR - parse["Parse JSON"] --> route[["Route"]] --> extract["Extract columns"] --> enrich[["Enrich"]] --> buffer["Buffer per table"] --> insert[["Insert"]] - end - - clickhouse[("ClickHouse")] - dlq[("DLQ")] - - kafka -->|records| loader - grpc -->|records| loader - loader -->|rows| clickhouse - loader -.->|dead letters| dlq - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class kafka,dlq bus - class grpc ext - class parse,route,extract,enrich,buffer,insert part - class clickhouse store - style loader fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 -``` - -*Figure 1. L3 Components -- dfe-loader. Double-bordered steps have a diagram each.* Up: top of dfe-loader. Down: [L4 Detail -- Loader table choice](pipeline/routing.md), [L4 Detail -- Enrichment](pipeline/enrichment.md), [L4 Detail -- Insert layer](clickhouse/README.md). +[architecture.md](architecture.md) has the loader's parts, its dependencies and the diagram of how a record flows through it. With `kafka.acknowledgements.enabled` (the default) Kafka offsets commit once per flush cycle, never past a row not yet placed. A pre-route filter, a dead letter no DLQ can hold and a revoked partition's rows are the exceptions, each counted, and [operations.md](operations.md) covers them. @@ -119,7 +34,7 @@ The docs, by area: | Area | Doc | Covers | |------|-----|--------| -| Architecture | [architecture.md](architecture.md) | Layers, the dynamic insert, the client crates | +| Architecture | [architecture.md](architecture.md) | The parts, the dependencies and why it is built this way | | Configuration | [configuration.md](configuration.md) | The cascade and the settings that change behaviour | | Operations | [operations.md](operations.md) | Running, config-check, metrics/top, hot-reload, DLQ | | Build features | [feature-flags.md](feature-flags.md) | Cargo features and the ClickHouse client crates | diff --git a/docs/architecture.md b/docs/architecture.md index 726b8c5..3b3f255 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,25 +1,16 @@ -# Architecture +# dfe-loader architecture -dfe-loader is a per-message hot path with a per-table batching tail. A message arrives as bytes, gets routed and promoted into a typed row, accumulates in a per-table buffer, and flushes to ClickHouse as a batch. Kafka offsets commit once per flush cycle, never past a row not placed yet -- at-least-once. - -Everything on the hot path follows one rule: do the cheap thing per -message, defer the expensive thing to flush. Parse once (SIMD), -promote only schema columns, keep the full payload as a reference-counted -`Arc<[u8]>`, and splice it as `_json` only at serialise time. - -## Layers - -The code sits in four layers. Orchestration calls the transform and the insert layers, and both call into the ClickHouse layer, never back up. +dfe-loader is the last hop of the DFE data path: it takes records from Kafka or a gRPC push and inserts them into ClickHouse tables in batches. It needs no mapping first, because it reads each table's columns from ClickHouse and keeps whatever doesn't fit in `_json`. ```mermaid --- @@ -67,30 +58,29 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR - accTitle: L4 Detail -- Loader layers - accDescr: Orchestration holds the Orchestrator and the MessageProcessor. It calls the transform layer, the Router, the HeaderExtractor and the EnrichmentPipeline, and the buffer and insert layer, the BufferManager and the Inserter. Both call the ClickHouse layer, the query client, the schema cache, the dynamic insert and the ChClient. +flowchart TB + accTitle: L3 Components -- dfe-loader + accDescr: Records arrive from Kafka topics or gRPC senders. Inside the loader the transport hands them to the pipeline, which parses, routes, extracts, enriches and buffers each record per table, and the ClickHouse insert layer writes the buffers to ClickHouse in batches. Rows ClickHouse refuses for good and records that fail processing go to the DLQ. - subgraph orchestration["1 Orchestration"] - direction TB - orchestrator["Orchestrator"] ~~~ processor["MessageProcessor"] - end - subgraph transform["2 Transform"] - direction TB - router["Router"] ~~~ extractor["HeaderExtractor"] ~~~ enrichment["EnrichmentPipeline"] - end - subgraph insertlayer["3 Buffer and insert"] - direction TB - buffers["BufferManager"] ~~~ inserter["Inserter"] - end - subgraph chlayer["4 ClickHouse layer"] - direction TB - queryclient["ClickHouseQueryClient"] ~~~ schemacache["SchemaCache"] ~~~ dynamic["dfe_clickhouse
dynamic"] ~~~ chclient["ChClient"] + kafka[("Kafka topics")] + grpc("gRPC senders
receiver, transforms") + subgraph loader["dfe-loader"] + direction LR + transport("Transport
Kafka or gRPC") + pipeline("Pipeline
parse, route, extract, enrich, buffer") + insert("ClickHouse insert layer
batched inserts, salvage") + transport --> pipeline --> insert end - orchestration --> transform - orchestration --> insertlayer - transform --> chlayer - insertlayer --> chlayer + clickhouse[("ClickHouse")] + dlq[("DLQ")] + kafka -->|records| loader + grpc -->|records| loader + loader -->|rows| clickhouse + loader -.->|dead letters| dlq + class kafka,dlq bus + class grpc ext + class transport,pipeline,insert part + class clickhouse store classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -102,225 +92,79 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class orchestrator,processor,router,extractor,enrichment,buffers,inserter,queryclient,schemacache,dynamic,chclient part - style orchestration fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 - style transform fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 - style insertlayer fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 - style chlayer fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style loader fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Loader layers. Each layer calls only the layers to its right.* Up: [L3 Components -- dfe-loader](README.md). +*Figure 1. L3 Components -- dfe-loader. The dotted arrow is the dead-letter path.* Up: [L2 Services -- DFE deployment](https://github.com/hyperi-io/dfe-infra/blob/main/docs/architecture.md). Down: [L4 Detail -- Transport](transport/README.md), [L4 Detail -- Pipeline](pipeline/README.md), [L4 Detail -- ClickHouse insert layer](clickhouse/README.md). -| Component | What it does | -|---|---| -| `Orchestrator` | Wiring, shutdown, hot-reload, the flush cycle and the offset commit | -| `MessageProcessor` | The per-message pass, run on the worker pool | -| `BufferManager` | Per-table rows, raw payloads and offsets | -| `Inserter` | Format dispatch, retry and salvage | -| `ClickHouseQueryClient` | Schema, comment and health queries | -| `ChClient` | The official `clickhouse::Client` over HTTP, or `dfe_clickhouse::TcpClient` over native | +## How data flows -A higher layer depends on lower layers, never the reverse. The insert path's -external dependencies are the official `clickhouse` crate's HTTP `Client`, and -`dfe-clickhouse` for the native-protocol `TcpClient` and for reading the -server's exception code out of an HTTP rejection so the inserter can classify it. +1. The transport is scalo's. On Kafka it consumes the configured topics. With `transport: grpc` it serves a gRPC listener on port 6000 that dfe-receiver and the transforms push to directly. Both hand the pipeline the same message, so nothing after them knows which one a record came on. +2. The pipeline refuses anything not JSON, parses it once with SIMD and routes it to a `db.table` from its routing fields. It promotes the fields that match the table's columns, adds the enrichment columns and puts the row in that table's buffer. +3. The ClickHouse insert layer flushes each table's buffer as a batched INSERT, RowBinary by default, so ClickHouse parses no row text. Rows ClickHouse refuses for good go to the DLQ. +4. With `kafka.acknowledgements.enabled` (the default) Kafka offsets commit once per flush cycle, never past a row not yet placed. Delivery is at-least-once. -## The dynamic insert +## Components -The `clickhouse` crate keeps insertion typed (`T: Row`, compile-time schema) -- -that is the right upstream default. dfe-loader inserts -`Map` rows with columns from `system.columns` at runtime. -That dynamic layer comes from `dfe-clickhouse`'s `dynamic` module, on top of the upstream -`Client`. The rows are encoded against the schema read at runtime, then sent as RowBinary through the official client over HTTP, or as Native blocks through `dfe_clickhouse::TcpClient` over native. +| Part | Role | Page | +|---|---|---| +| Transport | scalo's Kafka and gRPC transports, adapted to one message type | [transport/README.md](transport/README.md) | +| Pipeline | The per-message pass and the per-table buffers | [pipeline/README.md](pipeline/README.md) | +| ClickHouse insert layer | The dynamic insert, insert formats, the schema cache, retries and salvage | [clickhouse/README.md](clickhouse/README.md) | -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Dynamic insert - accDescr: The Inserter hands rows to dfe_clickhouse's dynamic module. It reads the table's schema from system.columns, encodes each row against it with DynamicRow, and DynamicInsert sends the RowBinary through the clickhouse crate's insert_formatted_with. +## External dependencies - inserter["Inserter"] +| Dependency | When it's needed | What can stand in | +|---|---|---| +| Kafka | `transport: kafka` (the default), and the default DLQ | Any broker scalo's Kafka transport reaches | +| gRPC senders | `transport: grpc` | dfe-receiver and the transforms, over scalo's Push | +| ClickHouse | Always | A server in the cluster over HTTP or the native protocol, or ClickHouse Cloud, where a host under `clickhouse.cloud` turns TLS on | +| The DFE tables | Always | dfe-engine creates them from the dfe-schemas manifest. The loader never runs DDL | +| GeoIP databases | `geoip.enabled` | MaxMind GeoLite2 or DB-IP files, downloaded at startup or mounted | - subgraph dynamic["dfe_clickhouse::dynamic"] - direction LR - schema["fetch_dynamic_schema"] --> encode["DynamicRow"] --> send["DynamicInsert"] - end +A missing GeoIP database leaves enrichment off and never stops the loader. A missing table holds its records until the schema arrives, up to `schema.pending_max_age_secs`, and then dead-letters them. - client["clickhouse::Client
insert_formatted_with"] +## Deployment variants - inserter -->|rows| dynamic - dynamic -->|RowBinary| client - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class inserter,schema,encode,send part - class client ext - style dynamic fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 -``` +Kafka is the default transport. With `transport: grpc` there are no broker offsets: a held batch is answered `UNAVAILABLE` to its sender instead of holding a partition's commit, and the DLQ is file only. -*Figure 2. L4 Detail -- Dynamic insert, over HTTP. The dashed box is the crates.io client.* Up: [L3 Components -- dfe-loader](README.md). +`kafka.acknowledgements.enabled: false` commits offsets at receipt. The buffered rows are then their only copy, so a kill loses them. -`ParsedType` parses each column's type string for the encoder. A JSONEachRow insert skips the encoder: `shape_for_type` rewrites the values the server would refuse and the rows go straight to `insert_formatted_with`. +The released chart scales on CPU, with its Kafka lag trigger off, because raw consumer lag rises when a stage downstream breaks. In a DFE deployment, dfe-infra's values set the CPU target to 70% and add the scaling-pressure trigger. The slim profile clears that trigger, so slim scales on CPU alone. [operations.md](operations.md#scaling) has the numbers. -Over HTTP every insert goes through `insert_formatted_with`: `FORMAT RowBinaryWithNamesAndTypes`, one INSERT for each set of defaulted columns the batch's rows supply, or `JSONEachRow`. With `clickhouse.protocol: native` the rows go out as `FORMAT Native` blocks through `TcpClient::insert_native`, and `JSONEachRow` is refused at startup. See [clickhouse/clickhouse-ext.md](clickhouse/clickhouse-ext.md). +## Design considerations -## Insert dispatch +**Reliability.** The insert tail degrades rather than stalls. A data error splits the batch down to the bad rows, which go to the DLQ, and the rest land. A failed insert never sends its batch to the DLQ: it holds the batch and its partition's commit, and retries with jittered backoff. A cycle in which every insert failed opens the circuit gate on the scaling signal, since more pods can't help a dead ClickHouse. [operations.md](operations.md#when-inserts-fail) has every case. -A failed RowBinary insert ends one of three ways: held for a later flush, retried against a fresh schema, or split so only its bad rows reach the DLQ. +**Performance.** Everything on the hot path follows one rule: do the cheap thing per message, and defer the expensive thing to flush. The payload is parsed once and kept as a reference-counted `Arc<[u8]>`. Only schema columns are promoted, and the payload is spliced in as `_json` only when the batch is serialised. [performance/overview.md](performance/overview.md) has the rest. -```mermaid ---- -config: - theme: base - look: neo - fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif - flowchart: - curve: linear - nodeSpacing: 40 - rankSpacing: 64 - padding: 18 - wrappingWidth: 280 - htmlLabels: true - elk: - preset: modelOrder - themeVariables: - fontSize: 15px - lineColor: "#64748B" - primaryTextColor: "#0F172A" - clusterBorder: "#0072B2" - edgeLabelBackground: "#FFFFFF" - themeCSS: >- - p{margin:0} - .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} - .node rect{rx:10px;ry:10px} - .cluster rect{rx:16px;ry:16px} - .node .label div{min-width:150px} - .node.decision .label div{min-width:0} - .cluster[data-look] rect{filter:none} - .nodeLabel{line-height:1.35} - .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} - .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} - .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} - .flowchart-link{stroke-width:1.75px} - .labelBkg{background:transparent!important} - span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} - span.edgeLabel p{background:transparent!important} - .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} - .ingest .label-icon{color:#0E8A7D} - .xform .label-icon{color:#7B4FB0} - .load .label-icon{color:#0072B2} - .bus .label-icon{color:#A8517F} - .store .label-icon{color:#00785A} - .ui .label-icon{color:#D9480F} - .ext .label-icon{color:#64748B} ---- -flowchart LR - accTitle: L4 Detail -- Insert outcome - accDescr: A successful insert lets its offsets join the flush cycle's commit. A timeout, a dropped connection or a server code on the retry list holds the batch. Any other refusal against a changed table is retried against the current schema, and one against an unchanged table goes to salvage and the DLQ. - - ok{"Insert
answered ok?
"} - retryable{"Retryable
failure?
"} - changed{"Table
changed?
"} - commit(["Offsets commit"]) - held(["Batch held"]) - retry(["Retry on
fresh schema
"]) - salvage(["Salvage
to DLQ
"]) - - ok -->|no| retryable - ok -->|yes| commit - retryable -->|no| changed - retryable -->|yes| held - changed -->|yes| retry - changed -->|no| salvage - classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A - classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A - classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A - classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A - classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A - classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A - classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A - classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A - classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A - classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A - classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class ok,retryable,changed decision - class commit,held,retry part - class salvage error -``` +**Operations.** `/readyz` stays 503 until ClickHouse accepts the configured user and answers reads of the database's tables and columns. Until then the loader joins no consumer group and opens no gRPC listener, so a pod with a wrong password takes no partitions from the pods that can write. An outage after that doesn't flip readiness, which would pull every loader out of its Service at once. `clickhouse_connected` reports it instead. + +**Security.** The loader has no listener an outsider can reach: its gRPC port takes pushes from dfe-receiver and the transforms. TLS to ClickHouse is `clickhouse.tls.enabled` with the CA in the trust store, and [clickhouse/tls.md](clickhouse/tls.md) has the private-CA case. + +## Why it's built this way + +**The insert is dynamic.** The `clickhouse` crate keeps insertion typed, with a compile-time schema, and that is the right upstream default. A loader that takes any table can't know its columns at compile time. So rows are `Map`, encoded against the schema read from `system.columns` at runtime by `dfe-clickhouse`'s `dynamic` module. + +**RowBinary is the default.** The rows go to ClickHouse already typed, so the server parses no row text. JSONEachRow stays as an HTTP fallback. -*Figure 3. L4 Detail -- Insert outcome. Orange is the end that dead-letters rows.* Up: [L3 Components -- dfe-loader](README.md). +**A rejection is read by its code, never its words.** A parse rejection echoes the offending row back, so the message text can carry anything. The loader reads the server's exception code and keeps at most 1 MiB of any error text. -`Inserter::dynamic_verdict` in `src/clickhouse/inserter.rs` makes the call. Retryable means a timeout, a dropped connection or a code on the retry list in `src/clickhouse/error.rs`. A stale or unreadable cached schema is retried on a fresh read without asking the table, and an encoder refusal of a value goes straight to salvage. +**The engine owns the schema.** The loader never creates or alters a table. A record for a table that isn't there yet waits for it, so a schema change has one owner. -The RowBinary path ships `FORMAT RowBinaryWithNamesAndTypes` through -`insert_formatted_with`, and the header's types make the server refuse rows encoded against a type the column no longer has. A refusal is blamed on the rows once a fresh read shows -the table unchanged since they were encoded, and otherwise the retry encodes -against the current schema. Over native the block declares the types. +**The payload is stored once.** Under the default `capture_mode: full` a JSON record lands in `_json`, beside the promoted columns, and nowhere else. [pipeline/capture-modes.md](pipeline/capture-modes.md) has the four modes. -A rejected insert is classified by the server's exception code, read with -`dfe_clickhouse::ServerException::parse` over HTTP and carried structurally -over native, never by the words of the message: -a parse rejection echoes the offending row back. The loader keeps at most 1 MiB -of any error text, because the client reads a whole error body into memory. +## Diagram map -`capture_mode` decides whether the full payload is kept beside the promoted columns, as `_json`, `_raw` or neither, resolved per table from the DDL tag, then per-table config, then the global default. Under the default `full` a JSON record is stored once, in `_json`. [pipeline/capture-modes.md](pipeline/capture-modes.md) has the four modes. +- Figure 1. L3 Components -- dfe-loader (this page) + - [L4 Detail -- Transport](transport/README.md), with [the gRPC transport](transport/grpc.md) and [the common header](transport/common-header.md) + - [L4 Detail -- Pipeline](pipeline/README.md), with [the stages](pipeline/overview.md), [routing](pipeline/routing.md), [enrichment](pipeline/enrichment.md), [capture modes](pipeline/capture-modes.md) and [parallelism](pipeline/parallelism.md) + - [L4 Detail -- ClickHouse insert layer](clickhouse/README.md), with [the dynamic insert](clickhouse/clickhouse-ext.md), [insert formats](clickhouse/insert-formats.md), [the schema cache](clickhouse/schema-cache.md), [DDL directives](clickhouse/ddl-directives.md) and [TLS](clickhouse/tls.md) +- [Configuration](configuration.md), [performance](performance/overview.md), [publishing](deployment/publishing.md) and [shared schemas](deployment/schemas.md) each have a diagram of their own -## Resilience +## What's next -- A data error binary-splits the batch down to the bad rows, which go to the DLQ, and the rest land. -- A flush cycle in which every insert failed opens the circuit gate on the KEDA scaling signal, and the next successful insert closes it. A failed insert never sends its batch to the DLQ. -- Drift, a column dropped or retyped by `ALTER`, drops the cached schema, and the retry encodes against a fresh read. An added column breaks no insert, because the INSERT names its columns, so it takes its default until the cached schema refreshes. -- Offsets commit once per cycle, and on each partition stop below the lowest offset not placed yet. A failed batch or a dead letter the DLQ refused is held and retried with jittered backoff, and holds its partition's commit until it lands. A dead letter no DLQ backend can ever hold is dropped and counted instead (`pipeline_dead_letters_dropped_total{reason}`). A held batch is retried against the default table once ClickHouse reports its table absent, a batch bound for `dlq_load.table` aside, which waits for its table. -- Rows of a partition a rebalance has moved away are discarded before any write, since the partition's next owner reads them again from the committed offset (`transport_revoke_discarded_total{stage="buffer"}`). See [operations.md](operations.md#rebalances). +- [configuration.md](configuration.md): the cascade and the settings that change behaviour +- [operations.md](operations.md): running, metrics, failed inserts, the DLQ and scaling +- [config-schema.yaml](config-schema.yaml): every setting and its default +- [CONTRIBUTING.md](../CONTRIBUTING.md#context): where the code lives and what tends to bite diff --git a/docs/clickhouse/README.md b/docs/clickhouse/README.md index 867fca7..04a8263 100644 --- a/docs/clickhouse/README.md +++ b/docs/clickhouse/README.md @@ -10,11 +10,17 @@ # ClickHouse insert layer -The dynamic insert path over the official `clickhouse` crate. The loader -reflects each table's schema at runtime and encodes `Map` rows -straight to binary -- no compile-time `Row` types. +The ClickHouse insert layer is the last part of [dfe-loader](../architecture.md): it writes each table's batch to ClickHouse and sends the rows ClickHouse refuses for good to the DLQ. -The inserter takes each table's batch from its buffer and sends it through the dynamic insert and the client to ClickHouse, and the rows ClickHouse rejects for good are the ones that leave for the DLQ. +| | | +|---|---| +| Repo | [hyperi-io/dfe-loader](https://github.com/hyperi-io/dfe-loader) | +| Image and chart | `ghcr.io/hyperi-io/dfe-loader`, and the thin chart released beside it | +| Config reference | `clickhouse`, `schema` and `buffer` in [config-schema.yaml](../config-schema.yaml) | +| Endpoints | None of its own. It dials ClickHouse over HTTP, or the native protocol with `clickhouse.protocol: native` | +| Metrics | `rows_inserted_total`, `batches_flushed_total`, `clickhouse_connected`, `scaling_circuit_open`, in [operations.md](../operations.md#observe) | + +No `Row` type is compiled in: the schema is read at runtime and each `Map` row is encoded straight to binary. ```mermaid --- @@ -62,7 +68,7 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Insert layer accDescr: The per-table buffer hands batches to the insert layer. Inside, the Inserter passes rows to dfe_clickhouse's DynamicInsert, which writes them through the ChClient. Rows land in ClickHouse, and rows ClickHouse refuses for good go to the DLQ. @@ -79,6 +85,9 @@ flowchart LR buffer -->|batches| insert insert -->|rows| clickhouse insert -.->|dead letters| dlq + class buffer,inserter,dynamic,chclient part + class clickhouse store + class dlq bus classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -90,16 +99,231 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class buffer,inserter,dynamic,chclient part - class clickhouse store - class dlq bus style insert fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Insert layer. The dotted arrow is the salvage path.* Up: [L3 Components -- dfe-loader](../README.md). +*Figure 1. L4 Detail -- Insert layer. The dotted arrow is the salvage path.* Up: [L3 Components -- dfe-loader](../architecture.md). Down: [L4 Detail -- Dynamic insert](#how-it-works), [L4 Detail -- Insert outcome](#scaling-and-failure). + +## How it works + +The `clickhouse` crate keeps insertion typed (`T: Row`, compile-time schema) -- +that is the right upstream default. dfe-loader inserts +`Map` rows with columns from `system.columns` at runtime. +That dynamic layer comes from `dfe-clickhouse`'s `dynamic` module, on top of the upstream +`Client`. The rows are encoded against the schema read at runtime, then sent as RowBinary through the official client over HTTP, or as Native blocks through `dfe_clickhouse::TcpClient` over native. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart TB + accTitle: L4 Detail -- Dynamic insert + accDescr: The Inserter hands rows to dfe_clickhouse's dynamic module. It reads the table's schema from system.columns, encodes each row against it with DynamicRow, and DynamicInsert sends the RowBinary through the clickhouse crate's insert_formatted_with. + + inserter["Inserter"] + + subgraph dynamic["dfe_clickhouse::dynamic"] + direction LR + schema["fetch_dynamic_schema"] --> encode["DynamicRow"] --> send["DynamicInsert"] + end + + client["clickhouse::Client
insert_formatted_with"] + + inserter -->|rows| dynamic + dynamic -->|RowBinary| client + class inserter,schema,encode,send part + class client ext + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style dynamic fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 2. L4 Detail -- Dynamic insert, over HTTP. The dashed box is the crates.io client.* Up: [L4 Detail -- ClickHouse insert layer](README.md). + +`ParsedType` parses each column's type string for the encoder. A JSONEachRow insert skips the encoder: `shape_for_type` rewrites the values the server would refuse and the rows go straight to `insert_formatted_with`. + +## Interfaces `ChClient` is the official `clickhouse::Client` over HTTP, or `dfe_clickhouse::TcpClient` with `clickhouse.protocol: native`. The schema cache both clients read is in [schema-cache.md](schema-cache.md). +Over HTTP every insert goes through `insert_formatted_with`: `FORMAT RowBinaryWithNamesAndTypes`, one INSERT for each set of defaulted columns the batch's rows supply, or `JSONEachRow`. With `clickhouse.protocol: native` the rows go out as `FORMAT Native` blocks through `TcpClient::insert_native`, and `JSONEachRow` is refused at startup. See [clickhouse-ext.md](clickhouse-ext.md). + +`capture_mode` decides whether the full payload is kept beside the promoted columns, as `_json`, `_raw` or neither, resolved per table from the DDL tag, then per-table config, then the global default. Under the default `full` a JSON record is stored once, in `_json`. [capture-modes.md](../pipeline/capture-modes.md) has the four modes. + +## Scaling and failure + +A failed RowBinary insert ends one of three ways: held for a later flush, retried against a fresh schema, or split so only its bad rows reach the DLQ. + +```mermaid +--- +config: + theme: base + look: neo + fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif + flowchart: + curve: linear + nodeSpacing: 40 + rankSpacing: 64 + padding: 18 + wrappingWidth: 280 + htmlLabels: true + elk: + preset: modelOrder + themeVariables: + fontSize: 15px + lineColor: "#64748B" + primaryTextColor: "#0F172A" + clusterBorder: "#0072B2" + edgeLabelBackground: "#FFFFFF" + themeCSS: >- + p{margin:0} + .node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))} + .node rect{rx:10px;ry:10px} + .cluster rect{rx:16px;ry:16px} + .node .label div{min-width:150px} + .node.decision .label div{min-width:0} + .cluster[data-look] rect{filter:none} + .nodeLabel{line-height:1.35} + .nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em} + .nodeLabel span{font-size:12.5px;font-weight:400;color:#334155} + .cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase} + .flowchart-link{stroke-width:1.75px} + .labelBkg{background:transparent!important} + span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500} + span.edgeLabel p{background:transparent!important} + .label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px} + .ingest .label-icon{color:#0E8A7D} + .xform .label-icon{color:#7B4FB0} + .load .label-icon{color:#0072B2} + .bus .label-icon{color:#A8517F} + .store .label-icon{color:#00785A} + .ui .label-icon{color:#D9480F} + .ext .label-icon{color:#64748B} +--- +flowchart TB + accTitle: L4 Detail -- Insert outcome + accDescr: A successful insert lets its offsets join the flush cycle's commit. A timeout, a dropped connection or a server code on the retry list holds the batch. Any other refusal against a changed table is retried against the current schema, and one against an unchanged table goes to salvage and the DLQ. + + subgraph outcome["Insert outcome"] + direction TB + subgraph R1["1 Insert"] + direction LR + ok{"Insert
answered ok?
"} + commit(["Offsets commit"]) + ok -->|yes| commit + end + subgraph R2["2 Failed"] + direction LR + retryable{"Retryable
failure?
"} + held(["Batch held"]) + retryable -->|yes| held + end + subgraph R3["3 Not retryable"] + direction LR + changed{"Table
changed?
"} + retry(["Retry on
fresh schema
"]) + salvage(["Salvage
to DLQ
"]) + changed -->|yes| retry + changed -->|no| salvage + end + R1 -->|no| R2 + R2 -->|no| R3 + end + class ok,retryable,changed decision + class commit,held,retry part + class salvage error + classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A + classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A + classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A + classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A + classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A + classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A + classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A + classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A + classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A + classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A + classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A + style outcome fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R3 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 +``` + +*Figure 3. L4 Detail -- Insert outcome. Orange is the end that dead-letters rows.* Up: [L4 Detail -- ClickHouse insert layer](README.md). + +`Inserter::dynamic_verdict` in `src/clickhouse/inserter.rs` makes the call. Retryable means a timeout, a dropped connection or a code on the retry list in `src/clickhouse/error.rs`. A stale or unreadable cached schema is retried on a fresh read without asking the table, and an encoder refusal of a value goes straight to salvage. + +The RowBinary path ships `FORMAT RowBinaryWithNamesAndTypes` through +`insert_formatted_with`, and the header's types make the server refuse rows encoded against a type the column no longer has. A refusal is blamed on the rows once a fresh read shows +the table unchanged since they were encoded, and otherwise the retry encodes +against the current schema. Over native the block declares the types. + +A rejected insert is classified by the server's exception code, read with +`dfe_clickhouse::ServerException::parse` over HTTP and carried structurally +over native, never by the words of the message: +a parse rejection echoes the offending row back. The loader keeps at most 1 MiB +of any error text, because the client reads a whole error body into memory. + +## Key metrics + +- `rows_inserted_total` and `batches_flushed_total`: what landed. +- `dfe_loader_permanent_reject_total`, per table: rows ClickHouse refused for good. +- `clickhouse_connected`: 0 once an insert went unanswered. +- `scaling_circuit_open`: 1 after a cycle in which every insert failed. + +## Related + Read in this order: - [clickhouse-ext.md](clickhouse-ext.md) -- the dynamic insert from @@ -113,5 +337,3 @@ Read in this order: - [ddl-directives.md](ddl-directives.md) -- the `@`-comment directives the loader reads off table DDL (capture mode, field mapping). - [tls.md](tls.md) -- TLS transport security and private-CA trust. - -dfe-loader inserts through crates.io `clickhouse` and reads rejection codes with `dfe-clickhouse`, with the dynamic encoding layer from `dfe-clickhouse`'s `dynamic` module. See [clickhouse-ext.md](clickhouse-ext.md). diff --git a/docs/clickhouse/ddl-directives.md b/docs/clickhouse/ddl-directives.md index 4e4cdda..51c92c5 100644 --- a/docs/clickhouse/ddl-directives.md +++ b/docs/clickhouse/ddl-directives.md @@ -82,22 +82,34 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Column directive resolution accDescr: A column marked skip is left out of the insert. Any other column reads the paths its source or renamed directive lists, else the field named like the column. A field found is written. A column with no field found takes its computed or default value. - skip{"@skip?"} - read["Listed paths
or column name
"] - left(["Left out"]) - found{"Field
found?
"} - value(["Field's value"]) - fill(["@computed or
@default value
"]) - - skip -->|no| read - skip -->|yes| left - read --> found - found -->|yes| value - found -->|no| fill + subgraph resolve["Column directive resolution"] + direction TB + subgraph R1["1 Skip"] + direction LR + skip{"@skip?"} + left(["Left out"]) + skip -->|yes| left + end + subgraph R2["2 Read"] + direction LR + read("Listed paths
or column name
") + found{"Field
found?
"} + value(["Field's value"]) + fill(["@computed or
@default value
"]) + read --> found + found -->|yes| value + found -->|no| fill + end + R1 -->|no| R2 + end + class skip,found decision + class read,left,value,fill part + class skip,found decision + class read,left,value,fill part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -109,8 +121,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class skip,found decision - class read,left,value,fill part + style resolve fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Column directive resolution. `_org_id` and `_timestamp` read fixed fields and skip this path.* Up: [ClickHouse insert layer](README.md). diff --git a/docs/clickhouse/insert-formats.md b/docs/clickhouse/insert-formats.md index 8756f78..52e801f 100644 --- a/docs/clickhouse/insert-formats.md +++ b/docs/clickhouse/insert-formats.md @@ -60,24 +60,39 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Insert format and protocol accDescr: A rowbinary insert goes as RowBinaryWithNamesAndTypes over HTTP or as Native blocks over the native protocol. A jsoneachrow insert goes as JSONEachRow over HTTP, and the pipeline refuses it at start over native. - format{"insert_format?"} - binproto{"protocol?"} - jsonproto{"protocol?"} - rowbinary(["RowBinaryWithNamesAndTypes
over HTTP
"]) - native(["Native blocks
over TCP
"]) - jsonrows(["JSONEachRow
over HTTP
"]) - refused(["Refused at
pipeline start
"]) - - format -->|rowbinary| binproto - format -->|jsoneachrow| jsonproto - binproto -->|http| rowbinary - binproto -->|native| native - jsonproto -->|http| jsonrows - jsonproto -->|native| refused + subgraph choice["Format and protocol"] + direction TB + format{"insert_format?"} + subgraph RB["rowbinary"] + direction LR + binproto{"protocol?"} + rowbinary(["RowBinaryWithNamesAndTypes
over HTTP
"]) + native(["Native blocks
over TCP
"]) + binproto -->|http| rowbinary + binproto -->|native| native + end + subgraph JE["jsoneachrow"] + direction LR + jsonproto{"protocol?"} + jsonrows(["JSONEachRow
over HTTP
"]) + refused(["Refused at
pipeline start
"]) + jsonproto -->|http| jsonrows + jsonproto -->|native| refused + end + format -->|rowbinary| RB + RB ~~~ JE + format -->|jsoneachrow| JE + end + class format,binproto,jsonproto decision + class rowbinary,native,jsonrows part + class refused error + class format,binproto,jsonproto decision + class rowbinary,native,jsonrows part + class refused error classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -89,9 +104,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class format,binproto,jsonproto decision - class rowbinary,native,jsonrows part - class refused error + style choice fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style RB fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style JE fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Insert format and protocol. Orange is the pair the loader rejects.* Up: [ClickHouse insert layer](README.md). diff --git a/docs/clickhouse/schema-cache.md b/docs/clickhouse/schema-cache.md index 0078f11..6571d9c 100644 --- a/docs/clickhouse/schema-cache.md +++ b/docs/clickhouse/schema-cache.md @@ -60,25 +60,38 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Schema cache read and recovery accDescr: An insert uses the cached schema when it is fresh, else reads system.columns. The rows are encoded and inserted. A drift error invalidates the table's entry so the retry reads the schema again, and any other answer ends the attempt. - fresh{"Cached and
fresh?
"} - cached["Cached schema"] - fetch["Read columns"] - insert["Encode, insert"] - drift{"Drift
error?
"} - done(["Attempt ends"]) - invalidate(["Invalidate"]) - - fresh -->|yes| cached - fresh -->|no| fetch - cached --> insert - fetch --> insert - insert --> drift - drift -->|no| done - drift -->|yes| invalidate + subgraph cache["Schema cache"] + direction TB + subgraph R1["1 Schema"] + direction LR + fresh{"Cached and
fresh?
"} + cached("Cached schema") + fetch("Read columns") + fresh -->|yes| cached + fresh -->|no| fetch + end + subgraph R2["2 Insert"] + direction LR + insert("Encode, insert") + drift{"Drift
error?
"} + done(["Attempt ends"]) + invalidate(["Invalidate"]) + insert --> drift + drift -->|no| done + drift -->|yes| invalidate + end + R1 -->|schema| R2 + end + class fresh,drift decision + class cached store + class fetch,insert,done,invalidate part + class fresh,drift decision + class cached store + class fetch,insert,done,invalidate part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -90,9 +103,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class fresh,drift decision - class cached store - class fetch,insert,done,invalidate part + style cache fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Schema cache read and recovery, with `schema.refresh_on_error` on.* Up: [ClickHouse insert layer](README.md). diff --git a/docs/clickhouse/tls.md b/docs/clickhouse/tls.md index c20798b..fc82a90 100644 --- a/docs/clickhouse/tls.md +++ b/docs/clickhouse/tls.md @@ -60,20 +60,32 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- TLS trust accDescr: clickhouse.tls.enabled turns TLS on. The system trust store, the image bundle or the file SSL_CERT_FILE or SSL_CERT_DIR names, supplies the CA certificates. The loader verifies the server's certificate chain and host name against them before it opens HTTPS or native TLS to ClickHouse. - enabled["clickhouse.tls.enabled"] - store["System trust store"] - verify["Verify chain
and host name
"] - conn["HTTPS or
native TLS
"] - clickhouse[("ClickHouse")] - - enabled -->|on| verify - store -->|CA certs| verify - verify -->|verified| conn - conn --> clickhouse + subgraph tls["TLS to ClickHouse"] + direction TB + subgraph R1["1 Verify"] + direction LR + enabled("clickhouse.tls.enabled") + store("System trust store") + verify("Verify chain
and host name
") + enabled -->|on| verify + store -->|CA certs| verify + end + subgraph R2["2 Connect"] + direction LR + conn("HTTPS or
native TLS
") + clickhouse[("ClickHouse")] + conn --> clickhouse + end + R1 -->|verified| R2 + end + class enabled,verify,conn part + class store,clickhouse store + class enabled,verify,conn part + class store,clickhouse store classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -85,8 +97,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class enabled,verify,conn part - class store,clickhouse store + style tls fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- TLS trust. The store is the image bundle, or what `SSL_CERT_FILE` or `SSL_CERT_DIR` names.* Up: [ClickHouse insert layer](README.md). diff --git a/docs/configuration.md b/docs/configuration.md index 8492138..27bc1ad 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -58,17 +58,30 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Config cascade accDescr: The loader starts from its hard-coded defaults, applies the YAML file over them, then the nested ENV variables, then the flat ENV variables, and validates the result. - defaults["Hard-coded
defaults
"] - file["YAML file"] - nested["Nested ENV"] - flat["Flat ENV"] + subgraph cascade["Config cascade, lowest first"] + direction TB + subgraph R1["1 Built in and file"] + direction LR + defaults("Hard-coded
defaults
") + file("YAML file") + defaults --> file + end + subgraph R2["2 Environment"] + direction LR + nested("Nested ENV") + flat("Flat ENV") + nested --> flat + end + R1 --> R2 + end config(["Validated config"]) - - defaults --> file --> nested --> flat --> config + cascade --> config + class defaults,file,nested,flat,config part + class defaults,file,nested,flat,config part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -80,7 +93,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class defaults,file,nested,flat,config part + style cascade fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Config cascade. Each layer overrides the one to its left.* Up: [dfe-loader docs](README.md). diff --git a/docs/deployment/publishing.md b/docs/deployment/publishing.md index a1f97c5..f3174bf 100644 --- a/docs/deployment/publishing.md +++ b/docs/deployment/publishing.md @@ -63,25 +63,37 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Release artefacts accDescr: A release request starts a CI build of linux amd64 and arm64 binaries. The Dockerfile wraps the binaries into an image, which is pushed with its version tags. The binary's generate-artefacts emits the deployment contract, from which a thin chart is assembled and pushed. Both land in GHCR. - tag(["Release request"]) - build["Build binaries"] - image["Build image"] - contract["Emit contract"] - pushimage["Push image tags"] - chart["Assemble and
push chart
"] + subgraph release["Release"] + direction TB + subgraph R1["1 Build"] + direction LR + tag(["Release request"]) + build("Build binaries") + tag --> build + end + subgraph R2["2 Package"] + direction LR + image("Build image") + pushimage("Push image tags") + contract("Emit contract") + chart("Assemble and
push chart
") + image --> pushimage + contract --> chart + end + R1 --> R2 + end ghcr[("GHCR")] - - tag --> build - build --> image - build --> contract - image --> pushimage - contract --> chart - pushimage --> ghcr - chart --> ghcr + release --> ghcr + class tag ext + class build,image,contract,pushimage,chart part + class ghcr store + class tag ext + class build,image,contract,pushimage,chart part + class ghcr store classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -93,9 +105,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class tag ext - class build,image,contract,pushimage,chart part - class ghcr store + style release fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Release artefacts. A release request is a merge with a `Release: true` trailer or a release dispatch.* Up: [Deployment](README.md). diff --git a/docs/deployment/schemas.md b/docs/deployment/schemas.md index 1ce43f2..03ebe35 100644 --- a/docs/deployment/schemas.md +++ b/docs/deployment/schemas.md @@ -66,18 +66,32 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Where a table's schema comes from accDescr: dfe-schemas holds the YAML definitions. dfe-engine applies the manifest at boot with CREATE and ALTER statements. The loader reads the result from ClickHouse system.columns into its schema cache, again each time an entry expires. - yaml["dfe-schemas
YAML"] - engine["dfe-engine"] - clickhouse[("ClickHouse
system.columns")] - loader["dfe-loader
SchemaCache"] - - yaml -->|manifest| engine - engine -->|CREATE, ALTER| clickhouse - clickhouse -->|columns| loader + subgraph path["Schema path"] + direction TB + subgraph R1["1 Define and apply"] + direction LR + yaml("dfe-schemas
YAML") + engine("dfe-engine") + yaml -->|manifest| engine + end + subgraph R2["2 Read"] + direction LR + clickhouse[("ClickHouse
system.columns")] + loader("dfe-loader
SchemaCache") + clickhouse -->|columns| loader + end + R1 -->|CREATE, ALTER| R2 + end + class yaml,clickhouse store + class engine control + class loader load + class yaml,clickhouse store + class engine control + class loader load classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -89,9 +103,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class yaml,clickhouse store - class engine control - class loader load + style path fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Where a table's schema comes from. Dashed boxes are other DFE repos.* Up: [Deployment](README.md). diff --git a/docs/performance/overview.md b/docs/performance/overview.md index f7535f2..0cefa8b 100644 --- a/docs/performance/overview.md +++ b/docs/performance/overview.md @@ -60,22 +60,39 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Where performance is set accDescr: hyperi-ci builds the binary with jemalloc, LTO, and on release PGO then BOLT. The batch flush settings in config tune the running loader. subgraph build["Build-time, hyperi-ci"] + direction TB + subgraph B1["Allocator and linking"] + direction LR + jemalloc("jemalloc") + lto("LTO") + jemalloc ~~~ lto + end + subgraph B2["Profile-guided"] + direction LR + pgo("PGO") + bolt("BOLT") + pgo ~~~ bolt + end + B1 ~~~ B2 + end + subgraph run["Run time"] direction LR - jemalloc["jemalloc"] ~~~ lto["LTO"] ~~~ pgo["PGO"] ~~~ bolt["BOLT"] + binary("Optimised binary") + batch("Batch settings
in config
") + loader(["Running loader"]) + binary --> loader + batch --> loader end - - binary["Optimised binary"] - batch["Batch settings
in config
"] - loader(["Running loader"]) - - build --> binary - binary --> loader - batch --> loader + build -->|compiles| run + class jemalloc,lto,pgo,bolt,binary,batch part + class loader load + class jemalloc,lto,pgo,bolt,binary,batch part + class loader load classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -87,9 +104,10 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class jemalloc,lto,pgo,bolt,binary,batch part - class loader load style build fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style B1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style B2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style run fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Where performance is set. BOLT runs on the PGO build.* Up: [Performance](README.md). diff --git a/docs/pipeline/README.md b/docs/pipeline/README.md index d58759e..1e04de9 100644 --- a/docs/pipeline/README.md +++ b/docs/pipeline/README.md @@ -10,7 +10,17 @@ # Pipeline -The hot path: bytes in, ClickHouse row out. The pass runs per message, and the batching is per table: each table has a separate buffer, flushed as INSERTs of its rows. +The pipeline is the hot path of [dfe-loader](../architecture.md): bytes in, a typed row in its table's buffer out. + +| | | +|---|---| +| Repo | [hyperi-io/dfe-loader](https://github.com/hyperi-io/dfe-loader) | +| Image and chart | `ghcr.io/hyperi-io/dfe-loader`, and the thin chart released beside it | +| Config reference | `payload`, `routing`, `field_mapping`, `computed_columns`, `enrichment`, `geoip`, `metadata` and `buffer` in [config-schema.yaml](../config-schema.yaml) | +| Endpoints | None of its own | +| Metrics | `messages_received_total`, `loader_messages_by_table_total`, `loader_buffer_rows_by_table`, `dfe_loader_unknown_table_fallback_total`, in [operations.md](../operations.md#observe) | + +The pass runs per message, and the batching is per table: each table has a separate buffer, flushed as INSERTs of its rows. ```mermaid --- @@ -58,23 +68,32 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Per-message pass, per-table batches accDescr: Each message goes through one pass that parses, routes, extracts and enriches it. The row lands in the buffer of the table it was routed to, such as dfe.main or dfe.auth. Each table's buffer flushes as INSERTs into ClickHouse. - pass["Per-message pass"] - - subgraph buffers["Per-table buffers"] + transport("Transport
Kafka or gRPC") + subgraph pipeline["Pipeline"] direction TB - main["dfe.main"] ~~~ auth["dfe.auth"] ~~~ other["Each other table"] + pass("Per-message pass
parse, route, extract, enrich") + subgraph buffers["Per-table buffers"] + direction LR + main("dfe.main") + auth("dfe.auth") + other("Each other table") + main ~~~ auth ~~~ other + end + pass -->|rows| buffers end - - insert["INSERTs
per table
"] + insert("INSERTs per table") clickhouse[("ClickHouse")] - - pass -->|rows| buffers - buffers -->|batches| insert + transport --> pipeline + pipeline -->|batches| insert insert --> clickhouse + class transport,pass,main,auth,other,insert part + class clickhouse store + class pass,main,auth,other,insert part + class clickhouse store classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -86,14 +105,38 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class pass,main,auth,other,insert part - class clickhouse store + style pipeline fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style buffers fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Per-message pass, per-table batches. Each buffer flushes on a row, byte or age trigger.* Up: [L3 Components -- dfe-loader](../README.md). +*Figure 1. L4 Detail -- Per-message pass, per-table batches. Each buffer flushes on a row, byte or age trigger.* Up: [L3 Components -- dfe-loader](../architecture.md). + +## How it works + +1. A message carrying several records, a JSON array or newline-separated objects, splits into one message per record. Each keeps its topic, partition and offset, so a batch still commits as one unit. +2. The pass parses each record once with sonic-rs and keeps the bytes as one reference-counted `Arc<[u8]>`. Anything not JSON is dead-lettered. +3. Routing picks `db.table` from `routing.table_fields` before any flattening, and falls back to `routing.default_table`. +4. The fields the table declares are promoted to typed columns. The rest stay in the payload, written as `_json` at serialise time. +5. Enrichment adds flat GeoIP, reputation and risk columns to the row, never to `_json`. +6. The row lands in its table's buffer, which flushes on `buffer.flush_rows`, `flush_bytes` or `flush_age_secs`. + +[overview.md](overview.md) has each step, and [parallelism.md](parallelism.md) which parts of a batch run in parallel. + +## Interfaces + +In: message bytes from the transport, with their topic, partition and offset. Out: per-table batches of typed rows for the ClickHouse insert layer, and dead letters for the DLQ. A record goes to the DLQ when it isn't JSON, is nested deeper than 64 levels, has `_org_id` or `_source` twice, or waits longer than `schema.pending_max_age_secs` for its table. + +## Scaling and failure + +The pass scales with the worker pool and with pods. Under memory pressure the loader pauses the consumer before the next Kafka poll rather than risk an OOM kill mid-batch. A routing miss is never a dead letter: it lands in the default table. [operations.md](../operations.md#memory-pressure) has the memory guard. + +## Key metrics + +- `messages_received_total`: records in, after batches are split. +- `loader_messages_by_table_total` and `loader_buffer_rows_by_table`: where records went, and what's waiting to flush. +- `dfe_loader_unknown_table_fallback_total`: records sent to the default table because theirs is gone. -## Docs +## Related - [overview.md](overview.md) -- the stages, in order. - [routing.md](routing.md) -- choosing `db.table` from message fields. @@ -104,4 +147,4 @@ flowchart LR For the buffer-and-insert tail see [../clickhouse/insert-formats.md](../clickhouse/insert-formats.md) and the -resilience model in [../architecture.md](../architecture.md#resilience). +failure handling in [../operations.md](../operations.md#when-inserts-fail). diff --git a/docs/pipeline/capture-modes.md b/docs/pipeline/capture-modes.md index be84689..1a74262 100644 --- a/docs/pipeline/capture-modes.md +++ b/docs/pipeline/capture-modes.md @@ -83,20 +83,32 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Capture mode precedence accDescr: A table whose COMMENT carries a capture_mode tag uses that mode. Otherwise a metadata.table_capture_modes entry for the table decides. Otherwise metadata.capture_mode applies. - tag{"@capture_mode
tag?
"} - entry{"table_capture_modes
entry?
"} - tagmode(["The tag's mode"]) - entrymode(["The entry's mode"]) - global(["metadata.capture_mode"]) - - tag -->|no| entry - tag -->|yes| tagmode - entry -->|yes| entrymode - entry -->|no| global + subgraph prec["Capture mode precedence"] + direction TB + subgraph R1["1 Table tag"] + direction LR + tag{"@capture_mode
tag?
"} + tagmode(["The tag's mode"]) + tag -->|yes| tagmode + end + subgraph R2["2 Config"] + direction LR + entry{"table_capture_modes
entry?
"} + entrymode(["The entry's mode"]) + global(["metadata.capture_mode"]) + entry -->|yes| entrymode + entry -->|no| global + end + R1 -->|no| R2 + end + class tag,entry decision + class tagmode,entrymode,global part + class tag,entry decision + class tagmode,entrymode,global part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -108,8 +120,9 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class tag,entry decision - class tagmode,entrymode,global part + style prec fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Capture mode precedence. The first question answered yes decides.* Up: [Pipeline](README.md). diff --git a/docs/pipeline/enrichment.md b/docs/pipeline/enrichment.md index ade746c..615d39d 100644 --- a/docs/pipeline/enrichment.md +++ b/docs/pipeline/enrichment.md @@ -62,22 +62,24 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Enrichment accDescr: The row from column extraction is looked up by its first IP field in GeoIP and in the reputation blocklists. Risk scoring weighs both results. The row goes on to its table's buffer with the geo, rep and risk columns added. - extract["Extract columns"] - + extract("Extract columns") subgraph lookups["Enrich"] direction LR - rep["Reputation
blocklists
"] --> risk["Risk scoring"] - geo["GeoIP"] --> risk + rep("Reputation
blocklists
") + geo("GeoIP") + risk("Risk scoring") + rep --> risk + geo --> risk end - - buffer["Buffer per table"] - + buffer("Buffer per table") extract -->|row| lookups lookups -->|row| buffer + class extract,geo,rep,risk,buffer part + class extract,geo,rep,risk,buffer part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -89,11 +91,10 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class extract,geo,rep,risk,buffer part style lookups fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Enrichment. Each provider is off until its setting turns it on.* Up: [L3 Components -- dfe-loader](../README.md). +*Figure 1. L4 Detail -- Enrichment. Each provider is off until its setting turns it on.* Up: [L3 Components -- dfe-loader](../architecture.md). ## Providers diff --git a/docs/pipeline/overview.md b/docs/pipeline/overview.md index 641afb4..695088d 100644 --- a/docs/pipeline/overview.md +++ b/docs/pipeline/overview.md @@ -64,15 +64,36 @@ flowchart TB accTitle: L4 Detail -- Loader pipeline accDescr: Two stages run in order. Read and route takes message bytes, splits batched messages, parses JSON and routes each record to a table. Extract and write extracts the header and schema columns, enriches, buffers per table, then inserts and commits offsets. - subgraph s1["1 Read and route"] - direction LR - bytes(["Message bytes"]) --> split["Split batches"] --> parse["Parse JSON"] --> route["Route to table"] + subgraph pass["Loader pipeline"] + direction TB + subgraph s1["1 Read and route"] + direction TB + subgraph s1a["Read"] + direction LR + bytes(["Message bytes"]) --> split("Split batches") --> parse("Parse JSON") + end + subgraph s1b["Route"] + direction LR + route("Route to table") + end + s1a --> s1b + end + subgraph s2["2 Extract and write"] + direction TB + subgraph s2a["Extract"] + direction LR + extract("Extract columns") --> enrich("Enrich") + end + subgraph s2b["Write"] + direction LR + buffer("Buffer per table") --> insert(["Insert and commit"]) + end + s2a --> s2b + end + s1 --> s2 end - subgraph s2["2 Extract and write"] - direction LR - extract["Extract columns"] --> enrich["Enrich"] --> buffer["Buffer per table"] --> insert(["Insert and commit"]) - end - s1 --> s2 + class bytes,split,parse,route,extract,enrich,buffer,insert part + class bytes,split,parse,route,extract,enrich,buffer,insert part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -84,9 +105,13 @@ flowchart TB classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class bytes,split,parse,route,extract,enrich,buffer,insert part + style pass fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style s1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style s1a fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style s1b fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style s2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style s2a fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style s2b fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` *Figure 1. L4 Detail -- Loader pipeline. Read each stage left to right, then down.* Up: [Pipeline](README.md). @@ -148,7 +173,7 @@ per-table config > global). See [capture-modes.md](capture-modes.md). ## 6. Insert and commit -The encoder reshapes only the values ClickHouse cannot take as sent: epoch vs ISO timestamps, hyphen-less UUIDs, integer IPv4, every geo_point form for a `Point`, and a single value for an `Array` (see [../clickhouse/types.md](../clickhouse/types.md)). Each table's buffer flushes on its triggers (`flush_rows` / `flush_bytes` / `flush_age_secs`). The batch is encoded and inserted through `dfe-clickhouse`'s `dynamic` module (RowBinary by default). Kafka offsets commit once per flush cycle, and on each partition stop below the lowest offset not placed yet -- at-least-once. Batch salvage isolates a row ClickHouse rejects for good and sends it to the DLQ. A batch that fails for any other reason is held and retried with jittered backoff until it lands. See [../architecture.md](../architecture.md#resilience) and [../clickhouse/insert-formats.md](../clickhouse/insert-formats.md). +The encoder reshapes only the values ClickHouse cannot take as sent: epoch vs ISO timestamps, hyphen-less UUIDs, integer IPv4, every geo_point form for a `Point`, and a single value for an `Array` (see [../clickhouse/types.md](../clickhouse/types.md)). Each table's buffer flushes on its triggers (`flush_rows` / `flush_bytes` / `flush_age_secs`). The batch is encoded and inserted through `dfe-clickhouse`'s `dynamic` module (RowBinary by default). Kafka offsets commit once per flush cycle, and on each partition stop below the lowest offset not placed yet -- at-least-once. Batch salvage isolates a row ClickHouse rejects for good and sends it to the DLQ. A batch that fails for any other reason is held and retried with jittered backoff until it lands. See [../operations.md](../operations.md#when-inserts-fail) and [../clickhouse/insert-formats.md](../clickhouse/insert-formats.md). ## Parallelism diff --git a/docs/pipeline/parallelism.md b/docs/pipeline/parallelism.md index 84680f3..97f51af 100644 --- a/docs/pipeline/parallelism.md +++ b/docs/pipeline/parallelism.md @@ -58,26 +58,33 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Batch phases accDescr: A received batch passes the BatchEngine's pre-route filters when a routing field is set. The worker pool runs MessageProcessor process on each message in parallel. The BatchCoordinator then walks the results in one thread, pushing each Ok row into its table's buffer, parking a message that waits on its schema and turning any other error into a dead letter. - recv(["Received batch"]) - preroute["Pre-route filters"] - - subgraph parallel["Parallel"] - process["MessageProcessor
process
"] + subgraph before["Before the batch splits"] + direction LR + recv(["Received batch"]) + preroute("Pre-route filters") + recv --> preroute end - - subgraph sequential["Sequential"] - push["Buffer push"] - dead["Dead letter"] + subgraph phases["Per message"] + direction LR + subgraph parallel["Parallel"] + process("MessageProcessor
process
") + end + subgraph sequential["Sequential"] + push("Buffer push") + dead("Dead letter") + end + process -->|Ok| push + process -.->|Err| dead end - - recv --> preroute - preroute --> process - process -->|Ok| push - process -.->|Err| dead + before --> phases + class recv,preroute,process,push part + class dead error + class recv,preroute,process,push part + class dead error classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -89,8 +96,8 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class recv,preroute,process,push part - class dead error + style before fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style phases fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style parallel fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 style sequential fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` diff --git a/docs/pipeline/routing.md b/docs/pipeline/routing.md index abd156d..785cbb8 100644 --- a/docs/pipeline/routing.md +++ b/docs/pipeline/routing.md @@ -60,24 +60,39 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Loader table choice accDescr: A message read from a dlq_load topic goes to dlq_load.table. Otherwise a table_fields value names the table. Otherwise a topic named for a source gives that source's table. Otherwise the message goes to default_table. - pin{"dlq_load
topic?
"} - field{"table_fields
value?
"} - topic{"Source
topic?
"} - replay(["dlq_load.table"]) - value(["Value's table"]) - source(["Source's table"]) - fallback(["default_table"]) - - pin -->|no| field - pin -->|yes| replay - field -->|no| topic - field -->|yes| value - topic -->|yes| source - topic -->|no| fallback + subgraph choice["Table choice"] + direction TB + subgraph R1["1 DLQ replay"] + direction LR + pin{"dlq_load
topic?
"} + replay(["dlq_load.table"]) + pin -->|yes| replay + end + subgraph R2["2 Routing field"] + direction LR + field{"table_fields
value?
"} + value(["Value's table"]) + field -->|yes| value + end + subgraph R3["3 Source"] + direction LR + topic{"Source
topic?
"} + source(["Source's table"]) + fallback(["default_table"]) + topic -->|yes| source + topic -->|no| fallback + end + R1 -->|no| R2 + R2 -->|no| R3 + end + class pin,field,topic decision + class replay,value,source,fallback store + class pin,field,topic decision + class replay,value,source,fallback store classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -89,11 +104,13 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class pin,field,topic decision - class replay,value,source,fallback store + style choice fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R1 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R2 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style R3 fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Loader table choice. Green ends are tables.* Up: [L3 Components -- dfe-loader](../README.md). +*Figure 1. L4 Detail -- Loader table choice. Green ends are tables.* Up: [L3 Components -- dfe-loader](../architecture.md). The database is the org's database when `org_routes` lists the tenant, else `default_db`. `source_to_table` then maps whichever table was chosen, a `dlq_load` one aside, and a table ClickHouse reports absent falls back to `default_table` as well. diff --git a/docs/transport/README.md b/docs/transport/README.md index f917635..27bdde6 100644 --- a/docs/transport/README.md +++ b/docs/transport/README.md @@ -10,9 +10,17 @@ # Transport -The transports, Kafka and gRPC, are scalo's. The loader does not implement them: it takes what scalo's `Transport` receives and feeds it into its parse stage, then on through routing, extraction and the per-table insert path. Whichever transport delivers a message, the bytes converge on the same hot path: refuse anything not JSON, parse, route to `db.table`, promote schema columns, and buffer for insert. A Memory transport exists behind the `transport-memory` cargo feature for unit tests only. +The transport is how records reach [dfe-loader](../architecture.md): scalo's Kafka or gRPC transport, adapted so the pipeline sees one message type. -Each transport has an adapter, and both adapters hand the pipeline the same `KafkaMessage`, so nothing after them knows which transport a record came on. +| | | +|---|---| +| Repo | [hyperi-io/dfe-loader](https://github.com/hyperi-io/dfe-loader) | +| Image and chart | `ghcr.io/hyperi-io/dfe-loader`, and the thin chart released beside it | +| Config reference | `transport`, `kafka` and `grpc` in [config-schema.yaml](../config-schema.yaml) | +| Endpoints | gRPC Push on port 6000 with `transport: grpc`. `/metrics`, `/livez` and `/readyz` on 9090 | +| Metrics | `messages_received_total`, `kafka_offsets_committed_total`, `pipeline_retries_total`, in [operations.md](../operations.md#observe) | + +The loader doesn't implement either transport. A Memory transport exists behind the `transport-memory` cargo feature for unit tests only. ```mermaid --- @@ -60,25 +68,31 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- Transport adapters accDescr: scalo's KafkaTransport feeds the loader's TransportAdapter and scalo's GrpcTransport feeds the GrpcTransportAdapter. Both adapters turn each record into a KafkaMessage, which goes to the JSON parse step. subgraph scalo["scalo transports"] - kafka["KafkaTransport"] - grpc["GrpcTransport"] + direction LR + kafka("KafkaTransport") + grpc("GrpcTransport") + kafka ~~~ grpc end - - kafkaadapter["TransportAdapter"] - grpcadapter["GrpcTransportAdapter"] - message["KafkaMessage"] - parse["Parse JSON"] - - kafka --> kafkaadapter - grpc --> grpcadapter - kafkaadapter --> message - grpcadapter --> message - message --> parse + subgraph transport["Transport"] + direction LR + kafkaadapter("TransportAdapter
from KafkaTransport") + grpcadapter("GrpcTransportAdapter
from GrpcTransport") + message("KafkaMessage") + kafkaadapter --> message + grpcadapter --> message + end + parse("Parse JSON
the pipeline") + scalo -->|each to its adapter| transport + transport --> parse + class kafka,grpc ext + class kafkaadapter,grpcadapter,message,parse part + class kafka,grpc ext + class kafkaadapter,grpcadapter,message,parse part classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -90,14 +104,33 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class kafka,grpc ext - class kafkaadapter,grpcadapter,message,parse part style scalo fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 + style transport fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- Transport adapters. Dashed boxes are scalo's.* Up: [L3 Components -- dfe-loader](../README.md). +*Figure 1. L4 Detail -- Transport adapters. Dashed boxes are scalo's.* Up: [L3 Components -- dfe-loader](../architecture.md). + +## How it works + +1. scalo's `KafkaTransport` consumes the configured topics, or its `GrpcTransport` serves Push on `grpc.listen`, port 6000 by default. +2. Each transport's adapter turns every record into the same `KafkaMessage`, with its topic, partition and offset. +3. The pipeline's parse step takes it from there, and nothing after it knows which transport a record came on. + +## Interfaces + +Kafka: the topics in `kafka.topics` or `kafka.topic_regex`, read as consumer group `kafka.group`. gRPC: scalo's Push on port 6000, which the deployment contract declares only when `transport: grpc`. The loader is a gRPC server only and never sends. + +## Scaling and failure + +A transport that can't start, such as Kafka with no broker reachable, is retried with backoff and counted in `pipeline_retries_total{stage="transport_start"}`. Until ClickHouse answers, the loader joins no consumer group and opens no gRPC listener. On gRPC a held batch is answered `UNAVAILABLE`, and the sender retries it. + +## Key metrics + +- `messages_received_total`: records in, on either transport. +- `kafka_offsets_committed_total`: flat on gRPC, which has no broker offsets. +- `pipeline_retries_total{stage="transport_start"}`: a transport that won't start. -## In this section +## Related - [grpc.md](grpc.md) -- the gRPC listener: how it answers a Push, the proto, its config and the deployment contract's port. - [common-header.md](common-header.md) -- the header columns every table starts with, and what the loader writes into each. diff --git a/docs/transport/grpc.md b/docs/transport/grpc.md index a3083d7..288db33 100644 --- a/docs/transport/grpc.md +++ b/docs/transport/grpc.md @@ -64,21 +64,26 @@ config: .ui .label-icon{color:#D9480F} .ext .label-icon{color:#64748B} --- -flowchart LR +flowchart TB accTitle: L4 Detail -- gRPC path accDescr: dfe-receiver pushes to the loader directly or through a transform, which is a gRPC server and client. The loader writes the rows to ClickHouse and answers each Push with a PushResponse. subgraph senders["gRPC senders"] direction LR - receiver["dfe-receiver"] -->|Push| transform["Transform"] + receiver("dfe-receiver") -->|Push| transform("Transform") end - - loader["dfe-loader"] + loader("dfe-loader") clickhouse[("ClickHouse")] - - senders -->|Push| loader + senders -->|Push, answered by PushResponse| loader loader -->|rows| clickhouse - loader -.->|PushResponse| senders + class receiver ingest + class transform xform + class loader load + class clickhouse store + class receiver ingest + class transform xform + class loader load + class clickhouse store classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A @@ -90,14 +95,10 @@ flowchart LR classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A - class receiver ingest - class transform xform - class loader load - class clickhouse store style senders fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4 ``` -*Figure 1. L4 Detail -- gRPC path. The dotted arrow is the answer.* Up: [Transport](README.md). +*Figure 1. L4 Detail -- gRPC path. The loader answers each Push with a PushResponse.* Up: [Transport](README.md). ## How the loader answers a Push