Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 66 additions & 17 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,38 +28,87 @@

Every record takes the same six steps whichever transport brought it, and normally leaves as a row in a batched ClickHouse insert or as a dead letter.

```mermaid

Check warning on line 31 in docs/README.md

View workflow job for this annotation

GitHub Actions / ci / Quality

hyperi-ci diagram-check: R9

frontmatter configuration
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart TB
accTitle: L3 Components -- dfe-loader
accDescr: Records arrive from Kafka topics or gRPC senders. Inside the loader each record is parsed, routed to a table, has its columns extracted, is enriched and is buffered per table, and the buffers are inserted into ClickHouse in batches. Rows ClickHouse refuses for good and records that fail processing go to the DLQ.

kafka[("Kafka topics")]
grpc["gRPC senders"]
kafka[("<b>Kafka topics</b>")]
grpc["<b>gRPC senders</b>"]

subgraph loader["dfe-loader"]
direction LR
parse["Parse JSON"] --> route[["Route"]] --> extract["Extract columns"] --> enrich[["Enrich"]] --> buffer["Buffer per table"] --> insert[["Insert"]]
parse["<b>Parse JSON</b>"] --> route[["<b>Route</b>"]] --> extract["<b>Extract columns</b>"] --> enrich[["<b>Enrich</b>"]] --> buffer["<b>Buffer per table</b>"] --> insert[["<b>Insert</b>"]]
end

clickhouse[("ClickHouse")]
dlq[("DLQ")]
clickhouse[("<b>ClickHouse</b>")]
dlq[("<b>DLQ</b>")]

kafka -->|records| loader
grpc -->|records| loader
loader -->|rows| clickhouse
loader -.->|dead letters| dlq

classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327
classDef async fill:#F6E6EF,stroke:#A8517F,stroke-width:1px,color:#3D1529
classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05
classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937
classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4
class kafka async
class grpc ext
class clickhouse store
class dlq error
class loader group
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
class kafka,dlq bus
class grpc ext
class parse,route,extract,enrich,buffer,insert part
class clickhouse store
style loader fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
```

*Figure 1. L3 Components -- dfe-loader. Double-bordered steps have a diagram each.* Up: top of dfe-loader. Down: [L4 Detail -- Loader table choice](pipeline/routing.md), [L4 Detail -- Enrichment](pipeline/enrichment.md), [L4 Detail -- Insert layer](clickhouse/README.md).
Expand Down
223 changes: 193 additions & 30 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,35 +21,92 @@

The code sits in four layers. Orchestration calls the transform and the insert layers, and both call into the ClickHouse layer, never back up.

```mermaid

Check warning on line 24 in docs/architecture.md

View workflow job for this annotation

GitHub Actions / ci / Quality

hyperi-ci diagram-check: R9

frontmatter configuration
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart LR
accTitle: L4 Detail -- Loader layers
accDescr: Orchestration holds the Orchestrator and the MessageProcessor. It calls the transform layer, the Router, the HeaderExtractor and the EnrichmentPipeline, and the buffer and insert layer, the BufferManager and the Inserter. Both call the ClickHouse layer, the query client, the schema cache, the dynamic insert and the ChClient.

subgraph orchestration["1 Orchestration"]
direction TB
orchestrator["Orchestrator"] ~~~ processor["MessageProcessor"]
orchestrator["<b>Orchestrator</b>"] ~~~ processor["<b>MessageProcessor</b>"]
end
subgraph transform["2 Transform"]
direction TB
router["Router"] ~~~ extractor["HeaderExtractor"] ~~~ enrichment["EnrichmentPipeline"]
router["<b>Router</b>"] ~~~ extractor["<b>HeaderExtractor</b>"] ~~~ enrichment["<b>EnrichmentPipeline</b>"]
end
subgraph insertlayer["3 Buffer and insert"]
direction TB
buffers["BufferManager"] ~~~ inserter["Inserter"]
buffers["<b>BufferManager</b>"] ~~~ inserter["<b>Inserter</b>"]
end
subgraph chlayer["4 ClickHouse layer"]
direction TB
queryclient["ClickHouseQueryClient"] ~~~ schemacache["SchemaCache"] ~~~ dynamic["dfe_clickhouse<br/>dynamic"] ~~~ chclient["ChClient"]
queryclient["<b>ClickHouseQueryClient</b>"] ~~~ schemacache["<b>SchemaCache</b>"] ~~~ dynamic["<b>dfe_clickhouse</b><br/><span>dynamic</span>"] ~~~ chclient["<b>ChClient</b>"]
end
orchestration --> transform
orchestration --> insertlayer
transform --> chlayer
insertlayer --> chlayer

classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4
class orchestration,transform,insertlayer,chlayer group
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
class orchestrator,processor,router,extractor,enrichment,buffers,inserter,queryclient,schemacache,dynamic,chclient part
style orchestration fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style transform fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style insertlayer fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style chlayer fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
```

*Figure 1. L4 Detail -- Loader layers. Each layer calls only the layers to its right.* Up: [L3 Components -- dfe-loader](README.md).
Expand All @@ -76,28 +133,81 @@
That dynamic layer comes from `dfe-clickhouse`'s `dynamic` module, on top of the upstream
`Client`. The rows are encoded against the schema read at runtime, then sent as RowBinary through the official client over HTTP, or as Native blocks through `dfe_clickhouse::TcpClient` over native.

```mermaid

Check warning on line 136 in docs/architecture.md

View workflow job for this annotation

GitHub Actions / ci / Quality

hyperi-ci diagram-check: R9

frontmatter configuration
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart LR
accTitle: L4 Detail -- Dynamic insert
accDescr: The Inserter hands rows to dfe_clickhouse's dynamic module. It reads the table's schema from system.columns, encodes each row against it with DynamicRow, and DynamicInsert sends the RowBinary through the clickhouse crate's insert_formatted_with.

inserter["Inserter"]
inserter["<b>Inserter</b>"]

subgraph dynamic["dfe_clickhouse::dynamic"]
direction LR
schema["fetch_dynamic_schema"] --> encode["DynamicRow"] --> send["DynamicInsert"]
schema["<b>fetch_dynamic_schema</b>"] --> encode["<b>DynamicRow</b>"] --> send["<b>DynamicInsert</b>"]
end

client["clickhouse::Client<br/>insert_formatted_with"]
client["<b>clickhouse::Client</b><br/><span>insert_formatted_with</span>"]

inserter -->|rows| dynamic
dynamic -->|RowBinary| client

classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937
classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4
class client ext
class dynamic group
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
class inserter,schema,encode,send part
class client ext
style dynamic fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
```

*Figure 2. L4 Detail -- Dynamic insert, over HTTP. The dashed box is the crates.io client.* Up: [L3 Components -- dfe-loader](README.md).
Expand All @@ -110,31 +220,84 @@

A failed RowBinary insert ends one of three ways: held for a later flush, retried against a fresh schema, or split so only its bad rows reach the DLQ.

```mermaid

Check warning on line 223 in docs/architecture.md

View workflow job for this annotation

GitHub Actions / ci / Quality

hyperi-ci diagram-check: R9

frontmatter configuration
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart LR
accTitle: L4 Detail -- Insert outcome
accDescr: A successful insert lets its offsets join the flush cycle's commit. A timeout, a dropped connection or a server code on the retry list holds the batch. Any other refusal against a changed table is retried against the current schema, and one against an unchanged table goes to salvage and the DLQ.

ok{"Insert<br/>answered ok?"}
retryable{"Retryable<br/>failure?"}
changed{"Table<br/>changed?"}
commit(["Offsets commit"])
held(["Batch held"])
retry(["Retry on<br/>fresh schema"])
salvage(["Salvage<br/>to DLQ"])
ok{"<b>Insert<br/>answered ok?</b>"}
retryable{"<b>Retryable<br/>failure?</b>"}
changed{"<b>Table<br/>changed?</b>"}
commit(["<b>Offsets commit</b>"])
held(["<b>Batch held</b>"])
retry(["<b>Retry on<br/>fresh schema</b>"])
salvage(["<b>Salvage<br/>to DLQ</b>"])

ok -->|no| retryable
ok -->|yes| commit
retryable -->|no| changed
retryable -->|yes| held
changed -->|yes| retry
changed -->|no| salvage

classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00
classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05
class ok,retryable,changed decision
class salvage error
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
class ok,retryable,changed decision
class commit,held,retry part
class salvage error
```

*Figure 3. L4 Detail -- Insert outcome. Orange is the end that dead-letters rows.* Up: [L3 Components -- dfe-loader](README.md).
Expand Down
Loading
Loading