Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
312 changes: 264 additions & 48 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,45 +25,100 @@ Pushed to `dfe-receiver` or pulled by `dfe-fetcher`, and out through `dfe-loader
into ClickHouse. Everything between is optional and profile-selected. On a Kafka profile, the receiver and fetcher produce to Kafka, and the loader reads the topics into ClickHouse.

```mermaid
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart LR
accTitle: L2 Services -- DFE Compose stack
accDescr: Push clients send events to dfe-receiver and dfe-fetcher pulls them from APIs, databases and files. Both produce to Kafka topics. dfe-loader reads the topics and writes ClickHouse, dfe-archiver writes each source's raw events to its archive, and a transform reads a source's topic and writes the transformed topic the loader then reads.

push["Push clients"]
pull["APIs, databases,<br/>files"]
push("<b>Push clients</b>")
pull("<b>APIs, databases, files</b>")

subgraph stack["DFE Compose stack"]
subgraph ingest["Ingest"]
receiver["dfe-receiver"]
fetcher["dfe-fetcher"]
subgraph ingestgrp["Ingest"]
receiver("<b>dfe-receiver</b><br/><span>terminates every protocol</span>")
fetcher("<b>dfe-fetcher</b><br/><span>pulls on a schedule</span>")
end
topics[("Kafka topics")]
loader[["dfe-loader"]]
transforms["Transforms,<br/>one per source"]
archiver["dfe-archiver"]
ch[("ClickHouse")]
archive[("Archive")]
topics[("<b>Kafka topics</b>")]
loader("<b>dfe-loader</b><br/><span>schema-free insert</span>")
transforms("<b>Transforms</b><br/><span>one per source</span>")
archiver("<b>dfe-archiver</b><br/><span>raw events to the archive</span>")
ch[("<b>ClickHouse</b>")]
archive[("<b>Archive</b>")]
end

push -->|events| receiver
pull -->|events| fetcher
ingest --> topics
ingestgrp --> topics
topics --> loader
topics --> transforms
topics --> archiver
loader --> ch
archiver --> archive
transforms -.-> topics

classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327
classDef async fill:#F6E6EF,stroke:#A8517F,stroke-width:1px,color:#3D1529
classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937
classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4
class push,pull ext
class topics async
class receiver,fetcher ingest
class topics bus
class loader,archiver load
class transforms xform
class ch,archive store
class stack,ingest group
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
style stack fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
style ingestgrp fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
```

*Figure 1. L2 Services -- DFE Compose stack. The dotted arrow is a transform writing `<source>_load` back to the broker.* Up: top of dfe-docker. Down: [L4 Detail -- Two listeners on port 6000](#component-repos-and-their-grpc-listeners), [L4 Detail -- Loader topic choice](known-issues.md#a-_load-topic-changes-what-the-loader-reads), [L4 Detail -- Loader and schema checks](troubleshooting.md#events-are-accepted-but-nothing-lands), [L4 Detail -- Topic and dead-letter checks](troubleshooting.md#events-are-accepted-but-nothing-lands).
Expand Down Expand Up @@ -100,25 +155,79 @@ A source with a transform stops sharing `main_land`: the receiver routes it to `
Compose profiles select INFRASTRUCTURE. `service_profiles.yaml` selects DFE SERVICES. `scripts/resolve_profile.py` reads the second and derives the first, writing `.tmp/profile.mk` for the Makefile. The resolver reads the profile and the environment, and the Makefile includes the `profile.mk` it writes.

```mermaid
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart LR
accTitle: L4 Detail -- Profile resolution
accDescr: resolve_profile.py reads the active profile from service_profiles.yaml and the DFE_PROFILE and footprint flags from the environment. It writes profile.mk, which the Makefile includes to pass the Compose profiles and DFE services to the targets that start or check a stack.


sp["service_profiles.yaml"]
env[".env flags"]
resolve["resolve_profile.py"]
mk[".tmp/profile.mk"]
make["make targets"]
sp("<b>service_profiles.yaml</b>")
env("<b>.env flags</b>")
resolve("<b>resolve_profile.py</b>")
mk("<b>.tmp/profile.mk</b>")
make("<b>make targets</b>")

sp -->|profile| resolve
env -->|overrides| resolve
resolve -->|writes| mk
mk --> make

classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F
class resolve core
class sp,env,mk store
class resolve,make part
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
```

*Figure 2. L4 Detail -- Profile resolution. The flags are `DFE_PROFILE` and the footprint toggles such as `DFE_CORE_ENABLED`.* Up: [dfe-docker documentation](README.md).
Expand All @@ -140,28 +249,82 @@ A profile declares its whole footprint through the optional `clickhouse`, `core`
The `core` footprint is the management plane, independent of transport. `dfe-proxy` gives the console and the engine API one origin, because the console's client calls the engine with a relative `/api/v1/...` base URL. dfe-proxy sends `/api/v1` to dfe-engine and every other path to dfe-ui.

```mermaid
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart LR
accTitle: L4 Detail -- One console origin
accDescr: A browser reaches dfe-proxy on host port 3000. dfe-proxy sends paths under /api/v1 to dfe-engine and every other path to dfe-ui. dfe-engine reads and writes ClickHouse.

browser["Browser"]
proxy["dfe-proxy"]
engine["dfe-engine :8000"]
ui["dfe-ui :3000"]
ch[("ClickHouse")]
browser("<b>Browser</b>")
proxy("<b>dfe-proxy</b>")
engine("<b>dfe-engine</b><br/><span>:8000</span>")
console("<b>dfe-ui</b><br/><span>:3000</span>")
ch[("<b>ClickHouse</b>")]

browser -->|host :3000| proxy
proxy -->|/api/v1| engine
proxy -->|other paths| ui
proxy -->|other paths| console
engine --> ch

classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F
classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327
classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937
class browser ext
class proxy core
class proxy part
class engine control
class console ui
class ch store
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
```

*Figure 3. L4 Detail -- One console origin. The ports in the boxes are container ports.* Up: [dfe-docker documentation](README.md).
Expand Down Expand Up @@ -200,27 +363,80 @@ dfe-transform-vector mounts `resources/enrichment/` at `/etc/dfe-transform-vecto
Two gRPC listeners share port 6000, and they are not the same thing. A `*-grpc` profile runs both.

```mermaid
---
config:
theme: base
look: neo
fontFamily: Inter, -apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif
flowchart:
curve: linear
nodeSpacing: 40
rankSpacing: 64
padding: 18
wrappingWidth: 280
htmlLabels: true
elk:
preset: modelOrder
themeVariables:
fontSize: 15px
lineColor: "#64748B"
primaryTextColor: "#0F172A"
clusterBorder: "#0072B2"
edgeLabelBackground: "#FFFFFF"
themeCSS: >-
p{margin:0}
.node[data-look] rect,.node[data-look] path,.node[data-look] polygon,.node[data-look] .outer-path{filter:drop-shadow(0 1px 2px rgba(15,23,42,.10)) drop-shadow(0 6px 14px rgba(15,23,42,.10))}
.node rect{rx:10px;ry:10px}
.cluster rect{rx:16px;ry:16px}
.node .label div{min-width:150px}
.node.decision .label div{min-width:0}
.cluster[data-look] rect{filter:none}
.nodeLabel{line-height:1.35}
.nodeLabel b{font-weight:600;font-size:15px;letter-spacing:-0.01em}
.nodeLabel span{font-size:12.5px;font-weight:400;color:#334155}
.cluster-label span.nodeLabel{display:inline-block;background:#FFFFFF;color:#0B2A3F;border:1.5px solid #0072B2;border-radius:999px;padding:2px 12px;font-size:11.5px;font-weight:700;letter-spacing:.09em;text-transform:uppercase}
.flowchart-link{stroke-width:1.75px}
.labelBkg{background:transparent!important}
span.edgeLabel{display:inline-block;background:#FFFFFF!important;color:#1E293B;border:1px solid #94A3B8;border-radius:999px;padding:2px 10px;font-size:12px;font-weight:500}
span.edgeLabel p{background:transparent!important}
.label-icon{width:20px;height:20px;vertical-align:-4px;margin-right:6px}
.ingest .label-icon{color:#0E8A7D}
.xform .label-icon{color:#7B4FB0}
.load .label-icon{color:#0072B2}
.bus .label-icon{color:#A8517F}
.store .label-icon{color:#00785A}
.ui .label-icon{color:#D9480F}
.ext .label-icon{color:#64748B}
---
flowchart LR
accTitle: L4 Detail -- Two listeners on port 6000
accDescr: An external Vector client can push to dfe-receiver's Vector protocol listener on port 6000 once it is enabled. dfe-receiver pushes to dfe-loader's internal Push listener, also on port 6000, over the compose network.

client["Vector client"]
client("<b>Vector client</b>")

subgraph net["Compose network"]
receiver["dfe-receiver"]
loader[["dfe-loader"]]
receiver("<b>dfe-receiver</b>")
loader("<b>dfe-loader</b>")
end

client -.->|Vector :6000| receiver
receiver -->|Push :6000| loader

classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F
classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937
classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4
class client ext
class loader core
class net group
class receiver ingest
class loader load
classDef ext fill:#F4F6F8,stroke:#94A3B8,stroke-width:1.5px,stroke-dasharray:5 4,color:#0F172A
classDef ingest fill:#FFFFFF,stroke:#0E8A7D,stroke-width:3px,color:#0F172A
classDef xform fill:#FFFFFF,stroke:#7B4FB0,stroke-width:3px,color:#0F172A
classDef load fill:#FFFFFF,stroke:#0072B2,stroke-width:3px,color:#0F172A
classDef bus fill:#FFFFFF,stroke:#A8517F,stroke-width:3px,color:#0F172A
classDef store fill:#FFFFFF,stroke:#00785A,stroke-width:3px,color:#0F172A
classDef ui fill:#FFFFFF,stroke:#D9480F,stroke-width:3px,color:#0F172A
classDef decision fill:#FFFFFF,stroke:#9A6700,stroke-width:3px,color:#0F172A
classDef error fill:#FFFFFF,stroke:#C2410C,stroke-width:3px,color:#0F172A
classDef control fill:#FFFFFF,stroke:#334155,stroke-width:3px,color:#0F172A
classDef part fill:#FFFFFF,stroke:#94A3B8,stroke-width:1.5px,color:#0F172A
style net fill:none,stroke:#0072B2,stroke-width:1.5px,stroke-dasharray:6 4
```

*Figure 4. L4 Detail -- Two listeners on port 6000. The dotted arrow is off until `config.grpc.enabled` is set.* Up: [L2 Services -- DFE Compose stack](#events-arrive-two-ways-and-leave-one).
Expand Down
Loading
Loading