Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ The query selects the log line as its first column. Sources are re-iterable (the

Pattern mode streams: lines flow through dedup and template mining as one generator pipeline, so the unique-line set is never materialised. Memory is set by `max_clusters`, `max_patterns` and `dedup_cache_size`, not by the input.

This diagram shows the two paths a line can take through the reducer.
Every line goes through exact dedup first, then down the pattern path or the anomaly path.

```mermaid
flowchart LR
Expand Down Expand Up @@ -116,7 +116,7 @@ Pattern mode takes the solid path, anomaly mode the dotted one, and hybrid both.

The anomaly path and temporal mode do not stream. Isolation Forest is batch ML, so cap its input with `anomaly_max_rows` on very high-cardinality sources. Temporal mode holds every line in memory to group them by time window.

This diagram shows which mode to pick.
Pattern suits recurring structure, anomaly rare events, temporal bursts in time, and hybrid fits when you're unsure.

```mermaid
flowchart LR
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ credential inside it. The adapters that DO own connections -- SQLAlchemy,
clickhouse-connect, confluent-kafka -- sit behind optional extras, and the core
never imports them.

This diagram shows the parts of logreducer and what passes between them.
Sources hand lines to the core on every pass, and the core returns the reduced lines and can write them to a sink.

```mermaid
flowchart LR
Expand Down
4 changes: 2 additions & 2 deletions docs/library-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Notes: SQLite has no seedable RNG, so a seeded sample raises `SamplingNotSupport

When you want *about N* representative lines rather than "reduce everything", `reduce_to_target` pulls fresh random batches and reduces each, accumulating distinct representatives until a stop condition fires. Peak memory is bounded to roughly one batch plus the accumulator.

This diagram shows one round of the loop.
Each round pulls a fresh random batch and keeps its new lines, and the loop stops on an empty batch or a stop condition.

```mermaid
flowchart LR
Expand Down Expand Up @@ -90,7 +90,7 @@ reducer = LogReducer(level="enhanced", mode="hybrid", max_clusters=50_000, anoma

The engine is built around two structural protocols, `Source` and `Sink`, and they are its whole IO surface.

This diagram shows the built-in sources and sinks on either side of the reducer.
Any built-in source, or a re-iterable you write, feeds the core, and the core writes to a built-in sink or one you write.

```mermaid
flowchart LR
Expand Down
Loading