Skip to content

Add graphdiff and depcluster algorithms #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Notgnoshi merged 6 commits into from
Feb 21, 2026
Merged

Add graphdiff and depcluster algorithms #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
983a7db
Upgrade petgraph to 0.8
Notgnoshi Feb 20, 2026
f23a171
Add graphdiff tool
Notgnoshi Feb 20, 2026
9f4a254
Add depcluster tool
Notgnoshi Feb 21, 2026
a5ae808
Add depcluster and graphdiff blurbs to README
Notgnoshi Feb 21, 2026
0283c84
Reorder sections in the README to group related tools
Notgnoshi Feb 21, 2026
b066f23
Add depfilter slice subcommand
Notgnoshi Feb 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,16 @@ either = "1"
eyre = "0.6.12"
globset = "0.4.18"
gnuplot = "0.0.46"
graphrs = "0.11"
hex = { version = "0.4.3", features = ["serde"] }
indexmap = "2"
itertools = "0.14.0"
mermaid-rs-renderer = { version = "0.1.2", default-features = false }
itertools-num = "0.1.3"
kernel-density-estimation = "0.2.0"
ordered-float = "5.0.0"
petgraph = "0.7"
petgraph = "0.8"
rand = "0.9"
regex = "1"
pathdiff = "0.2.3"
serde = { version = "1.0.217", features = ["derive"] }
Expand Down
241 changes: 145 additions & 96 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,21 @@ Gizmos for working with CSVs
* [csvplot](#csvplot) -- line and scatter plots from CSV files
* [csvstats](#csvstats) -- histograms and summary statistics for CSV files
* [csvcat](#csvcat) -- concatenate CSV files
* [can2k](#can2k) -- parse NMEA 2000 GPS data from CAN logs
* [qgsdir](#qgsdir) -- generate QGIS projects from directories of CSV files
* [csvdelta](#csvdelta) -- calculate inter-row deltas for CSV files
* [minpath](#minpath) -- shorten file paths to minimal unique suffixes
* [depconv](#depconv) -- convert dependency graphs between formats
* [depfilter](#depfilter) -- filter or select subsets of dependency graphs
* [deptransform](#deptransform) -- transform dependency graphs
* [can2k](#can2k) -- parse NMEA 2000 GPS data from CAN logs
* [can2csv](#can2csv) -- parse CAN logs into CSV files
* [canspam](#canspam) -- generate random CAN traffic
* [canstruct](#canstruct) -- reconstruct NMEA 2000 Fast Packet / ISO 11783-3 Transport Protocol
sessions
* [qgsdir](#qgsdir) -- generate QGIS projects from directories of CSV files
* [depconv](#depconv) -- convert dependency graphs between formats
* [depfilter](#depfilter) -- filter or select subsets of dependency graphs
* [deptransform](#deptransform) -- transform dependency graphs
* [depquery](#depquery) -- query properties of dependency graphs
* [depcluster](#depcluster) -- cluster dependency graphs using community detection
* [graphdiff](#graphdiff) -- compare two dependency graphs
* [bbclasses](#bbclasses) -- generate BitBake recipe inheritance diagrams
* [minpath](#minpath) -- shorten file paths to minimal unique suffixes

# Philosophy

Expand Down Expand Up @@ -123,6 +125,27 @@ foo,bar,baz
7,8,9
```

## csvdelta

Calculate the inter-row deltas for a CSV column. Useful for understanding the time between events.
Also supports mean-centering a column, or centering it around a specific value.

```sh
$ csvdelta --column foo <<EOF
foo,bar
0,a
1,b
3,c
5,d
EOF

foo,bar,foo-deltas
0,a,
1,b,1
3,c,2
5,d,2
```

## can2k

Parse NMEA 2000 GPS data out of a candump into a CSV file that QGIS can load with minimal effort.
Expand All @@ -140,59 +163,65 @@ src,seq_id,longitude_deg,latitude_deg,altitude_m,sog_mps,cog_deg_cwfn,cog_ref,me
>
> If you want to use `can2k` together with `qgsdir`, you need to use `can2k --wkt`.

## qgsdir
## can2csv

Generate a QGIS project from a directory of CSV layer files. Each CSV file is assumed to have a
column of WKT geometries named `geometry` (QGIS's geometry heuristics don't appear to be exposed via
their Python API).
Parse basic data from a CAN frame into a CSV record. Faster than `sed`, and also parses the canid.
Useful in conjunction with `csvdelta` to understand message timing.

`can2csv` is not a real CAN parser, and does not understand any of the data transmitted via CAN.

```sh
$ can2k --wkt ./data/n2k-sample.log ./data/n2k.csv
$ qgsdir --open ./data/n2k.csv
$ head -n 3 data/candump-random-data.log | can2csv
timestamp,interface,canid,dlc,priority,src,dst,pgn,data
1739229594.465994,can0,0xE9790B5,8,3,0xB5,0x90,0x29700,CA3F871A5A6EE75F
1739229594.467052,can0,0xD15F192,8,3,0x92,0xF1,0x11500,500B3766CB2DED7C
```

You may pass directories or files. If you pass a directory, the script is able to group layers by
subdirectory, leading to an easier-to-use layer tree.
If you pass `--reconstruct`, then `can2csv` will reconstruct any transport layer sessions it can
understand. Right now that's just NMEA 2000 Fast Packet, but ISO-11783 Transport Protocol is
planned.

## csvdelta
## canspam

Calculate the inter-row deltas for a CSV column. Useful for understanding the time between events.
Also supports mean-centering a column, or centering it around a specific value.
The [canspam](./scripts/canspam) script can generate random CAN traffic on a Linux CAN device. It's
useful for inflating busload, or for generating random traffic to test `can2csv` against ;)

## canstruct

The `canstruct` tool is a NMEA 2000 Fast Packet / ISO 11783-3 Transport Protocol transport session
reconstruction tool. That is, you give it the individual 8-byte frames, and it gives you the
reconstructed messages.

```sh
$ csvdelta --column foo <<EOF
foo,bar
0,a
1,b
3,c
5,d
EOF
$ cat data/abort-then-full.log
(1750963033.251412) can0 18EC2A1C#101600040400EF00 // TP.CM_RTS
(1750963033.270725) can0 18EC1C2A#FF01FFFFFF00EF00 // TP.Conn_Abort
(1750963079.757877) can0 18EC2A1C#101600040400EF00 // TP.CM_RTS
(1750963079.775206) can0 18EC1C2A#110401FFFF00EF00 // TP.CM_CTS
(1750963079.778342) can0 14EB2A1C#0111111111111111 // TP.DT
(1750963079.779468) can0 14EB2A1C#0222222222222222 // TP.DT
(1750963079.780613) can0 14EB2A1C#0333333333333333 // TP.DT
(1750963079.781778) can0 14EB2A1C#0444FFFFFFFFFFFF // TP.DT
(1750963079.795905) can0 18EC1C2A#13160004FF00EF00 // TP.CM_EndofMsgACK

foo,bar,foo-deltas
0,a,
1,b,1
3,c,2
5,d,2
$ canstruct data/abort-then-full.log
2025-06-28T15:36:19.051620Z WARN csvizmo::can::tp: TP.CM_ABRT 0x1C <- 0x2A reason ExistingTransportSession pgn 0xEF00
(1750963079.795905) can0 18EF2A1C#11111111111111222222222222223333333333333344
```

## minpath
## qgsdir

Shorten file paths to the minimal unique suffix. Useful for displaying lists of files in a compact
way while keeping them distinguishable.
Generate a QGIS project from a directory of CSV layer files. Each CSV file is assumed to have a
column of WKT geometries named `geometry` (QGIS's geometry heuristics don't appear to be exposed via
their Python API).

```sh
$ minpath <<EOF
/home/user/project/src/main.rs
/home/user/project/src/lib.rs
/home/user/project/tests/main.rs
EOF

src/main.rs
lib.rs
tests/main.rs
$ can2k --wkt ./data/n2k-sample.log ./data/n2k.csv
$ qgsdir --open ./data/n2k.csv
```

Multiple options are available to customize and tune the output. See `minpath --help` for details.
You may pass directories or files. If you pass a directory, the script is able to group layers by
subdirectory, leading to an easier-to-use layer tree.

## depconv

Expand Down Expand Up @@ -251,13 +280,6 @@ Converting from a rich format (DOT, cargo metadata) to a simpler one (TGF, depfi
unsupported attributes. Converting in the other direction preserves graph topology but cannot
recover lost metadata.

> [!NOTE]
>
> DOT parsing requires building with `--features dot`, which pulls in the `dot-parser` crate
> (GPL-2.0). The default build does not include this feature and is MIT-licensed. When built with
> `--features dot`, the resulting binary is GPL-2.0. DOT _emitting_ is always available (custom
> string formatting, no GPL dependency).

## depfilter

Filter or select subsets of dependency graphs. Works on the same formats as `depconv`, and is
Expand All @@ -282,11 +304,6 @@ digraph {
}
```

> [!NOTE]
>
> The `depfilter` tool shares the same GPL-2.0 license caveat as `depconv` with respect to DOT
> parsing.

## deptransform

Structural transformations on dependency graphs. Works on the same formats as `depconv`, and is
Expand All @@ -309,11 +326,6 @@ $ cat data/depconv/bitbake.curl.task-depends.dot |
deptransform sub --key=node:label 's/.*//' |
```

> [!NOTE]
>
> The `deptransform` tool shares the same GPL-2.0 license caveat as `depconv` with respect to DOT
> parsing.

## depquery

Query properties of dependency graphs. Lists nodes, edges, and computes graph metrics. Supports the
Expand All @@ -333,55 +345,73 @@ tracing-subscriber 10
The `depquery` tool supports outputting `nodes`, `edges`, and `metrics`. The output is intended to
be machine-readable, and is tab-separated.

> [!NOTE]
>
> The `depquery` tool shares the same GPL-2.0 license caveat as `depconv` with respect to DOT
> parsing.

## can2csv

Parse basic data from a CAN frame into a CSV record. Faster than `sed`, and also parses the canid.
Useful in conjunction with `csvdelta` to understand message timing.
## depcluster

`can2csv` is not a real CAN parser, and does not understand any of the data transmitted via CAN.
Run community detection on a dependency graph to identify clusters of related nodes. Each cluster
becomes a subgraph in the output, with cross-cluster edges at the top level. Supports Louvain
(default), Leiden, and Label Propagation algorithms.

```sh
$ head -n 3 data/candump-random-data.log | can2csv
timestamp,interface,canid,dlc,priority,src,dst,pgn,data
1739229594.465994,can0,0xE9790B5,8,3,0xB5,0x90,0x29700,CA3F871A5A6EE75F
1739229594.467052,can0,0xD15F192,8,3,0x92,0xF1,0x11500,500B3766CB2DED7C
$ echo -e "a b\na c\nb c\nd e\nd f\ne f\n#\na b\na c\nb c\nd e\nd f\ne f" |
depcluster -I tgf -O mermaid
```

If you pass `--reconstruct`, then `can2csv` will reconstruct any transport layer sessions it can
understand. Right now that's just NMEA 2000 Fast Packet, but ISO-11783 Transport Protocol is
planned.
```mermaid
flowchart LR
subgraph cluster_0
a
b
c
a --> b
a --> c
b --> c
end
subgraph cluster_1
d
e
f
d --> e
d --> f
e --> f
end
```

## canspam
## graphdiff

The [canspam](./scripts/canspam) script can generate random CAN traffic on a Linux CAN device. It's
useful for inflating busload, or for generating random traffic to test `can2csv` against ;)
Compare two dependency graphs and report what changed. Nodes are matched by ID, and edges by their
endpoints.

## canstruct
`graphdiff` supports several subcommands:

The `canstruct` tool is a NMEA 2000 Fast Packet / ISO 11783-3 Transport Protocol transport session
reconstruction tool. That is, you give it the individual 8-byte frames, and it gives you the
reconstructed messages.
* `graphdiff annotate` -- output the combined graph with changes highlighted (added, removed,
changed nodes/edges get distinct attributes)
* `graphdiff list` -- tab-delimited list of changes (`+` added, `-` removed, `~` changed, `>` moved)
* `graphdiff summary` -- tab-delimited counts of each change type
* `graphdiff subtract` -- set difference: nodes and edges only in the first graph

```sh
$ cat data/abort-then-full.log
(1750963033.251412) can0 18EC2A1C#101600040400EF00 // TP.CM_RTS
(1750963033.270725) can0 18EC1C2A#FF01FFFFFF00EF00 // TP.Conn_Abort
(1750963079.757877) can0 18EC2A1C#101600040400EF00 // TP.CM_RTS
(1750963079.775206) can0 18EC1C2A#110401FFFF00EF00 // TP.CM_CTS
(1750963079.778342) can0 14EB2A1C#0111111111111111 // TP.DT
(1750963079.779468) can0 14EB2A1C#0222222222222222 // TP.DT
(1750963079.780613) can0 14EB2A1C#0333333333333333 // TP.DT
(1750963079.781778) can0 14EB2A1C#0444FFFFFFFFFFFF // TP.DT
(1750963079.795905) can0 18EC1C2A#13160004FF00EF00 // TP.CM_EndofMsgACK
$ cat before.tgf
a Alpha
b Bravo
#
a b

$ canstruct data/abort-then-full.log
2025-06-28T15:36:19.051620Z WARN csvizmo::can::tp: TP.CM_ABRT 0x1C <- 0x2A reason ExistingTransportSession pgn 0xEF00
(1750963079.795905) can0 18EF2A1C#11111111111111222222222222223333333333333344
$ cat after.tgf
b Bravo
c Charlie
#
b c

$ graphdiff annotate before.tgf after.tgf -O mermaid
```

```mermaid
flowchart LR
b["Bravo"]
c["+ Charlie"]
a["- Alpha"]
b --> c
a --> b
```

## bbclasses
Expand Down Expand Up @@ -453,3 +483,22 @@ flowchart LR
poky/meta/conf/distro/include/ptest-packagelists.inc -->|"require"| poky/meta/classes-recipe/ptest.bbclass
poky/meta/recipes-support/curl/curl_8.7.1.bb -->|"appends"| meta-work/recipes-support/curl/curl__.bbappend
```

## minpath

Shorten file paths to the minimal unique suffix. Useful for displaying lists of files in a compact
way while keeping them distinguishable.

```sh
$ minpath <<EOF
/home/user/project/src/main.rs
/home/user/project/src/lib.rs
/home/user/project/tests/main.rs
EOF

src/main.rs
lib.rs
tests/main.rs
```

Multiple options are available to customize and tune the output. See `minpath --help` for details.
2 changes: 2 additions & 0 deletions crates/csvizmo-depgraph/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,11 @@ dot-parser = { workspace = true, optional = true }
either = { workspace = true, optional = true }
eyre.workspace = true
globset.workspace = true
graphrs.workspace = true
indexmap.workspace = true
csvizmo-minpath.workspace = true
petgraph.workspace = true
rand.workspace = true
regex.workspace = true
mermaid-rs-renderer.workspace = true
serde.workspace = true
Expand Down
Loading