Repository navigation
W-23900223 : Hyper Query Plan - Replace raw property dumps with tidy node rows. - #173
rbrahmachary wants to merge 2 commits into
Conversation
f5c2c86 to
5667285
Compare
|
The tooltip support landed via #212; rebasing this PR on top of latest |
5667285 to
2c04c60
Compare
2c04c60 to
7a84e0a
Compare
vogelsgesang
left a comment
There was a problem hiding this comment.
partial review; not completely through, yet
|
|
||
| // Render a map of properties, one row each: scalars as `ScalarRow`, nested | ||
| // groups as the expandable `GroupRow`. Used by node bodies and the plan metadata. | ||
| export function PropertyList({properties, expandGroups}: {properties: Map<string, PropertyEntry>; expandGroups?: boolean}) { |
There was a problem hiding this comment.
PropertyList should go into a separate, reusable file (given it's used by both QueryNode and the metadata panel now)
Also split QueryNode.css accordingly
| getEstimatedCardinality?(rawNode: JsonObject): number | undefined; | ||
| /** Prefer the actual cardinality when available and compare it with the estimate. */ | ||
| getActualCardinality?(rawNode: JsonObject): number | undefined; | ||
| /** Adjust the fully converted node (for example to curate its tooltip rows). */ |
There was a problem hiding this comment.
"curate its tooltip rows" - what do you mean by this? There are no tooltips on the nodes currently?
Rephrase comment accordingly
| // Let the format curate the fully assembled node (e.g. Hyper adds metric | ||
| // rows read from its `statistics` group). |
There was a problem hiding this comment.
Remove example. This example will almost certainly get outdated very soon
| // Let the format curate the fully assembled node (e.g. Hyper adds metric | |
| // rows read from its `statistics` group). | |
| // Let the format curate the fully assembled node |
| // billion, trillion) rather than the metric `G`. Scaled values below 100 keep | ||
| // one decimal (`1.2B`), so large counts stay distinguishable; unscaled values | ||
| // keep three significant digits (`57.5`, `0.476`), so small fractions survive. | ||
| export function formatCount(x: number): string { |
There was a problem hiding this comment.
many of those functions are in the wrong file.
The idea is that loader-utils.ts should contain utility functions used internally by the loaders only. They should be implementation details.
However, many of the functions here are used also by the rendering layer. Those don't belong into loader-utils.ts.
Instead, I think we should move the formatter functions (formatCount, formatBytes, formatSeconds, formatExact, formatNumber) into query-graphs/src/number-formats.ts (outside the loaders folder, next to tree-description.ts).
The export type NumberFormat = ... should also be moved into that new number-formats.ts.
formatPropertyValue doesn't fit into number-formats.ts, though since it depends on tree-description and we would then have a circular dependency between number-formats.ts and tree-description.ts. Instead, I would recommend moving formatPropertyValue directly into PropertyList.tsx
| } | ||
|
|
||
| // Read a scalar property as a string; groups have no single value. | ||
| export function getScalarProperty(properties: Map<string, PropertyEntry> | undefined, key: string): string | undefined { |
There was a problem hiding this comment.
this loads on the already loaded tree representation and not the JSON input.
Those type of utilities (which will also be used by the plan-insights component in the future) shouldn't live in the loader-internal loader-utils.ts. Instead, they are traditionally rather in tree-description.ts, next to visitTreeNodes and similar utilities to access a converted tree
Same for getPropertyEntry, getScalarProperty.
| const recommended = ["statistics", "analyze"].some( | ||
| (block) => tryGetPropertyPath(rawNode, [block, "index-recommender", "should-recommend-candidate"]) === true, | ||
| ); | ||
| // Report only which index a scan used, noting a covering scan; the rest of `used-index` is not shown. |
There was a problem hiding this comment.
| // Report only which index a scan used, noting a covering scan; the rest of `used-index` is not shown. | |
| // Report only which index a scan used, noting a covering scan; the rest of `used-index` is not shown. | |
| // TODO: Preserve the complete index usage group once property presentation can choose which | |
| // nested entries are initially visible. |
| setMetric("execution-time", statistics["execution-time"]); | ||
|
|
||
| // Show the index-recommendation candidate only when the recommender actually suggests it. | ||
| const recommended = ["statistics", "analyze"].some( |
There was a problem hiding this comment.
large parts of this code here seem to be for backwards-compatibility. In general, we don't want to complicate the code for backwards compatibility. In the cases were we still do it, we explicitly document that this is legacy behavior and should be removed in the future. See the pre-existing TODO(2026-12-18) in this file.
Similar re "used-index" vs "usedIndex" and "index-recommendation-candidate" vs "indexRecommendationCandidate" further down.
Either remove the backwards compat code, or explain which parts are backwards compatibility and add a dated TODO comment.
| } | ||
| } | ||
|
|
||
| for (const key of ["index-recommendation-candidate", "indexRecommendationCandidate"]) { |
There was a problem hiding this comment.
| for (const key of ["index-recommendation-candidate", "indexRecommendationCandidate"]) { | |
| // Only display accepted index recommendations; remove rejected recommendations. | |
| // TODO: Preserve all recommendation candidates and move accepted/rejected classification into enrichment or Plan Insights. | |
| for (const key of ["index-recommendation-candidate", "indexRecommendationCandidate"]) { |
| "usedIndex", | ||
| "indexRecommendationCandidate", |
There was a problem hiding this comment.
mark via TODO (<date>) comment as backwards compat changes
| // Table metadata for lakehouse scans, complete, with the identifying keys first. | ||
| const tableMetadata = rawNode["table-metadata"]; | ||
| if (isJsonObject(tableMetadata)) { | ||
| const group = jsonToPropertyEntry(tableMetadata).value as Map<string, PropertyEntry>; | ||
| properties.set("table-metadata", {value: reorderProperties(group, ["identifier", "partitioned-by", "sort-order"])}); | ||
| } |
There was a problem hiding this comment.
- use a clearer comment
- don't re-convert from JSON
| // Table metadata for lakehouse scans, complete, with the identifying keys first. | |
| const tableMetadata = rawNode["table-metadata"]; | |
| if (isJsonObject(tableMetadata)) { | |
| const group = jsonToPropertyEntry(tableMetadata).value as Map<string, PropertyEntry>; | |
| properties.set("table-metadata", {value: reorderProperties(group, ["identifier", "partitioned-by", "sort-order"])}); | |
| } | |
| // Table metadata for lakehouse scans. | |
| // `alwaysPropertyKeys` already added it as properties, but we want to sort `identifier`, `partitioned-by` and `sort-order` first | |
| const tableMetadata = properties.get("table-metadata"); | |
| if (tableMetadata instanceof Map) { | |
| properties.set("table-metadata", {value: reorderProperties(tableMetadata, ["identifier", "partitioned-by", "sort-order"])}); | |
| } |
|
|
||
| // Row counts and CPU cycles, shown at an operator's top level (nested ones are | ||
| // copied there), so they read without opening any group. | ||
| const hyperRows: SurfaceRowOptions = { |
There was a problem hiding this comment.
The "row" naming is a bit confusing. Especially, we are talking about databases, which usually have a very different meaning for "rows". Rename to "properties" to stay in sync with the terminology we also use in tree-description.ts, PropertyList etc.
| const hyperRows: SurfaceRowOptions = { | |
| const hyperSurfacedProperties: SurfacePropertyOptions = { |
Same also in all other places which currently refer to this concept as "rows":
- RowProperty → PropertyPath
- rowProps → propertyPaths
- surfaceNodeRowAttributes → surfaceNodeProperties
- surfaceRowAttributes → surfaceProperties
- hyperRows → hyperPropertiesToSurface
| // Curated operator tooltip rows, in display order. Unlisted rows such as | ||
| // `operator-id` keep their order after these; groups always come last. | ||
| const operatorRowOrder = [ |
There was a problem hiding this comment.
| // Curated operator tooltip rows, in display order. Unlisted rows such as | |
| // `operator-id` keep their order after these; groups always come last. | |
| const operatorRowOrder = [ | |
| // Curated operator tooltip properties, in display order. Unlisted properties such as | |
| // `operator-id` keep their order after these; groups always come last. | |
| const operatorPropertyOrder = [ |
| } | ||
| } catch { | ||
| // The converter produced this JSON, but keep malformed manually constructed trees harmless. | ||
| const value = readStatistic(node, ["statistics", "cpu-cycles"]); |
There was a problem hiding this comment.
should be replaced by getNumericProperty.
To do so:
- change
getNumericPropertyandgetScalarPropertyto accept a path, same asgetPropertyPath getNumericProperty/ andgetScalarPropertyshould internally delegate togetPropertyPathto resolve the path; then check the type- now you can use
getNumericPropertyhere
| export interface SurfaceRowOptions { | ||
| // The plan format's row counts (and similar figures, such as CPU cycles), | ||
| // by their exact names. | ||
| rowProps: readonly RowProperty[]; | ||
| } |
There was a problem hiding this comment.
given this only accepts a single option, you don't need an interface here. Just directly pass propertyPath: RowProperty[] to surfaceNodeRowAttributes
| // Show a node's row counts without opening any group: nested ones are copied to the top level under their own | ||
| // name, right after the last top-level row count. On a name clash, the | ||
| // top-level row wins, then the first listed path. Groups are never surfaced. |
There was a problem hiding this comment.
the comment claims you would be using this to "show row counts". In fact, you are also using it for cpu-cycles etc.
In general, I highly recommend to keep the comments generic and not talk about what the callers are doing with them. You will end up with incorrect comments otherwise, and will go through unnecessary review cycles due to incorrect comments...
| assert.equal([...(tree.root.properties?.keys() ?? [])].at(-1), "statistics"); | ||
| }); | ||
|
|
||
| test("Hyper lists every property group after the scalar rows", () => { |
There was a problem hiding this comment.
I don't understand what you mean by "lists every property group after the scalar rows".
What are "rows" here? "statistics.output-rows"? But they aren't first in the property map?
| assert.equal(root.properties?.has("index-recommendation-candidate"), false); | ||
| }); | ||
|
|
||
| test("Hyper reports the complete table metadata, identifying keys first", () => { |
There was a problem hiding this comment.
what are "identifying keys"? Reword, so it's easily understandable
| test("Hyper reports the complete table metadata, identifying keys first", () => { | |
| test("Hyper puts the table identifier before other table metadata", () => { |
| // Read a scalar tooltip property as rendered, following a path into nested property groups. | ||
| export function propValue(node: TreeNode | undefined, ...path: string[]): string | undefined { |
There was a problem hiding this comment.
rename to formatProperty to make it clear that this returns the formatted property and not its value (the name propValue would suggest it returns the PropertyEntry.value)
| } | ||
|
|
||
| // Flatten a property group's scalar members, as rendered, into a plain record. | ||
| export function propGroup(node: TreeNode | undefined, key: string): Record<string, string> | undefined { |
There was a problem hiding this comment.
- also rename accordingly
- make it consistent with
propValue/formatPropertyand accept a...path: string[]. It's surprising to have two similarly named functions accepting different parameters
| export function propGroup(node: TreeNode | undefined, key: string): Record<string, string> | undefined { | |
| export function formatPropertyGroup(node: TreeNode | undefined, ...path: string[]): Record<string, string> | undefined { |
| assert.deepEqual(group.get("list"), {value: "[]"}); | ||
| }); | ||
|
|
||
| test("scalar, numeric and nested property accessors", () => { |
There was a problem hiding this comment.
when you split up loader-utils.ts, make sure to also split this test file accordingly
| export type QueryGraphNode = Node<TreeNode, "querynode">; | ||
|
|
||
| // Render a scalar property as `name: value`, colored by its emphasis. | ||
| function ScalarRow({name, entry}: {name: string; entry: PropertyEntry}) { |
There was a problem hiding this comment.
also rename on the UI level:
ScalarRow→ScalarPropertyGroupRow→PropertyGrouprows→propertiesornestedProperties
| // Inside a graph node, resize the node smoothly and keep it in place. | ||
| const nodeElement = e.currentTarget.closest<HTMLElement>(".qg-graph-node"); | ||
| if (nodeId !== undefined && animateGraphChange !== null && nodeElement !== null) { | ||
| animateGraphChange(toggle, {resizingNodes: [{nodeId, nodeElement}]}); | ||
| } else { | ||
| toggle(); | ||
| } |
There was a problem hiding this comment.
Since PropertyList is shared with the metadata panel, it should not depend directly on graph node IDs, AnimateGraphChangeContext, or the .qg-graph-node DOM structure.
Please let QueryNode provide an optional callback that wraps a group state change in the graph resize animation. The metadata panel can omit that callback. This keeps the reusable property renderer independent of graph layout.
For example, the component could accept something conceptually like:
animateChange?: (
applyChange: () => void,
sourceElement: HTMLElement,
) => void;
But the exact API is up to you :)
| <div className={cc(["qg-prop-group", {"qg-prop-recommended": entry.recommended}])}> | ||
| <button | ||
| type="button" | ||
| className={cc(["qg-prop-group-header", {"qg-prop-highlighted": entry.highlighted, "qg-expanded": expanded}])} |
There was a problem hiding this comment.
the rendering of qg-prop-recommended, qg-prop-highlighted and qg-prop-informational is inconsistent.
Why are you applying recommended to a different element than highlighted?
Why are you completely ignoring informational here?
| // Render a map of properties, one row each: scalars as `ScalarRow`, nested | ||
| // groups as the expandable `GroupRow`. Used by node bodies and the plan metadata. |
There was a problem hiding this comment.
- avoid "row" terminology; prefer "property"
- don't talk about implementation details (
ScalarRow); focus on describing the user-observable behavior instead
| // Render a map of properties, one row each: scalars as `ScalarRow`, nested | |
| // groups as the expandable `GroupRow`. Used by node bodies and the plan metadata. | |
| // Render scalar properties and expandable nested property groups. | |
| // Used by graph nodes and the plan metadata panel. |
| /* A nested, expandable property group (e.g. the raw statistics block). | ||
| A <button>, so it is keyboard operable, reset to look like a plain row. | ||
| `:where` keeps the reset below the emphasis classes (e.g. a highlighted header). */ | ||
| :where(.qg-prop-group-header) { |
There was a problem hiding this comment.
this :where here is unnecessarily complex.
:where appears to be used only to lower the specificity of the button reset so that the .qg-prop-{informational,recommended,highlighted} win despite appearing earlier in the CSS file.
Please put the base .qg-prop-group-header styles before the emphasis variants instead. Then this can use a normal class selector, and the intended cascade is clear from the source order.
You can then also merge this with the existing .qg-prop-group-header section below
| border-top: .3em solid transparent; | ||
| border-bottom: .3em solid transparent; | ||
| border-left: .35em solid hsl(0, 0%, 50%); | ||
| transition: transform 0.12s ease; |
There was a problem hiding this comment.
respect reduced-motion user preference by using a media-query
| text-align: left; | ||
| } | ||
|
|
||
| .qg-prop-group-header { |
There was a problem hiding this comment.
the styling of highlights looks inconsistent...
Note how
- the arrow on the left is squeezed very closely to the right side of the entry
- the right sides of the
pipeline-statsandcpu-cyclesare not aligned
Please also do some visual testing yourself while addressing this. I guess there might be more, similar visual inconsistencies
7a84e0a to
6372e20
Compare
Node properties used to be flat strings, so structured values such as Hyper's statistics were dumped as raw JSON text. A property is now a PropertyEntry whose value is either a scalar or a nested map of further entries, with optional highlighted, defaultVisible and collapsed flags. jsonToPropertyEntry converts arbitrary JSON into this shape, and every loader (Hyper, Postgres, DuckDB, Umbra, Tableau, XML) produces it.
The node UI renders groups as expandable headers. Every group, at any depth and for every plan format, starts collapsed and is listed after the node's plain rows. Opening a group also opens every group nested inside it.
Row counts and CPU cycles are surfaced generically by name rather than through a hard-coded list of Hyper attributes. Any attribute whose name mentions 'rows' or 'cpu cycles', even when nested inside a collapsed group, is shown at the top level of the node by default.
Numbers are shown compactly with K/M/B/T suffixes (2.3B rather than 2345678901): row counts, edge labels, and every value inside Hyper's property groups. Memory sizes use binary units (KiB/MiB/GiB). Hovering an edge spells out the exact figures, always actual before estimated ('Actual rows: 1,234,567 / Estimated rows: 980').
For Hyper, the operator body now shows the table or alias, the row counts, and the memory, CPU and time metrics. Lakehouse scans also get a table-metadata group. Hotspot nodes highlight their cpu-cycles and pipeline-stats rows to match the node's heat color.
Shared helpers (formatCount, formatBytes, reorderProperties, and the row surfacing and group collapsing passes) live in loader-utils.ts, with new unit tests in loader-utils.test.ts.
… Hyper's statistics were dumped as raw JSON text. A property is now a PropertyEntry whose value is a string, a number or a nested map of further entries, with optional highlighted, recommended and informational flags and an optional numberFormat. jsonToPropertyEntry converts arbitrary JSON into this shape (empty objects and arrays become the text {} and []), and every loader (Hyper, Postgres, DuckDB, Umbra, Tableau, XML) produces it. Plan metadata uses the same shape and is rendered with the same PropertyList component as node bodies.
The node UI renders groups as expandable headers. Headers are buttons with aria-expanded, so they work from the keyboard. Groups start collapsed (recommended ones open) and are listed after the node's plain rows. Opening a group also opens every group nested inside it, and a group remembers its state when a parent group or subtree is closed and reopened. Opening or closing a group resizes the graph node smoothly and keeps it in place. A collapsed node body is inert, so the tab key skips its hidden buttons. When a node body scrolls, a chevron at the top or bottom edge shows that more rows are hidden there.
Numbers stay numeric in the tree and are formatted only when rendered: rounded (K/M/B/T, e.g. 2.3B rather than 2345678901), exact (thousand separators), memory-bytes (KiB/MiB/GiB) or time-seconds (µs/ms/s/min/h). The format is inferred from the words of the property name: rows and cycles are rounded, memory and bytes are memory-bytes, time, elapsed, clock and seconds are time-seconds. A group named for a measure passes its format on to the numbers inside it. Names that state another unit (peak-memory-mb, r_total_time_ms) and all other numbers are shown as is, so ids, codes and offsets are never abbreviated. Hyper and DuckDB use every format; Postgres only rounded, since it reports times in milliseconds and sizes in kB.
Row surfacing is explicit rather than name-based. The Hyper loader lists its row counts by exact path (estimated-rows, output-rows, processed-rows and rows-matching-restrictions from the operator statistics, rows-matching-restrictions from the pipeline scan-stats, and the pipeline cpu-cycles), and these are copied to the top level of the node so they are visible without opening a group. Other loaders already report their row counts at the top level.
For Hyper, the operator body shows the table or alias, the row counts, and the memory, CPU and time metrics first. Lakehouse scans also get a table-metadata group. Hotspot nodes highlight their cpu-cycles and pipeline-stats rows to match the node's heat color. debug-name is kept unless it becomes the table-or-alias row.
Edge labels use the same K/M/B/T rounding. Hovering an edge spells out the exact figures, always actual before estimated ('Actual rows: 1,234,567 / Estimated rows: 980').
Shared helpers (formatCount, formatBytes, formatSeconds, formatPropertyValue, applyNumberFormats, surfaceRowAttributes, getScalarProperty, getNumericProperty, getPropertyPath, jsonToPropertyEntry) live in loader-utils.ts and replace the per-loader copies. New unit tests in loader-utils.test.ts cover the formatters, format inference, row surfacing, the accessors, JSON conversion and group ordering, plus Hyper tests for debug-name and number rendering.
6372e20 to
955f78b
Compare
Node properties used to be flat strings, so structured values such as Hyper's statistics were dumped as raw JSON text. A property is now a PropertyEntry whose value is a string, a number or a nested map of further entries, with optional highlighted, recommended and informational flags and an optional numberFormat. jsonToPropertyEntry converts arbitrary JSON into this shape (empty objects and arrays become the text {} and []), and every loader (Hyper, Postgres, DuckDB, Umbra, Tableau, XML) produces it. Plan metadata uses the same shape and is rendered with the same PropertyList component as node bodies.
The node UI renders groups as expandable headers. Headers are buttons with aria-expanded, so they work from the keyboard. Groups start collapsed (recommended ones open) and are listed after the node's plain rows. Opening a group also opens every group nested inside it, and a group remembers its state when a parent group or subtree is closed and reopened. Opening or closing a group resizes the graph node smoothly and keeps it in place. A collapsed node body is inert, so the tab key skips its hidden buttons. When a node body scrolls, a chevron at the top or bottom edge shows that more rows are hidden there.
Numbers stay numeric in the tree and are formatted only when rendered: rounded (K/M/B/T, e.g. 2.3B rather than 2345678901), exact (thousand separators), memory-bytes (KiB/MiB/GiB) or time-seconds (µs/ms/s/min/h). The format is inferred from the words of the property name: rows and cycles are rounded, memory and bytes are memory-bytes, time, elapsed, clock and seconds are time-seconds. A group named for a measure passes its format on to the numbers inside it. Names that state another unit (peak-memory-mb, r_total_time_ms) and all other numbers are shown as is, so ids, codes and offsets are never abbreviated. Hyper and DuckDB use every format; Postgres only rounded, since it reports times in milliseconds and sizes in kB.
Row surfacing is explicit rather than name-based. The Hyper loader lists its row counts by exact path (estimated-rows, output-rows, processed-rows and rows-matching-restrictions from the operator statistics, rows-matching-restrictions from the pipeline scan-stats, and the pipeline cpu-cycles), and these are copied to the top level of the node so they are visible without opening a group. Other loaders already report their row counts at the top level.
For Hyper, the operator body shows the table or alias, the row counts, and the memory, CPU and time metrics first. Lakehouse scans also get a table-metadata group. Hotspot nodes highlight their cpu-cycles and pipeline-stats rows to match the node's heat color. debug-name is kept unless it becomes the table-or-alias row.
Edge labels use the same K/M/B/T rounding. Hovering an edge spells out the exact figures, always actual before estimated ('Actual rows: 1,234,567 / Estimated rows: 980').
Shared helpers (formatCount, formatBytes, formatSeconds, formatPropertyValue, applyNumberFormats, surfaceRowAttributes, getScalarProperty, getNumericProperty, getPropertyPath, jsonToPropertyEntry) live in loader-utils.ts and replace the per-loader copies. New unit tests in loader-utils.test.ts cover the formatters, format inference, row surfacing, the accessors, JSON conversion and group ordering, plus Hyper tests for debug-name and number rendering.
Query Graph node rows:-

