Skip to content

aitools: stop discover-schema column list at DESCRIBE metadata sections - #7677

Merged
denik merged 4 commits into
mainfrom
denik/aitools-describe-partition-dups
Nov 16, 2026
Merged

denik merged 4 commits into
mainfrom
denik/aitools-describe-partition-dups

Conversation

@denik

@denik denik commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Changes

discover-schema stops reading DESCRIBE TABLE columns at the first blank or # row.

Why

DESCRIBE TABLE repeats partition and clustering columns under # Partition Information / # Clustering Information. The parser skipped the header rows but kept reading, so those columns were listed twice and their null-count expression was duplicated in the generated SQL.

Tests

New acceptance test acceptance/experimental/aitools/tools/discover-schema-partitioned with real-shaped DESCRIBE output; the unit-test fixture is updated to the real layout.

Found by analyzing test coverage gaps: the code path was not covered by any test.

This pull request and its description were written by Isaac.

denik and others added 3 commits November 15, 2026 09:19
Records current behavior: partition columns are listed twice under
COLUMNS and get a duplicate null-count expression, because
parseDescribeResult reads past the '# Partition Information' section of
DESCRIBE TABLE.

Co-authored-by: Isaac <no-reply@databricks.com>
DESCRIBE TABLE lists partition (and clustering) columns in the main
column section and then repeats them under '# Partition Information' /
'# Clustering Information'. parseDescribeResult skipped the '#' header
rows but kept reading, so those columns were reported twice and got
duplicate null-count expressions. Stop at the first blank or '#' row.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Commit: d4a76ecc9 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (bf8caad) latest (v1.26.0)
TestAccept/experimental/aitools/tools/discover-schema-partitioned ✅ ❌ ❌
TestAccept/experimental/aitools/tools/discover-schema-partitioned: branch ✅ | main (bf8caad) ❌ | latest (v1.26.0) ❌

main (bf8caad):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: ">>> [CLI] experimental aitools tools discover-schema main.default.events\nCOLUMNS:\n  id: bigint\n  #tag: string\n  region: string\n\n\n=== wire SQL\n\n>>> print_requests.py --sort //api/2.0/sql/statements\nDESCRIBE TABLE `main`.`default`.`events`\nSELECT * FROM `main`.`default`.`events` LIMIT 5\nSELECT COUNT(*) AS total_rows, SUM(CASE WHEN `id` IS NULL THEN 1 ELSE 0 END) AS `id_nulls`, SUM(CASE WHEN `#tag` IS NULL THEN 1 ELSE 0 END) AS `#tag_nulls`, SUM(CASE WHEN `region` IS NULL THEN 1 ELSE 0 END) AS `region_nulls` FROM `main`.`default`.`events`\n"
        	            	actual  : ">>> [CLI] experimental aitools tools discover-schema main.default.events\nCOLUMNS:\n  id: bigint\n  region: string\n  region: string\n\n\n=== wire SQL\n\n>>> print_requests.py --sort //api/2.0/sql/statements\nDESCRIBE TABLE `main`.`default`.`events`\nSELECT * FROM `main`.`default`.`events` LIMIT 5\nSELECT COUNT(*) AS total_rows, SUM(CASE WHEN `id` IS NULL THEN 1 ELSE 0 END) AS `id_nulls`, SUM(CASE WHEN `region` IS NULL THEN 1 ELSE 0 END) AS `region_nulls`, SUM(CASE WHEN `region` IS NULL THEN 1 ELSE 0 END) AS `region_nulls` FROM `main`.`default`.`events`\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -3,3 +3,3 @@
        	            	   id: bigint
        	            	-  #tag: string
        	            	+  region: string
        	            	   region: string
        	            	@@ -12,3 +12,3 @@
        	            	 SELECT * FROM `main`.`default`.`events` LIMIT 5
        	            	-SELECT COUNT(*) AS total_rows, SUM(CASE WHEN `id` IS NULL THEN 1 ELSE 0 END) AS `id_nulls`, SUM(CASE WHEN `#tag` IS NULL THEN 1 ELSE 0 END) AS `#tag_nulls`, SUM(CASE WHEN `region` IS NULL THEN 1 ELSE 0 END) AS `region_nulls` FROM `main`.`default`.`events`
        	            	+SELECT COUNT(*) AS total_rows, SUM(CASE WHEN `id` IS NULL THEN 1 ELSE 0 END) AS `id_nulls`, SUM(CASE WHEN `region` IS NULL THEN 1 ELSE 0 END) AS `region_nulls`, SUM(CASE WHEN `region` IS NULL THEN 1 ELSE 0 END) AS `region_nulls` FROM `main`.`default`.`events`
        	            	 
        	Test:       	TestAccept/experimental/aitools/tools/discover-schema-partitioned
        	Messages:   	experimental/aitools/tools/discover-schema-partitioned/output.txt vs /tmp/TestAcceptexperimentalaitoolstoolsdiscover-schema-partitione3844949869/002/output.txt
FAIL

latest (v1.26.0): same output as main (bf8caad)

@pietern pietern added the aitools AI tools (databricks aitools) related changes label Nov 16, 2026

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

Delta allows column names that start with '#', so the name can't mark the
end of the column list; a real column such as '#tag' cut the list short.
The blank separator and the section headers are the only rows without a
data_type. Also use the fake workspace's default warehouse in the test.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik

denik commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Follow-ups (out of scope for this PR), from an adversarial review:

  • No cloud test confirms the real DESCRIBE TABLE layout for partitioned and liquid-clustered tables; one would retire the assumption that the column list ends at the first row without a type.

@denik
denik added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 330df98 Nov 16, 2026
23 checks passed
@denik
denik deleted the denik/aitools-describe-partition-dups branch November 16, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aitools AI tools (databricks aitools) related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants