Skip to content

fix(spec): gate --strict on failures outside a known-failures list - #140

Merged
ohaibbq merged 1 commit into
mainfrom
fix/spec-gate-known-failures
Nov 13, 2026
Merged

ohaibbq merged 1 commit into
mainfrom
fix/spec-gate-known-failures

Conversation

@ohaibbq

@ohaibbq ohaibbq commented Nov 13, 2026

Copy link
Copy Markdown
Member

Fixes #139

The spec job has failed on every run of main. This PR makes zig build spec -- --strict exit 0 on the current tree while still failing on a new regression, a new failing case, or a stale unsupported_in_bigquery annotation.

This PR:

  • Adds an ordinal to the baseline key when a description repeats within a file (caseKeys in entrypoints/spec/spec_runner.zig). Many compliance files hold several cases described as prepare_database; they shared one key, so one passing case made every failing sibling a regression. That accounted for 102 of the 103 regressions. First occurrences keep their existing key.
  • Makes --strict fail only on a failing case that testdata/spec_known_failures.txt does not list. --update-baseline now rewrites that list alongside testdata/spec_baseline.txt.
  • Moves one case out of compliance/pico_timestamp, whose file-level annotation says BigQuery rejects it.

Everything else is the regenerated lists and the docs that describe them.

What BigQuery does

SELECT PARSE_TIMESTAMP("%Y-%m-%dT%H:%M:%E*S%Ez", "2017-01-01T13:54:59.123456789012345Z") returns 2017-01-01 13:54:59.123456 UTC (recidiviz-bigquery-sandbox, 2026-11-12). BigQuery accepts the query and drops the digits past the microsecond.

What the emulator did

The engine already returned that instant. The case was reported as stale because it sat under an annotation covering picosecond literals, the precision named argument and braced STRUCT constructors, none of which it uses.

Spec cases

  • Added: testdata/specs/zigquery/functions/timestamp/parse_timestamp.zon, "parse timestamp with %E*S drops fractional digits past the microsecond".
  • Removed: testdata/specs/googlesql/compliance/pico_timestamp.zon, parse_timestamp_truncates_beyond_precision.

Where to look

  • The known-failures list records all 1,541 cases that fail today, so this PR accepts them as known rather than fixing them. They include clusters in areas the engine implements: IN UNNEST over NULL and empty arrays, FLOAT64 NaN and infinity read back as NULL, RANGE frames over NUMERIC and BIGNUMERIC, and EXCEPT ALL / INTERSECT ALL. Each needs its own issue.
  • The baseline drops in_queries :: in_clause_not_in_null_selector_empty_unnest_is_null. It passed only while NOT grouped incorrectly; 3195f05 fixed the grouping and exposed the IN UNNEST bug above.
  • A known failure that starts passing is reported as newly passing and does not fail the run, the same as a baseline addition.
  • If tools/spec_import.py re-imports the compliance suite it may restore the removed pico_timestamp case. I did not check how the importer treats removed cases.

Checks

  • zig build test
  • zig build test-server
  • zig build spec -- --strict (5413 pass, 0 regressions, 0 unlisted failures, 0 stale)
  • zig fmt --check build.zig entrypoints (with tools included it trips locally on the git-ignored vendored ephor package under tools/ephor/zig-pkg/)
  • ephor reports zero findings

🤖 Generated with Claude Code

The spec job failed on every run of main for three reasons, each fatal on
its own.

Cases that share a description within a file shared one baseline key, so a
single passing prepare_database case made each failing sibling a
regression. The second and later cases with a description now carry their
ordinal in the key.

--strict failed the run on any failing case. It now fails only on a
failing case that testdata/spec_known_failures.txt does not list, and
--update-baseline rewrites that list alongside the baseline.

compliance/pico_timestamp held a case BigQuery accepts under a file-level
unsupported_in_bigquery annotation. The case moves to the engine's
PARSE_TIMESTAMP spec with the value measured from BigQuery.

Fixes #139

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ohaibbq
ohaibbq merged commit fc36a9d into main Nov 13, 2026
18 of 19 checks passed
@ohaibbq
ohaibbq deleted the fix/spec-gate-known-failures branch November 13, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spec job fails on main: duplicate case names, --strict on every failure, one stale annotation

2 participants