Repository navigation
Translation checks avoid repeated row and interval scans - #834
Merged
Merged
Conversation
Index normalized table rows while preserving first-match offsets and unmatched row order. Use prefix maximum ends for protected-line lookups. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #786
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
Translation validation indexes protected ranges and table rows to avoid repeated scans on large reference pages. Request boundaries, labels and source offsets are preserved.
Why
List protection scans every protected interval for each line. Table reconciliation repeatedly normalizes remaining candidate cells when the two Markdown dialects produce different rows. Large reference tables multiply that work during translation validation.
Scope and risk
Two pure parser paths change. Protected-line lookups use sorted starts and prefix maximum ends, retaining overlapping and unsorted intervals and exact-start behavior. Table matching queues source-row indexes by their normalized labels, preserving the first matching duplicate, source offsets, unmatched dialect rows and original remaining-row order.
Regression cases check those semantics and bound interval visits and cell copies as input grows. Existing container, table-reflow and request-boundary cases remain in the selected checks.
Compatibility, rollout and rollback
There is no wire, schema, dependency, configuration or provider change. Operators need no migration or configuration step. Rolling back restores the previous parser traversal.
Tests
Six new regression cases produce two expected failures before the repair. Afterward, 18 selected directly related cases pass. Ruff and format checks pass for both files. Full suites run in PR CI and must pass before merge.
A bounded profile on the existing 214,207-byte memory-operation reference compares old and new functions on the same input. All 82 request strings and all 6,777 labels, including offsets and metadata, are exactly equal. Under cProfile, request planning takes 2.81 versus 0.72 seconds and table-label extraction 8.18 versus 1.51 seconds. These are one-input observations, not production latency guarantees. No provider calls run.
Profile the two paths on each checkout with:
python -c 'import cProfile; from pathlib import Path; from scripts.i18n import chunks, translate; text = Path("docs/reference/operations/memory.md").read_text(); cProfile.runctx("translate.english_requests(text); chunks.table_labels(text)", globals(), locals(), sort="cumulative")'Reproduce the new semantic and work-bound regressions with
pytest tests/unit/test_i18n_request_bounds.py -q -k 'site_list_boundaries or site_lists_do_not_revisit or table_labels'.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff).
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lint,make typecheckandmake test-unitpass; the integration suite passes if server behaviour changed.docs/reference/configuration.mdand thedocs/reference/tables (python scripts/codegen/docs_reference.py), the reach manifest,schemas/schema.sql(make schema-dump) and the Codex plugin copies (python3 scripts/codegen/agent_plugin_sync.py).main; no merged migration is edited.hooks/,lib/,bin/,skills/,.mcp.json, the Codexplugin-defaults.json) bumps both plugin manifests' version, and no other open PR uses that version.npm --prefix frontend run build) and its tests pass, iffrontend/changed.docs/describe the system as it now is, and this description names any operator steps and wire changes.tests/unit/test_public_copy.py).git commit -s), as CONTRIBUTING.md asks.