Repository navigation
fix: docling reruns replace their files, keyword bullets kept, sidecar rm checked - #876
vlad-perevezentsev wants to merge 11 commits into
Conversation
The README section parser checked every line for "use case", "application" or "when to use" before it checked for a bullet, so a bullet such as "- Web applications" under "## Use cases" was taken for a section header and dropped, and the same bullet outside a use-case section opened one and pulled in the bullets after it. A bullet (a list marker and a space) is now always an item; only a line that is not one opens a section, so "**Use cases**" still does. The PyGithub path had its own copy of the parser; it now calls the one parser the gh CLI path uses. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
stop_proxy() raised only when `docker rm -f` timed out; a removal that exited non-zero (a daemon error) still set _stopped, so close() skipped its retry and the egress sidecar, which holds the upstream proxy credentials, could stay on the network while the verifier ran. verifier_sidecar() had the same flag after a failed start. `docker rm -f` of a container that is already gone exits 0, so a non-zero exit now means the sidecar may still run: stop_proxy() raises RuntimeError before the verifier (the cell errors, as for any failed docker step), stop_proxy_quietly() logs it, and close() retries the removal before it drops the network. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
materialize_markdown() took any existing <stem>-<ext>.md as someone else's, so a rerun of `data import docling` on the same inputs wrote <stem>-<ext>-2.md beside the first file, and a docs dir filled with copies of each document. The output directory now keeps .dokimi-docling.json, a new docling_sources.v1 artifact naming each file's source (relative to the directory). A source the record names gets its file back and replaces it through write_text_atomic; another source with the same name gets the next number, in one run or across runs. A file no record names, a directory and a link of any kind are still never written over (intel#545), and neither is a link at the record's own name. The claim is saved before the file is written, so a failed save leaves no file the next run cannot place. A record that cannot be read is named in a warning and its files are kept. The changelog fragment covers the three intel#687 fixes of this branch. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
_source_id() resolved the source, so a link and its target were one source: an input dir holding a.pdf and link.pdf -> a.pdf wrote a-pdf.md twice and no link-pdf.md, and a link repointed to a new version was a new source whose rerun wrote cur-pdf-2.md beside a stale cur-pdf.md. The source and the output directory are now made absolute without following links (os.path.abspath), so each path discover_documents returns is a source of its own. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The docling_sources.v1 name pattern refused `\`, a name character on POSIX, so `raw/a\b.pdf` failed the record's validation: the import stopped with a traceback, wrote no file and left .dokimi-docling.json.rejected, where main wrote `a\b-pdf.md`. The pattern now refuses only `/`. On Windows a `\` is a separator, so a recorded name that is not a plain name there is not used. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
_dest() looked only at the file system for a free name, so once a file was deleted, a new source with the same name took it and the record dropped the old source's claim: over [v0, v1, v2] of m.pdf, v0 got m-pdf.md and v1 moved to m-pdf-3.md, and m-pdf.md then held another document's text. The free-name search now also skips the names the record gives another source, so a deleted file's name stays with its source. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…imeout does A timed-out removal fails the task (exit 124, reward 0.0); a failed one ends as an errored cell (exit 1, no reward), as any failed docker step. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
_analyze_with_pygithub() held the README already, then _extract_use_cases() and _extract_api_patterns() each fetched it again (two more GitHub API calls) and ran a copy of the text parsers the gh CLI path uses. Both now get the README text; the two wrappers are gone, and their tests call the text parsers. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
Since 8488fbc3 the free-name search skips every name in the record, and no entry was ever dropped: with a-pdf.md deleted and raw/ renamed to corpus/, the rerun wrote a-pdf-2.md into a directory with no other .md, and the record kept a-pdf.md for a source that no longer exists. An entry now holds its name only while its file or its source exists; the others are dropped when the record is read, so the next save leaves them out. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The source id is a lexical path relative to the output directory, but the "source still exists" check joined it to out_dir and let the kernel resolve `..` after following an out-dir link. With work/md -> data/md, a recorded work/raw/v1/m.pdf counted as gone, so a new v0/m.pdf took v1's m-pdf.md once that file was deleted. The check now normalizes the joined path first, the rule _source_id built the id with. Refs intel#687 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
|
🔍 OpenCodeReview found 1 issue(s) in this PR. CI at `dc147f4`: 5 failing, 20 passing, 3 skippedNot fully read: job Test Linux Python 3.11: HTTP 404: The specified blob does not exist.; job Tests (py3.11): HTTP 404: The specified blob does not exist.; job Tests (py3.14): HTTP 404: The specified blob does not exist.
By severity: 1 medium
Review details
|
| return path | ||
| base = f"{src.stem}-{src.suffix.lstrip('.').lower()}" | ||
| names = itertools.chain([f"{base}.md"], (f"{base}-{i}.md" for i in itertools.count(2))) | ||
| return next(out_dir / n for n in names if n not in owners and not os.path.lexists(out_dir / n)) |
There was a problem hiding this comment.
Must-fix — preserve reservations using filesystem filename identity. Trigger: on a case-insensitive filesystem (Windows or default macOS), import x/A.pdf, delete A-pdf.md while retaining its source, then import y/a.pdf. Evidence: n not in owners treats a-pdf.md as different from the reserved A-pdf.md, and lexists is false because the output was deleted. Both names are then recorded for different sources, but refer to one physical file; rerunning x/A.pdf accepts that file in the own-file lookup and atomically replaces y/a.pdf's Markdown. Impact: one source silently overwrites another's converted output. Compare reservations using filesystem-aware name equivalence and add a regression covering this sequence. Workaround: use distinct stems beyond letter case or separate output directories. Severity: medium because this causes bounded corruption of regenerable Markdown, not loss of original documents.
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi runs list` parameters changed:
~ changed option `--kind`: choices: ["'answers'", "'arms'", "'catalog_eval'", "'checklist'", "'cli_routes'", "'cross_judge'", "'dokimi_config'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'grounding'", "'guidance_reach'", "'headroom'", "'judge_validity'", "'judge_wording'", "'leak'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'pairwise'", "'persona_analysis'", "'personas'", "'plan'", "'plugin_delta'", "'question_panel'", "'question_quality'", "'questions'", "'ragas'", "'reach_matrix'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_ledger'", "'repo_readiness'", "'report_qa'", "'result'", "'run_context'", "'security_report'", "'skill_coverage'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"] -> ["'answers'", "'arms'", "'catalog_eval'", "'checklist'", "'cli_routes'", "'cross_judge'", "'docling_sources'", "'dokimi_config'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'grounding'", "'guidance_reach'", "'headroom'", "'judge_validity'", "'judge_wording'", "'leak'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'pairwise'", "'persona_analysis'", "'personas'", "'plan'", "'plugin_delta'", "'question_panel'", "'question_quality'", "'questions'", "'ragas'", "'reach_matrix'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_ledger'", "'repo_readiness'", "'report_qa'", "'result'", "'run_context'", "'security_report'", "'skill_coverage'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"]Updated on every push; the |
Summary
Three of the #687 Low items outside the
repos/*files, each a state that lies after a rerun or a failed step. Refs #687.data import doclingreruns replace their own files.materialize_markdowntook any existing<stem>-<ext>.mdas someone else's, so a rerun on the same inputs wrote<stem>-<ext>-2.mdbeside the first file, and a docs dir filled with copies. The output directory now keeps.dokimi-docling.json, a newdocling_sources.v1artifact that names each file's source:write_text_atomic(fix(io): publish pins, locks, atoms, probes, cache entries and task.json through one atomic write #697).x/a.pdf→a-pdf.md, theny/a.pdf→a-pdf-2.md, and each rerun keeps its own).cur.pdf -> v3.pdfrepointed tov4.pdfis stillcur-pdf.md. The same lexical rule finds a recorded source when--out-diris itself a link.SCHEMA_VERSIONS,runs/kinds.py(soruns list --kind docling_sourcesand the CLI snapshot list it), a fixture,docs/ingestion-docling.md.eval personas discoverkeeps a use-case bullet that names its section. The README parser checked every line foruse case,applicationorwhen to usebefore it checked for a bullet, so- Web applicationsunder## Use caseswas taken for a header and dropped, and the same bullet outside such a section opened one. A bullet (a list marker and a space) is now always an item;**Use cases**still opens a section. The PyGithub path had its own copies of the use-case and API-pattern parsers and fetched the README two more times; it now passes the README it holds to the one set of parsers.stop_proxy()raised only whendocker rm -ftimed out; a non-zero exit still set_stopped, so the verifier ran beside the egress sidecar (which holds the upstream proxy credentials) andclose()skipped its retry.verifier_sidecar()had the same flag after a failed start.docker rm -fof a container that is already gone exits 0 (checked on Docker 29.1.3), so a non-zero exit now raises before the verifier: the cell errors as for any failed docker step (exit 1, no reward), and teardown retries the removal before it drops the network. fix(repos): Ctrl-C kills the cell's container by name #776 (open) does not touchharnesses/container_agent.py.The fourth item,
repos/conformance.py:331("failed and skipped rows can be duplicated"), is not a defect on main:docs/reach-matrix.md:107-108says failed and skipped runs "hold no cells and are kept as they are", andtest_merge_keeps_two_failed_runs_of_one_harnesspins it. Within one matrix a case has one row; onlymergekeeps two failed runs as two. Collapsing them changes documented behaviour, so it is left out.#687 boxes this PR covers, for the maintainer to tick (none ticked here):
ingest/docling_loader.py:81: a rerun writes-2.mdduplicates.personas/analyzer.py:181-256: the section parser drops bullets that contain the keyword.harnesses/container_agent.py:342:_stoppedis set even whenrmfails.Not covered, see above:
conformance.py:331.Changed output
dokimi data import docling --input raw --out-dir docsrun twice onraw/a.pdfandraw/sub/a.pdf:A directory converted before this change has no record, so its first rerun writes one more numbered copy of each file; after that the names stay put.
runs list --kindacceptsdocling_sources(the CLI snapshot changes by that one choice).A Docker cell whose sidecar removal fails now ends as an errored cell (exit 1,
rewardnull,metrics.error=RuntimeError: docker rm dokimi-egress-… failed (exit 1); see run.log) instead of running the verifier.Evidence
tests/test_687_low.py, 21 tests. On main's code 18 fail; the other 3 pass there and guard what already worked (a file no record names is kept, a directory at a recorded name is taken,**Use cases**opens a section). On the branch all 21 pass. Examples on main: two runs overa.pdf,sub/a.pdfreturna-pdf-3.md, a-pdf-4.mdinstead ofa-pdf.md, a-pdf-2.md;- Web applicationsunder## Use casesgives['batch jobs']instead of['web applications', 'batch jobs']; a failed sidecarrmgives(0, 1.0, True)instead of(1, None, False).resolve()); the free-name search taking another source's name; stale entries kept (or kept only while the file exists); the source looked up through an--out-dirlink; the warning dropped; bullets detected by any leading*; a heading no longer closing a section;stop_proxy()not raising on a non-zerorm;stop_proxy_quietly()not catching it (the network is then never removed); the verifier sidecar's flag ignoring the exit code.materialize_markdown, with Docling faked: an empty record, invalid JSON, a record whosenameis../../evil.md(refused by the schema): each is named in a warning, a new file is written, nothing outside--out-dir; a directory at the record's name raisesOSError, whichdata import doclingreports with exit 2, as for any unwritable--out-dir.make ci-local --offlineon main b768023: 0 failed, 23 passed, 20 skipped (network-only steps); pytest 22512 passed, 79 skipped; changed-line coverage passes (docling_loader.py95.7%: the two lines left are the other-Windows-drive fallback).tasks runwithdocker-oracle,docker-probeononetbb-histogram-local-aggregation): 1/1 pass each; the probe ran behind the egress sidecar and its teardown removed it.--out-dirlink; the changelog comparing the sidecar error to a timeout; the artifacts docstring; the README fetched three times; the benchflow site below.convert_to_markdown).Not in this PR
repos/conformance.py:331(Implicit state machines: correctness defects (cache replay, false passes, non-atomic markers, resume cost) #687 Low): documented behaviour, see Summary.harnesses/benchflow.py:646remove_rollout_containers: the same bug class as the sidecar flag; it reports a compose project removed whendocker rm -fexits non-zero.Checklist
pytestpasses:tests/test_687_low.py,test_personas_analyzer.py,test_runs_scan.py,test_runs_show.py; 22512 passedruff,mypy, and schema checks pass:make ci-local --offlinestatic and typecheck rows,schema_compatincludeddocs/ingestion-docling.md, CLI snapshotAGENTS.md: mutations, bad inputs and other sites aboveraise/exit and test names the doc, contract or issue it traces to: thestop_proxy()raise names Implicit state machines: correctness defects (cache replay, false passes, non-atomic markers, resume cost) #687; the tests cite Implicit state machines: correctness defects (cache replay, false passes, non-atomic markers, resume cost) #687 anddocs/ingestion-docling.md.github/.licenserc.yaml): the test file and the fragment; the schema and the JSON fixture are JSON, like the other schemas and fixtures