Skip to content

fix: docling reruns replace their files, keyword bullets kept, sidecar rm checked - #876

Open
vlad-perevezentsev wants to merge 11 commits into
intel:mainfrom
vlad-perevezentsev:fix/687-low-outside-repos
Open

vlad-perevezentsev wants to merge 11 commits into
intel:mainfrom
vlad-perevezentsev:fix/687-low-outside-repos

Conversation

@vlad-perevezentsev

Copy link
Copy Markdown
Contributor

Summary

Three of the #687 Low items outside the repos/* files, each a state that lies after a rerun or a failed step. Refs #687.

  • data import docling reruns replace their own files. materialize_markdown took any existing <stem>-<ext>.md as someone else's, so a rerun on the same inputs wrote <stem>-<ext>-2.md beside the first file, and a docs dir filled with copies. The output directory now keeps .dokimi-docling.json, a new docling_sources.v1 artifact that names each file's source:
    • A source the record names gets its own file back, replaced through write_text_atomic (fix(io): publish pins, locks, atoms, probes, cache entries and task.json through one atomic write #697).
    • Another source with the same name gets the next number, in one run or across runs (x/a.pdf → a-pdf.md, then y/a.pdf → a-pdf-2.md, and each rerun keeps its own).
    • A source is its path, relative to the output directory, links not followed: cur.pdf -> v3.pdf repointed to v4.pdf is still cur-pdf.md. The same lexical rule finds a recorded source when --out-dir is itself a link.
    • A deleted file's name stays with its source while that source exists; an entry whose file and source are both gone is dropped.
    • Never written over, as before (ingest: materialize_markdown writes through a dangling symlink it takes for a free output name #545): a file the record does not name, a directory, a link at a file's name. Nor is a link at the record's own name read or written through.
    • The claim is saved before the file is written, so a failed save leaves no file the next run cannot place. A record that cannot be read is named in a warning; its files stay and the run writes numbered ones beside them.
    • The new kind is registered like the others: schema, SCHEMA_VERSIONS, runs/kinds.py (so runs list --kind docling_sources and the CLI snapshot list it), a fixture, docs/ingestion-docling.md.
  • eval personas discover keeps a use-case bullet that names its section. The README parser checked every line for use case, application or when to use before it checked for a bullet, so - Web applications under ## Use cases was taken for a header and dropped, and the same bullet outside such a section opened one. A bullet (a list marker and a space) is now always an item; **Use cases** still opens a section. The PyGithub path had its own copies of the use-case and API-pattern parsers and fetched the README two more times; it now passes the README it holds to the one set of parsers.
  • Docker harness cells: a sidecar whose removal failed is no longer taken as gone. stop_proxy() raised only when docker rm -f timed out; a non-zero exit still set _stopped, so the verifier ran beside the egress sidecar (which holds the upstream proxy credentials) and close() skipped its retry. verifier_sidecar() had the same flag after a failed start. docker rm -f of a container that is already gone exits 0 (checked on Docker 29.1.3), so a non-zero exit now raises before the verifier: the cell errors as for any failed docker step (exit 1, no reward), and teardown retries the removal before it drops the network. fix(repos): Ctrl-C kills the cell's container by name #776 (open) does not touch harnesses/container_agent.py.

The fourth item, repos/conformance.py:331 ("failed and skipped rows can be duplicated"), is not a defect on main: docs/reach-matrix.md:107-108 says failed and skipped runs "hold no cells and are kept as they are", and test_merge_keeps_two_failed_runs_of_one_harness pins it. Within one matrix a case has one row; only merge keeps two failed runs as two. Collapsing them changes documented behaviour, so it is left out.

#687 boxes this PR covers, for the maintainer to tick (none ticked here):

  • ingest/docling_loader.py:81: a rerun writes -2.md duplicates.
  • personas/analyzer.py:181-256: the section parser drops bullets that contain the keyword.
  • harnesses/container_agent.py:342: _stopped is set even when rm fails.

Not covered, see above: conformance.py:331.

Changed output

dokimi data import docling --input raw --out-dir docs run twice on raw/a.pdf and raw/sub/a.pdf:

 docs/
   a-pdf.md
   a-pdf-2.md
-  a-pdf-3.md
-  a-pdf-4.md
+  .dokimi-docling.json

A directory converted before this change has no record, so its first rerun writes one more numbered copy of each file; after that the names stay put. runs list --kind accepts docling_sources (the CLI snapshot changes by that one choice).

A Docker cell whose sidecar removal fails now ends as an errored cell (exit 1, reward null, metrics.error = RuntimeError: docker rm dokimi-egress-… failed (exit 1); see run.log) instead of running the verifier.

Evidence

  • tests/test_687_low.py, 21 tests. On main's code 18 fail; the other 3 pass there and guard what already worked (a file no record names is kept, a directory at a recorded name is taken, **Use cases** opens a section). On the branch all 21 pass. Examples on main: two runs over a.pdf, sub/a.pdf return a-pdf-3.md, a-pdf-4.md instead of a-pdf.md, a-pdf-2.md; - Web applications under ## Use cases gives ['batch jobs'] instead of ['web applications', 'batch jobs']; a failed sidecar rm gives (0, 1.0, True) instead of (1, None, False).
  • Mutations, each caught: the own-file lookup dropped; the link, directory and record-link guards dropped; the claim saved after the write; links followed in the source id (resolve()); the free-name search taking another source's name; stale entries kept (or kept only while the file exists); the source looked up through an --out-dir link; the warning dropped; bullets detected by any leading *; a heading no longer closing a section; stop_proxy() not raising on a non-zero rm; stop_proxy_quietly() not catching it (the network is then never removed); the verifier sidecar's flag ignoring the exit code.
  • Bad inputs to materialize_markdown, with Docling faked: an empty record, invalid JSON, a record whose name is ../../evil.md (refused by the schema): each is named in a warning, a new file is written, nothing outside --out-dir; a directory at the record's name raises OSError, which data import docling reports with exit 2, as for any unwritable --out-dir.
  • make ci-local --offline on main b768023: 0 failed, 23 passed, 20 skipped (network-only steps); pytest 22512 passed, 79 skipped; changed-line coverage passes (docling_loader.py 95.7%: the two lines left are the other-Windows-drive fallback).
  • Docker smoke (tasks run with docker-oracle,docker-probe on onetbb-histogram-local-aggregation): 1/1 pass each; the probe ran behind the egress sidecar and its teardown removed it.
  • An independent review found nine points, all addressed in separate commits: links followed in the source id; a backslash in a source name aborting the import; a new source taking a deleted file's name; names reserved forever; a recorded source missed through an --out-dir link; the changelog comparing the sidecar error to a timeout; the artifacts docstring; the README fetched three times; the benchflow site below.
  • Not run: Windows (the link and backslash tests skip there); Docling itself (not installed here; every test fakes convert_to_markdown).

Not in this PR

Checklist

The README section parser checked every line for "use case",
"application" or "when to use" before it checked for a bullet, so a
bullet such as "- Web applications" under "## Use cases" was taken for
a section header and dropped, and the same bullet outside a use-case
section opened one and pulled in the bullets after it.

A bullet (a list marker and a space) is now always an item; only a line
that is not one opens a section, so "**Use cases**" still does. The
PyGithub path had its own copy of the parser; it now calls the one
parser the gh CLI path uses.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
stop_proxy() raised only when `docker rm -f` timed out; a removal that
exited non-zero (a daemon error) still set _stopped, so close() skipped
its retry and the egress sidecar, which holds the upstream proxy
credentials, could stay on the network while the verifier ran.
verifier_sidecar() had the same flag after a failed start.

`docker rm -f` of a container that is already gone exits 0, so a
non-zero exit now means the sidecar may still run: stop_proxy() raises
RuntimeError before the verifier (the cell errors, as for any failed
docker step), stop_proxy_quietly() logs it, and close() retries the
removal before it drops the network.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
materialize_markdown() took any existing <stem>-<ext>.md as someone
else's, so a rerun of `data import docling` on the same inputs wrote
<stem>-<ext>-2.md beside the first file, and a docs dir filled with
copies of each document.

The output directory now keeps .dokimi-docling.json, a new
docling_sources.v1 artifact naming each file's source (relative to the
directory). A source the record names gets its file back and replaces it
through write_text_atomic; another source with the same name gets the
next number, in one run or across runs. A file no record names, a
directory and a link of any kind are still never written over (intel#545),
and neither is a link at the record's own name. The claim is saved
before the file is written, so a failed save leaves no file the next run
cannot place. A record that cannot be read is named in a warning and its
files are kept.

The changelog fragment covers the three intel#687 fixes of this branch.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
_source_id() resolved the source, so a link and its target were one
source: an input dir holding a.pdf and link.pdf -> a.pdf wrote a-pdf.md
twice and no link-pdf.md, and a link repointed to a new version was a
new source whose rerun wrote cur-pdf-2.md beside a stale cur-pdf.md.

The source and the output directory are now made absolute without
following links (os.path.abspath), so each path discover_documents
returns is a source of its own.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The docling_sources.v1 name pattern refused `\`, a name character on
POSIX, so `raw/a\b.pdf` failed the record's validation: the import
stopped with a traceback, wrote no file and left
.dokimi-docling.json.rejected, where main wrote `a\b-pdf.md`.

The pattern now refuses only `/`. On Windows a `\` is a separator, so a
recorded name that is not a plain name there is not used.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
_dest() looked only at the file system for a free name, so once a file
was deleted, a new source with the same name took it and the record
dropped the old source's claim: over [v0, v1, v2] of m.pdf, v0 got
m-pdf.md and v1 moved to m-pdf-3.md, and m-pdf.md then held another
document's text.

The free-name search now also skips the names the record gives another
source, so a deleted file's name stays with its source.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…imeout does

A timed-out removal fails the task (exit 124, reward 0.0); a failed one
ends as an errored cell (exit 1, no reward), as any failed docker step.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
_analyze_with_pygithub() held the README already, then
_extract_use_cases() and _extract_api_patterns() each fetched it again
(two more GitHub API calls) and ran a copy of the text parsers the gh
CLI path uses. Both now get the README text; the two wrappers are gone,
and their tests call the text parsers.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
Since 8488fbc3 the free-name search skips every name in the record, and
no entry was ever dropped: with a-pdf.md deleted and raw/ renamed to
corpus/, the rerun wrote a-pdf-2.md into a directory with no other .md,
and the record kept a-pdf.md for a source that no longer exists.

An entry now holds its name only while its file or its source exists;
the others are dropped when the record is read, so the next save leaves
them out.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The source id is a lexical path relative to the output directory, but
the "source still exists" check joined it to out_dir and let the kernel
resolve `..` after following an out-dir link. With work/md -> data/md, a
recorded work/raw/v1/m.pdf counted as gone, so a new v0/m.pdf took
v1's m-pdf.md once that file was deleted.

The check now normalizes the joined path first, the rule _source_id
built the id with.

Refs intel#687

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
@vlad-perevezentsev
vlad-perevezentsev requested a review from a team as a code owner November 17, 2026 17:19
@vlad-perevezentsev vlad-perevezentsev self-assigned this Nov 17, 2026
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🔍 OpenCodeReview found 1 issue(s) in this PR.

CI at `dc147f4`: 5 failing, 20 passing, 3 skipped

Not fully read: job Test Linux Python 3.11: HTTP 404: The specified blob does not exist.; job Tests (py3.11): HTTP 404: The specified blob does not exist.; job Tests (py3.14): HTTP 404: The specified blob does not exist.

Five checks report failure; the other 23 are passing or skipped. After reading the diff, affected files, and available logs, none of the five failures can be attributed reliably: three job logs are unavailable, and the two wheel logs contain only partial test progress without a failure diagnostic. Local lint and type checking passed; no evidence-backed fix or changed-line anchor is available.

Check Status Assessment
Test Linux Python 3.11 failing Cause unclear. The check reports failure, but its job log could not be retrieved. Without the failed step or diagnostic, it is not possible to distinguish a regression from a pre-existing or infrastructure failure.
Test Linux Wheels Python 3.11 failing Cause unclear. The available log contains partial pytest progress around 27–31%, but no actual failing-test traceback or termination diagnostic. Searches returned passing tests whose names mention errors or failures, not evidence of this job's failure.
Test Linux Wheels Python 3.14 failing Cause unclear. The available log contains partial pytest progress around 27–31%, with no actual failure summary, traceback, or termination diagnostic. This does not establish whether the change caused the reported failure.
Tests (py3.11) failing Cause unclear. The check reports failure, but the job log is unavailable and no failure diagnostic was returned. The reviewed code and passing local static checks are insufficient to identify the cause.
Tests (py3.14) failing Cause unclear. The check reports failure, but the job log is unavailable and no failure diagnostic was returned. There is no evidence to classify this as a regression, pre-existing failure, or infrastructure issue.

By severity: 1 medium

Severity Category Location Finding
medium bug docling_loader.py:127 Must-fix — preserve reservations using filesystem filename identity.
Review details
  • Commit reviewed: dc147f4 against merge base b768023
  • Coverage: 15 files in 2 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 505k tokens (311k from cache), 1m15s
  • Tool calls: 33 (23 file_read, 5 code_search, 2 file_read_diff, 1 ci_status, 1 code_comment, 1 local_check_results)
  • Docling source ingestion, artifact schema, and run visibility: 8 files
  • Low-rerun analyzer and container-agent behavior: 7 files

return path
base = f"{src.stem}-{src.suffix.lstrip('.').lower()}"
names = itertools.chain([f"{base}.md"], (f"{base}-{i}.md" for i in itertools.count(2)))
return next(out_dir / n for n in names if n not in owners and not os.path.lexists(out_dir / n))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — preserve reservations using filesystem filename identity. Trigger: on a case-insensitive filesystem (Windows or default macOS), import x/A.pdf, delete A-pdf.md while retaining its source, then import y/a.pdf. Evidence: n not in owners treats a-pdf.md as different from the reserved A-pdf.md, and lexists is false because the output was deleted. Both names are then recorded for different sources, but refer to one physical file; rerunning x/A.pdf accepts that file in the own-file lookup and atomically replaces y/a.pdf's Markdown. Impact: one source silently overwrites another's converted output. Compare reservations using filesystem-aware name equivalence and add a regression covering this sequence. Workaround: use distinct stems beyond letter case or separate output directories. Severity: medium because this causes bounded corruption of regenerable Markdown, not loss of original documents.

@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi runs list` parameters changed:
    ~ changed option `--kind`: choices: ["'answers'", "'arms'", "'catalog_eval'", "'checklist'", "'cli_routes'", "'cross_judge'", "'dokimi_config'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'grounding'", "'guidance_reach'", "'headroom'", "'judge_validity'", "'judge_wording'", "'leak'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'pairwise'", "'persona_analysis'", "'personas'", "'plan'", "'plugin_delta'", "'question_panel'", "'question_quality'", "'questions'", "'ragas'", "'reach_matrix'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_ledger'", "'repo_readiness'", "'report_qa'", "'result'", "'run_context'", "'security_report'", "'skill_coverage'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"] -> ["'answers'", "'arms'", "'catalog_eval'", "'checklist'", "'cli_routes'", "'cross_judge'", "'docling_sources'", "'dokimi_config'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'grounding'", "'guidance_reach'", "'headroom'", "'judge_validity'", "'judge_wording'", "'leak'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'pairwise'", "'persona_analysis'", "'personas'", "'plan'", "'plugin_delta'", "'question_panel'", "'question_quality'", "'questions'", "'ragas'", "'reach_matrix'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_ledger'", "'repo_readiness'", "'report_qa'", "'result'", "'run_context'", "'security_report'", "'skill_coverage'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"]

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants