diff --git a/README.md b/README.md index 7e83498..2860d4e 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,7 @@ Leaderboard Dataset GitHub - arXiv coming soon - Paper PDF + arXiv

@@ -220,7 +219,7 @@ must record. ScienceVision/ ├── harness/ vendored eval harness: Harbor routes, agents, MCP server, viewer, scorer ├── sandbox/ deterministic replay: contract export, tree build, gold self-test -├── paper/ the compiled paper PDF +├── paper/ paper publication note (the PDF lives on arXiv) ├── resources/ pinned dataset, harness, paper, and image references ├── docs/ evaluation, dataset, leaderboard, containers, project notes ├── schemas/ JSON schemas for the release records @@ -248,7 +247,7 @@ be repointed. ```bash docker login ghcr.io docker pull ghcr.io/unipat-ai/sciencevision-codex-mcp@sha256:852f66cc924105c61e5efe6f85e0b5b389e01c416f7e6c8a09fbda0a225a1c98 -docker pull ghcr.io/unipat-ai/sciencevision-scorer@sha256:b2d62b237db316b4e668b146c1195bf52d77c6fd4215b915672c73849d0fbeb3 +docker pull ghcr.io/unipat-ai/sciencevision-scorer@sha256:19e2058e6bf35724c6aa6c26355c8fd580e292fb199479d80c1c5fee1c8ce540 ``` The [container guide](docs/containers.md) has every version with its digest, the @@ -256,12 +255,10 @@ bundled-versus-external distinction, and the scorer versions. ## Citation and license -The paper is readable as a [PDF](paper/sciencevision.pdf); that copy is a working build, and the -[paper directory](paper/README.md) records its status and how a new build replaces it. The arXiv -posting, citation, and BibTeX will be published here and on the -[project page](https://unipat.ai/blog/ScienceVision) once settled. Until then, please describe -results by the setting and scorer version you used rather than by a single headline number. The -[comparison notes](docs/leaderboard.md) explain why that matters here. +The paper is published on [arXiv](https://arxiv.org/abs/2611.04087). The repository no longer +carries a compiled PDF; [`paper/README.md`](paper/README.md) records the publication status. +Please describe results by the setting and scorer version you used rather than by a single +headline number. The [comparison notes](docs/leaderboard.md) explain why that matters here. The dataset package declares **MIT** in its card, `MANIFEST.json`, and `LICENSE`; that declaration does not grant rights to the third-party assets inside it. Documentation, schemas, diff --git a/docs/containers.md b/docs/containers.md index 27d539f..68ca09c 100644 --- a/docs/containers.md +++ b/docs/containers.md @@ -24,7 +24,7 @@ The harness needs an **agent image** for the route being run, and for the extern | Bundled MCP tools, Claude Code | [`sciencevision-claude-code-mcp`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-claude-code-mcp) | `server-20261015-h20` | | MCP client, Claude Code | [`sciencevision-claude-code-mcp-client`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-claude-code-mcp-client) | `release-20261002-h20` | | External HTTP MCP worker | [`sciencevision-mcp-worker`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-mcp-worker) | `server-20260924-h20` | -| Deterministic scorer | [`sciencevision-scorer`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer) | `server-20260930-h20` | +| Deterministic scorer | [`sciencevision-scorer`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer) | `server-20261104-h20` | The naming is positional, not generational. For each agent engine, the name without a suffix is the raw-file runtime, `-2d` consumes rendered views, `-mcp` bundles the CLI together with an MCP @@ -49,12 +49,13 @@ in [evaluation.md](evaluation.md#evaluation-settings). | Version | Use | |---|---| -| `server-20261022-h20` | Reference scorer. Grades the comparators and accepted spellings the 250-task package declares | -| `server-20260930-h20` | Default of the pinned evaluation harness | +| `server-20261104-h20` | In-container scorer. Installs `harbor_adapters`, so the dataset's `harbor/templates/score.py` can import `harbor_adapters.sci3d._score_core`; the dataset's builder pins this digest | +| `server-20261022-h20` | Host-side CLI reference scorer. Grades the comparators and accepted spellings, but is a `scorer_bundle` image without `harbor_adapters`; not the separate verifier image | +| `server-20260930-h20` | Historical harness default; superseded for in-container scoring by `server-20261104-h20` | | `server-20261005-h20` | Later platform-package scorer; select it when a prepared task package specifies it | | `server-20260928b-h20` | Historical release retained for earlier task contracts | -The four have different config digests, so **a newer date is not evidence that rescoring will +These images have different config digests, so **a newer date is not evidence that rescoring will preserve a result**. Choose the scorer the run contract names and record its digest. A scorer self-test that feeds ground truth back in is not a model benchmark result. @@ -67,7 +68,7 @@ addressing is what makes a run reproducible; a tag can be repointed. docker login ghcr.io docker pull ghcr.io/unipat-ai/sciencevision-codex-mcp@sha256:852f66cc924105c61e5efe6f85e0b5b389e01c416f7e6c8a09fbda0a225a1c98 docker pull ghcr.io/unipat-ai/sciencevision-claude-code-mcp@sha256:74a05ae987097140b97ca976a406cf272866c9edbb6fa2313e897a25dab36df5 -docker pull ghcr.io/unipat-ai/sciencevision-scorer@sha256:b2d62b237db316b4e668b146c1195bf52d77c6fd4215b915672c73849d0fbeb3 +docker pull ghcr.io/unipat-ai/sciencevision-scorer@sha256:19e2058e6bf35724c6aa6c26355c8fd580e292fb199479d80c1c5fee1c8ce540 ``` Avoid `latest` for anything you intend to report. Package access is separate from repository diff --git a/docs/dataset.md b/docs/dataset.md index 166a528..01683e5 100644 --- a/docs/dataset.md +++ b/docs/dataset.md @@ -71,12 +71,12 @@ turns the package into Harbor task directories: python harbor/build_tasks.py --out /tmp/sciencevision-harbor --route cc-mcp --limit 3 ``` -That builder stages a valid Harbor tree, but it does not fold in the per-field tolerances the -reference scorer requires. Scored against the package's own ground truth, a tree built this way -gives 200 of 250 golds at 1.0: 46 tasks return `invalid_task_contract` for a missing tolerance, -and four grade partially on a contract normalization the builder does not apply. -`sandbox/build_trees.py` and `scripts.harbor.qa_gen_to_tasks` apply the derivation in -`harness/scripts/harbor/_tolerance_rules.py`; score only a tree built through them. +That builder folds the per-field tolerances the reference scorer requires into every task, using +the derivation in `harness/scripts/harbor/_tolerance_rules.py`. Scored against the package's own +ground truth, a tree built this way gives **250 of 250 golds at 1.0**. If you build with an older +package revision, apply the same derivation from `sandbox/build_trees.py` or +`scripts.harbor.qa_gen_to_tasks`; a tree without tolerances returns `invalid_task_contract` for +numeric fields instead of grading them. ## What this release is diff --git a/docs/project.md b/docs/project.md index 0326c46..c6e81fc 100644 --- a/docs/project.md +++ b/docs/project.md @@ -57,5 +57,5 @@ The dataset's declared limitations and the provenance notes in a given score covers. The project has a paper. Its citation, author list, and published figures come from the final -version; the [current build](../paper/sciencevision.pdf) is a placeholder until the arXiv posting. +version; the published paper is on [arXiv](https://arxiv.org/abs/2611.04087). This repository records the release artifacts and the comparison those figures were drawn from. diff --git a/paper/README.md b/paper/README.md index a06f755..d5ba1a5 100644 --- a/paper/README.md +++ b/paper/README.md @@ -1,43 +1,13 @@ # Paper -`paper/sciencevision.pdf` is the single copy of the compiled paper: -**ScienceVision: Can AI Observe and Reason Like a Scientist?** (UniPat AI, Elorian AI). +**ScienceVision: Can AI Observe and Reason Like a Scientist?** (UniPat AI, Elorian AI) is +published on arXiv: [2611.04087](https://arxiv.org/abs/2611.04087). -The filename is stable so links from the [repository README](../README.md), the citation section, and -any page outside this repository keep resolving across rebuilds: a new build replaces the bytes, not -the path. GitHub renders the PDF in the browser and offers a download. +The repository no longer carries a compiled PDF. arXiv is the canonical copy, so there is no +local file to hash or replace. [`resources/paper.json`](../resources/paper.json) records the +publication status and the arXiv identifier. -| | | -|---|---| -| Status | `placeholder` — a working build, not the camera-ready | -| arXiv | not posted; [`resources/paper.json`](../resources/paper.json) records `"arxiv": null` | -| Pages | 29 | -| Built | 2026-10-30T07:27:24+01:00 (XeLaTeX / xdvipdfmx 0.1) | -| Size | 1,407,201 bytes | -| SHA-256 | `2026aef0ac556c1e02b59a8fc67968460d760f9d9c32ba1774cdbb3d79904d21` | +## Updating the record -Open [`sciencevision.pdf`](sciencevision.pdf), or the Paper badge at the top of the README. Nothing -here is generated at build time: what is committed is what a reader downloads. - -## Publishing a new build - -1. Overwrite `paper/sciencevision.pdf`. Keep the name; do not put a date or version in the filename. -2. Update [`resources/paper.json`](../resources/paper.json): `status`, `arxiv`, `pages`, `built_at`, - `bytes`, `sha256`. `sha256sum paper/sciencevision.pdf` prints the last one. -3. Commit the two files together. Nothing else needs regenerating. - -CI hashes the PDF against the recorded value and checks that `status` and `arxiv` agree -(`scripts/validate_repository.py`). A replaced PDF with a stale record fails validation; that is what -the record is for, not an obstacle to work around. Setting `arxiv` to an identifier and `status` to -`published` is the whole switch from placeholder to posted paper. - -## What belongs here - -- The compiled PDF and this note. `resources/paper.json` is the machine-readable record; keep the two - in step. -- LaTeX sources, figures, and review notes stay in the paper workspace. A source tree here would - drift from the committed PDF, and nothing in this repository would notice. - -You may commit a build before the paper is settled, provided it is cleared for the repository's -current visibility. Git history keeps every replaced build, so a draft committed to a public -repository stays readable after the final PDF replaces it. +When a new arXiv version is posted, update `resources/paper.json` (`verified_at`, `arxiv`, `url`, +and `note`) and this note. No PDF is committed here. diff --git a/paper/sciencevision.pdf b/paper/sciencevision.pdf deleted file mode 100644 index 1dd7cb4..0000000 Binary files a/paper/sciencevision.pdf and /dev/null differ diff --git a/resources/dataset.json b/resources/dataset.json index 0a1f22c..88ed9cc 100644 --- a/resources/dataset.json +++ b/resources/dataset.json @@ -1,17 +1,17 @@ { "schema_version": "sciencevision.dataset_reference.v1", - "verified_at": "2026-10-29T17:05:00+00:00", + "verified_at": "2026-11-04T07:35:00+00:00", "status": "published_release_pinned", "published_name": "ScienceVision", "source": { "repository": "UnipatAI/ScienceVision", "url": "https://huggingface.co/datasets/UnipatAI/ScienceVision", "revision_type": "commit", - "revision_name": "a63a513da98ca34d7efba3c547255c8be54effb1", - "commit": "a63a513da98ca34d7efba3c547255c8be54effb1", - "private": true, - "gated": false, - "snapshot_note": "The canonical dataset repository now exists; this record pins the package as uploaded. Its MANIFEST names the package ScienceVision, version 1.0, created 2026-10-20; the upload landed 2026-10-21. The 2026-10-29 uploads added the release-blog links to the card and the citation, repaired the integrity manifest against the served files, and reconciled the build tree to the served bytes; data/tasks.jsonl and the asset payload are unchanged, so the index hash still applies." + "revision_name": "e4ee384776d4c38d208ad22b6e2b11f80e81957e", + "commit": "e4ee384776d4c38d208ad22b6e2b11f80e81957e", + "private": false, + "gated": true, + "snapshot_note": "The record pins the 2026-11-04 upload. That upload changed only harbor/build_tasks.py, harbor/manifests.jsonl, harbor/README.md, and CHECKSUMS.sha256: the builder now writes per-field tolerances, uses Harbor's no-network mode, declares /logs/agent artifacts and workdir=/tests, and honors staged_name. The task payload and data/tasks.jsonl are unchanged, so the index hash still applies. The generated tree's gold self-test is 250/250 at 1.0." }, "canonical_repository": "UnipatAI/ScienceVision", "canonical_repository_status": "published; this record pins the uploaded package", @@ -24,7 +24,7 @@ "medical": 50 }, "file_count": 19301, - "logical_file_bytes": 9418292906, + "logical_file_bytes": 9418301906, "index_sha256": "d7a52e611b3e36a05ecf961904163206b4993a87aeaceedc9b1702b9e4709f4e", "layout": { "index": "data/tasks.jsonl", @@ -81,7 +81,7 @@ "verification": { "file_inventory_and_total_bytes": true, "index_rows_and_hash": true, - "served_files_byte_identical_to_build_staging": false, + "served_files_byte_identical_to_build_staging": true, "task_counts": 250, "full_asset_payload_checksums": false, "full_harness_reproduction": false diff --git a/resources/harness.json b/resources/harness.json index c50629f..266d09a 100644 --- a/resources/harness.json +++ b/resources/harness.json @@ -46,8 +46,10 @@ "extent_voxel_tolerance_v1" ], "dataset_compatibility": { - "dataset_commit": "a63a513da98ca34d7efba3c547255c8be54effb1", - "status": "task_builder_shipped_not_end_to_end_validated", + "dataset_commit": "e4ee384776d4c38d208ad22b6e2b11f80e81957e", + "status": "task_builder_shipped_and_gold_validated", + "gold_self_test": "250/250 at 1.0", + "in_container_scorer_smoke": "3/3 at 1.0 (string, numeric scalar plus numeric list, numeric vector)", "index_is_runnable_input": false }, "revision_note": "The complete eval-baseline branch is vendored under harness/ at source commit 6fd95252e81fb153432add0f2400d9ee1eea6697. UniPat-authored code in that subtree is distributed under the MIT License (harness/LICENSE); the upstream branch as published was Apache-2.0, which source_license_at_pinned_revision records. The reported results were produced on b47f13ac5abb54b9e287ff3131ae7e29771ec53e, which is recorded above as reported_results_commit. No second checkout is required to run the harness.", diff --git a/resources/images.json b/resources/images.json index 799fe2c..35fcf94 100644 --- a/resources/images.json +++ b/resources/images.json @@ -1,8 +1,8 @@ { "schema_version": "sciencevision.image_inventory.v1", - "verified_at": "2026-10-16T01:08:15.049088+00:00", + "verified_at": "2026-11-04T07:35:00+00:00", "status": "private_canonical_packages_verified", - "verification": "Re-read from GHCR on 2026-10-16 by scripts/verify_images.py, which reads every manifest and its config blob and compares the recorded digests against what the registry serves. A manifest digest addresses one immutable artifact; the config digest identifies the image content, so equal config digests establish equal declared uncompressed rootfs layer identities and runtime configuration. The pass did not download layer payloads or run containers. The experimental oracle runtime is an archived local build rather than a registry release, and its provenance is recorded separately on its entry.", + "verification": "Re-read from GHCR on 2026-10-16 by scripts/verify_images.py, which reads every manifest and its config blob and compares the recorded digests against what the registry serves. A manifest digest addresses one immutable artifact; the config digest identifies the image content, so equal config digests establish equal declared uncompressed rootfs layer identities and runtime configuration. The pass did not download layer payloads or run containers. The experimental oracle runtime is an archived local build rather than a registry release, and its provenance is recorded separately on its entry. On 2026-11-04 the server-20261104-h20 in-container scorer was pushed to GHCR and verified: its manifest and config were read from the registry, and the image was started locally to import harbor_adapters.sci3d._score_core and run the 3-task gold smoke.", "images": [ { "role": "raw-file Codex", @@ -141,7 +141,7 @@ "visibility": "private", "linked_repository": "UniPat-AI/ScienceVision", "package_url": "https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer", - "usage": "Default of the pinned eval-baseline harness; also used by oracle validation." + "usage": "Historical harness default; superseded for in-container scoring by server-20261104-h20." }, { "role": "platform package scorer", @@ -167,7 +167,7 @@ "visibility": "private", "linked_repository": "UniPat-AI/ScienceVision", "package_url": "https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer", - "usage": "Grades the comparators and accepted spellings the ScienceVision package declares. Built from harness/scorer_bundle at 273e32d, scorer manifest sha256 b32b1fa1e87ddcf86d8ea3b21eda37f31655e4cca4ef418a5c7170d3e0764f22." + "usage": "Host-side CLI gold self-test only. This is a scorer_bundle image: it contains scorer_cli.py and the deterministic scorers, but not harbor_adapters.sci3d._score_core, so it cannot serve as the separate verifier image that harbor/templates/score.py requires. Built from harness/scorer_bundle at 273e32d, scorer manifest sha256 b32b1fa1e87ddcf86d8ea3b21eda37f31655e4cca4ef418a5c7170d3e0764f22." }, { "role": "experimental oracle MCP Claude Code", @@ -231,6 +231,19 @@ "config_digest": "sha256:45826f37cf1c8f8a01515f268278d95dc7661ad2c262cfa1096a81f44394cd4c", "rootfs_diff_id_count": 23, "platform": "linux/amd64" + }, + { + "role": "in-container scorer", + "canonical": "ghcr.io/unipat-ai/sciencevision-scorer:server-20261104-h20", + "canonical_manifest_digest": "sha256:19e2058e6bf35724c6aa6c26355c8fd580e292fb199479d80c1c5fee1c8ce540", + "canonical_digest_reference": "ghcr.io/unipat-ai/sciencevision-scorer@sha256:19e2058e6bf35724c6aa6c26355c8fd580e292fb199479d80c1c5fee1c8ce540", + "config_digest": "sha256:c4abdb0f13cd57b4868123233490429ed6383b2396f8c916ce4623a08da24f81", + "rootfs_diff_id_count": 10, + "platform": "linux/amd64", + "visibility": "public", + "linked_repository": "UniPat-AI/ScienceVision", + "package_url": "https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer", + "usage": "Required by the dataset's harbor/templates/score.py: installs harbor_adapters.sci3d._score_core. Built from harness/scripts/harbor/templates/Dockerfile at f6b0929. Verified by importing the scoring core in the image and by a 3-task in-container gold smoke (3/3 at 1.0)." } ], "repository_linked_at": "2026-10-06", diff --git a/resources/paper.json b/resources/paper.json index 70be6d4..f4c65e6 100644 --- a/resources/paper.json +++ b/resources/paper.json @@ -1,17 +1,14 @@ { "schema_version": "sciencevision.paper_reference.v1", - "verified_at": "2026-10-30T06:30:00+00:00", + "verified_at": "2026-11-04T07:45:00+00:00", "title": "ScienceVision: Can AI Observe and Reason Like a Scientist?", "authors": [ "UniPat AI", "Elorian AI" ], - "path": "paper/sciencevision.pdf", - "status": "placeholder", - "arxiv": null, - "pages": 29, - "built_at": "2026-10-30T07:27:24+01:00", - "bytes": 1407201, - "sha256": "2026aef0ac556c1e02b59a8fc67968460d760f9d9c32ba1774cdbb3d79904d21", - "note": "Working build committed as the placeholder until the arXiv version is posted. The path is stable across rebuilds; this record is what CI hashes against." + "path": null, + "status": "published", + "arxiv": "2611.04087", + "url": "https://arxiv.org/abs/2611.04087", + "note": "Published on arXiv; the repository no longer carries a compiled PDF. arXiv is the canonical copy." } diff --git a/scripts/validate_repository.py b/scripts/validate_repository.py index a5dccf1..e8b0260 100644 --- a/scripts/validate_repository.py +++ b/scripts/validate_repository.py @@ -146,11 +146,16 @@ def validate_references() -> None: require(paper["status"] in {"placeholder", "published"}, "unknown paper status") require((paper["arxiv"] is not None) == (paper["status"] == "published"), "paper status disagrees with the recorded arXiv identifier") - paper_path = ROOT / paper["path"] - require(paper_path.is_file(), f"missing paper PDF: {paper['path']}") - require(paper_path.stat().st_size == paper["bytes"], "paper byte count mismatch") - require(hashlib.sha256(paper_path.read_bytes()).hexdigest() == paper["sha256"], - "paper hash mismatch") + if paper["status"] == "published": + require(str(paper.get("url", "")).startswith("https://arxiv.org/abs/"), + "published paper must record its arXiv URL") + paper_path_value = paper.get("path") + if paper_path_value: + paper_path = ROOT / paper_path_value + require(paper_path.is_file(), f"missing paper PDF: {paper_path_value}") + require(paper_path.stat().st_size == paper["bytes"], "paper byte count mismatch") + require(hashlib.sha256(paper_path.read_bytes()).hexdigest() == paper["sha256"], + "paper hash mismatch") references = set() for item in images["images"]: expected = item["canonical"].rsplit(":", 1)[0] + "@" + item["canonical_manifest_digest"]