Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 7 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@
<a href="https://unipat.ai/blog/ScienceVision#leaderboard"><img alt="Leaderboard" src="https://img.shields.io/badge/Leaderboard-4285F4?style=for-the-badge&logo=google-chrome&logoColor=green"></a>
<a href="https://huggingface.co/datasets/UnipatAI/ScienceVision"><img alt="Dataset" src="https://img.shields.io/badge/Dataset-4285F4?style=for-the-badge&logo=huggingface&logoColor=yellow"></a>
<a href="https://github.com/UniPat-AI/ScienceVision"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-24292F?style=for-the-badge&logo=github&logoColor=white"></a>
<img alt="arXiv coming soon" src="https://img.shields.io/badge/arXiv-coming%20soon-red?style=for-the-badge&logo=arxiv&logoColor=white">
<a href="paper/sciencevision.pdf"><img alt="Paper PDF" src="https://img.shields.io/badge/Paper-PDF-4285F4?style=for-the-badge&logo=latex&logoColor=white"></a>
<a href="https://arxiv.org/abs/2611.04087"><img alt="arXiv" src="https://img.shields.io/badge/arXiv-2611.04087-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white"></a>
</p>
</div>

Expand Down Expand Up @@ -220,7 +219,7 @@ must record.
ScienceVision/
├── harness/ vendored eval harness: Harbor routes, agents, MCP server, viewer, scorer
├── sandbox/ deterministic replay: contract export, tree build, gold self-test
├── paper/ the compiled paper PDF
├── paper/ paper publication note (the PDF lives on arXiv)
├── resources/ pinned dataset, harness, paper, and image references
├── docs/ evaluation, dataset, leaderboard, containers, project notes
├── schemas/ JSON schemas for the release records
Expand Down Expand Up @@ -248,20 +247,18 @@ be repointed.
```bash
docker login ghcr.io
docker pull ghcr.io/unipat-ai/sciencevision-codex-mcp@sha256:852f66cc924105c61e5efe6f85e0b5b389e01c416f7e6c8a09fbda0a225a1c98
docker pull ghcr.io/unipat-ai/sciencevision-scorer@sha256:b2d62b237db316b4e668b146c1195bf52d77c6fd4215b915672c73849d0fbeb3
docker pull ghcr.io/unipat-ai/sciencevision-scorer@sha256:19e2058e6bf35724c6aa6c26355c8fd580e292fb199479d80c1c5fee1c8ce540
```

The [container guide](docs/containers.md) has every version with its digest, the
bundled-versus-external distinction, and the scorer versions.

## Citation and license

The paper is readable as a [PDF](paper/sciencevision.pdf); that copy is a working build, and the
[paper directory](paper/README.md) records its status and how a new build replaces it. The arXiv
posting, citation, and BibTeX will be published here and on the
[project page](https://unipat.ai/blog/ScienceVision) once settled. Until then, please describe
results by the setting and scorer version you used rather than by a single headline number. The
[comparison notes](docs/leaderboard.md) explain why that matters here.
The paper is published on [arXiv](https://arxiv.org/abs/2611.04087). The repository no longer
carries a compiled PDF; [`paper/README.md`](paper/README.md) records the publication status.
Please describe results by the setting and scorer version you used rather than by a single
headline number. The [comparison notes](docs/leaderboard.md) explain why that matters here.

The dataset package declares **MIT** in its card, `MANIFEST.json`, and `LICENSE`; that
declaration does not grant rights to the third-party assets inside it. Documentation, schemas,
Expand Down
11 changes: 6 additions & 5 deletions docs/containers.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The harness needs an **agent image** for the route being run, and for the extern
| Bundled MCP tools, Claude Code | [`sciencevision-claude-code-mcp`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-claude-code-mcp) | `server-20261015-h20` |
| MCP client, Claude Code | [`sciencevision-claude-code-mcp-client`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-claude-code-mcp-client) | `release-20261002-h20` |
| External HTTP MCP worker | [`sciencevision-mcp-worker`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-mcp-worker) | `server-20260924-h20` |
| Deterministic scorer | [`sciencevision-scorer`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer) | `server-20260930-h20` |
| Deterministic scorer | [`sciencevision-scorer`](https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer) | `server-20261104-h20` |

The naming is positional, not generational. For each agent engine, the name without a suffix is
the raw-file runtime, `-2d` consumes rendered views, `-mcp` bundles the CLI together with an MCP
Expand All @@ -49,12 +49,13 @@ in [evaluation.md](evaluation.md#evaluation-settings).

| Version | Use |
|---|---|
| `server-20261022-h20` | Reference scorer. Grades the comparators and accepted spellings the 250-task package declares |
| `server-20260930-h20` | Default of the pinned evaluation harness |
| `server-20261104-h20` | In-container scorer. Installs `harbor_adapters`, so the dataset's `harbor/templates/score.py` can import `harbor_adapters.sci3d._score_core`; the dataset's builder pins this digest |
| `server-20261022-h20` | Host-side CLI reference scorer. Grades the comparators and accepted spellings, but is a `scorer_bundle` image without `harbor_adapters`; not the separate verifier image |
| `server-20260930-h20` | Historical harness default; superseded for in-container scoring by `server-20261104-h20` |
| `server-20261005-h20` | Later platform-package scorer; select it when a prepared task package specifies it |
| `server-20260928b-h20` | Historical release retained for earlier task contracts |

The four have different config digests, so **a newer date is not evidence that rescoring will
These images have different config digests, so **a newer date is not evidence that rescoring will
preserve a result**. Choose the scorer the run contract names and record its digest. A scorer
self-test that feeds ground truth back in is not a model benchmark result.

Expand All @@ -67,7 +68,7 @@ addressing is what makes a run reproducible; a tag can be repointed.
docker login ghcr.io
docker pull ghcr.io/unipat-ai/sciencevision-codex-mcp@sha256:852f66cc924105c61e5efe6f85e0b5b389e01c416f7e6c8a09fbda0a225a1c98
docker pull ghcr.io/unipat-ai/sciencevision-claude-code-mcp@sha256:74a05ae987097140b97ca976a406cf272866c9edbb6fa2313e897a25dab36df5
docker pull ghcr.io/unipat-ai/sciencevision-scorer@sha256:b2d62b237db316b4e668b146c1195bf52d77c6fd4215b915672c73849d0fbeb3
docker pull ghcr.io/unipat-ai/sciencevision-scorer@sha256:19e2058e6bf35724c6aa6c26355c8fd580e292fb199479d80c1c5fee1c8ce540
```

Avoid `latest` for anything you intend to report. Package access is separate from repository
Expand Down
12 changes: 6 additions & 6 deletions docs/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,12 @@ turns the package into Harbor task directories:
python harbor/build_tasks.py --out /tmp/sciencevision-harbor --route cc-mcp --limit 3
```

That builder stages a valid Harbor tree, but it does not fold in the per-field tolerances the
reference scorer requires. Scored against the package's own ground truth, a tree built this way
gives 200 of 250 golds at 1.0: 46 tasks return `invalid_task_contract` for a missing tolerance,
and four grade partially on a contract normalization the builder does not apply.
`sandbox/build_trees.py` and `scripts.harbor.qa_gen_to_tasks` apply the derivation in
`harness/scripts/harbor/_tolerance_rules.py`; score only a tree built through them.
That builder folds the per-field tolerances the reference scorer requires into every task, using
the derivation in `harness/scripts/harbor/_tolerance_rules.py`. Scored against the package's own
ground truth, a tree built this way gives **250 of 250 golds at 1.0**. If you build with an older
package revision, apply the same derivation from `sandbox/build_trees.py` or
`scripts.harbor.qa_gen_to_tasks`; a tree without tolerances returns `invalid_task_contract` for
numeric fields instead of grading them.

## What this release is

Expand Down
2 changes: 1 addition & 1 deletion docs/project.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,5 @@ The dataset's declared limitations and the provenance notes in
a given score covers.

The project has a paper. Its citation, author list, and published figures come from the final
version; the [current build](../paper/sciencevision.pdf) is a placeholder until the arXiv posting.
version; the published paper is on [arXiv](https://arxiv.org/abs/2611.04087).
This repository records the release artifacts and the comparison those figures were drawn from.
46 changes: 8 additions & 38 deletions paper/README.md
Original file line number Diff line number Diff line change
@@ -1,43 +1,13 @@
# Paper

`paper/sciencevision.pdf` is the single copy of the compiled paper:
**ScienceVision: Can AI Observe and Reason Like a Scientist?** (UniPat AI, Elorian AI).
**ScienceVision: Can AI Observe and Reason Like a Scientist?** (UniPat AI, Elorian AI) is
published on arXiv: [2611.04087](https://arxiv.org/abs/2611.04087).

The filename is stable so links from the [repository README](../README.md), the citation section, and
any page outside this repository keep resolving across rebuilds: a new build replaces the bytes, not
the path. GitHub renders the PDF in the browser and offers a download.
The repository no longer carries a compiled PDF. arXiv is the canonical copy, so there is no
local file to hash or replace. [`resources/paper.json`](../resources/paper.json) records the
publication status and the arXiv identifier.

| | |
|---|---|
| Status | `placeholder` — a working build, not the camera-ready |
| arXiv | not posted; [`resources/paper.json`](../resources/paper.json) records `"arxiv": null` |
| Pages | 29 |
| Built | 2026-10-30T07:27:24+01:00 (XeLaTeX / xdvipdfmx 0.1) |
| Size | 1,407,201 bytes |
| SHA-256 | `2026aef0ac556c1e02b59a8fc67968460d760f9d9c32ba1774cdbb3d79904d21` |
## Updating the record

Open [`sciencevision.pdf`](sciencevision.pdf), or the Paper badge at the top of the README. Nothing
here is generated at build time: what is committed is what a reader downloads.

## Publishing a new build

1. Overwrite `paper/sciencevision.pdf`. Keep the name; do not put a date or version in the filename.
2. Update [`resources/paper.json`](../resources/paper.json): `status`, `arxiv`, `pages`, `built_at`,
`bytes`, `sha256`. `sha256sum paper/sciencevision.pdf` prints the last one.
3. Commit the two files together. Nothing else needs regenerating.

CI hashes the PDF against the recorded value and checks that `status` and `arxiv` agree
(`scripts/validate_repository.py`). A replaced PDF with a stale record fails validation; that is what
the record is for, not an obstacle to work around. Setting `arxiv` to an identifier and `status` to
`published` is the whole switch from placeholder to posted paper.

## What belongs here

- The compiled PDF and this note. `resources/paper.json` is the machine-readable record; keep the two
in step.
- LaTeX sources, figures, and review notes stay in the paper workspace. A source tree here would
drift from the committed PDF, and nothing in this repository would notice.

You may commit a build before the paper is settled, provided it is cleared for the repository's
current visibility. Git history keeps every replaced build, so a draft committed to a public
repository stays readable after the final PDF replaces it.
When a new arXiv version is posted, update `resources/paper.json` (`verified_at`, `arxiv`, `url`,
and `note`) and this note. No PDF is committed here.
Binary file removed paper/sciencevision.pdf
Binary file not shown.
16 changes: 8 additions & 8 deletions resources/dataset.json
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
{
"schema_version": "sciencevision.dataset_reference.v1",
"verified_at": "2026-10-29T17:05:00+00:00",
"verified_at": "2026-11-04T07:35:00+00:00",
"status": "published_release_pinned",
"published_name": "ScienceVision",
"source": {
"repository": "UnipatAI/ScienceVision",
"url": "https://huggingface.co/datasets/UnipatAI/ScienceVision",
"revision_type": "commit",
"revision_name": "a63a513da98ca34d7efba3c547255c8be54effb1",
"commit": "a63a513da98ca34d7efba3c547255c8be54effb1",
"private": true,
"gated": false,
"snapshot_note": "The canonical dataset repository now exists; this record pins the package as uploaded. Its MANIFEST names the package ScienceVision, version 1.0, created 2026-10-20; the upload landed 2026-10-21. The 2026-10-29 uploads added the release-blog links to the card and the citation, repaired the integrity manifest against the served files, and reconciled the build tree to the served bytes; data/tasks.jsonl and the asset payload are unchanged, so the index hash still applies."
"revision_name": "e4ee384776d4c38d208ad22b6e2b11f80e81957e",
"commit": "e4ee384776d4c38d208ad22b6e2b11f80e81957e",
"private": false,
"gated": true,
"snapshot_note": "The record pins the 2026-11-04 upload. That upload changed only harbor/build_tasks.py, harbor/manifests.jsonl, harbor/README.md, and CHECKSUMS.sha256: the builder now writes per-field tolerances, uses Harbor's no-network mode, declares /logs/agent artifacts and workdir=/tests, and honors staged_name. The task payload and data/tasks.jsonl are unchanged, so the index hash still applies. The generated tree's gold self-test is 250/250 at 1.0."
},
"canonical_repository": "UnipatAI/ScienceVision",
"canonical_repository_status": "published; this record pins the uploaded package",
Expand All @@ -24,7 +24,7 @@
"medical": 50
},
"file_count": 19301,
"logical_file_bytes": 9418292906,
"logical_file_bytes": 9418301906,
"index_sha256": "d7a52e611b3e36a05ecf961904163206b4993a87aeaceedc9b1702b9e4709f4e",
"layout": {
"index": "data/tasks.jsonl",
Expand Down Expand Up @@ -81,7 +81,7 @@
"verification": {
"file_inventory_and_total_bytes": true,
"index_rows_and_hash": true,
"served_files_byte_identical_to_build_staging": false,
"served_files_byte_identical_to_build_staging": true,
"task_counts": 250,
"full_asset_payload_checksums": false,
"full_harness_reproduction": false
Expand Down
6 changes: 4 additions & 2 deletions resources/harness.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,10 @@
"extent_voxel_tolerance_v1"
],
"dataset_compatibility": {
"dataset_commit": "a63a513da98ca34d7efba3c547255c8be54effb1",
"status": "task_builder_shipped_not_end_to_end_validated",
"dataset_commit": "e4ee384776d4c38d208ad22b6e2b11f80e81957e",
"status": "task_builder_shipped_and_gold_validated",
"gold_self_test": "250/250 at 1.0",
"in_container_scorer_smoke": "3/3 at 1.0 (string, numeric scalar plus numeric list, numeric vector)",
"index_is_runnable_input": false
},
"revision_note": "The complete eval-baseline branch is vendored under harness/ at source commit 6fd95252e81fb153432add0f2400d9ee1eea6697. UniPat-authored code in that subtree is distributed under the MIT License (harness/LICENSE); the upstream branch as published was Apache-2.0, which source_license_at_pinned_revision records. The reported results were produced on b47f13ac5abb54b9e287ff3131ae7e29771ec53e, which is recorded above as reported_results_commit. No second checkout is required to run the harness.",
Expand Down
21 changes: 17 additions & 4 deletions resources/images.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"schema_version": "sciencevision.image_inventory.v1",
"verified_at": "2026-10-16T01:08:15.049088+00:00",
"verified_at": "2026-11-04T07:35:00+00:00",
"status": "private_canonical_packages_verified",
"verification": "Re-read from GHCR on 2026-10-16 by scripts/verify_images.py, which reads every manifest and its config blob and compares the recorded digests against what the registry serves. A manifest digest addresses one immutable artifact; the config digest identifies the image content, so equal config digests establish equal declared uncompressed rootfs layer identities and runtime configuration. The pass did not download layer payloads or run containers. The experimental oracle runtime is an archived local build rather than a registry release, and its provenance is recorded separately on its entry.",
"verification": "Re-read from GHCR on 2026-10-16 by scripts/verify_images.py, which reads every manifest and its config blob and compares the recorded digests against what the registry serves. A manifest digest addresses one immutable artifact; the config digest identifies the image content, so equal config digests establish equal declared uncompressed rootfs layer identities and runtime configuration. The pass did not download layer payloads or run containers. The experimental oracle runtime is an archived local build rather than a registry release, and its provenance is recorded separately on its entry. On 2026-11-04 the server-20261104-h20 in-container scorer was pushed to GHCR and verified: its manifest and config were read from the registry, and the image was started locally to import harbor_adapters.sci3d._score_core and run the 3-task gold smoke.",
"images": [
{
"role": "raw-file Codex",
Expand Down Expand Up @@ -141,7 +141,7 @@
"visibility": "private",
"linked_repository": "UniPat-AI/ScienceVision",
"package_url": "https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer",
"usage": "Default of the pinned eval-baseline harness; also used by oracle validation."
"usage": "Historical harness default; superseded for in-container scoring by server-20261104-h20."
},
{
"role": "platform package scorer",
Expand All @@ -167,7 +167,7 @@
"visibility": "private",
"linked_repository": "UniPat-AI/ScienceVision",
"package_url": "https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer",
"usage": "Grades the comparators and accepted spellings the ScienceVision package declares. Built from harness/scorer_bundle at 273e32d, scorer manifest sha256 b32b1fa1e87ddcf86d8ea3b21eda37f31655e4cca4ef418a5c7170d3e0764f22."
"usage": "Host-side CLI gold self-test only. This is a scorer_bundle image: it contains scorer_cli.py and the deterministic scorers, but not harbor_adapters.sci3d._score_core, so it cannot serve as the separate verifier image that harbor/templates/score.py requires. Built from harness/scorer_bundle at 273e32d, scorer manifest sha256 b32b1fa1e87ddcf86d8ea3b21eda37f31655e4cca4ef418a5c7170d3e0764f22."
},
{
"role": "experimental oracle MCP Claude Code",
Expand Down Expand Up @@ -231,6 +231,19 @@
"config_digest": "sha256:45826f37cf1c8f8a01515f268278d95dc7661ad2c262cfa1096a81f44394cd4c",
"rootfs_diff_id_count": 23,
"platform": "linux/amd64"
},
{
"role": "in-container scorer",
"canonical": "ghcr.io/unipat-ai/sciencevision-scorer:server-20261104-h20",
"canonical_manifest_digest": "sha256:19e2058e6bf35724c6aa6c26355c8fd580e292fb199479d80c1c5fee1c8ce540",
"canonical_digest_reference": "ghcr.io/unipat-ai/sciencevision-scorer@sha256:19e2058e6bf35724c6aa6c26355c8fd580e292fb199479d80c1c5fee1c8ce540",
"config_digest": "sha256:c4abdb0f13cd57b4868123233490429ed6383b2396f8c916ce4623a08da24f81",
"rootfs_diff_id_count": 10,
"platform": "linux/amd64",
"visibility": "public",
"linked_repository": "UniPat-AI/ScienceVision",
"package_url": "https://github.com/orgs/UniPat-AI/packages/container/package/sciencevision-scorer",
"usage": "Required by the dataset's harbor/templates/score.py: installs harbor_adapters.sci3d._score_core. Built from harness/scripts/harbor/templates/Dockerfile at f6b0929. Verified by importing the scoring core in the image and by a 3-task in-container gold smoke (3/3 at 1.0)."
}
],
"repository_linked_at": "2026-10-06",
Expand Down
15 changes: 6 additions & 9 deletions resources/paper.json
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
{
"schema_version": "sciencevision.paper_reference.v1",
"verified_at": "2026-10-30T06:30:00+00:00",
"verified_at": "2026-11-04T07:45:00+00:00",
"title": "ScienceVision: Can AI Observe and Reason Like a Scientist?",
"authors": [
"UniPat AI",
"Elorian AI"
],
"path": "paper/sciencevision.pdf",
"status": "placeholder",
"arxiv": null,
"pages": 29,
"built_at": "2026-10-30T07:27:24+01:00",
"bytes": 1407201,
"sha256": "2026aef0ac556c1e02b59a8fc67968460d760f9d9c32ba1774cdbb3d79904d21",
"note": "Working build committed as the placeholder until the arXiv version is posted. The path is stable across rebuilds; this record is what CI hashes against."
"path": null,
"status": "published",
"arxiv": "2611.04087",
"url": "https://arxiv.org/abs/2611.04087",
"note": "Published on arXiv; the repository no longer carries a compiled PDF. arXiv is the canonical copy."
}
Loading
Loading