Skip to content

Evals for a closed product (oneMKL): execution-graded questions and tasks without a repository #808

Open

Description

@napetrov

Problem

dokimi can already build a question set for a product with no repository (eval personas discover --description, eval questions generate on doc chunks, refine/review/redteam), but every generated question is graded by a judge. Executable tasks come only from a repository (eval tasks generate --from graph|prs|replay). For a closed binary product such as oneMKL (registered in products.yaml, coverage_status: planned) there is no path to execution-graded evals.

Gaps (surveyed on main 47f89ed)

# Gap Why it matters for a closed product
G1 Executable check for answer-track questions: compile/run the answer's code in a product image, compare output with tolerance Link lines, ILP64, threading and numeric results are checkable without a judge
G2 eval tasks generate --from examples (and doc snippets): blank part of a shipped example, its output is the oracle Products ship examples/; no git history needed
G3 identifiers veto from C headers / exported symbols (nm -D), not only --help Vetoes invented APIs (cblas_dgemmx) and verifies answer keys
G4 Import forum threads / support tickets as candidate questions (cluster, keep the ones with a checkable answer) Real user task distribution
G5 Task environment = product image (apt/conda package), no source repo Repo tracks assume a source checkout
G6 Generated-task validation: oracle passes and the unchanged start / a naive answer fails Keeps synthetic tasks honest; redteam covers text checks only

Also: the onemkl entry's doc URL points at the 2024-2 reference, and repo: uxlfoundation/oneMKL is the open interfaces project, not the MKL binary.

Plan

  • G3: PR from napetrov/dokimi (in progress).
  • Seed set: ~30 hand-validated Harbor tasks for oneMKL (linking, threading, API calls, numerics, Python wrappers) as the reference for G1/G2/G5.
  • G1, G2, G5, G6, G4 after that, in this order.

Activity

  1. napetrov commented on Nov 16, 2026

    @napetrov
    ContributorAuthor

    G3: #811. Seed set done locally: 30 oneMKL Harbor tasks, 6 classes; oracle passes 30/30, unchanged start fails 30/30; tasks run --harnesses docker-oracle 30/30.
    Hit while building it:

    • no null/baseline mode in docker-oracle (G6) — needed a separate script;
    • images always tagged dokimi/<task>:latest, no prefix option;
    • tasks run --help harness list omits docker-oracle;
    • image build uses --network host with hard-coded proxy build args.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions