You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Evals for a closed product (oneMKL): execution-graded questions and tasks without a repository #808
dokimi can already build a question set for a product with no repository (eval personas discover --description, eval questions generate on doc chunks, refine/review/redteam), but every generated question is graded by a judge. Executable tasks come only from a repository (eval tasks generate --from graph|prs|replay). For a closed binary product such as oneMKL (registered in products.yaml, coverage_status: planned) there is no path to execution-graded evals.
Activity
napetrov commentedon Nov 16, 2026
G3: #811. Seed set done locally: 30 oneMKL Harbor tasks, 6 classes; oracle passes 30/30, unchanged start fails 30/30;
tasks run --harnesses docker-oracle30/30.Hit while building it:
docker-oracle(G6) — needed a separate script;dokimi/<task>:latest, no prefix option;tasks run --helpharness list omitsdocker-oracle;--network hostwith hard-coded proxy build args.