feat(text-metrics): split oneig_alignment#646
feat(text-metrics): split oneig_alignment#646davidberenstein1957 wants to merge 2 commits intofeat/vlm-pr-3a-qa-accuracyfrom
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 2627d78. Configure here.
| Benchmark( | ||
| name="OneIG Portrait", | ||
| description="OneIG subset: people and portraits.", | ||
| metrics=["oneig_alignment"], |
There was a problem hiding this comment.
OneIG subset benchmarks fail to register at import
High Severity
The new OneIG benchmarks create lookup keys (e.g., OneIGAnimeStylization) that are missing from base_datasets. This causes BenchmarkRegistry._register to raise a ValueError, failing module import for pruna.evaluation.benchmarks and its dependents. The original OneIG entry was also removed.
Reviewed by Cursor Bugbot for commit 2627d78. Configure here.
| assert "21" not in record["questions"] | ||
| assert "21" not in record["dependencies"] | ||
| assert record["questions"] == {"1": "Is there a cat?"} | ||
| assert record["dependencies"] == {"1": [0]} |
There was a problem hiding this comment.
OneIG record test calls function with wrong signature
Medium Severity
The test_to_oneig_record_strips_null_questions_and_dependencies test calls _to_oneig_record with an incorrect number of arguments, causing a TypeError. The test also asserts that null-valued questions and dependencies are stripped, a behavior not implemented in _to_oneig_record.
Reviewed by Cursor Bugbot for commit 2627d78. Configure here.
| metrics=["oneig_alignment"], | ||
| task_type="text_to_image", | ||
| reference="https://arxiv.org/abs/2506.07977", | ||
| ), |
There was a problem hiding this comment.
OneIG Multilingualism subset has no alignment questions
Medium Severity
The OneIG Multilingualism benchmark wires the oneig_alignment metric, but _CATEGORY_TO_QD in prompt.py only maps Anime_Stylization, Portrait, and General_Object to Q_D files. Multilingualism rows therefore receive an empty questions dict in _to_oneig_record, and OneIGAlignmentMetric.update raises ValueError whenever questions is empty. Running this benchmark would error out on every sample rather than produce alignment scores.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 2627d78. Configure here.
Isolates qa_accuracy metric implementation and GenEval benchmark wiring so it can be reviewed independently before stacking the remaining text metrics. Made-with: Cursor
Adds oneig_alignment metric implementation, its focused tests, and benchmark subset wiring while keeping reasoning and text-rendering metrics for later stacked PRs. Made-with: Cursor
davidberenstein1957
force-pushed
the
feat/vlm-pr-3a-qa-accuracy
branch
from
15db155 to
04ab2e5
Compare
davidberenstein1957
force-pushed
the
feat/vlm-pr-3b-oneig-alignment
branch
from
2627d78 to
c51653e
Compare
davidberenstein1957
force-pushed
the
feat/vlm-pr-3a-qa-accuracy
branch
from
04ab2e5 to
161223e
Compare
1 participant
Summary
Splits
oneig_alignmentinto its own stacked PR, addsOneIGAlignmentMetric, and wires OneIG alignment subset benchmarks.Stack Position
feat/vlm-pr-3a-qa-accuracy)feat/vlm-pr-3c-text-score-pair)feat/vlm-pr-5-e2e-tests)feat/metrics-vlm-support)Files
src/pruna/evaluation/metrics/metric_oneig_alignment.pysrc/pruna/evaluation/benchmarks.pytests/evaluation/test_text_metrics.pyTest Plan
Review Focus
Review Flow (Order)
Review the stack in this exact order:
This PR in the flow (4/10)