[Kaap10]

Description

This PR implements regression runs and CI gating for dynaevals, enabling evaluation runs to be compared against baseline runs with configurable thresholds, failing CI builds with a non-zero exit code when quality regresses. It also adds atomic cross-version trend persistence to EvalRunStore and documents an end-to-end GitHub Actions workflow.

Related issue

Fixes #305

Changes

• CI Regression Engine (src/dynavec/eval/gate.py):

  • Implemented check_regression() and assert_no_regression() for evaluating run metrics against a baseline.
  • Metric polarity awareness: Quality metrics (recall@k, mrr, ndcg@k, faithfulness, relevance) must not drop, while latency metrics (p50_latency_ms, p95_latency_ms, error_rate) must not spike.
  • Configurable thresholds: Supports global --max-drop (relative or absolute) and per-metric threshold overrides.
  • Robust edge cases: Handled division-by-zero (baseline == 0.0), NaN detection, and floating-point epsilon tolerance (1e-9).

• Cross-Version Trend Persistence (src/dynavec/eval/trend.py):

  • Added EvalRun.from_dict().
  • Added EvalRunStore.save_json() with atomic file writes (NamedTemporaryFile + os.replace) to guard against file corruption.
  • Added EvalRunStore.load_json() supporting seamless bootstrapping on fresh repos/branches.

• CLI Gating Subcommand (src/dynavec/cli.py):

  • Added dynavec eval check --current <file> --baseline <file> [--max-drop <float>] [--threshold <METRIC=VAL>] [--store <file>] [--tag <KEY=VAL>] [--json] [--allow-missing-baseline].
  • Exits with 0 on passing runs, 1 on regressions (blocking CI), and 2 on bad inputs.

• Documentation & Workflow Guide (docs/eval_ci_gating.md):

  • Documented complete GitHub Actions workflow using actions/cache and baseline promotion on merge to main.
  • Added CLI and Python API usage references.

• Comprehensive Test Suite:

  • Added tests/test_eval_gate.py (21 unit tests covering thresholds, polarity, NaN checks, and edge cases).
  • Extended tests/test_eval_trend.py with JSON persistence and atomic write tests.
  • Extended tests/test_cli.py with CLI command tests asserting exit codes and output formatting.

Testing

• Tests pass locally (pytest tests/test_eval_gate.py tests/test_eval_trend.py tests/test_cli.py tests/test_eval.py - 93 passed)
• Ruff checks pass (ruff check src benchmarks)
• Documentation updated, if applicable (docs/eval_ci_gating.md)

Checklist

• My changes are focused and relevant to this pull request.
• I have added or updated tests where appropriate.
• I have reviewed my changes for unrelated modifications.
• I have updated documentation where necessary.