You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Expand the disconnected-cluster guide with per-benchmark asset requirements and example S3 prefixes for all 24 curated benchmark entries across nine collections.
Document staged-data layouts and tokenizer handling for LM Evaluation Harness, LightEval, RULER, and Inspect (HLE, DocVQA, CyberSecEval 2, and BFCL), plus target/judge connectivity and adapter image verification.
Correct the Crows Pairs mapping to jannalu/crows_pairs_multilingual with the English subset.
This repository does not receive automatic reviews because it has fewer than 10 stars.
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: a90d0cf8-867e-4897-920d-234e6c0e364f
No actionable comments were generated in the recent review.
Walkthrough
The disconnected-cluster guide now lists curated benchmark assets and prefixes, updates LM Evaluation Harness bundle examples, and adds staging and connectivity instructions for LightEval, RULER, and Inspect benchmarks.
Changes: Disconnected evaluation assets
Layer / File(s)
Summary
Benchmark inventory and access requirements src/content/docs/guides/disconnected-cluster.mdx
The guide lists curated benchmarks, example S3 prefixes, and required assets. It distinguishes staged assets from target, judge, and image registry connectivity requirements.
LM Evaluation Harness bundles and references src/content/docs/guides/disconnected-cluster.mdx
Dataset and tokenizer examples now use benchmark-specific bundles and S3 prefixes. The guide updates prefix syncing and BFCL references to use the disconnected Tool Use v1 prefix.
The documentation expands disconnected asset-staging guidance. No actionable merge-blocking issue was established; it is ready to merge after normal checks.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title clearly and concisely describes the main change: expanding the disconnected curated collections guide.
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description check
The description clearly explains the documentation changes, motivation, related issue, and validation results. It does not use the template headings for Type or Breaking changes, and it does not expli…
Full details: Description check
Explanation
The description clearly explains the documentation changes, motivation, related issue, and validation results. It does not use the template headings for Type or Breaking changes, and it does not explicitly state whether breaking changes exist, but the description is otherwise complete and relevant.
Finishing Touches
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: b859e61a-f466-46a9-87e9-a1531dbb88c6
Commits
Reviewing files that changed from the base of the PR and between be8b9cc and 5b35db5.
Files selected for processing: 1
src/content/docs/guides/disconnected-cluster.mdx
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
| Tool Use v1 | First `inspect/bfcl` entry / Inspect | `disconnected/tool-use-v1/bfcl/` | Complete `BFCL/` cache for the selected categories, including the `simple_python` sentinel and category support files |
| Tool Use v1 | Second `inspect/bfcl` entry / Inspect | `disconnected/tool-use-v1/bfcl/` | Same complete `BFCL/` cache; this repeated entry still needs its own `test_data_ref` |
The reason will be displayed to describe this comment to others. Learn more.
Could we change this directory to truthful_qa--multiple_choice/? The stable truthfulqa_mc1 task derives the offline slug from dataset_path: truthful_qa and dataset_name: multiple_choice. The download source is different; the LMES stager in #1239 already handles this distinction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
jannalu/crows_pairs_multilingualwith the English subset.Related: RHAI-16267
Validation
astro check: 0 errors, warnings, or hints.astro build: passed (the repository reports existing component-override and bundle-size warnings).git diff --check.Summary by CodeRabbit