Add multi-turn RL support for iterative kernel refinement#5
Open
nataliakokoromyti wants to merge 2 commits intoScalingIntelligence:mainfrom
Open
Add multi-turn RL support for iterative kernel refinement#5nataliakokoromyti wants to merge 2 commits intoScalingIntelligence:mainfrom
nataliakokoromyti wants to merge 2 commits intoScalingIntelligence:mainfrom
Conversation
nataliakokoromyti
force-pushed
the
add-multi-turn
branch
5 times, most recently
from
7c139d6 to
cc20524
Compare
nataliakokoromyti
force-pushed
the
add-multi-turn
branch
from
cc20524 to
2f219a9
Compare
Adds a second evaluator backend (in-process subprocess on the local GPU)
alongside the existing Modal cloud backend, plus a dispatch layer that
selects between them via eval_config.evaluator_backend ("modal" | "local").
The local backend exists because KernelBench compiles every kernel through
torch.utils.cpp_extension.load_inline, which leaks memory across compiles
and OOMs an MI350X after a few hundred problems. Each kernel now runs in
a fresh `python -c` worker so the leak dies with the worker, mirroring the
trick used by KernelBench's eval_hip.py.
Changes:
- Add EvalConfig.evaluator_backend / gpu_arch / local_timeout fields.
- New kernelbench_tinker.local.evaluator with subprocess-isolated worker
that calls set_gpu_arch() before importing torch and uses a per-process
TORCH_EXTENSIONS_DIR to dodge stale lock files from killed predecessors.
- New evaluator_dispatch module so call sites stay agnostic; both
ModalKernelEvaluator and LocalKernelEvaluator expose the same async API.
- Wire dispatch through KernelBenchDatasetBuilder, kernelbench_client, and
the eval_kernel_rl script. Add multi-level iteration to eval_kernel_rl.
- Add rl_kernelbench_hip.yaml: backend=hip, evaluator_backend=local,
gpu_arch=[gfx950], levels=[1,2,3], multiturn enabled, dataset_src=local.
The Modal backend remains the default, so existing CUDA configs are
unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.