[Multi-GPU Polars] Introduce Ray mode for multi-GPU cudf-polars execution#21746
[Multi-GPU Polars] Introduce Ray mode for multi-GPU cudf-polars execution#21746rapids-bot[bot] merged 38 commits intorapidsai:mainfrom
Conversation
madsbk
added
improvement
github-actions
Bot
added
Python
madsbk
force-pushed
the
rapidsmpf-ray
branch
3 times, most recently
from
2d3e98f to
69eea3c
Compare
| self._comm = None | ||
| self._mr = None |
There was a problem hiding this comment.
exit_actor tears down the actor process though, so that should delete the class and hence all these objects.
Co-authored-by: Tom Augspurger <tom.augspurger88@gmail.com> Co-authored-by: Lawrence Mitchell <wence@gmx.li>
madsbk
added
breaking
|
Needs #21787 first. |
rjzamora
left a comment
rjzamora
left a comment
There was a problem hiding this comment.
Flushing some comments - Still planning to read through this a bit more.
rjzamora
left a comment
rjzamora
left a comment
There was a problem hiding this comment.
Thanks for working on this Mads!
| assert config_options.executor.runtime == "rapidsmpf", "Runtime must be rapidsmpf" | ||
|
|
||
| # Lower the IR graph on the client process (for now). | ||
| ir, partition_info, stats = lower_ir_graph(ir, config_options) |
There was a problem hiding this comment.
Just a note that we may use the GPU during this lowering stage, so the client does technically need a GPU. (I don't think you say otherwise anywhere in this PR, but I think you may have in some discussion about/within #21769)
There was a problem hiding this comment.
Good point. Can we avoid the GPU requirement by calling lower_ir_graph on each worker instead of on the client?
There was a problem hiding this comment.
Yes, I think we can/should move in that direction. I don't think it will be difficult, but its not completely trivial either. If we just move the lowering to the worker "as is", the workers will all sample the same parquet metadata/row-groups redundantly. We probably want to tweak the logic so that the workers target different information and allgather the results (so the workers sample more efficiently and make consistent decisions). I don't think it's a problem to do this after you have the initial frontends in place.
|
/merge |
6 participants
Introduces a Ray-based execution frontend for cudf-polars backed by the RapidsMPF streaming engine, complementing the existing SPMD mode.