Question: stability of FFI_TableProvider

[westonpace]

Not really a concrete issue, just looking for clarification / advice.

I created an FFI_TableProvider and it broke (violently, with segmentation faults) when I upgraded my rust to DF 50 but left my datafusion python package at 48. What is the expected stability for FFI_TableProvider? Do I need to have rust & python aligned? Or are these growing pains?

Right now lance offers an FFI_TableProvider in our python. In the pyproject.toml we have a loose pin on datafusion (e.g. datafusion > X). I'm wondering if we need to change this into a hard pin or not.

Also, just to be clear, this is in no way a complaint. If these are growing pains then I think we can help expose them / report them / etc. If this is expected then I don't want to hassle you all. I really appreciate the FFI_TableProvider and just want to calibrate my expectations.

[timsaucer]

Related to apache/datafusion#17374

Right now the FFI implementation is not stable. The best advice I can give is that you should keep your rust datafusion version and your python datafusion version at the same major level. Not every version has had a breaking change, but I do expect more to come. In fact, I have a PR I've been working on for a week or so that will be a big change, but in the long term a very useful one IMO.

I'm trying to think of a better way to convey what are the breaking changes between FFI versions, but I haven't come up with any great ideas yet. I'm open to any suggestions.

At rerun we've gone the route of adding a checking for versions:
https://github.com/rerun-io/rerun/blob/e6411c2eb1cfcf0a675fc5f69051fb3be1f955ad/rerun_py/rerun_sdk/rerun/catalog.py#L31
https://github.com/rerun-io/rerun/blob/e6411c2eb1cfcf0a675fc5f69051fb3be1f955ad/rerun_py/src/catalog/catalog_client.rs#L37

[kevinjqliu]

hey @westonpace i ran into the same issue when upgrading. Here's the other thread for context, #1217

TLDR, rust FFI_TableProvider introduced a breaking change from 48 -> 49 and another from 49 -> 50. The breaking change forces rust and python to use the same version.

For iceberg-rust and iceberg-python, we've taken a stance to make sure the versions are aligned, by pinning the datafusion version. Otherwise, python resolver will use the latest version

[westonpace]

Ok, not a problem at all. For now we will keep it pinned and if things change in the future or users start to complain then we can revisit! Thanks for the input!