What version(s) of Substrait are/should be supported?

[ingomueller-net]

This project seems to upgrade to the latest version of Substrait at a pretty regular basis. This is nice for users who want to use the latest version but also makes it difficult to find and use the version of substrait-python for a specific version of Substrait.

For example, substrait-mlir uses Substrait v0.42.1 and substrait-python v0.12.1 for testing some end-to-end Python use cases. Now, we can't easily upgrade to pyarrow v16 or higher because those versions require substrait-python v0.15 or higher (due to how the generated proto files are packaged, AFAIU), which includes a different version of Substrait than what we use.

Note that Substrait regularly introduces breaking changes, in particular, to the text format, so it is not generally possible to use the protobuf definitions of a newer version.

To remedy the situation, I can imagine three things:

1. Add a table to the README and/or PyPI that shows the correspondance between the version(s) of substrait-python and Substrait. This would make it easier to find the right version of substrait-python. In some situations, that may be all that is necessary. Currently, the only way I know how to do this is to install several versions of substrait-python and print substrait.__substrait_version__, which isn't ideal.
2. We might want to investigate whether it isn't possible to package several versions of Substrait inside of substrait-python, for example, under substrait.v0_74_0 and substrait.v0_42_1, possibly with the latest version living in substrait as well.
3. Another possibility would be to separate the packaging of the generated proto files from the other functionality of this repository. substrait-mlir and, AFAIU, pyarrow only need the former. Such a package could either aim to be "released once" such that, for every version of Substrait, there would be a release of that package that would never need to be updated (ideally with the same versioning scheme as Substrait itself), or use the multi-version scheme from the previous point.

[tokoko]

Another possibility would be to separate the packaging of the generated proto files from the other functionality of this repository. substrait-mlir and, AFAIU, pyarrow only need the former. Such a package could either aim to be "released once" such that, for every version of Substrait, there would be a release of that package that would never need to be updated (ideally with the same versioning scheme as Substrait itself), or use the multi-version scheme from the previous point.

This seems a lot cleaner than option 2. For me the only concern with it would be that other language implementations don't do the same, so I'm not sure why python should be the odd one out. I also think that if the goal is for the new package (let's call it substrait-proto for example) to match upstream substrait versions, it makes more sense for the build/publish steps to be part of substrait rather than substrait-python.

This is probably something that should be discussed in the main repo, but I can see a setup where a new substrait release means publishing a set of proto-generated barebones "libraries" in all main different languages. The language repos would then depend on those libraries and add some additional utilities on top.

[ingomueller-net]

Actually, there is already substrait-io/substrait-protobuf, which seems to correspond quite exactly to what you describe. Maybe we should indeed implement proto packaging logic for Python there and use (and thus contribute to) that approach?

Someone should probably also ping the developers of the repositories of the other languages to do the same... 😉

[tokoko]

We can do that. Having a separate repo for it seems very much a go-inspired decision 😄 because you need git as a destination for artifacts. For other languages, doing the releases from the main repo sounds better. With the current approach one would be forced to essentially do a release twice. I don't really mind either approach, I'm not the one doing releases. Probably best to continue the discussion on slack.. I'll post a message there

[vbarua]

Did this ever get continued in the main repo?

We can do that. Having a separate repo for it seems very much a go-inspired decision 😄 because you need git as a destination for artifacts.

That is 100% accurate.

This is probably something that should be discussed in the main repo, but I can see a setup where a new substrait release means publishing a set of proto-generated barebones "libraries" in all main different languages. The language repos would then depend on those libraries and add some additional utilities on top.

I've had this in the back of my brain when I created substrait-protobuf, dedicated protobuf libraries that the language libraries depend on. For substrait-go, this allows end-user to use a more recent version of the specification (barring breaking changes in the protobufs) than used in the library.

I don't have strong opinions about where they should live / be generated (substrait or substrait-python repo)

[tokoko]

I think there's a slack thread, but didn't take it further.

I don't have strong opinions about where they should live / be generated (substrait or substrait-python repo)

I'd prefer it to be in substrait for the sole reason that the release of substrait-protobuf library should always coincide with the main substrait release. Doing it as part of a single release process feels right. On the other hand, the actual release in the main repo will become a lot more involved and complicated if we add other languages as well.

[vbarua]

On the other hand, the actual release in the main repo will become a lot more involved and complicated if we add other languages as well.

Besides Golang specifically needing a repository for code, that was the other reason to do it in a separate repo. There'a already a ton of stuff in the core spec repository.

for the sole reason that the release of substrait-protobuf library should always coincide with the main substrait release.

I agree with this. I've been meaning to automate the Golang protobuf releases in substrait-protobuf to trigger off of releases in Substrait, and this has given me the impetus to start that work.

If that automation is in place, it should be feasible to do the same for Python (and other languages as well). Would that lessen your concerns about doing it in a separate repo?