Skip to content

Migrate to prost#9

Merged
DougLau merged 1 commit intoDougLau:masterfrom
hhirtz:prost
Apr 13, 2026
Merged

Migrate to prost#9
DougLau merged 1 commit intoDougLau:masterfrom
hhirtz:prost

Conversation

@hhirtz
Copy link
Copy Markdown
Contributor

@hhirtz hhirtz commented Apr 10, 2026

Hey, i'd like to improve the compile time of a project using mvt https://github.com/OpenRailAssociation/osrd/tree/dev/editoast The project uses prost for other reasons, so i'd like to change mvt's protocol buffers implementation to prost if it's alright...

This patch improves compile times from ~3.5s to ~2.0s on my linux machine. Additionally, some benchmarks have shown that prost is also faster than protobuf at decoding:
georust/geozero#118 (comment)

This patch changes the signature of the Error type.

prost doesn't expose a way to write directly to a std::io::Write, so this makes Tile::write_to write to a Vec first.

Some generated fields are now options, while others aren't anymore. It doesn't show in the public API.

protobuf's compute_size seemed to include the length delimiter while prost's encoded_len does not. However, the encoding was always done without the length delimiter, so it makes more sense to not include it? If you want backwards compatibility it can be easily added.

This patch improves compile times from ~3.5s to ~2.0s on my linux
machine. Additionally, some benchmarks have shown that prost is also
faster than protobuf at decoding:
georust/geozero#118 (comment)

This patch changes the signature of the `Error` type.

prost doesn't expose a way to write directly to a `std::io::Write`, so
this makes `Tile::write_to` write to a `Vec` first.

Some generated fields are now options, while others aren't anymore. It
doesn't show in the public API.

protobuf's `compute_size` seemed to include the length delimiter while
prost's `encoded_len` does not. However, the encoding was always done
without the length delimiter, so it makes more sense to not include it?
If you want backwards compatibility it can be easily added.
@DougLau
Copy link
Copy Markdown
Owner

DougLau commented Apr 10, 2026

This seems like the way to go, since the protobuf crate is no longer maintained. Thanks!

I tested the simple example program, and the output has changed:

Output with v0.12:
encoded 62 bytes: [26, 60, 120, 2, 10, 11, 70, 105, 114, 115, 116, 32, 76, 97, 121, 101, 114, 18, 26, 8, 1, 18, 2, 0, 0, 24, 2, 34, 16, 9, 0, 0, 34, 128, 16, 0, 0, 128, 32, 128, 16, 0, 0, 128, 32, 26, 3, 107, 101, 121, 34, 7, 10, 5, 118, 97, 108, 117, 101, 40, 128, 32]

Output with prost:
encoded 62 bytes: [26, 60, 10, 11, 70, 105, 114, 115, 116, 32, 76, 97, 121, 101, 114, 18, 26, 8, 1, 18, 2, 0, 0, 24, 2, 34, 16, 9, 0, 0, 34, 128, 16, 0, 0, 128, 32, 128, 16, 0, 0, 128, 32, 26, 3, 107, 101, 121, 34, 7, 10, 5, 118, 97, 108, 117, 101, 40, 128, 32, 120, 2]

I haven't looked into it too deeply, but maybe these are two different ways of encoding the same data. Any ideas?

@hhirtz
Copy link
Copy Markdown
Contributor Author

hhirtz commented Apr 13, 2026

Yeah prost encodes fields by ascending tag values:
https://docs.rs/prost-derive/0.14.3/src/prost_derive/lib.rs.html#87-91

while protobuf encodes them by the order in which they appear in the proto file:
https://github.com/DougLau/mvt/pull/9/changes#diff-99f14677bbf5f612c35ca273f6914cce0ab720cf7a3a1d171296720e731d6ca7L801-L803

so the version (120 2) now appears last (tag 15 in the proto file -> shift by 3 to get 120, append 2 as a varint) 👍

@DougLau DougLau merged commit 2a2fe49 into DougLau:master Apr 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants