Wishlist

[abarciauskas-bgse]

• Add repack preprocessing times to test_workflow.ipynb
• Add flatgeobuf preprocessing times to test_workflow.ipynb
• Track fix to sliderule and re-run processing times notebook when sliderule is working
• Complete creating tests for the ArrMean and SubsetMean for remaining data formats
• Scale running tests + report something statistically significant. Investigate if S3 caching is causing any impact.
• Provide instructions on how to update the dataset list (say we want to run the test for different ATL03 granules or even need to produce the test data in a different bucket)
• Integrate h5coro as a backend to xarray and include that option in the tests
• Investigate why repacking makes results slower than the original h5

[weiji14]

• Integrate h5coro as a backend to xarray and include that option in the tests

Besides h5coro (C++ based), I'd also like to add hidefix (Rust-based) into the benchmark comparison. It looks like there's a backend engine already at https://github.com/gauteh/hidefix/blob/v0.6.3/python/hidefix/xarray.py#L40, related issue at pydata/xarray#7446. Will be easier to test once the conda-forge package is ready.

[asteiker]

@andypbarrett @betolink @asteiker Are reviewing this Issue to determine what else is needed to provide robust recommendations for ATL03 reformatting. Should we break out the relevant tasks into separate Issues? Other notes:

• The last task has already been broken out into Issue Investigate why access performance isn't improved uniformly by repacking metadata #19
• "Integrate h5coro as a backend to xarray and include that option in the tests" - Already in work by @rwegener2 @jpswinski
• Fifth task ("Scale running tests + report something statistically significant. Investigate if S3 caching is causing any impact") Is still important to determine. @jpswinski May have some recommendations here?

[jpswinski]

In SlideRule performance testing I've seen a 25% performance difference between hitting an object for the first time in S3 and then subsequent reads of the same object. I've also seen a 10% to 20% performance swing based on the time of day we are hitting S3. So, for our performance testing, we try to run the tests at the same time (early morning seems to be more stable), and we run the tests multiple times until the timing numbers come down and stabilize. Doing that doesn't capture the performance most people experience (because they will typically hit our system in the middle of the day to late afternoon, and because the resulting data pull by SlideRule is usually reading the data after the S3 caches have cooled), but it does allow us to compare apples to apples when we make changes in the code and want to know how the changes affected performance.