Add optional ZFile index compression by williswus · Pull Request #465 · containerd/overlaybd

williswus · 2026-10-03T03:03:30Z

ZFile already compresses data blocks, but stores the block-length index uncompressed. This PR adds optional index compression using the existing format’s flag bit 5.

Changes:

Add CompressArgs::compress_index and the --compress-index CLI flag, disabled by default.
Share index writing across the whole-file writer and both streaming builders.
Support LZ4 and ZSTD, retaining raw indexes when compression provides no space savings or exceeds codec limits.
Validate metadata, sizes, checksums and decoded block lengths before exposing the file for reads.
Handle short source reads and enforce index limits before buffering additional input.

Format choices:

With bit 5 clear, index_size remains the entry count.
With bit 5 set, it records compressed bytes; the decoded entry count is derived from the original file size and block size.
The index checksum covers the stored bytes.

Existing raw indexes remain supported. Older readers cannot read compressed indexes, so compression remains opt-in. I have not compared these format choices with the historical ease implementation.

Validation:

Debug build with QAT disabled.
Full zfile_test suite: all 12 tests passed.
Coverage includes both codecs, all writers, header/trailer paths, raw fallback, short reads and malformed inputs.
git diff --check passed.

lihuiba · 2026-10-03T03:56:46Z

+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+#pragma once


merge this file into zfile.cpp

lihuiba · 2026-10-03T04:19:58Z

+}
+
+// Shared by the whole-file writer and both streaming builders.
+static int write_index(IFile *file, const std::vector<uint32_t> &lengths,


why static? why not let it be a member of ZFileBuilderBase?

lihuiba · 2026-10-03T04:23:19Z

-                 pht->opt.use_dict);
-
-        if (index_bytes > trailer_offset - pht->index_offset)
-            LOG_ERROR_RETURN(0, false, "invalid index bytes or size. ");


is it necessary to change so much here?

Copilot

Copilot review overview

🔵 Needs a closer look

An unresolved out-of-bounds read needs correction, and historical format compatibility still requires human review.

Review effort: Balanced
Findings: 1

Open (1)

Reject stored blocks exceeding MAX_READ_SIZE before table construction · New

What changed in this PR

Adds opt-in ZFile index compression while preserving raw indexes by default.

Changes:

Adds LZ4/ZSTD index compression shared by all writers.
Exposes API and CLI options.
Strengthens validation and adds round-trip and malformed-input tests.

File	Description
src/tools/overlaybd-zfile.cpp	Adds `--compress-index`.
src/overlaybd/zfile/zfile.cpp	Integrates index compression and validation.
src/overlaybd/zfile/test/test.cpp	Tests writers, codecs, and invalid inputs.
src/overlaybd/zfile/index_compression.h	Implements index codec helpers.
src/overlaybd/zfile/format_spec.md	Documents compressed-index storage and compatibility.
src/overlaybd/zfile/compressor.h	Adds the opt-in API setting.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

lihuiba · 2026-10-12T05:35:08Z


+// Whole-index codecs use CPU routines: data-block/QAT buffer limits do not apply.
+// Return 1 for compressed output, 0 to keep the original, or -1 on codec error.
+static int compress_index_buffer(const void *src, size_t size, uint8_t algo,


make it a member of ZFileBuilderBase.

lihuiba · 2026-10-12T05:35:14Z

+}
+
+// A successful decode must restore exactly the expected index byte count.
+static int decompress_index_buffer(const void *src, size_t size, void *dst,


make it a member of ZFileBuilderBase.

Copilot

🔵 Needs a closer look

Reusing on-disk flag bit 5 requires human compatibility review against historical compressed-index implementations.

No open issues 🎉

1 resolved since last review

Reject stored blocks exceeding MAX_READ_SIZE before table construction

_{Review effort: Balanced}

lihuiba · 2026-10-19T02:04:04Z

 }

+// Check before buffering or queueing data, not after the index has already grown.
+static int check_input_size(uint64_t current_size, size_t count, uint32_t block_size) {


put it into ZFileBuilderBase, not necessarily static.

lihuiba · 2026-10-19T02:04:11Z

-    virtual int fini() = 0;
+    // Whole-index codecs use CPU routines: data-block/QAT buffer limits do not apply.
+    // Return 1 for compressed output, 0 to keep the original, or -1 on codec error.
+    static int compress_index_buffer(const void *src, size_t size, uint8_t algo,


not necessarily static

lihuiba · 2026-10-19T02:04:16Z

+    }
+
+    // A successful decode must restore exactly the expected index byte count.
+    static int decompress_index_buffer(const void *src, size_t size, void *dst,


not necessarily static

Copilot

🔵 Needs a closer look

The size-limit check can block indefinitely on an open pipe after overflow is already known because it waits for a full batch or EOF instead of checking after each successful read.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Check input size after each read to avoid blocking on overflow

src/overlaybd/zfile/zfile.cpp:1375

The size-limit check runs only after a full batch or EOF. If the accepted input has reached the maximum and a pipe supplies one extra byte but remains open, this loop attempts another read and can block indefinitely instead of returning EFBIG, even though the overflow is already known. Check the accumulated batch size after each successful read before requesting more input.

_{Review effort: Balanced}

BigVan · 2026-11-04T03:07:28Z

 | flags   |      32        |   uint64_t   | bits for flags* (see later for details) |
 | index_offset | 40        |   uint64_t   | index offset |
-| index_size   | 48        |   uint64_t   | size of the index section, possibly compressed base on flags |
+| index_size   | 48        |   uint64_t   | entry count when bit 5 is 0; compressed byte count when bit 5 is 1 |


bit 5 --> FLAG_SHIFT_IDX_COMP ?

Allow ZFile writers to compress the block-length index with LZ4 or ZSTD when requested. Keep the raw index when compression does not reduce its size. Validate stored and decoded indexes before use. Signed-off-by: Willis Wu <willis060510@gmail.com>

lihuiba reviewed Oct 3, 2026

View reviewed changes

liulanzheng requested a balanced review from Copilot October 3, 2026 05:28

Copilot started reviewing on behalf of liulanzheng October 3, 2026 05:28

View session

Copilot AI reviewed Oct 3, 2026

View reviewed changes

Comment thread src/overlaybd/zfile/zfile.cpp Resolved

lihuiba reviewed Oct 12, 2026

View reviewed changes

liulanzheng requested a balanced review from Copilot October 18, 2026 09:55

Copilot started reviewing on behalf of liulanzheng October 18, 2026 09:55

View session

Copilot AI reviewed Oct 18, 2026

View reviewed changes

lihuiba reviewed Oct 19, 2026

View reviewed changes

liulanzheng requested a balanced review from Copilot November 3, 2026 06:22

Copilot started reviewing on behalf of liulanzheng November 3, 2026 06:22

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

BigVan reviewed Nov 4, 2026

View reviewed changes

Add optional ZFile index compression

9a5e7f2

Allow ZFile writers to compress the block-length index with LZ4 or ZSTD when requested. Keep the raw index when compression does not reduce its size. Validate stored and decoded indexes before use. Signed-off-by: Willis Wu <willis060510@gmail.com>

williswus force-pushed the feat/zfile-index-compression branch from 0438df3 to 9a5e7f2 Compare November 7, 2026 16:05

liulanzheng approved these changes Nov 8, 2026

View reviewed changes

liulanzheng merged commit f084bf5 into containerd:main Nov 8, 2026
3 checks passed

Conversation

williswus commented Oct 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants