HBASE-29875 Handle null-reader store files in compaction cleanup and selection#7972
Open
tr0k wants to merge 2 commits intoapache:branch-2.5from
Open
HBASE-29875 Handle null-reader store files in compaction cleanup and selection#7972tr0k wants to merge 2 commits intoapache:branch-2.5from
tr0k wants to merge 2 commits intoapache:branch-2.5from
Conversation
…eededFiles DefaultStoreFileManager.getUnneededFiles assumes HStoreFile.getReader() is always non-null. In production we observed NullPointerException in the expired-file cleanup path, which aborts compaction requests and can lead to unbounded storefile growth for affected stores. Guard against null readers and skip such files instead of throwing. Add a focused unit test covering the null-reader case.
…ompaction selection After guarding getUnneededFiles, null-reader HStoreFiles could still reach compaction policy evaluation and interfere with minor compaction selection. Filter null-reader files out of the eligible candidate list before compaction selection so healthy files can still be compacted. Add a unit test covering the selection path with null-reader files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
This PR fixes HBASE-29875 in two commits:
DefaultStoreFileManager.getUnneededFilesagainst null readersSortedCompactionPolicyWhy
In our 2.5.10 production environment, some stores reached a state
where
HStoreFile.getReader()returned null. This caused compactionpaths to fail and blocked compaction progress for affected stores.
Validation
Validated on a downstream 2.5.10-based build with Java 11 and with:
Observed result:
Notes
This is a defensive fix around unexpected null-reader state. It does not
attempt to solve the root cause of why some
HStoreFileinstances end upwith
reader == null.