Repository navigation
Hash only files whose size another file shares [patch] - #180
Merged
Merged
Conversation
Two files of different lengths can never be duplicates, but every verb read and SHA-256-hashed every scanned file in full. A new FileHasher.HashPossibleDuplicates groups the scan by size first and hashes only the files whose size is shared, so a file with a unique size is never read. Scan, DryRun, Deduplicate and Stats all go through it. Empty files still group with each other, and a file that cannot be sized is still hashed so it is reported as unreadable as before. Stats counts the files skipped for a unique size as unique and includes them in the total size. Fixes #147 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtvW1bPVVfCTF7fCWpeyRE
Scan, DryRun and Deduplicate each carried the same discover, hash and group block, and Stats a near copy of it. The size pre-filter changed a line in every copy, which SonarCloud then counted as new duplicated code. DuplicateScan.Run now does those steps once, printing the same progress lines, and all four verbs read the files, hashes, unique-size files and groups from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtvW1bPVVfCTF7fCWpeyRE
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #147
Problem
Every verb read and SHA-256-hashed every file it scanned, but two files of different lengths can never be duplicates. In a tree where most sizes are unique, such as a media library on a cold disk, USB drive or network share, most of that reading was wasted.
Change
FileHasher.HashPossibleDuplicatesgroups the scanned files by size and hashes only those whose size another file shares. It also returns the files it skipped because their size was unique.DuplicateScan.Runhandles the discover, hash and group steps thatScan,DryRunandDeduplicateeach had a copy of, withStatsholding a near copy. It prints the same progress lines, and all four verbs now get their data from it. This is the one helper the issue asks for. It also removes the duplicated block SonarCloud flagged.EmptyFilesAreDuplicatesOfEachOtherrequires. A file whose size can't be read is still hashed, so it is reported as unreadable as before.Statscounts the skipped files as unique and includes them in the total size. Before this change it took "Unique files" fromhashGroups.Count.Deduplicator.TryGetSizeis nowinternalso the size check can share it.Tests
HashPossibleDuplicatesSkipsFilesWithAUniqueSize: files that all have different sizes produce no hashes.HashPossibleDuplicatesHashesEveryFileThatSharesASize: files of the same size but different content, and empty files, are hashed. A file with its own size is not.ScanHashesNothingWhenEveryFileHasItsOwnSize:Scanprints noHashed:lines for such a tree.StatsCountsFilesWithAUniqueSizeAsUnique:Statsreports the right totals without hashing.AHashingErrorNamesTheFullPathchanged: the locked file now has the same length as its sibling. Otherwise it would never be read, and the error the test checks would not occur.All four new tests fail when
HashPossibleDuplicatesis temporarily made to hash every file, and pass with the filter on. The full suite passes locally on Linux: 94 passed, 3 skipped. The skipped tests need a non-root user.dotnet format --verify-no-changesreports nothing to change.🤖 Generated with Claude Code
https://claude.ai/code/session_01FtvW1bPVVfCTF7fCWpeyRE