Repository navigation
test: prune and rewrite SDK tests flagged by the current-main audit rerun #2142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
devin-ai-integration
wants to merge
5
commits into
devin/1793236744-remove-gated-git-tests
from
devin/1794604210-prune-sdk-tests-rerun
+301
−508
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
2e0fd2f
test(js-sdk): enforce retainOutput type assertions with vitest typecheck
devin-ai-integration[bot] 1ee3da1
test(js-sdk): drop tests that check their own fake or duplicate a sib…
devin-ai-integration[bot] 3ce3ccf
test(python-sdk): cover stream_input pty data and deadline at the pub…
devin-ai-integration[bot] 56755c4
test(python-sdk): rewrite private-helper and mocked-factory tests at …
devin-ai-integration[bot] cb548ce
test(python-sdk): drop duplicate, tautological and internal-state tests
devin-ai-integration[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 41 additions & 0 deletions
41
packages/js-sdk/tests/sandbox/commands/retainOutput.test-d.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| import { expectTypeOf, test } from 'vitest' | ||
|
|
||
| import type { CommandStartOpts, Sandbox } from '../../../src/index.js' | ||
|
|
||
| declare const sandbox: Sandbox | ||
| declare const opts: CommandStartOpts | ||
|
|
||
| test('retained output types stdout as a string', async () => { | ||
| expectTypeOf( | ||
| (await sandbox.commands.run('echo')).stdout | ||
| ).toEqualTypeOf<string>() | ||
| expectTypeOf( | ||
| (await sandbox.commands.run('echo', {})).stdout | ||
| ).toEqualTypeOf<string>() | ||
| expectTypeOf( | ||
| (await sandbox.commands.run('echo', { retainOutput: true })).stdout | ||
| ).toEqualTypeOf<string>() | ||
| }) | ||
|
|
||
| test('retainOutput: false types stdout as undefined', async () => { | ||
| expectTypeOf( | ||
| (await sandbox.commands.run('echo', { retainOutput: false })).stdout | ||
| ).toEqualTypeOf<undefined>() | ||
| expectTypeOf( | ||
| ( | ||
| await sandbox.commands.run('echo', { | ||
| background: true, | ||
| retainOutput: false, | ||
| }) | ||
| ).stdout | ||
| ).toEqualTypeOf<undefined>() | ||
| expectTypeOf( | ||
| (await sandbox.commands.connect(1, { retainOutput: false })).stdout | ||
| ).toEqualTypeOf<undefined>() | ||
| }) | ||
|
|
||
| test('non-literal retainOutput types stdout as possibly undefined', async () => { | ||
| expectTypeOf((await sandbox.commands.run('echo', opts)).stdout).toEqualTypeOf< | ||
| string | undefined | ||
| >() | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| { | ||
| "extends": "./tsconfig.json", | ||
| "compilerOptions": { | ||
| "noEmit": true | ||
| }, | ||
| "include": ["src", "tests/**/*.test-d.ts"] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 (optional) After this change, no test in either SDK pins the template files-hash value. A recipe change that silently invalidates every user's build cache would now pass CI. The new multi-chunk test only checks that flipping the last byte changes the hash. It still passes if earlier chunks are skipped or if
hashFileContenthashes the wholebufferinstead ofsubarray(0, bytesRead). Fix: give both SDKs deterministic content spanning more than one 1 MiB chunk and assert one hard-coded hex digest, shared by JS and Python. The Python twin is at test_get_all_files_in_path.py:576-599.Why this was flagged
The removed tests recomputed the sha256 recipe (COPY line, relative path, mode, size, content) in both getAllFilesInPath.test.ts and test_get_all_files_in_path.py. They pinned the cache key and kept the two SDKs computing the same digest. The replacement test (getAllFilesInPath.test.ts:584-602, test_get_all_files_in_path.py:576-599) uses random content and only compares hashes against each other. Three regressions now pass in both languages: a loop that hashes only the final chunk, a change to field order or encoding, and in JS
hash.update(buffer)withoutsubarray(0, bytesRead)at src/template/utils.ts:297. That last one leaks stale bytes from the reusedBuffer.allocUnsafe(HASH_CHUNK_SIZE), so small-file hashes become nondeterministic. Grep finds no 64-hex golden digest under either SDK's tests/template, so nothing else catches these. The base branch failed on all of them. A literal golden digest comes from an independent source, so it does not have the tautology problem that the PR set out to remove.Verification: The diff deletes the only assertions in either SDK that compared the files hash against an independently computed sha256 of the full content. The new test (getAllFilesInPath.test.ts:584-602; test_get_all_files_in_path.py:576-599) uses random content. It flips only
content[content.length - 1]. No hex digest or sha256 recomputation remains anywhere.