Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/app-security-suppressions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@shopify/app': minor
'@shopify/cli': minor
---

Add support for suppressing app security findings.
62 changes: 58 additions & 4 deletions docs-shopify.dev/generated/generated_docs_data_v2.json

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ const blockingLevels: AppSecurityBlockingLevel[] = ['high', 'medium', 'low', 'no
export const appSecurityBlockingFlag = {
blocking: Flags.custom<AppSecurityBlockingLevel>({
description:
'The minimum finding severity that causes a non-zero exit code: high, medium, or low. Defaults to none, which never fails on findings.',
'The minimum check severity that causes a non-zero exit code when the check has active findings: high, medium, or low. Suppressed and superseded findings never count. Defaults to none, which never fails on findings.',
options: blockingLevels,
default: 'none',
env: 'SHOPIFY_FLAG_APP_SECURITY_BLOCKING',
Expand Down
933 changes: 927 additions & 6 deletions packages/app/src/cli/commands/app/security/check.integration.test.ts

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions packages/app/src/cli/commands/app/security/check.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,16 @@ describe('app security check command', () => {
expect(SecurityCheck.descriptionWithMarkdown).not.toMatch(/--findings|--clean|compile|trace/)
})

test('points the coding agent to review --suppress to add entries, because record never changes the file', () => {
const help = SecurityCheck.descriptionWithMarkdown

expect(help).toContain(
'ask your coding agent to run `shopify app security review --suppress` for a suppression that you approve',
)
expect(help).toContain('`shopify app security record` never changes the file')
expect(help).not.toContain('have your coding agent record a suppression')
})

test('documents what is scanned, --exclude and --no-git-ignore', () => {
expect(SecurityCheck.flags.exclude.multiple).toBe(true)
expect(SecurityCheck.flags.exclude.description).toBe(
Expand Down
7 changes: 6 additions & 1 deletion packages/app/src/cli/commands/app/security/check.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,11 @@ Use \`--list-files\` to check the scope before scanning: it prints the files the

In interactive terminals, the command offers to copy the coding-agent instructions, print them, or choose nothing; copying is the default. In CI and other non-interactive environments, instructions aren't offered unless you pass \`--print-instructions\`, which prints them. You can also run \`shopify app security instructions\` to print, copy, or write them later. With \`--json\`, the command offers to include the instructions in the result, with or without copying them, and \`--print-instructions\` includes them.

With \`--json\`, the result describes the scan and the files it wrote, and whether \`--blocking\` passed, but not the findings: run \`shopify app security review --json\` for those.
Findings that an entry of \`shopify.app-security.json\` in the app directory matches are suppressed: they don't count for \`--blocking\`, and the output lists each one with its kind and reason. An entry that matches no finding is stale, and an entry whose code matches more than one finding is ambiguous. Neither suppresses anything, and the output lists both, so you can remove or update them. When an edit to the flagged code, such as a rename, makes an entry stale, the output names the location of its likely finding: run \`shopify app security review\` to see that finding's number, and \`shopify app security review --suppress\` with it to re-attach the entry. It names none when the entry's code or the finding's line holds a redacted secret, because redaction hides the only text that tells two credentials apart, or when the entry's code holds a secret that the CLI would redact. Agent entries aren't evaluated, because the check reads no agent results. An invalid file makes the check fail with every problem. To add entries, run \`shopify app security review --interactive\` or \`shopify app security review --suppress\`, or ask your coding agent to run \`shopify app security review --suppress\` for a suppression that you approve, then review and commit the file. \`shopify app security record\` never changes the file: a suppression in the agent findings applies only to them.

With \`--json\`, the result describes the scan and the files it wrote, whether \`--blocking\` passed, and how the entries of \`shopify.app-security.json\` applied, but not the findings: run \`shopify app security review --json\` for those.

Findings that an entry of \`shopify.app-security.json\` in the app directory suppresses don't count for \`--blocking\`. The results files still hold every finding. Suppressions apply to all app configurations and client-ID runs. An entry's \`code\` matches the code excerpt that the results file saves with the finding. A results file that would be larger than 5 MB with excerpts is saved without them; then an entry with \`code\` matches only the finding's message, so it can go stale and leave the finding active. The check reads \`shopify.app-security.json\` before it scans, so an invalid file stops the command before any results file is replaced.

Reports the number of findings to Shopify as usage telemetry, but not finding details. See [usage reporting](https://shopify.dev/docs/api/shopify-cli#usage-reporting) for details.`

Expand All @@ -39,6 +43,7 @@ Reports the number of findings to Shopify as usage telemetry, but not finding de
'<%= config.bin %> <%= command.id %> --path app --include-dir . --exclude marketing --exclude "**/generated"',
'<%= config.bin %> <%= command.id %> --no-git-ignore --exclude local-data',
'<%= config.bin %> <%= command.id %> --include-dir ../backend --list-files',
'<%= config.bin %> <%= command.id %> --blocking high',
]

static flags = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,14 @@ describe('app security instructions command', () => {
expect(SecurityInstructions.jsonOutputSchema).toBe(appSecurityInstructionsJsonOutputSchema)
})

test('says that record never changes shopify.app-security.json, and that review --suppress saves a suppression', () => {
const help = SecurityInstructions.descriptionWithMarkdown

expect(help).toContain('`record` never changes `shopify.app-security.json`')
expect(help).toContain('the agent runs `shopify app security review --suppress`')
expect(help).not.toMatch(/record`, which saves/)
})

test('defines the selection flags as check does', () => {
expect(SecurityInstructions.flags.path).toBe(appFlags.path)
expect(SecurityInstructions.flags.config).toBe(appFlags.config)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ export default class SecurityInstructions extends BaseCommand {

static descriptionWithMarkdown = `Prints the complete workflow that a coding agent should follow to review app security check results.

By default, the instructions are printed to stdout. Use \`--copy\` to copy them to the clipboard or \`--write\` to write them to a file. Standalone instructions always start by running \`shopify app security check\`; only that invocation's generated review pack is trusted as workflow input.`
By default, the instructions are printed to stdout. Use \`--copy\` to copy them to the clipboard or \`--write\` to write them to a file. Standalone instructions always start by running \`shopify app security check\`; only that invocation's generated review pack is trusted as workflow input.

The agent records its findings with \`shopify app security record\`. A suppression in those findings applies only to them, until the next \`record\`: \`record\` never changes \`shopify.app-security.json\`. To save a suppression that you approve to that file, the agent runs \`shopify app security review --suppress\`. Review and commit the file.`

static jsonOutputSchema = appSecurityInstructionsJsonOutputSchema

Expand Down
2 changes: 2 additions & 0 deletions packages/app/src/cli/commands/app/security/record.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ export default class SecurityRecord extends BaseCommand {

The document must include a \`scope\` with the \`include_dirs\`, \`excludes\` and \`no_git_ignore\` values of the \`check\` run it describes, exactly as typed. It's recorded as reported and never compared with the scan's files.

Each finding is stored with the redacted lines of code that it cites, read from the cited files that \`check\` scans with that scope: files in the app directory and its extension and web directories, and in the \`--include-dir\` directories that the latest \`check\` recorded when its \`include_dirs\` are the same. Files that \`--exclude\` or Git ignores and files in \`.git\` are never read, and binary files and files that hold secrets, such as \`.env\`, are never quoted. When the lines would make \`agent-findings.json\` larger than 5 MB, it's written without them and the command warns.

The document is recorded all or nothing: if anything is invalid, the command fails with every error, writes nothing, and exits with a non-zero code. It needs the results directory that \`shopify app security check\` creates.

Reports the number of findings to Shopify as usage telemetry, but not finding details. See [usage reporting](https://shopify.dev/docs/api/shopify-cli#usage-reporting) for details.`
Expand Down
Loading
Loading