Skip to content

Move secrets in HTTP base URLs into encrypted variables - #1778

Closed
helpcodeai wants to merge 1 commit into
mainfrom
fix/webhook-url-secrets-v2
Closed

helpcodeai wants to merge 1 commit into
mainfrom
fix/webhook-url-secrets-v2

Conversation

@helpcodeai

Copy link
Copy Markdown
Contributor

Summary

A REST connector built by hand with a webhook URL as its base URL kept the credential in plaintext in connectors.base_url and showed it in full to admins and the owner. The typical case is a Bitrix24 inbound webhook, https://portal.bitrix24.com/rest/<user>/<code>/, where the code works like a password. The catalog adapter avoids this with {{BITRIX24_WEBHOOK_URL}}; hand-made connectors went through the secret-literal extraction on save, but its rules did not know these shapes (a 16-character lower-case code is not "high entropy").

This PR teaches the extraction the common webhook URL shapes and user:password@, repairs stored rows at start-up the way #1742 does for DATABASE connection strings, and masks whatever is left in API output.

Changes

  • Detector (common/secret-literals.util.ts, new kinds webhook_token and url_password). Each rule is tied to the vendor's host and the exact path it issues:
    • Bitrix24 /rest/<digits>/<code> on *.bitrix24.<tld>. On any host (self-hosted portals) only a 16-character [a-z0-9] code with letters and digits that is not hex-only.
    • Slack hooks.slack.com/services|workflows|triggers/…/<secret>
    • Discord /api[/vN]/webhooks/<id>/<token>
    • Microsoft Teams …webhook.office.com/…/IncomingWebhook/<rest> and the legacy outlook.office.com host
    • Zapier hooks.zapier.com/hooks/catch|standard/<id>/<code>
    • Make hook.<zone>.make.com|integromat.com|make.celonis.com/<token>
    • the password in scheme://user:password@host (any HTTP type; {{VAR}}, ******** and user-only userinfo are left alone)
    • Already covered by the existing rules, now with tests: Power Automate / Logic Apps sig=, Google Chat key=/token=, Telegram bot<token>, and generic token/key/apikey/api_key/secret/sig/signature/access_token query parameters.
    • About 50 negative cases (Jira/Confluence /rest/api/2, Magento /rest/V1, /rest/1/crm.deal.list, hex ids, docs examples, ?email=a@b, host:port, placeholders, masks). The test that checks the whole catalog for false positives still finds nothing.
  • Save (SecretExtractor, unchanged entry points in create/update/import/restore). Only the secret segment moves, into <NAME>_WEBHOOK_TOKEN / <NAME>_PASSWORD. Both names are ones isSecretName recognises, so the value is encrypted, masked and never returned. I chose the segment over the whole URL because:
    • the host stays readable for the connector page, error hints, hostFromUrl and log lines;
    • it is the mechanism Telegram and query keys already use (one code path);
    • an existing variable holding the same value is reused.
  • How calls keep working. Every engine path interpolates {{VAR}} in the base URL verbatim before the request: interpolateConnectorConfig / interpolateString (no encoding) in DynamicMcpTools, testConnection, MCP discovery and OAuth discovery, and WSDL import via resolvedAddress. normalizeConnectorBaseUrl returns the string as typed, so the resolved URL is byte-for-byte the one stored before. Tests check this for every case, including a real DynamicMcpTools.executeTool call whose RestEngine.execute gets the original Bitrix URL.
  • Start-up repair ConnectorsService.repairBaseUrlSecrets(). It is chained after repairDatabaseUrlCredentials in onApplicationBootstrap, is not awaited and never throws.
    • Rules: a save's rules minus the long-random-token heuristic. A save tells the user what moved; this pass acts silently on every instance, so it only moves certain shapes.
    • Variables: read raw and decrypted strictly. A row whose variables cannot be decrypted is never written.
    • Writes: updateMany with a compare-and-swap on baseUrl and updatedAt, so a concurrent save or a second instance wins. The at-rest extension encrypts the variables. config.baseUrlBaseline gets the same variable.
    • Log: one summary line with ids and counts, never a URL or value: Base URL secrets: secrets of N connectors moved into encrypted variables (…); M left as is (…); K failed (…). A second run changes nothing.
  • API output (toPublicConnector, so connector reads, lists, update answers and exports without secrets). A secret still in an HTTP base URL or its baseline shows as ********, to everyone, admins included.
    • On update, the masked URL sent back unchanged keeps the stored URL, and the save then moves the secret into a variable.
    • A URL edited around ******** is refused with a 400 asking for the full address.
  • Docs: docs/connectors/rest.md gets a new section "Secrets typed into the base URL". docs/api-reference.md notes the export masking. llms-full.txt is regenerated.

After upgrading

On its first start, v0.55.4 moves secrets still written in the base URL of REST, GraphQL, SOAP, OData and MCP connectors into encrypted connector variables. These are webhook codes and tokens (Bitrix24, Slack, Discord, Microsoft Teams, Zapier, Make), Telegram bot tokens, ?api_key=-style values and user:password@. Only the secret part of the URL is replaced, by a {{<NAME>_WEBHOOK_TOKEN}}-style reference, and calls go to exactly the same address as before. Nothing needs to be configured, and it works the same on Cloud and self-hosted. The backend log shows one line, Base URL secrets: secrets of N connectors moved into encrypted variables (…); M left as is; K failed, with connector ids only. "Failed" connectors (variables that cannot be decrypted with the current ENCRYPTION_KEY) stay unchanged, their secret is shown as ********, and the next start tries again. scripts/secret-literals/report.ts lists anything left, including tool paths and headers, which this pass does not touch.

Type

  • Bug fix
  • New feature
  • Documentation
  • Refactoring
  • Performance improvement
  • Other (describe below)

Testing

  • Added/updated unit tests
  • Tested manually (describe steps below)
  • No tests needed (explain why)

All commands were run in packages/backend after npm ci (root) and npx prisma generate with a dummy DATABASE_URL:

  • npx jest src/common/secret-literals.util.spec.ts --maxWorkers=2: 146 passed. It includes the detector positives and negatives and the catalog false-positive check.
  • npx jest src/connectors/base-url-secrets.util.spec.ts src/connectors/secret-extraction.util.spec.ts --maxWorkers=2: 40 passed at the time of the run.
  • npx jest src/connectors/connectors.service.base-url-secrets-repair.spec.ts src/connectors/connectors.service.database-credentials-repair.spec.ts --maxWorkers=2: 50 passed. This covers the repair (moved, left, failed, idempotency, no secret in logs, the boot order after the database pass) and the existing database pass.
  • npx jest src/connectors/secret-guard.flows.spec.ts --maxWorkers=2: 29 passed. This covers create and update through the controller, the masked round trip, the refusal, and a tool call that resolves the original URL.
  • npx jest src/adapters/connector-setup.service.spec.ts src/adapters/mcp-adapter.spec.ts src/common src/connectors src/ee/support/support-connector-setup.service.spec.ts src/mcp-server --maxWorkers=2 --silent: 232 suites passed, 7 skipped (they are env-gated integration and live suites), 6221 tests passed, 0 failed.
  • npx jest src/adapters/[a-z]*.spec.ts src/ee/support/support-triage.service.spec.ts --maxWorkers=2 --silent: 32 suites, 22135 tests passed.
  • npm run lint: 0 errors. The 11 warnings are all in files this PR does not touch, and --fix changed nothing outside this diff.
  • npx tsc --noEmit -p tsconfig.json: no errors.
  • node scripts/generate-llms-txt.mjs (from the root) regenerated llms-full.txt.
  • Real database check on a scratch postgres:17-alpine, removed afterwards:
    • Setup: prisma migrate deploy, then rows for a Bitrix24 REST connector, a Slack REST connector with plaintext variables, a clean REST connector and a DATABASE connector.
    • First run of repairBaseUrlSecrets(): moved c-bitrix, c-slack. Both env_vars columns now hold {"$enc": …}, updated_at was bumped, and each stored template resolves to the original URL.
    • Second run: 0 moved; 0 left as is; 0 failed.
    • The clean and DATABASE rows were not touched.

I did not run the full npm test (the shared machine has limited resources). The suites above cover every module that imports the changed code.

Checklist

  • My code follows the existing code style
  • I have tested my changes locally
  • I have updated documentation (if applicable)
  • All existing tests pass (npm test). Only the targeted suites listed above were run.
  • This PR has a descriptive title

Related Issues

Same approach as #1742 (DATABASE connection strings, #1245) for HTTP base URLs.

Left for later:

  • The start-up pass only touches base URLs. Tool paths, headers and tool names are still cleaned on save and by scripts/secret-literals/repair.ts.
  • user@host with a token as the user name (no password) is not treated as a secret.
  • The setup chat tools that write connectors without going through the connector controller were not reviewed here.

A hand-made REST connector whose base URL carries its credential in the
path (a Bitrix24 inbound webhook, /rest/<user>/<code>/) kept the code in
plaintext in connectors.base_url and returned it to admins and the
owner. The secret-literal extraction on save did not know these shapes.

- Detector: vendor-anchored rules for Bitrix24 (cloud and self-hosted),
  Slack, Discord, Microsoft Teams, Zapier and Make webhook URLs, and the
  password in scheme://user:password@ for every HTTP connector type.
  Power Automate sig=, Google Chat key/token and Telegram were already
  covered by the query and token rules.
- Save: only the secret segment moves into a variable named so that
  isSecretName masks it; calls interpolate {{VAR}} verbatim, so the URL
  sent is unchanged.
- Start-up pass repairBaseUrlSecrets, run after the database pass: same
  rules minus the long-random-token heuristic, compare-and-swap on
  base URL and updatedAt, strict decryption of the variables, one
  summary log line with ids and counts only.
- API output masks a secret still in an HTTP base URL (and the catalog
  baseline) as ********; a form that sends the masked URL back keeps
  the stored one, a URL edited around the mask is refused.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant