Repository navigation
Move secrets in HTTP base URLs into encrypted variables - #1778
Closed
helpcodeai wants to merge 1 commit into
Closed
helpcodeai wants to merge 1 commit into
helpcodeai wants to merge 1 commit into
Conversation
A hand-made REST connector whose base URL carries its credential in the
path (a Bitrix24 inbound webhook, /rest/<user>/<code>/) kept the code in
plaintext in connectors.base_url and returned it to admins and the
owner. The secret-literal extraction on save did not know these shapes.
- Detector: vendor-anchored rules for Bitrix24 (cloud and self-hosted),
Slack, Discord, Microsoft Teams, Zapier and Make webhook URLs, and the
password in scheme://user:password@ for every HTTP connector type.
Power Automate sig=, Google Chat key/token and Telegram were already
covered by the query and token rules.
- Save: only the secret segment moves into a variable named so that
isSecretName masks it; calls interpolate {{VAR}} verbatim, so the URL
sent is unchanged.
- Start-up pass repairBaseUrlSecrets, run after the database pass: same
rules minus the long-random-token heuristic, compare-and-swap on
base URL and updatedAt, strict decryption of the variables, one
summary log line with ids and counts only.
- API output masks a secret still in an HTTP base URL (and the catalog
baseline) as ********; a form that sends the masked URL back keeps
the stored one, a URL edited around the mask is refused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A REST connector built by hand with a webhook URL as its base URL kept the credential in plaintext in
connectors.base_urland showed it in full to admins and the owner. The typical case is a Bitrix24 inbound webhook,https://portal.bitrix24.com/rest/<user>/<code>/, where the code works like a password. The catalog adapter avoids this with{{BITRIX24_WEBHOOK_URL}}; hand-made connectors went through the secret-literal extraction on save, but its rules did not know these shapes (a 16-character lower-case code is not "high entropy").This PR teaches the extraction the common webhook URL shapes and
user:password@, repairs stored rows at start-up the way #1742 does for DATABASE connection strings, and masks whatever is left in API output.Changes
common/secret-literals.util.ts, new kindswebhook_tokenandurl_password). Each rule is tied to the vendor's host and the exact path it issues:/rest/<digits>/<code>on*.bitrix24.<tld>. On any host (self-hosted portals) only a 16-character[a-z0-9]code with letters and digits that is not hex-only.hooks.slack.com/services|workflows|triggers/…/<secret>/api[/vN]/webhooks/<id>/<token>…webhook.office.com/…/IncomingWebhook/<rest>and the legacyoutlook.office.comhosthooks.zapier.com/hooks/catch|standard/<id>/<code>hook.<zone>.make.com|integromat.com|make.celonis.com/<token>scheme://user:password@host(any HTTP type;{{VAR}},********and user-only userinfo are left alone)sig=, Google Chatkey=/token=, Telegrambot<token>, and generictoken/key/apikey/api_key/secret/sig/signature/access_tokenquery parameters./rest/api/2, Magento/rest/V1,/rest/1/crm.deal.list, hex ids, docs examples,?email=a@b,host:port, placeholders, masks). The test that checks the whole catalog for false positives still finds nothing.SecretExtractor, unchanged entry points in create/update/import/restore). Only the secret segment moves, into<NAME>_WEBHOOK_TOKEN/<NAME>_PASSWORD. Both names are onesisSecretNamerecognises, so the value is encrypted, masked and never returned. I chose the segment over the whole URL because:hostFromUrland log lines;{{VAR}}in the base URL verbatim before the request:interpolateConnectorConfig/interpolateString(no encoding) inDynamicMcpTools,testConnection, MCP discovery and OAuth discovery, and WSDL import viaresolvedAddress.normalizeConnectorBaseUrlreturns the string as typed, so the resolved URL is byte-for-byte the one stored before. Tests check this for every case, including a realDynamicMcpTools.executeToolcall whoseRestEngine.executegets the original Bitrix URL.ConnectorsService.repairBaseUrlSecrets(). It is chained afterrepairDatabaseUrlCredentialsinonApplicationBootstrap, is not awaited and never throws.updateManywith a compare-and-swap onbaseUrlandupdatedAt, so a concurrent save or a second instance wins. The at-rest extension encrypts the variables.config.baseUrlBaselinegets the same variable.Base URL secrets: secrets of N connectors moved into encrypted variables (…); M left as is (…); K failed (…). A second run changes nothing.toPublicConnector, so connector reads, lists, update answers and exports without secrets). A secret still in an HTTP base URL or its baseline shows as********, to everyone, admins included.********is refused with a 400 asking for the full address.docs/connectors/rest.mdgets a new section "Secrets typed into the base URL".docs/api-reference.mdnotes the export masking.llms-full.txtis regenerated.After upgrading
On its first start, v0.55.4 moves secrets still written in the base URL of REST, GraphQL, SOAP, OData and MCP connectors into encrypted connector variables. These are webhook codes and tokens (Bitrix24, Slack, Discord, Microsoft Teams, Zapier, Make), Telegram bot tokens,
?api_key=-style values anduser:password@. Only the secret part of the URL is replaced, by a{{<NAME>_WEBHOOK_TOKEN}}-style reference, and calls go to exactly the same address as before. Nothing needs to be configured, and it works the same on Cloud and self-hosted. The backend log shows one line,Base URL secrets: secrets of N connectors moved into encrypted variables (…); M left as is; K failed, with connector ids only. "Failed" connectors (variables that cannot be decrypted with the currentENCRYPTION_KEY) stay unchanged, their secret is shown as********, and the next start tries again.scripts/secret-literals/report.tslists anything left, including tool paths and headers, which this pass does not touch.Type
Testing
All commands were run in
packages/backendafternpm ci(root) andnpx prisma generatewith a dummyDATABASE_URL:npx jest src/common/secret-literals.util.spec.ts --maxWorkers=2: 146 passed. It includes the detector positives and negatives and the catalog false-positive check.npx jest src/connectors/base-url-secrets.util.spec.ts src/connectors/secret-extraction.util.spec.ts --maxWorkers=2: 40 passed at the time of the run.npx jest src/connectors/connectors.service.base-url-secrets-repair.spec.ts src/connectors/connectors.service.database-credentials-repair.spec.ts --maxWorkers=2: 50 passed. This covers the repair (moved, left, failed, idempotency, no secret in logs, the boot order after the database pass) and the existing database pass.npx jest src/connectors/secret-guard.flows.spec.ts --maxWorkers=2: 29 passed. This covers create and update through the controller, the masked round trip, the refusal, and a tool call that resolves the original URL.npx jest src/adapters/connector-setup.service.spec.ts src/adapters/mcp-adapter.spec.ts src/common src/connectors src/ee/support/support-connector-setup.service.spec.ts src/mcp-server --maxWorkers=2 --silent: 232 suites passed, 7 skipped (they are env-gated integration and live suites), 6221 tests passed, 0 failed.npx jest src/adapters/[a-z]*.spec.ts src/ee/support/support-triage.service.spec.ts --maxWorkers=2 --silent: 32 suites, 22135 tests passed.npm run lint: 0 errors. The 11 warnings are all in files this PR does not touch, and--fixchanged nothing outside this diff.npx tsc --noEmit -p tsconfig.json: no errors.node scripts/generate-llms-txt.mjs(from the root) regeneratedllms-full.txt.postgres:17-alpine, removed afterwards:prisma migrate deploy, then rows for a Bitrix24 REST connector, a Slack REST connector with plaintext variables, a clean REST connector and a DATABASE connector.repairBaseUrlSecrets(): movedc-bitrix, c-slack. Bothenv_varscolumns now hold{"$enc": …},updated_atwas bumped, and each stored template resolves to the original URL.0 moved; 0 left as is; 0 failed.I did not run the full
npm test(the shared machine has limited resources). The suites above cover every module that imports the changed code.Checklist
npm test). Only the targeted suites listed above were run.Related Issues
Same approach as #1742 (DATABASE connection strings, #1245) for HTTP base URLs.
Left for later:
scripts/secret-literals/repair.ts.user@hostwith a token as the user name (no password) is not treated as a secret.