diff --git a/docs/api-reference.md b/docs/api-reference.md index df944d7c0..ce1005b3a 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -100,7 +100,9 @@ endpoint. `authConfig` (OAuth and other stored credentials) is never exported: re-authorize the connector after importing it. Secret environment variables and headers are included only for admins signed in to the dashboard or using a `MANAGEMENT_ADMIN` key. For other roles, and for any `MANAGEMENT_READ` key, they -are empty and named in `maskedEnvVars` / `maskedHeaders`. Per-tool `useProxy`, annotation +are empty and named in `maskedEnvVars` / `maskedHeaders`, and a secret still written in an HTTP +base URL (a webhook token, a key, `user:password@`) shows as `********`; see +[Secrets typed into the base URL](connectors/rest.md#secrets-typed-into-the-base-url). Per-tool `useProxy`, annotation overrides and role access are not exported. ```bash diff --git a/docs/connectors/rest.md b/docs/connectors/rest.md index 5b62fb6ed..7623515ce 100644 --- a/docs/connectors/rest.md +++ b/docs/connectors/rest.md @@ -467,6 +467,54 @@ curl -s -X PUT http://localhost:4000/api/connectors/$CONNECTOR_ID/env-vars \ Use `{{VAR_NAME}}` in endpoint mappings. Parameters matching env var names are automatically stripped from the tool schema. +### Secrets typed into the base URL + +Some services put the credential in the address itself: a Bitrix24 inbound webhook +(`https://portal.bitrix24.com/rest/1//`), a Slack, Discord, Microsoft Teams, Zapier or Make +webhook URL, a Telegram bot URL (`https://api.telegram.org/bot`), a key in the query string +(`?api_key=…`, `?token=…`, `?sig=…`), or `https://user:password@host`. Whoever holds such a URL can +use it like a password. + +When you save a REST, GraphQL, SOAP, OData or MCP connector with such a base URL, AnythingMCP moves +only the secret part into an encrypted variable and stores the rest of the URL with a reference: + +``` +typed: https://portal.bitrix24.com/rest/1/k2bx…vqgo/ +stored: https://portal.bitrix24.com/rest/1/{{BITRIX_CRM_WEBHOOK_TOKEN}}/ +``` + +- The variable is named after the connector (`_WEBHOOK_TOKEN`, `_PASSWORD`, + `_API_KEY`, `TELEGRAM_BOT_TOKEN`) or reuses a variable that already holds the same value. + Its value is encrypted at rest and never returned by the connector API, to admins neither (only an + admin export with secrets carries it, like every other variable); the dashboard shows it as set. The save answers with `secretsMoved` and `secretsNotice`, naming the variable, + never the secret. +- Calls are unchanged: every call replaces `{{VAR}}` in the base URL with the variable's value as + stored, so the request goes to exactly the URL you typed. To change the secret, edit the variable, + or type the full URL again. +- The host and the rest of the path stay readable, so the connector page, error hints and logs still + show where the connector points. +- Headers and tool paths get the same treatment (`Authorization: Bearer ` becomes + `Bearer {{_TOKEN}}`). + +Connectors saved before a rule existed keep the secret in the stored base URL until they are +saved again. Until then the API shows it as `********` +(`https://portal.bitrix24.com/rest/1/********/`); saving the form with the URL as shown keeps the +stored secret and moves it into a variable, and a URL edited around `********` is refused, so type +the full address again. The backend also moves these at every start, the same way a save does, +except for long random path segments with no known shape (a save moves those and says so; the +start-up pass leaves them alone). This needs no setting and works the same on AnythingMCP Cloud and +self-hosted. The backend log shows the result with connector ids and counts, never a URL: + +``` +Base URL secrets: secrets of 2 connectors moved into encrypted variables (cm1…, cm2…); 0 left as is; 0 failed +``` + +"Left as is" are connectors that were saved while the pass ran; the next start checks them again. +"Failed" names connectors whose variables could not be decrypted (a changed `ENCRYPTION_KEY`) or +written; they stay unchanged, still masked in the API, and the next start tries again. +`scripts/secret-literals/report.ts` lists what is left, and `repair.ts` also cleans tool paths, +headers and tool names (dry run by default, with a backup table before `--apply`). + --- ## Example: Full REST API Setup diff --git a/llms-full.txt b/llms-full.txt index eb30de10a..fa469518e 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -3378,6 +3378,54 @@ curl -s -X PUT http://localhost:4000/api/connectors/$CONNECTOR_ID/env-vars \ Use `{{VAR_NAME}}` in endpoint mappings. Parameters matching env var names are automatically stripped from the tool schema. +### Secrets typed into the base URL + +Some services put the credential in the address itself: a Bitrix24 inbound webhook +(`https://portal.bitrix24.com/rest/1//`), a Slack, Discord, Microsoft Teams, Zapier or Make +webhook URL, a Telegram bot URL (`https://api.telegram.org/bot`), a key in the query string +(`?api_key=…`, `?token=…`, `?sig=…`), or `https://user:password@host`. Whoever holds such a URL can +use it like a password. + +When you save a REST, GraphQL, SOAP, OData or MCP connector with such a base URL, AnythingMCP moves +only the secret part into an encrypted variable and stores the rest of the URL with a reference: + +``` +typed: https://portal.bitrix24.com/rest/1/k2bx…vqgo/ +stored: https://portal.bitrix24.com/rest/1/{{BITRIX_CRM_WEBHOOK_TOKEN}}/ +``` + +- The variable is named after the connector (`_WEBHOOK_TOKEN`, `_PASSWORD`, + `_API_KEY`, `TELEGRAM_BOT_TOKEN`) or reuses a variable that already holds the same value. + Its value is encrypted at rest and never returned by the connector API, to admins neither (only an + admin export with secrets carries it, like every other variable); the dashboard shows it as set. The save answers with `secretsMoved` and `secretsNotice`, naming the variable, + never the secret. +- Calls are unchanged: every call replaces `{{VAR}}` in the base URL with the variable's value as + stored, so the request goes to exactly the URL you typed. To change the secret, edit the variable, + or type the full URL again. +- The host and the rest of the path stay readable, so the connector page, error hints and logs still + show where the connector points. +- Headers and tool paths get the same treatment (`Authorization: Bearer ` becomes + `Bearer {{_TOKEN}}`). + +Connectors saved before a rule existed keep the secret in the stored base URL until they are +saved again. Until then the API shows it as `********` +(`https://portal.bitrix24.com/rest/1/********/`); saving the form with the URL as shown keeps the +stored secret and moves it into a variable, and a URL edited around `********` is refused, so type +the full address again. The backend also moves these at every start, the same way a save does, +except for long random path segments with no known shape (a save moves those and says so; the +start-up pass leaves them alone). This needs no setting and works the same on AnythingMCP Cloud and +self-hosted. The backend log shows the result with connector ids and counts, never a URL: + +``` +Base URL secrets: secrets of 2 connectors moved into encrypted variables (cm1…, cm2…); 0 left as is; 0 failed +``` + +"Left as is" are connectors that were saved while the pass ran; the next start checks them again. +"Failed" names connectors whose variables could not be decrypted (a changed `ENCRYPTION_KEY`) or +written; they stay unchanged, still masked in the API, and the next start tries again. +`scripts/secret-literals/report.ts` lists what is left, and `repair.ts` also cleans tool paths, +headers and tool names (dry run by default, with a backup table before `--apply`). + --- ## Example: Full REST API Setup @@ -8558,7 +8606,9 @@ endpoint. `authConfig` (OAuth and other stored credentials) is never exported: re-authorize the connector after importing it. Secret environment variables and headers are included only for admins signed in to the dashboard or using a `MANAGEMENT_ADMIN` key. For other roles, and for any `MANAGEMENT_READ` key, they -are empty and named in `maskedEnvVars` / `maskedHeaders`. Per-tool `useProxy`, annotation +are empty and named in `maskedEnvVars` / `maskedHeaders`, and a secret still written in an HTTP +base URL (a webhook token, a key, `user:password@`) shows as `********`; see +[Secrets typed into the base URL](). Per-tool `useProxy`, annotation overrides and role access are not exported. ```bash diff --git a/packages/backend/src/common/secret-literals.util.spec.ts b/packages/backend/src/common/secret-literals.util.spec.ts index f1aba790b..89c02e102 100644 --- a/packages/backend/src/common/secret-literals.util.spec.ts +++ b/packages/backend/src/common/secret-literals.util.spec.ts @@ -128,6 +128,155 @@ describe('findSecretLiterals', () => { }); }); +// Webhook credentials of each vendor's shape, assembled at runtime too. +const BITRIX_CODE = ['k2bx', 'a8g5', 'n4gh', 'vqgo'].join(''); +const SLACK_HOOK = ['AbCdEfGh', 'IjKlMnOp', 'QrStUv12'].join(''); +const DISCORD_TOKEN = ['Xy1_Ab2-Cd3Ef4Gh5', 'Ij6Kl7Mn8Op9Qr0St', 'Uv1Wx2Yz3Ab4Cd5Ef', 'Gh6Ij7Kl8Mn9Op0Q'].join(''); +const TEAMS_HEX = ['0123456789ab', 'cdef01234567', '89abcdef'].join(''); +const TEAMS_GUID = '1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d'; +const ZAPIER_CODE = ['ab', 'c1d', '2e'].join(''); +const MAKE_TOKEN = ['h7k2m9p4q1', 'w8x3y6z5a0', 'b2c4d6e8f1'].join(''); +const SIG = ['aB3dE5gH7jK9mN1pQ3', 'sT5vW7yZ9bC1eF3hJ5', 'kL7n'].join(''); + +describe('webhook URLs and URL passwords', () => { + /** The URL, the secret it carries, and the kind of finding. */ + const POSITIVE: Array<[label: string, url: string, secret: string, kind: string]> = [ + ['Bitrix24 cloud portal', `https://portal.bitrix24.com/rest/1/${BITRIX_CODE}/`, BITRIX_CODE, 'webhook_token'], + ['Bitrix24 .de portal with a method', `https://acme.bitrix24.de/rest/27/${BITRIX_CODE}/crm.deal.list.json`, BITRIX_CODE, 'webhook_token'], + ['Bitrix24 .com.br portal, no trailing slash', `https://acme.bitrix24.com.br/rest/3/${BITRIX_CODE}`, BITRIX_CODE, 'webhook_token'], + ['Bitrix24 self-hosted portal', `https://crm.example.com/rest/12/${BITRIX_CODE}/`, BITRIX_CODE, 'webhook_token'], + ['Bitrix24 self-hosted, relative path', `/rest/12/${BITRIX_CODE}/crm.lead.add`, BITRIX_CODE, 'webhook_token'], + ['Slack incoming webhook', `https://hooks.slack.com/services/T0A1B2C3D/B0E1F2G3H/${SLACK_HOOK}`, SLACK_HOOK, 'webhook_token'], + ['Slack workflow trigger', `https://hooks.slack.com/workflows/T0A1B2C3D/A0E1F2G3H/456789012/${SLACK_HOOK}`, SLACK_HOOK, 'webhook_token'], + ['Slack triggers URL', `https://hooks.slack.com/triggers/T0A1B2C3D/456789012/${SLACK_HOOK}`, SLACK_HOOK, 'webhook_token'], + ['Discord webhook', `https://discord.com/api/webhooks/123456789012345678/${DISCORD_TOKEN}`, DISCORD_TOKEN, 'webhook_token'], + ['Discord webhook, versioned, old host', `https://discordapp.com/api/v10/webhooks/123456789012345678/${DISCORD_TOKEN}/slack`, DISCORD_TOKEN, 'webhook_token'], + [ + 'Teams Office 365 connector', + `https://acme.webhook.office.com/webhookb2/${TEAMS_GUID}@${TEAMS_GUID}/IncomingWebhook/${TEAMS_HEX}/${TEAMS_GUID}`, + `${TEAMS_HEX}/${TEAMS_GUID}`, + 'webhook_token', + ], + [ + 'Teams connector, legacy host, V2 suffix', + `https://outlook.office.com/webhook/${TEAMS_GUID}@${TEAMS_GUID}/IncomingWebhook/${TEAMS_HEX}/${TEAMS_GUID}/V2abcDEF123/`, + `${TEAMS_HEX}/${TEAMS_GUID}/V2abcDEF123`, + 'webhook_token', + ], + [ + 'Power Automate workflow (sig=)', + `https://prod-12.westeurope.logic.azure.com:443/workflows/0a1b2c3d4e5f/triggers/manual/paths/invoke?api-version=2016-06-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=${SIG}`, + SIG, + 'secret_query_param', + ], + [ + 'Power Platform workflow (sig=)', + `https://default0a1b.2c.environment.api.powerplatform.com:443/powerautomate/automations/direct/workflows/0a1b2c3d/triggers/manual/paths/invoke?api-version=1&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=${SIG}`, + SIG, + 'secret_query_param', + ], + ['Zapier catch hook', `https://hooks.zapier.com/hooks/catch/1234567/${ZAPIER_CODE}/`, ZAPIER_CODE, 'webhook_token'], + ['Zapier catch hook, silent', `https://hooks.zapier.com/hooks/catch/1234567/${ZAPIER_CODE}/silent/`, ZAPIER_CODE, 'webhook_token'], + ['Make webhook', `https://hook.eu1.make.com/${MAKE_TOKEN}`, MAKE_TOKEN, 'webhook_token'], + ['Integromat webhook', `https://hook.integromat.com/${MAKE_TOKEN}`, MAKE_TOKEN, 'webhook_token'], + ['Make on Celonis', `https://hook.us2.make.celonis.com/${MAKE_TOKEN}`, MAKE_TOKEN, 'webhook_token'], + ['REST API with user:password@', 'https://apiuser:Pa55w0rd-1@api.example.com/v1', 'Pa55w0rd-1', 'url_password'], + ['user:password@ with a port and an encoded character', 'https://svc:p%40ss42@erp.example.com:8443/odata/', 'p%40ss42', 'url_password'], + ['password next to a {{VAR}} user name', 'https://{{API_USER}}:hunter2pass@api.example.com', 'hunter2pass', 'url_password'], + ]; + + it.each(POSITIVE)('finds the secret of a %s', (_label, url, secret, kind) => { + const found = findSecretLiterals(url); + expect(found).toHaveLength(1); + expect(found[0].kind).toBe(kind); + expect(found[0].value).toBe(secret); + expect(url.slice(found[0].start, found[0].end)).toBe(secret); + expect(found[0].pattern).not.toContain(secret); + }); + + it('also finds the Telegram token and the Google Chat key and token it already knew', () => { + expect(findSecretLiterals(`https://api.telegram.org/bot${TG}`)[0].kind).toBe('telegram_bot_token'); + const chatToken = ['3D', 'aBcDeFgHiJkLmNoPqRsT', '_uVwXyZ0123456789%3D'].join(''); + const chat = findSecretLiterals( + `https://chat.googleapis.com/v1/spaces/AAAAbCdEfGh/messages?key=${GOOGLE}&token=${chatToken}`, + ); + expect(chat.map((f) => f.value)).toEqual([GOOGLE, chatToken]); + }); + + it('describes a finding without the secret', () => { + const [bitrix] = findSecretLiterals(`https://portal.bitrix24.com/rest/1/${BITRIX_CODE}/`); + expect(bitrix.pattern).toBe('Bitrix24 webhook ***(16)'); + const [password] = findSecretLiterals('https://apiuser:Pa55w0rd-1@api.example.com'); + expect(password.pattern).toBe('user:***(10)@'); + }); + + it.each([ + // plain API base URLs + 'https://api.example.com/v1', + 'https://api.example.com/rest/v1/users', + 'https://jira.example.com/rest/api/2/issue', + 'https://acme.atlassian.net/rest/agile/1.0/board/123', + 'https://shop.example.com/rest/V1/products', + 'https://shop.example.com/rest/default/V1/orders', + 'https://erp.example.com/rest/2.0/', + 'https://erp.example.com/rest/1/crm.deal.list', + 'https://erp.example.com/rest/1/abcdefghijklmnop/', + 'https://erp.example.com/rest/1/0123456789abcdef/', + 'https://erp.example.com/rest/1/1234567890123456/', + 'https://erp.example.com/rest/1/k2bxa8g5n4ghvqgoextra/', + 'https://portal.bitrix24.com/rest/', + 'https://portal.bitrix24.com/rest/1/', + 'https://portal.bitrix24.com/rest/api/crm.deal.list', + 'https://portal.bitrix24.com/rest/1/department/', + 'https://portal.bitrix24.com/rest/1/12345678/', + 'https://notbitrix24.example.com/rest/1/abc/', + 'https://slack.com/api/chat.postMessage', + 'https://hooks.slack.com/services/', + // Slack's own docs example, assembled so that push protection does not take it for a leak + `https://hooks.slack.com/services/T00000000/B00000000/${'X'.repeat(24)}`, + 'https://hooks.slack.com/services/T0A1B2C3D/B0E1F2G3H/short', + 'https://discord.com/api/v10/channels/123456789012345678/messages', + 'https://discord.com/api/webhooks/123456789012345678', + 'https://discord.com/api/webhooks/123456789012345678/short', + 'https://graph.microsoft.com/v1.0/teams/abc/channels', + 'https://acme.webhook.office.com/webhookb2/abc', + 'https://www.example.com/docs/IncomingWebhook/how-to-set-up-one', + 'https://outlook.office.com/api/v2.0/me/messages', + 'https://hooks.zapier.com/hooks/catch/', + 'https://hooks.zapier.com/hooks/catch/1234567/', + 'https://zapier.com/hooks/catch/1234567/abc1d2e/', + 'https://www.make.com/en/help/tools/webhooks', + 'https://eu1.make.com/api/v2/scenarios', + 'https://hook.eu1.make.com/short', + 'https://hook.eu1.make.com/{{MAKE_WEBHOOK_TOKEN}}', + 'https://logic.azure.com/workflows/abc/triggers/manual/paths/invoke?api-version=2016-06-01', + // placeholders, masks and user names alone + 'https://portal.bitrix24.com/rest/1/{{BITRIX24_WEBHOOK_CODE}}/', + '{{BITRIX24_WEBHOOK_URL}}', + 'https://{{API_USER}}:{{API_PASSWORD}}@api.example.com', + 'https://apiuser:{{API_PASSWORD}}@api.example.com', + 'https://apiuser:********@api.example.com', + 'https://apiuser@api.example.com', + 'https://apiuser:@api.example.com', + 'https://portal.bitrix24.com/rest/1/********/', + // an @ or a colon that is not userinfo + 'https://api.example.com:8443?email=someone@example.com', + 'https://api.example.com/users/someone@example.com', + 'https://api.example.com/v1?from=a:b@c', + 'mailto:someone@example.com', + 'someone:secretish@example.com', + ])('does not flag %s', (url) => { + expect(findSecretLiterals(url)).toEqual([]); + }); + + it('redacts a webhook URL quoted in free text', () => { + const text = `POST https://hooks.slack.com/services/T0A1B2C3D/B0E1F2G3H/${SLACK_HOOK} failed`; + expect(redactSecretLiterals(text)).toBe( + 'POST https://hooks.slack.com/services/T0A1B2C3D/B0E1F2G3H/[REDACTED] failed', + ); + }); +}); + describe('looksHighEntropy', () => { it('rejects UUIDs, hex and short tokens', () => { expect(looksHighEntropy('123e4567e89b12d3a456426614174000')).toBe(false); diff --git a/packages/backend/src/common/secret-literals.util.ts b/packages/backend/src/common/secret-literals.util.ts index 2692657f7..1fbb0bfa2 100644 --- a/packages/backend/src/common/secret-literals.util.ts +++ b/packages/backend/src/common/secret-literals.util.ts @@ -25,6 +25,8 @@ export type SecretKind = | 'jwt' | 'secret_query_param' | 'credential_header' + | 'webhook_token' + | 'url_password' | 'high_entropy'; export interface SecretFinding { @@ -301,6 +303,10 @@ function redactShape(kind: SecretKind, value: string, label: string): string { return `${label}=***(${n})`; case 'credential_header': return `${label}: ***(${n})`; + case 'webhook_token': + return `${label} webhook ***(${n})`; + case 'url_password': + return `user:***(${n})@`; case 'high_entropy': default: return `***(${n} chars, high-entropy)`; @@ -333,6 +339,167 @@ function dedupe(findings: SecretFinding[]): SecretFinding[] { return out; } +/** + * Webhook URLs whose path is the credential: whoever holds the URL can post + * as the account, so the token in it is a password, not an id. Each rule is + * tied to the vendor's own host and the exact path shape the vendor issues, + * so an ordinary API base URL never matches. Only the secret segment is + * returned; the host and the ids around it stay readable. + * + * Incoming webhooks that carry their secret in a query parameter (Power + * Automate and Logic Apps `sig=`, Google Chat `key=`/`token=`) are found by + * the query rule, and Telegram's `bot` by its own rule above. + */ +interface WebhookRule { + label: string; + host: RegExp; + /** The secret's offsets within the path, or null. */ + match: (path: string) => { start: number; end: number } | null; +} + +/** + * `prefix` then `secret` (then `/` or the end of the path): the secret's + * offsets. `accept`, when given, must also say yes to the secret. + */ +function segmentAfter(re: RegExp, accept?: (secret: string) => boolean): WebhookRule['match'] { + return (path) => { + const m = re.exec(path); + if (!m || (accept && !accept(m[2]))) return null; + const start = (m.index ?? 0) + m[1].length; + return { start, end: start + m[2].length }; + }; +} + +const WEBHOOK_RULES: WebhookRule[] = [ + // Bitrix24 inbound webhook: /rest///. The + // code is random letters and digits; a word there (`/rest/1/department/`) + // is not one. + { + label: 'Bitrix24', + host: /(?:^|\.)bitrix24\.[a-z]{2,3}(?:\.[a-z]{2})?$/, + match: segmentAfter( + /(\/rest\/\d{1,10}\/)([A-Za-z0-9_]{8,64})(?=\/|$)/, + (code) => /[A-Za-z]/.test(code) && /[0-9]/.test(code), + ), + }, + // Slack incoming webhook (/services/T…/B…/) and workflow triggers. + { + label: 'Slack', + host: /^hooks\.slack(?:-gov)?\.com$/, + match: segmentAfter( + /^(\/(?:services|workflows|triggers)\/(?:[A-Za-z0-9]{1,32}\/){1,4})([A-Za-z0-9]{16,128})(?=\/?$)/, + ), + }, + // Discord: /api[/v10]/webhooks// + { + label: 'Discord', + host: /^(?:(?:ptb|canary)\.)?discord(?:app)?\.com$/, + match: segmentAfter(/^(\/api(?:\/v\d{1,2})?\/webhooks\/\d{15,22}\/)([A-Za-z0-9_-]{20,128})(?=\/|$)/), + }, + // Microsoft Teams (Office 365 connector): …/IncomingWebhook//[/V2…] + { + label: 'Microsoft Teams', + host: /(?:^|\.)webhook\.office\.com$|^outlook\.office(?:365)?\.com$/, + match: (path) => { + const marker = '/IncomingWebhook/'; + const at = path.indexOf(marker); + if (at === -1) return null; + const start = at + marker.length; + const secret = path.slice(start).replace(/\/+$/, ''); + if (!/^[A-Za-z0-9@._/-]{16,512}$/.test(secret)) return null; + return { start, end: start + secret.length }; + }, + }, + // Zapier catch hook: /hooks/catch/// + { + label: 'Zapier', + host: /^hooks\.zapier\.com$/, + match: segmentAfter(/^(\/hooks\/(?:catch|standard)\/\d{1,15}\/)([A-Za-z0-9]{4,64})(?=\/|$)/), + }, + // Make (formerly Integromat) custom webhook: hook..make.com/ + { + label: 'Make', + host: /^hook\.(?:[a-z0-9-]{1,20}\.)?(?:make\.com|integromat\.com|make\.celonis\.com)$/, + match: segmentAfter(/^(\/)([A-Za-z0-9]{16,64})(?=\/|$)/), + }, +]; + +/** + * A Bitrix24 webhook on a self-hosted portal (any host): `/rest//` + * followed by the 16-character webhook code, lower case letters and digits, + * with both and not hex only (a hex id is a resource, not a code). + */ +const BITRIX_ANY_HOST = /\/rest\/\d{1,10}\/([a-z0-9]{16})(?=[/?#\s"'`]|$)/g; + +function isBitrixCode(code: string): boolean { + return /[a-z]/.test(code) && /[0-9]/.test(code) && !HEX.test(code); +} + +/** `XXXXXXXX…`, as in vendor docs: an example, not a credential. */ +function isRepeatedChar(value: string): boolean { + return /^(.)\1+$/.test(value); +} + +/** An absolute URL in free text: scheme, `://`, then up to the next space or quote. */ +const ABSOLUTE_URL = /(?`]{1,4096}/g; + +/** What a masked password slot holds (database-url-credentials.util.ts URL_PASSWORD_MASK). */ +const MASKED_PASSWORD = /^\*+$/; + +/** + * Secrets in an absolute URL's userinfo and in a known webhook path. + * `masked` has its placeholders blanked out; offsets refer to `text`. + */ +function findUrlSecrets(text: string, masked: string): SecretFinding[] { + const found: SecretFinding[] = []; + for (const m of masked.matchAll(ABSOLUTE_URL)) { + const url = m[0]; + const offset = m.index ?? 0; + const authStart = url.indexOf('://') + 3; + const rest = url.slice(authStart); + // The authority ends at the path, the query or the fragment: an `@` in + // `?email=a@b.example` is not userinfo. + const stop = rest.search(/[/?#]/); + const authority = stop === -1 ? rest : rest.slice(0, stop); + const at = authority.lastIndexOf('@'); + + // scheme://user:password@host: the password, never the user name. + if (at > 0) { + const userinfo = authority.slice(0, at); + const colon = userinfo.indexOf(':'); + const password = colon === -1 ? '' : userinfo.slice(colon + 1); + if (password && !password.includes(FILLER) && !MASKED_PASSWORD.test(password)) { + const start = offset + authStart + colon + 1; + found.push(finding('url_password', text, start, start + password.length, 'password')); + } + } + + const host = (at === -1 ? authority : authority.slice(at + 1)).replace(/:\d*$/, '').toLowerCase(); + if (host.includes(FILLER)) continue; + const tail = rest.slice(authority.length); + const queryAt = tail.search(/[?#]/); + const path = queryAt === -1 ? tail : tail.slice(0, queryAt); + const pathOffset = offset + authStart + authority.length; + for (const rule of WEBHOOK_RULES) { + if (!rule.host.test(host)) continue; + const span = rule.match(path); + if (!span) continue; + const secret = path.slice(span.start, span.end); + if (secret.includes(FILLER) || isRepeatedChar(secret)) continue; + found.push( + finding('webhook_token', text, pathOffset + span.start, pathOffset + span.end, rule.label), + ); + } + } + + for (const m of masked.matchAll(BITRIX_ANY_HOST)) { + if (!isBitrixCode(m[1])) continue; + const start = (m.index ?? 0) + m[0].length - m[1].length; + found.push(finding('webhook_token', text, start, start + m[1].length, 'Bitrix24')); + } + return found; +} + export interface FindOptions { /** Include the high-entropy heuristic (default true). */ highEntropy?: boolean; @@ -348,7 +515,9 @@ export function findSecretLiterals(input: string, opts: FindOptions = {}): Secre if (typeof input !== 'string' || input.length < 8) return []; const text = input.length > MAX_SCAN_LENGTH ? input.slice(0, MAX_SCAN_LENGTH) : input; const masked = maskPlaceholders(text); - const found: SecretFinding[] = []; + // First, so that a webhook token the long-random-token rule would also + // catch is reported (and named) as what it is. + const found: SecretFinding[] = findUrlSecrets(text, masked); for (const m of masked.matchAll(TELEGRAM)) { // Without `bot` in front, `:` is also an AWS ARN tail diff --git a/packages/backend/src/connectors/base-url-secrets.util.spec.ts b/packages/backend/src/connectors/base-url-secrets.util.spec.ts new file mode 100644 index 000000000..eb87a4ee0 --- /dev/null +++ b/packages/backend/src/connectors/base-url-secrets.util.spec.ts @@ -0,0 +1,97 @@ +import { + BASE_URL_SECRET_MASK, + baseUrlSecrets, + hasHttpBaseUrl, + keepMaskedBaseUrl, + maskBaseUrlSecrets, +} from './base-url-secrets.util'; +import { toPublicConnector } from './connector-secrets.util'; +import { toExportEntry } from './connector-export.util'; + +const BITRIX = ['k2bx', 'a8g5', 'n4gh', 'vqgo'].join(''); +const BITRIX_URL = `https://portal.bitrix24.com/rest/1/${BITRIX}/`; +const OPAQUE = 'q8ZtW3nRk2LxV9pMbH4cJs7YdF6gTe1A'; + +describe('base URL secrets', () => { + it('checks the HTTP connector types only', () => { + for (const type of ['REST', 'GRAPHQL', 'SOAP', 'MCP', 'ODATA']) expect(hasHttpBaseUrl(type)).toBe(true); + for (const type of ['DATABASE', 'EMAIL', 'FILE_SEARCH', 'WEBHOOK', undefined]) expect(hasHttpBaseUrl(type)).toBe(false); + }); + + it('uses every rule of a save but the long-random-token one', () => { + expect(baseUrlSecrets(BITRIX_URL)).toHaveLength(1); + expect(baseUrlSecrets('https://maps.example.com/api?key=a1b2c3d4e5f6')).toHaveLength(1); + expect(baseUrlSecrets(`https://api.example.com/files/x/${OPAQUE}`)).toEqual([]); + expect(baseUrlSecrets(null)).toEqual([]); + }); + + it('masks only the secret, and leaves references and clean URLs alone', () => { + expect(maskBaseUrlSecrets(BITRIX_URL)).toBe(`https://portal.bitrix24.com/rest/1/${BASE_URL_SECRET_MASK}/`); + expect(maskBaseUrlSecrets('https://u:Pa55w0rd-1@api.example.com/?token=abc123def456')).toBe( + 'https://u:********@api.example.com/?token=********', + ); + expect(maskBaseUrlSecrets('https://portal.bitrix24.com/rest/1/{{B_WEBHOOK_TOKEN}}/')).toBe( + 'https://portal.bitrix24.com/rest/1/{{B_WEBHOOK_TOKEN}}/', + ); + expect(maskBaseUrlSecrets('https://api.example.com/v1')).toBe('https://api.example.com/v1'); + }); + + describe('an edit made against the masked view', () => { + it('keeps the stored URL when the masked one comes back unchanged', () => { + expect(keepMaskedBaseUrl(maskBaseUrlSecrets(BITRIX_URL), BITRIX_URL)).toBe(BITRIX_URL); + }); + + it('takes a retyped URL as sent', () => { + const other = `https://portal.bitrix24.com/rest/2/${BITRIX}/`; + expect(keepMaskedBaseUrl(other, BITRIX_URL)).toBe(other); + }); + + it('cannot rebuild a URL edited around the mask', () => { + expect(keepMaskedBaseUrl(`https://other.bitrix24.com/rest/1/${BASE_URL_SECRET_MASK}/`, BITRIX_URL)).toBeNull(); + }); + + it('changes nothing when nothing stored was masked', () => { + expect(keepMaskedBaseUrl('https://api.example.com/v2', 'https://api.example.com/v1')).toBe( + 'https://api.example.com/v2', + ); + }); + }); + + describe('API output', () => { + const connector = { + id: 'c1', + name: 'Bitrix', + type: 'REST', + baseUrl: BITRIX_URL, + headers: null, + envVars: { BITRIX_WEBHOOK_TOKEN: BITRIX, PORTAL: 'portal' }, + config: { adapterSlug: 'custom', baseUrlBaseline: BITRIX_URL }, + authConfig: 'encrypted', + tools: [], + }; + + it('never returns the secret of a base URL, to anyone', () => { + const out = toPublicConnector(connector); + expect(out.baseUrl).toBe(`https://portal.bitrix24.com/rest/1/${BASE_URL_SECRET_MASK}/`); + expect((out.config as any).baseUrlBaseline).toBe(out.baseUrl); + expect((out.config as any).adapterSlug).toBe('custom'); + expect(out.envVars).toEqual({ BITRIX_WEBHOOK_TOKEN: '', PORTAL: 'portal' }); + expect(out.maskedEnvVars).toEqual(['BITRIX_WEBHOOK_TOKEN']); + expect(JSON.stringify(out)).not.toContain(BITRIX); + }); + + it('nor in an export without secrets', () => { + expect(JSON.stringify(toExportEntry(connector, false))).not.toContain(BITRIX); + }); + + it('shows a repaired connector as it is stored', () => { + const repaired = { ...connector, baseUrl: 'https://portal.bitrix24.com/rest/1/{{BITRIX_WEBHOOK_TOKEN}}/', config: null }; + expect(toPublicConnector(repaired).baseUrl).toBe(repaired.baseUrl); + }); + + it('leaves a DATABASE connection string to its own masking', () => { + const db = { ...connector, type: 'DATABASE', baseUrl: 'postgresql://u:s3cret@db.example.com/x', config: null }; + expect(toPublicConnector(db).baseUrl).toBe('postgresql://u:********@db.example.com/x'); + }); + }); +}); diff --git a/packages/backend/src/connectors/base-url-secrets.util.ts b/packages/backend/src/connectors/base-url-secrets.util.ts new file mode 100644 index 000000000..75f8d84da --- /dev/null +++ b/packages/backend/src/connectors/base-url-secrets.util.ts @@ -0,0 +1,61 @@ +import { findSecretLiterals, type FindOptions, type SecretFinding } from '../common/secret-literals.util'; +import { SECRET_GUARDED_TYPES } from './secret-extraction.util'; + +/** + * A secret written into an HTTP connector's base URL: a Bitrix24, Slack, + * Discord, Teams, Zapier or Make webhook URL, a Telegram bot token, a + * `?api_key=` style parameter, a known key shape, or `user:password@`. + * + * Saving a connector moves such a secret into an encrypted variable and + * stores `{{VAR}}` in its place (SecretExtractor). This file covers what was + * stored before that, or what a save could not move: + * + * - the start-up repair (`ConnectorsService.repairBaseUrlSecrets`) moves it, + * with these rules; + * - until then, the API shows the secret as `********` (`toPublicConnector`), + * and a form that sends the masked URL back unchanged keeps the stored one. + * + * These rules are every rule a save applies except the long-random-token + * heuristic: a save tells the user what moved, a start-up pass does it + * silently on every instance, so it only acts on shapes that are certain. + */ +export const BASE_URL_SECRET_RULES: FindOptions = { highEntropy: false }; + +/** What the API shows in place of a secret in a base URL. */ +export const BASE_URL_SECRET_MASK = '********'; + +/** Connector types whose base URL is an HTTP address (and so is checked). */ +export function hasHttpBaseUrl(type: unknown): boolean { + return typeof type === 'string' && SECRET_GUARDED_TYPES.has(type); +} + +/** The secrets in a base URL, by the rules above. */ +export function baseUrlSecrets(url: unknown): SecretFinding[] { + if (typeof url !== 'string' || !url) return []; + return findSecretLiterals(url, BASE_URL_SECRET_RULES); +} + +/** The base URL with every secret replaced by the mask. `{{VAR}}` references stay. */ +export function maskBaseUrlSecrets(url: string): string { + const found = baseUrlSecrets(url); + let out = url; + for (const f of [...found].sort((a, b) => b.start - a.start)) { + out = `${out.slice(0, f.start)}${BASE_URL_SECRET_MASK}${out.slice(f.end)}`; + } + return out; +} + +/** + * Apply an edit made against the masked view. The masked URL sent back as it + * was shown means "unchanged": the stored URL is kept (and the save then + * moves its secret into a variable). A URL that still holds the mask but was + * edited elsewhere cannot be put back together: null, and the caller asks for + * the full address. Anything else is taken as sent. + */ +export function keepMaskedBaseUrl(incoming: string, stored: string): string | null { + if (typeof incoming !== 'string' || typeof stored !== 'string') return incoming; + const masked = maskBaseUrlSecrets(stored); + if (masked === stored) return incoming; + if (incoming === masked) return stored; + return incoming.includes(BASE_URL_SECRET_MASK) ? null : incoming; +} diff --git a/packages/backend/src/connectors/connector-export.util.ts b/packages/backend/src/connectors/connector-export.util.ts index 2760dea0d..8b7595236 100644 --- a/packages/backend/src/connectors/connector-export.util.ts +++ b/packages/backend/src/connectors/connector-export.util.ts @@ -9,8 +9,9 @@ export const EXPORT_FORMAT_VERSION = '1.0'; * One connector as it appears in an export file. authConfig is never * exported. Secret env vars and headers are kept only when `includeSecrets` * is set (admins); otherwise they are left empty and named in - * `maskedEnvVars` / `maskedHeaders`, and a DATABASE connection string comes - * back with its password masked. + * `maskedEnvVars` / `maskedHeaders`, a DATABASE connection string comes + * back with its password masked, and a secret in an HTTP base URL as + * `********`. */ export function toExportEntry(c: any, includeSecrets: boolean) { const masked = includeSecrets ? null : toPublicConnector(c); @@ -22,7 +23,7 @@ export function toExportEntry(c: any, includeSecrets: boolean) { authType: c.authType, specUrl: c.specUrl, headers: masked ? masked.headers : c.headers, - config: c.config, + config: masked ? masked.config : c.config, envVars: masked ? masked.envVars : c.envVars, ...(masked ? { maskedEnvVars: masked.maskedEnvVars, maskedHeaders: masked.maskedHeaders } diff --git a/packages/backend/src/connectors/connector-secrets.util.ts b/packages/backend/src/connectors/connector-secrets.util.ts index a7a5edfff..0aa2fc3eb 100644 --- a/packages/backend/src/connectors/connector-secrets.util.ts +++ b/packages/backend/src/connectors/connector-secrets.util.ts @@ -1,5 +1,6 @@ import { getAdapter } from '../adapters/catalog'; import { maskUrlPassword } from './database-url-credentials.util'; +import { hasHttpBaseUrl, maskBaseUrlSecrets } from './base-url-secrets.util'; /** * Which connector environment variables and headers hold credentials, and how @@ -266,7 +267,9 @@ export function maskHeaders( * non-secret OAuth settings have their own endpoint (`GET :id/oauth-config`). * A DATABASE connection string comes back with its password masked; new * connectors no longer store one there (database-url-credentials.util.ts), - * this covers the ones saved before. + * this covers the ones saved before. Likewise a secret in an HTTP base URL + * (a webhook token, a key, `user:password@`) comes back as `********` + * (base-url-secrets.util.ts). */ export function toPublicConnector< T extends ConnectorLike & { envVars?: unknown; authConfig?: unknown }, @@ -285,12 +288,23 @@ export function toPublicConnector< webhookSecret: _webhookSecret, ...rest } = connector as T & { authStatusRef?: unknown; webhookSecret?: unknown }; - const { type, baseUrl } = connector as { type?: unknown; baseUrl?: unknown }; + const { type, baseUrl, config } = connector as { type?: unknown; baseUrl?: unknown; config?: unknown }; + // An HTTP base URL can carry a webhook token, a key or `user:password@` + // that was stored before saves moved it into a variable (the start-up + // repair moves it too): shown masked to everyone, admins included. + const httpMasked = hasHttpBaseUrl(type) && typeof baseUrl === 'string' ? maskBaseUrlSecrets(baseUrl) : null; + const baseline = (config as { baseUrlBaseline?: unknown } | null | undefined)?.baseUrlBaseline; + const maskedBaseline = + hasHttpBaseUrl(type) && typeof baseline === 'string' ? maskBaseUrlSecrets(baseline) : null; return { ...rest, ...(type === 'DATABASE' && typeof baseUrl === 'string' ? { baseUrl: maskUrlPassword(baseUrl) } : {}), + ...(httpMasked !== null && httpMasked !== baseUrl ? { baseUrl: httpMasked } : {}), + ...(maskedBaseline !== null && maskedBaseline !== baseline + ? { config: { ...(config as Record), baseUrlBaseline: maskedBaseline } } + : {}), envVars: env.envVars, headers: hdr.headers, maskedEnvVars: env.maskedEnvVars, diff --git a/packages/backend/src/connectors/connectors.controller.ts b/packages/backend/src/connectors/connectors.controller.ts index 3549295d9..870ac3fae 100644 --- a/packages/backend/src/connectors/connectors.controller.ts +++ b/packages/backend/src/connectors/connectors.controller.ts @@ -118,6 +118,7 @@ import { mergeSummaries, type SecretMoveSummary, } from './secret-extraction.util'; +import { hasHttpBaseUrl, keepMaskedBaseUrl } from './base-url-secrets.util'; import { MemberVisibilityService, canSeeConnector, @@ -1632,6 +1633,18 @@ export class ConnectorsController { // The connection string was shown with its password masked. data.baseUrl = keepMaskedUrlPassword(dto.baseUrl, connector.baseUrl); } + if (hasHttpBaseUrl(connector.type) && dto.baseUrl !== undefined) { + // A secret still stored in the base URL was shown as ********: the + // URL sent back as shown keeps the stored one (and the extraction + // below then moves the secret into a variable). + const kept = keepMaskedBaseUrl(dto.baseUrl, connector.baseUrl); + if (kept === null) { + throw new BadRequestException( + 'The base URL still contains a hidden secret (********). Type the full address again, secret included: it is then stored in an encrypted variable.', + ); + } + data.baseUrl = kept; + } if ((connector.type as string) === 'EMAIL') { data.authConfig = this.mergeEmailAuthConfig(connector.authConfig, dto.authConfig); } diff --git a/packages/backend/src/connectors/connectors.service.base-url-secrets-repair.spec.ts b/packages/backend/src/connectors/connectors.service.base-url-secrets-repair.spec.ts new file mode 100644 index 000000000..5fadd1e38 --- /dev/null +++ b/packages/backend/src/connectors/connectors.service.base-url-secrets-repair.spec.ts @@ -0,0 +1,350 @@ +import { Logger } from '@nestjs/common'; +import { ConnectorsService } from './connectors.service'; +import { openEnvVars, openEnvVarsStrict, sealEnvVars } from '../common/crypto/env-vars-at-rest'; +import { encrypt } from '../common/crypto/encryption.util'; +import { interpolateConnectorConfig } from '../common/env-interpolation.util'; +import { isSecretName } from './connector-secrets.util'; + +/** + * The boot pass for HTTP connectors whose base URL still carries a secret + * (a webhook token, a key, `user:password@`): it moves what a save would + * move into an encrypted variable, leaves `{{VAR}}` in the URL so calls are + * unchanged, and says nothing secret in the log. + */ +describe('ConnectorsService.repairBaseUrlSecrets', () => { + const KEY = 'k'.repeat(48); + const BITRIX = ['k2bx', 'a8g5', 'n4gh', 'vqgo'].join(''); + const SLACK = ['AbCdEfGh', 'IjKlMnOp', 'QrStUv12'].join(''); + const TG = `7123456789:${['AAH', 'k3jd9QwErTyUiOp', 'AsDfGhJkLzXcVbN', 'm12'].join('')}`; + const OPAQUE = 'q8ZtW3nRk2LxV9pMbH4cJs7YdF6gTe1A'; + const MCP_KEY = 'k3y4bc12def9'; + const PASSWORD = 'Pa55w0rd-1'; + + type Row = { + id: string; + name: string; + type: string; + baseUrl: string; + config: Record | null; + /** As stored: sealed, plain, or null. */ + envVars: unknown; + updatedAt: Date; + }; + + const at = new Date('2026-11-01T10:00:00.000Z'); + const make = (r: Omit & Partial): Row => ({ + config: null, + envVars: null, + updatedAt: at, + ...r, + }); + + /** Stored row, the base URL it ends with, and the variables it then has. */ + const MOVED: Array<[label: string, row: () => Row, baseUrl: string, envVars: Record]> = [ + [ + 'a Bitrix24 webhook URL', + () => make({ id: 'c-bitrix', name: 'Bitrix CRM', type: 'REST', baseUrl: `https://portal.bitrix24.com/rest/1/${BITRIX}/` }), + 'https://portal.bitrix24.com/rest/1/{{BITRIX_CRM_WEBHOOK_TOKEN}}/', + { BITRIX_CRM_WEBHOOK_TOKEN: BITRIX }, + ], + [ + 'a Slack webhook URL, keeping the variables it has', + () => + make({ + id: 'c-slack', + name: 'Alerts', + type: 'REST', + baseUrl: `https://hooks.slack.com/services/T0A1B2C3D/B0E1F2G3H/${SLACK}`, + envVars: sealEnvVars({ CHANNEL: 'general', LIMIT: 5 }), + }), + 'https://hooks.slack.com/services/T0A1B2C3D/B0E1F2G3H/{{ALERTS_WEBHOOK_TOKEN}}', + { CHANNEL: 'general', LIMIT: 5, ALERTS_WEBHOOK_TOKEN: SLACK }, + ], + [ + 'a key in an MCP server URL', + () => make({ id: 'c-mcp', name: 'Gateway', type: 'MCP', baseUrl: `https://mcp.example.com/mcp?api_key=${MCP_KEY}` }), + 'https://mcp.example.com/mcp?api_key={{GATEWAY_API_KEY}}', + { GATEWAY_API_KEY: MCP_KEY }, + ], + [ + 'user:password@ in an OData URL, with plain stored variables', + () => + make({ + id: 'c-odata', + name: 'ERP', + type: 'ODATA', + baseUrl: `https://svc:${PASSWORD}@erp.example.com/odata/`, + envVars: { COMPANY: 'acme' }, + }), + 'https://svc:{{ERP_PASSWORD}}@erp.example.com/odata/', + { COMPANY: 'acme', ERP_PASSWORD: PASSWORD }, + ], + [ + 'a Telegram token a variable already holds', + () => + make({ + id: 'c-reuse', + name: 'Bot', + type: 'REST', + baseUrl: `https://api.telegram.org/bot${TG}`, + envVars: sealEnvVars({ TELEGRAM_BOT_TOKEN: TG }), + }), + 'https://api.telegram.org/bot{{TELEGRAM_BOT_TOKEN}}', + { TELEGRAM_BOT_TOKEN: TG }, + ], + ]; + + /** Never written: clean, references, the left-out heuristic, another type. */ + const UNTOUCHED: Array<() => Row> = [ + () => make({ id: 'c-clean', name: 'Clean', type: 'REST', baseUrl: 'https://api.example.com/v1' }), + () => + make({ + id: 'c-ref', + name: 'Ref', + type: 'REST', + baseUrl: 'https://portal.bitrix24.com/rest/1/{{B_WEBHOOK_TOKEN}}/', + envVars: sealEnvVars({ B_WEBHOOK_TOKEN: BITRIX }), + }), + () => make({ id: 'c-entropy', name: 'Opaque', type: 'REST', baseUrl: `https://api.example.com/hooks/${OPAQUE}` }), + () => make({ id: 'c-db', name: 'Db', type: 'DATABASE', baseUrl: `postgresql://u:${PASSWORD}@db.example.com/x` }), + ]; + + const SECRETS = [BITRIX, SLACK, TG.split(':')[1], MCP_KEY, PASSWORD, 'portal.bitrix24.com', '://']; + + let rows: Row[]; + let prisma: { connector: { findMany: jest.Mock; updateMany: jest.Mock }; $queryRaw: jest.Mock }; + let service: ConnectorsService; + let logged: string[]; + const row = (id: string) => rows.find((r) => r.id === id)!; + const envOf = (id: string) => openEnvVarsStrict(row(id).envVars); + + beforeEach(() => { + process.env.ENCRYPTION_KEY = KEY; + rows = [...MOVED.map(([, r]) => r()), ...UNTOUCHED.map((r) => r())]; + prisma = { + connector: { + findMany: jest.fn(async ({ where }: any) => + rows + .filter((r) => where.type.in.includes(r.type)) + .sort((a, b) => a.id.localeCompare(b.id)) + .map(({ id, name, type, baseUrl, config, updatedAt }) => ({ id, name, type, baseUrl, config, updatedAt })), + ), + // The guarded write, with the at-rest extension's encryption. + updateMany: jest.fn(async ({ where, data }: any) => { + const hit = rows.filter( + (r) => r.id === where.id && r.baseUrl === where.baseUrl && r.updatedAt.getTime() === where.updatedAt.getTime(), + ); + for (const r of hit) { + Object.assign(r, data, { updatedAt: new Date(r.updatedAt.getTime() + 1000) }); + if ('envVars' in data) r.envVars = sealEnvVars(data.envVars); + } + return { count: hit.length }; + }), + }, + $queryRaw: jest.fn(async (_strings: TemplateStringsArray, ids: string[]) => + rows.filter((r) => ids.includes(r.id)).map((r) => ({ id: r.id, env_vars: r.envVars })), + ), + }; + service = new ConnectorsService(prisma as any, { get: () => KEY } as any, {} as any, {} as any, {} as any, {} as any, {} as any); + logged = []; + for (const level of ['log', 'warn', 'error', 'debug', 'verbose'] as const) { + jest.spyOn(Logger.prototype, level).mockImplementation((...args: unknown[]) => { + logged.push(`${level}: ${args.map(String).join(' ')}`); + }); + } + }); + + afterEach(() => jest.restoreAllMocks()); + + it('reads the HTTP connectors, and the variables of the candidates only', async () => { + await service.repairBaseUrlSecrets(); + + expect(prisma.connector.findMany).toHaveBeenCalledTimes(1); + expect(prisma.connector.findMany.mock.calls[0][0].where.type.in.sort()).toEqual(['GRAPHQL', 'MCP', 'ODATA', 'REST', 'SOAP']); + expect(prisma.$queryRaw).toHaveBeenCalledTimes(1); + expect([...prisma.$queryRaw.mock.calls[0][1]].sort()).toEqual(['c-bitrix', 'c-mcp', 'c-odata', 'c-reuse', 'c-slack']); + }); + + it.each(MOVED)('moves %s into an encrypted variable', async (_label, stored, baseUrl, envVars) => { + const id = stored().id; + await service.repairBaseUrlSecrets(); + + expect(row(id).baseUrl).toBe(baseUrl); + expect(envOf(id)).toEqual(envVars); + // Encrypted at rest, and named so that the dashboard masks it. + for (const value of Object.values(envVars)) { + if (typeof value === 'string') expect(JSON.stringify(row(id).envVars)).not.toContain(value); + } + for (const name of Object.keys(envVars)) { + if (!['CHANNEL', 'LIMIT', 'COMPANY'].includes(name)) expect(isSecretName(name)).toBe(true); + } + }); + + it.each(MOVED)('keeps the URL a call sends for %s', async (_label, stored) => { + const before = stored().baseUrl; + const id = stored().id; + await service.repairBaseUrlSecrets(); + + const { config } = interpolateConnectorConfig( + { baseUrl: row(id).baseUrl }, + { method: 'GET', path: '/' }, + openEnvVars(row(id).envVars) as Record, + ); + expect(config.baseUrl).toBe(before); + }); + + it('does not rewrite the variables when one already holds the secret', async () => { + const sealed = row('c-reuse').envVars; + await service.repairBaseUrlSecrets(); + + const call = prisma.connector.updateMany.mock.calls.find(([arg]) => arg.where.id === 'c-reuse')![0]; + expect(call.data).toEqual({ baseUrl: 'https://api.telegram.org/bot{{TELEGRAM_BOT_TOKEN}}' }); + expect(row('c-reuse').envVars).toBe(sealed); + }); + + it('cleans the install-time baseline of a catalog connector with the same variable', async () => { + const url = `https://portal.bitrix24.com/rest/1/${BITRIX}/`; + row('c-bitrix').config = { adapterSlug: 'acme-crm', baseUrlBaseline: url, adapterVersion: '1.0.0' }; + + await service.repairBaseUrlSecrets(); + + expect(row('c-bitrix').config).toEqual({ + adapterSlug: 'acme-crm', + baseUrlBaseline: 'https://portal.bitrix24.com/rest/1/{{BITRIX_CRM_WEBHOOK_TOKEN}}/', + adapterVersion: '1.0.0', + }); + expect(Object.keys(envOf('c-bitrix')!)).toEqual(['BITRIX_CRM_WEBHOOK_TOKEN']); + }); + + it('leaves everything else exactly as stored', async () => { + const before = UNTOUCHED.map((r) => ({ ...row(r().id) })); + await service.repairBaseUrlSecrets(); + + for (const stored of before) expect(row(stored.id)).toEqual(stored); + const written = prisma.connector.updateMany.mock.calls.map(([arg]) => arg.where.id); + expect(written).toEqual(expect.not.arrayContaining(before.map((r) => r.id))); + }); + + it('reports what it moved in one line with ids and counts', async () => { + const result = await service.repairBaseUrlSecrets(); + + expect(result).toEqual({ + moved: ['c-bitrix', 'c-mcp', 'c-odata', 'c-reuse', 'c-slack'], + leftAsIs: [], + failed: [], + }); + expect(logged).toEqual([ + 'log: Base URL secrets: secrets of 5 connectors moved into encrypted variables ' + + '(c-bitrix, c-mcp, c-odata, c-reuse, c-slack); 0 left as is; 0 failed', + ]); + }); + + it('changes nothing on a second run', async () => { + await service.repairBaseUrlSecrets(); + const after = rows.map((r) => ({ ...r })); + prisma.connector.updateMany.mockClear(); + prisma.$queryRaw.mockClear(); + + const second = await service.repairBaseUrlSecrets(); + + expect(prisma.connector.updateMany).not.toHaveBeenCalled(); + expect(prisma.$queryRaw).not.toHaveBeenCalled(); + expect(rows).toEqual(after); + expect(second).toEqual({ moved: [], leftAsIs: [], failed: [] }); + expect(logged.at(-1)).toBe( + 'log: Base URL secrets: secrets of 0 connectors moved into encrypted variables; 0 left as is; 0 failed', + ); + }); + + it('never writes a row whose variables it cannot decrypt', async () => { + row('c-slack').envVars = { $enc: encrypt(JSON.stringify({ CHANNEL: 'general' }), 'x'.repeat(48), 'connectors.env_vars') }; + const stored = { ...row('c-slack') }; + + const result = await service.repairBaseUrlSecrets(); + + expect(result.failed).toEqual(['c-slack']); + expect(row('c-slack')).toEqual(stored); + expect(result.moved).toContain('c-bitrix'); + expect(logged.at(-1)).toMatch(/^warn: Base URL secrets: .*; 1 failed \(c-slack\)$/); + }); + + it('leaves a row that was saved between the read and the write', async () => { + const read = prisma.connector.findMany.getMockImplementation()!; + prisma.connector.findMany.mockImplementation(async (arg: any) => { + const found = await read(arg); + row('c-bitrix').updatedAt = new Date(at.getTime() + 5000); + row('c-bitrix').envVars = sealEnvVars({ EDITED: 'yes' }); + return found; + }); + + const result = await service.repairBaseUrlSecrets(); + + expect(result.leftAsIs).toEqual(['c-bitrix']); + expect(row('c-bitrix').baseUrl).toBe(`https://portal.bitrix24.com/rest/1/${BITRIX}/`); + expect(envOf('c-bitrix')).toEqual({ EDITED: 'yes' }); + }); + + it('goes on with the other rows when one write fails', async () => { + const write = prisma.connector.updateMany.getMockImplementation()!; + prisma.connector.updateMany.mockImplementation(async (arg: any) => { + if (arg.where.id === 'c-mcp') { + // A Prisma message quotes the arguments of the call. + throw Object.assign(new Error(`Invalid invocation: where: { baseUrl: "${arg.where.baseUrl}" }`), { code: 'P2028' }); + } + return write(arg); + }); + + const result = await service.repairBaseUrlSecrets(); + + expect(result.failed).toEqual(['c-mcp']); + expect(result.moved).toEqual(['c-bitrix', 'c-odata', 'c-reuse', 'c-slack']); + expect(row('c-mcp').baseUrl).toBe(`https://mcp.example.com/mcp?api_key=${MCP_KEY}`); + expect(logged[0]).toBe( + 'warn: Connector c-mcp: moving the secret in its base URL failed (P2028), the connector is left as is', + ); + }); + + it('does not throw when the query fails, and says only what kind of error it was', async () => { + prisma.connector.findMany.mockRejectedValue( + Object.assign(new Error(`connect failed https://portal.bitrix24.com/rest/1/${BITRIX}/`), { code: 'P1001' }), + ); + + await expect(service.repairBaseUrlSecrets()).resolves.toEqual({ moved: [], leftAsIs: [], failed: [] }); + expect(logged).toEqual(['error: Base URL secrets not checked: P1001']); + }); + + it('runs at start-up after the database pass, without holding the start up', async () => { + const read = prisma.connector.findMany.getMockImplementation()!; + const order: string[] = []; + prisma.connector.findMany.mockImplementation(async (arg: any) => { + // The database pass filters on one type; this pass on a list. + if (typeof arg.where.type === 'string') { + order.push('database'); + return []; + } + order.push('base url'); + return read(arg); + }); + + expect(service.onApplicationBootstrap()).toBeUndefined(); + expect(order).toEqual(['database']); + for (let i = 0; i < 10; i++) await new Promise((resolve) => setImmediate(resolve)); + + expect(order).toEqual(['database', 'base url']); + expect(row('c-bitrix').baseUrl).toBe('https://portal.bitrix24.com/rest/1/{{BITRIX_CRM_WEBHOOK_TOKEN}}/'); + }); + + it('logs no URL, host or secret, whatever happens', async () => { + row('c-slack').envVars = { $enc: encrypt('{}', 'x'.repeat(48), 'connectors.env_vars') }; + const write = prisma.connector.updateMany.getMockImplementation()!; + prisma.connector.updateMany.mockImplementation(async (arg: any) => { + if (arg.where.id === 'c-odata') throw new Error(`update failed for ${arg.where.baseUrl} -> ${arg.data.baseUrl}`); + return write(arg); + }); + + await service.repairBaseUrlSecrets(); + + const text = logged.join('\n'); + for (const secret of SECRETS) expect(text).not.toContain(secret); + }); +}); diff --git a/packages/backend/src/connectors/connectors.service.ts b/packages/backend/src/connectors/connectors.service.ts index 542bdf1a4..a96550923 100644 --- a/packages/backend/src/connectors/connectors.service.ts +++ b/packages/backend/src/connectors/connectors.service.ts @@ -19,6 +19,8 @@ import { SoapEngine, WsdlReadError } from './engines/soap.engine'; import { wsdlUrlForLog } from './parsers/wsdl.parser'; import { connectorSecretValues } from './connector-secrets.util'; import { moveUrlCredentials, urlUserinfo } from './database-url-credentials.util'; +import { BASE_URL_SECRET_RULES, baseUrlSecrets } from './base-url-secrets.util'; +import { SECRET_GUARDED_TYPES, SecretExtractor } from './secret-extraction.util'; import { GraphqlEngine } from './engines/graphql.engine'; import { DatabaseEngine, assertSafeSqlIdentifierVars } from './engines/database.engine'; import { McpClientEngine } from './engines/mcp-client.engine'; @@ -74,6 +76,12 @@ function errorKind(err: unknown): string { return typeof name === 'string' && name ? name : 'Error'; } +/** A catalog connector's install-time base URL (`config.baseUrlBaseline`), or null. */ +function baselineOf(config: unknown): string | null { + const value = (config as { baseUrlBaseline?: unknown } | null | undefined)?.baseUrlBaseline; + return typeof value === 'string' ? value : null; +} + /** Connector ids for a log line: the first few, then how many more. */ function idsForLog(ids: string[], max = 20): string { if (ids.length === 0) return ''; @@ -111,10 +119,14 @@ export class ConnectorsService implements OnApplicationBootstrap { onApplicationBootstrap() { // Not awaited, so a large table never holds up the start. The pass // catches its own errors; the catch here is for anything it did not - // foresee, so that nothing it does can stop the boot. - void this.repairDatabaseUrlCredentials().catch((err) => - this.logger.error(`Database connection strings not checked: ${errorKind(err)}`), - ); + // foresee, so that nothing it does can stop the boot. The base URL pass + // runs after the database one, so the two never read the table at once. + void this.repairDatabaseUrlCredentials() + .catch((err) => + this.logger.error(`Database connection strings not checked: ${errorKind(err)}`), + ) + .then(() => this.repairBaseUrlSecrets()) + .catch((err) => this.logger.error(`Base URL secrets not checked: ${errorKind(err)}`)); } async findAll(): Promise { @@ -445,6 +457,137 @@ export class ConnectorsService implements OnApplicationBootstrap { return { moved, leftAsIs, failed }; } + /** + * Move the secrets still written in stored HTTP base URLs (a Bitrix24, + * Slack, Discord, Teams, Zapier or Make webhook token, a Telegram bot + * token, a `?api_key=` value, `user:password@`) into the connector's + * encrypted variables, leaving `{{VAR}}` in the URL. A save has done this + * since the extraction existed (SecretExtractor), and for webhook URLs and + * userinfo since its rules learned them; connectors stored before kept the + * secret in the plaintext `base_url` until someone saved them. This does + * it for them at boot, like `repairDatabaseUrlCredentials`. + * + * - The rules are a save's, minus the long-random-token heuristic + * (base-url-secrets.util.ts): nobody sees a notice for what this pass + * moves, so it only moves shapes that are certain. Only the secret + * segment moves; host and path stay readable. `config.baseUrlBaseline` + * (catalog connectors) holds the same URL and gets the same variable. + * - Calls are unchanged: `{{VAR}}` in a base URL is replaced verbatim from + * the variables on every call (env-interpolation.util.ts), so the URL + * sent is byte for byte the one stored before. A variable that already + * holds the value is reused. + * - Variables are read as stored and decrypted strictly: a row whose + * variables cannot be decrypted is never written (its secret stays, and + * stays masked on read). + * - Instance-level maintenance, outside `tenantTx`, like the database + * pass. Each row is written by id, and only if its base URL and + * `updatedAt` are still what was read, so a save in between, or a + * second instance running the same pass, wins. A second run finds + * nothing to move. + * - Never throws. Logs connector ids and counts only: no URL, variable + * value or error message. + */ + async repairBaseUrlSecrets(): Promise<{ + moved: string[]; + leftAsIs: string[]; + failed: string[]; + }> { + const moved: string[] = []; + const leftAsIs: string[] = []; + const failed: string[] = []; + type Row = { + id: string; + name: string; + type: string; + baseUrl: string; + config: unknown; + updatedAt: Date; + }; + let candidates: Row[]; + let storedEnv: Map; + try { + const rows: Row[] = await this.prisma.connector.findMany({ + where: { type: { in: [...SECRET_GUARDED_TYPES] as ConnectorType[] } }, + select: { id: true, name: true, type: true, baseUrl: true, config: true, updatedAt: true }, + orderBy: { id: 'asc' }, + }); + candidates = rows.filter( + (r) => baseUrlSecrets(r.baseUrl).length > 0 || baseUrlSecrets(baselineOf(r.config)).length > 0, + ); + storedEnv = new Map(); + if (candidates.length > 0) { + // Raw on purpose: what is stored, before the at-rest extension, so + // that "cannot be decrypted" is told apart from "no variables". + const ids = candidates.map((r) => r.id); + const envRows = await this.prisma.$queryRaw>` + SELECT id, env_vars FROM connectors WHERE id = ANY(${ids})`; + for (const r of envRows) storedEnv.set(r.id, r.env_vars); + } + } catch (err) { + this.logger.error(`Base URL secrets not checked: ${errorKind(err)}`); + return { moved, leftAsIs, failed }; + } + + for (const row of candidates) { + const env = openEnvVarsStrict(storedEnv.get(row.id)); + if (env === null) { + failed.push(row.id); + this.logger.warn( + `Connector ${row.id}: its variables could not be read (wrong ENCRYPTION_KEY?), the secret in its base URL is left as is`, + ); + continue; + } + try { + const strings = Object.fromEntries( + Object.entries(env).filter(([, v]) => typeof v === 'string'), + ) as Record; + const secrets = new SecretExtractor(row.name, strings); + const baseUrl = secrets.extractUrl(row.baseUrl, 'baseUrl', BASE_URL_SECRET_RULES); + const baseline = baselineOf(row.config); + const cleanedBaseline = + baseline === null + ? null + : secrets.extractUrl(baseline, 'config.baseUrlBaseline', BASE_URL_SECRET_RULES); + if (baseUrl === row.baseUrl && cleanedBaseline === baseline) { + leftAsIs.push(row.id); + continue; + } + const data: Prisma.ConnectorUpdateManyMutationInput = { baseUrl }; + if (Object.keys(secrets.newEnvVars).length > 0) { + // Every stored key is kept, non-string values included; the + // extension encrypts the whole object on write. + data.envVars = { ...env, ...secrets.newEnvVars } as Prisma.InputJsonValue; + } + if (cleanedBaseline !== baseline) { + data.config = { + ...(row.config as Record), + baseUrlBaseline: cleanedBaseline, + } as Prisma.InputJsonValue; + } + const { count } = await this.prisma.connector.updateMany({ + where: { id: row.id, baseUrl: row.baseUrl, updatedAt: row.updatedAt }, + data, + }); + // No row: it was saved, or repaired by another instance, in between. + (count > 0 ? moved : leftAsIs).push(row.id); + } catch (err) { + failed.push(row.id); + this.logger.warn( + `Connector ${row.id}: moving the secret in its base URL failed (${errorKind(err)}), the connector is left as is`, + ); + } + } + + const summary = + `Base URL secrets: secrets of ${moved.length} connector${moved.length === 1 ? '' : 's'} ` + + `moved into encrypted variables${idsForLog(moved)}; ` + + `${leftAsIs.length} left as is${idsForLog(leftAsIs)}; ` + + `${failed.length} failed${idsForLog(failed)}`; + if (failed.length > 0) this.logger.warn(summary); + else this.logger.log(summary); + return { moved, leftAsIs, failed }; + } + /** * Merges `patch` into the connector's existing (decrypted) authConfig and * re-encrypts it, instead of replacing the whole object. Used by the OAuth diff --git a/packages/backend/src/connectors/secret-extraction.util.spec.ts b/packages/backend/src/connectors/secret-extraction.util.spec.ts index e98208edb..016da2999 100644 --- a/packages/backend/src/connectors/secret-extraction.util.spec.ts +++ b/packages/backend/src/connectors/secret-extraction.util.spec.ts @@ -6,6 +6,7 @@ import { } from './secret-extraction.util'; import { interpolateConnectorConfig } from '../common/env-interpolation.util'; import { CurlParser } from './parsers/curl.parser'; +import { isSecretName } from './connector-secrets.util'; const TG = `7123456789:${['AAH', 'k3jd9QwErTyUiOp', 'AsDfGhJkLzXcVbN', 'm12'].join('')}`; const GOOGLE = ['AI', 'zaSyB1x2y3z4A5B6C7D8E9F0gHiJkLmNoPqRs'].join(''); @@ -242,3 +243,80 @@ describe('SecretExtractor', () => { expect(mergeSummaries([{}, {}])).toEqual({}); }); }); + +describe('SecretExtractor: webhook URLs and URL passwords', () => { + const BITRIX = ['k2bx', 'a8g5', 'n4gh', 'vqgo'].join(''); + const SLACK = ['AbCdEfGh', 'IjKlMnOp', 'QrStUv12'].join(''); + const DISCORD = ['Xy1_Ab2-Cd3Ef4Gh5', 'Ij6Kl7Mn8Op9Qr0St', 'Uv1Wx2Yz3Ab4Cd5Ef', 'Gh6Ij7Kl8Mn9Op0Q'].join(''); + const GUID = '1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d'; + const HEX = ['0123456789ab', 'cdef01234567', '89abcdef'].join(''); + + /** Connector name, base URL as typed, what is stored, the variable and its value. */ + const CASES: Array<[name: string, typed: string, stored: string, variable: string, value: string]> = [ + [ + 'Bitrix CRM', + `https://portal.bitrix24.com/rest/1/${BITRIX}/`, + 'https://portal.bitrix24.com/rest/1/{{BITRIX_CRM_WEBHOOK_TOKEN}}/', + 'BITRIX_CRM_WEBHOOK_TOKEN', + BITRIX, + ], + [ + 'Slack Webhook', + `https://hooks.slack.com/services/T0A1B2C3D/B0E1F2G3H/${SLACK}`, + 'https://hooks.slack.com/services/T0A1B2C3D/B0E1F2G3H/{{SLACK_WEBHOOK_TOKEN}}', + 'SLACK_WEBHOOK_TOKEN', + SLACK, + ], + [ + 'Alerts', + `https://discord.com/api/webhooks/123456789012345678/${DISCORD}`, + 'https://discord.com/api/webhooks/123456789012345678/{{ALERTS_WEBHOOK_TOKEN}}', + 'ALERTS_WEBHOOK_TOKEN', + DISCORD, + ], + [ + 'Teams', + `https://acme.webhook.office.com/webhookb2/${GUID}@${GUID}/IncomingWebhook/${HEX}/${GUID}/`, + `https://acme.webhook.office.com/webhookb2/${GUID}@${GUID}/IncomingWebhook/{{TEAMS_WEBHOOK_TOKEN}}/`, + 'TEAMS_WEBHOOK_TOKEN', + `${HEX}/${GUID}`, + ], + [ + 'Legacy ERP', + 'https://apiuser:Pa55w0rd-1@erp.example.com:8443/api/', + 'https://apiuser:{{LEGACY_ERP_PASSWORD}}@erp.example.com:8443/api/', + 'LEGACY_ERP_PASSWORD', + 'Pa55w0rd-1', + ], + ]; + + it.each(CASES)('%s: only the secret segment moves, into a variable the dashboard masks', (name, typed, stored, variable, value) => { + const x = new SecretExtractor(name, {}); + expect(x.extractUrl(typed, 'baseUrl')).toBe(stored); + expect(x.newEnvVars).toEqual({ [variable]: value }); + expect(isSecretName(variable)).toBe(true); + expect(JSON.stringify(x.summary())).not.toContain(value); + }); + + it.each(CASES)('%s: a call resolves the base URL exactly as typed', (name, typed) => { + const x = new SecretExtractor(name, {}); + const stored = x.extractUrl(typed, 'baseUrl'); + const { config } = interpolateConnectorConfig( + { baseUrl: stored }, + { method: 'POST', path: '/' }, + { OTHER: 'x', ...x.newEnvVars }, + ); + expect(config.baseUrl).toBe(typed); + }); + + it('leaves out the long-random-token rule when asked to (start-up repair)', () => { + const opaque = 'q8ZtW3nRk2LxV9pMbH4cJs7YdF6gTe1A'; + const x = new SecretExtractor('Acme', {}); + expect(x.extractUrl(`https://api.example.com/hooks/${opaque}`, 'baseUrl', { highEntropy: false })).toBe( + `https://api.example.com/hooks/${opaque}`, + ); + expect(x.extractUrl(`https://api.example.com/hooks/${opaque}`, 'baseUrl')).toBe( + 'https://api.example.com/hooks/{{ACME_SECRET_1}}', + ); + }); +}); diff --git a/packages/backend/src/connectors/secret-extraction.util.ts b/packages/backend/src/connectors/secret-extraction.util.ts index 07dd662ba..4620c7bf8 100644 --- a/packages/backend/src/connectors/secret-extraction.util.ts +++ b/packages/backend/src/connectors/secret-extraction.util.ts @@ -1,4 +1,5 @@ import { + type FindOptions, findSecretInHeader, findSecretInQueryParam, findSecretLiterals, @@ -224,6 +225,14 @@ export class SecretExtractor { case 'known_prefix': case 'jwt': return `${this.prefix}_${PREFIX_SUFFIX[f.label] ?? 'TOKEN'}`; + case 'webhook_token': + // `SLACK_WEBHOOK_TOKEN`, not `SLACK_WEBHOOK_WEBHOOK_TOKEN`. Either + // way the name holds WEBHOOK and TOKEN, so isSecretName masks it. + return /(^|_)WEBHOOK$/.test(this.prefix) + ? `${this.prefix}_TOKEN` + : `${this.prefix}_WEBHOOK_TOKEN`; + case 'url_password': + return `${this.prefix}_PASSWORD`; case 'high_entropy': default: return `${this.prefix}_SECRET_${++this.secretCounter}`; @@ -246,10 +255,13 @@ export class SecretExtractor { return out; } - /** A base URL, a path or a URL-valued setting. */ - extractUrl(url: string, field: string): string { + /** + * A base URL, a path or a URL-valued setting. `opts` narrows the rules + * (the start-up repair leaves out the long-random-token heuristic). + */ + extractUrl(url: string, field: string, opts?: FindOptions): string { if (typeof url !== 'string' || !url) return url; - return this.rewrite(url, findSecretLiterals(url), field); + return this.rewrite(url, findSecretLiterals(url, opts), field); } /** diff --git a/packages/backend/src/connectors/secret-guard.flows.spec.ts b/packages/backend/src/connectors/secret-guard.flows.spec.ts index fdc4c6302..88c075518 100644 --- a/packages/backend/src/connectors/secret-guard.flows.spec.ts +++ b/packages/backend/src/connectors/secret-guard.flows.spec.ts @@ -629,3 +629,90 @@ describe('logs', () => { expect(stripQuery('https://api.example.com/v1/users/123')).toBe('https://api.example.com/v1/users/123'); }); }); + +describe('a webhook URL as base URL (Bitrix24)', () => { + const CODE = ['k2bx', 'a8g5', 'n4gh', 'vqgo'].join(''); + const URL_TYPED = `https://portal.bitrix24.com/rest/1/${CODE}/`; + const URL_STORED = 'https://portal.bitrix24.com/rest/1/{{BITRIX_CRM_WEBHOOK_TOKEN}}/'; + + it('create: stores {{VAR}} and an encrypted variable, and answers without the code', async () => { + const { controller, connectorsService } = buildConnectors(); + const out: any = await controller.create(req(), { name: 'Bitrix CRM', type: 'REST' as any, baseUrl: URL_TYPED }); + const data = connectorsService.create.mock.calls[0][2]; + expect(data.baseUrl).toBe(URL_STORED); + expect(data.envVars).toEqual({ BITRIX_CRM_WEBHOOK_TOKEN: CODE }); + expect(out.baseUrl).toBe(URL_STORED); + expect(out.maskedEnvVars).toEqual(['BITRIX_CRM_WEBHOOK_TOKEN']); + expect(out.secretsNotice).toMatch(/BITRIX_CRM_WEBHOOK_TOKEN/); + expect(JSON.stringify(out)).not.toContain(CODE); + }); + + it('update: a retyped webhook URL moves the same way', async () => { + const { controller, connectorsService } = buildConnectors({ name: 'Bitrix CRM', baseUrl: URL_STORED, envVars: {} }); + await controller.update(req(), 'c1', { baseUrl: URL_TYPED } as any); + const data = connectorsService.update.mock.calls[0][1]; + expect(data.baseUrl).toBe(URL_STORED); + expect(data.envVars).toEqual({ BITRIX_CRM_WEBHOOK_TOKEN: CODE }); + }); + + it('update: an older connector saved with the masked URL keeps its code, now in a variable', async () => { + const { controller, connectorsService } = buildConnectors({ name: 'Bitrix CRM', baseUrl: URL_TYPED, envVars: {} }); + const out: any = await controller.update(req(), 'c1', { + name: 'Bitrix CRM', + baseUrl: 'https://portal.bitrix24.com/rest/1/********/', + } as any); + const data = connectorsService.update.mock.calls[0][1]; + expect(data.baseUrl).toBe(URL_STORED); + expect(data.envVars).toEqual({ BITRIX_CRM_WEBHOOK_TOKEN: CODE }); + expect(JSON.stringify(out)).not.toContain(CODE); + }); + + it('update: a URL edited around the mask is refused, without saving anything', async () => { + const { controller, connectorsService } = buildConnectors({ name: 'Bitrix CRM', baseUrl: URL_TYPED, envVars: {} }); + await expect( + controller.update(req(), 'c1', { baseUrl: 'https://other.bitrix24.com/rest/1/********/' } as any), + ).rejects.toThrow(/hidden secret/); + expect(connectorsService.update).not.toHaveBeenCalled(); + }); + + it('a tool call through the stored connector still goes to the URL as typed', async () => { + const tool = { + id: 't1', + connectorId: 'c1', + organizationId: 'org1', + name: 'crm_deal_list', + description: 'x', + parameters: { type: 'object', properties: {} }, + connectorType: 'REST', + connectorConfig: { + baseUrl: URL_STORED, + authType: 'NONE', + envVars: { BITRIX_CRM_WEBHOOK_TOKEN: CODE }, + }, + endpointMapping: { method: 'POST', path: 'crm.deal.list.json' }, + } as RegisteredTool; + const execute = jest.fn().mockResolvedValue({ result: [] }); + const executor = new DynamicMcpTools( + { getTool: () => tool, getToolForOrg: () => tool } as any, + { logInvocation: jest.fn().mockResolvedValue(undefined) } as any, + { get: jest.fn().mockResolvedValue(null), set: jest.fn() } as any, + { checkLicenseActive: jest.fn() } as any, + { isCloud: () => false } as any, + {} as any, + { execute } as any, + {} as any, + {} as any, + {} as any, + {} as any, + { scheduleObservationalIngest: jest.fn() } as any, + { recentEvents: jest.fn().mockResolvedValue([]) } as any, + ); + + const res = await executor.executeTool('crm_deal_list', {}); + + expect(res.isError).toBeFalsy(); + expect(execute).toHaveBeenCalledTimes(1); + expect(execute.mock.calls[0][0].baseUrl).toBe(URL_TYPED); + expect(execute.mock.calls[0][1].path).toBe('crm.deal.list.json'); + }); +});