diff --git a/CHANGELOG.md b/CHANGELOG.md index bf985dc27a..9d81b86479 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Features +### The router counts its own work for an app + +A scheduled task's occurrence (`scheduling`), a campaign's dial (`phone_sms`) and a simulation run (`evals`) are counted in the activity record under their app, once each, refused with why when the occurrence failed or was missed, the dial placed no call or the run errored. A signed MCP connection event counts under `hooks_in` and a 10DLC campaign report under `phone_sms` (#2071). + ### The router posts a daily digest of Agents activity Each morning, after 00:05 UTC, the router posts yesterday's (UTC) Agents activity to the money notices' Slack channel (`ROUTER_LIMITS_SLACK_CHANNEL`). The digest lists the organizations whose first request was yesterday, with how far each got. It lists the organizations whose every request was refused, with the refusal code. It also gives the number of active organizations, with the day before for comparison. One router posts the digest once per day, whatever the number of routers and restarts. A new table, `activity_digests`, holds the per-day marker (an additive migration). The ladder of `furthest_step` on `GET /v1/ops/orgs/{org_id}/activity` now has `workers`, `voice_video` and `phone_sms` (#2075). @@ -29,6 +33,7 @@ While a reply's OpenUI streams, `TurnUI.content` now holds only its finished sta ### The JavaScript SDK compiles `ui.format: a2ui` `loadFolder` accepts `format: a2ui` in agent.yaml's `ui:` and writes the same A2UI prompt as the Go SDK: the agent is told to answer with A2UI 1.0 in a ```` ```ui:a2ui@1.0 ```` block, with the catalog `getstream.io:app-catalog` that stream-chat-swift's renderer draws. `keep` names the basic catalog's components and functions, `patch_mode` is refused with `a2ui`, and a blueprint `example` is checked so it renders as written. Both SDKs are held to the same fixture, byte for byte. + ### A conversation carried on keeps its tool timeout and declared tools, and resume can change them A text conversation carried on after it ended in another router process, by a message, a watch or a resume, used to lose the caller's `tool_timeout_ms` and the tools it declared: it waited 30 seconds and offered the model none of them. The session row now keeps both (`tool_timeout_ms`, and `caller_tools`, the declarations without results; an additive migration), so it carries on as it ran. `POST /v1/agents/sessions/{id}/resume` takes `tool_timeout_ms` and `tools` to write over them, as a fork does: the caller's tools win, and an empty list offers none. A conversation still running takes the timeout for the tool calls that start after it and keeps its tools. Both are kept for the kind of caller that opened the conversation: a device carrying on its backend's conversation neither gets nor changes the backend's tools and timeout. Concurrent requests carrying one conversation on now all get the session that runs, where the losers were refused with 400 "conversation is already open". JavaScript's `sessions.resume(id, { tool_timeout_ms, tools })` and Go's `Sessions.Resume(ctx, id, client.ResumeOptions{ToolTimeout})` send it when there is something to say, the agent's own functions declared; with nothing to say they read the session as before, and JavaScript declares no tools for a resume with `watch: false`. Deploy the router before releasing these SDKs. A fork of a conversation that ended still has the default 30 seconds (AI-3201). @@ -48,9 +53,11 @@ A call through a customer's own SIP trunk on a shared Stream line no longer ends ### List a modality's routes from the Go and JavaScript SDKs, and image generation is documented `client.Routes(ctx, acceleration.ModalityImage)` in Go and `api.router().routes("image")` in JavaScript list a modality's shortcuts, default first, with the models each resolves to: `image-fast` and `image-quality` for pictures. JavaScript callers no longer write `client.get("/v1/{modality}/routes", ...)` by hand. The router README documents `POST /v1/image/generations` (its options, routes, time, limits and billing) and lists `image-fast` and `image-quality` among the default groups (AI-3199). + ### A fork takes tool_timeout_ms, and the default tool timeout is documented as 30 seconds `POST /v1/agents/sessions/{id}/fork` takes `tool_timeout_ms`, as `createSession` does: how long the fork's model waits for a tool the caller or a worker runs. Left out, a fork keeps a running parent's, now also when the fork names another config, and a fork of a conversation that ended has the default. The default is documented: 30 seconds, and a tool's own `timeout_ms` replaces it for that tool, which is how a slow tool such as one that draws a picture gets longer alone. Go has `ForkOptions.ToolTimeout`; JavaScript sends `tool_timeout_ms` from `fork()` (AI-3201). + ### An agent draws pictures without a backend route of its own `image` on an agent config (`POST`, `PUT` and `PATCH /v1/agents/configs`, and `image:` in agent.yaml through a sync) names what the agent draws with, as a provider/model or a shortcut such as `image-fast`. A text agent whose conversation has a Chat channel is then offered `generate_image(prompt, aspect_ratio?)`. The tool answers at once and the turn ends: the picture is drawn in the background and posted into the conversation as the agent's own message, with the picture attached, a minute or two later. A picture that cannot be drawn posts a short message saying why (the safety filter, a spending limit, a shape it cannot draw, or a failure), never nothing. The job is kept in Postgres (`image_jobs`) and drawn by whichever router claims it, at most two at a time per router (FAL's own limit), so four pictures in one reply are all drawn. A router stopping with a drain (`shutdown.drain`) finishes the pictures it is drawing; one stopped without a drain gives them back at once, and one that crashed leaves them to the next router once their 5-minute lease runs out, each picture drawn at most twice. A picture is kept the moment it is drawn and never drawn or billed again: a failed upload or post is tried again over about two and a half hours, and an upload Stream Chat refuses for good posts that the picture could not be posted. A message already written is not posted twice. `aspect_ratio` is one of `1:1`, `4:3`, `3:4`, `16:9` and `9:16`, and an `image` target the deployment cannot draw with is refused when the config is saved. A draw is billed as `POST /v1/image/generations` is, when it is drawn, under the session's attribution, so app, user, team and session spend limits all hold it; the money it may cost is held for the whole draw, which has 240 seconds, the endpoint's own limit, now in the image router. `tool_timeout_ms` never bounds it. Each FAL picture's usage row records `queued_ms` and `drawn_ms`, the time in FAL's queue and the drawing. One reply draws at most four pictures. There is no default and no per-session override, since a session can be opened from a device: an agent draws only when its config says so. With `image` set, the built-in replaces a caller tool named `generate_image`, the workaround that drew through a backend route of its own; without it, that tool runs as before (AI-3200, AI-3201). @@ -62,6 +69,7 @@ A call through a customer's own SIP trunk on a shared Stream line no longer ends Three new operations, open to devices and to the session's owner alone: `listResponseSiblings` (`GET /v1/agents/sessions/{id}/responses/{response_id}/siblings?of=reply|question`, paged by cursor) lists a response's regenerations or the edits of its question, with each one's index and whether it is active; `getResponseBranch` (`GET .../branch`, paged) reads a hidden branch's messages as Chat showed them, and answers `active: true` with no items for the branch on the conversation; `activateResponse` (`POST .../activate`) brings a branch back, re-sending its messages with `skip_push` and `custom.resent_from`, and answers `already_active` when nothing changed. New error codes: `branch_limit_reached` (409: the conversation has used its 100 edit and regenerate points), `branch_too_long` (409: more than 100 messages to send again) and `branch_unavailable` (410: the branch's copies were deleted, as a memory truncate deletes them). Go has `Responses.Regenerate`, `Edit`, `Siblings`, `Branch` and `Activate` (`sdks/go/client/responses.go`) (AI-3198). + ### The router counts a resent Twilio alert once A Monitor alert whose `Sid` was counted in the last hour is answered 204 and adds nothing to `vendor_account_events_total`. The mark is kept in the router's Redis, so a resend to another replica is caught too. When Redis fails, or a deployment has none, the alert is counted again with a warning rather than dropped. Usage triggers are counted on every callback. (#2000) @@ -132,6 +140,7 @@ Every SDK takes the filter and the sort in place of its flat query fields, with ### A session keeps the team its usage is counted under, and is found by it `Session` gains `team`: the team its usage is counted under, from the caller's token, else the session's `team`, else its agent config's `tags: team`. It is kept with the session from when it opens, so an ended session answers it too. `querySessions` takes `filter.team` (a bare value or `{"$eq": ...}`). A caller whose token is signed for a team may only ask for that team, and asking for another is refused with a 400. Sessions kept before this have their team filled in by the migration from the `team:` usage category they were counted under. Every SDK's session query now takes the same fields as JavaScript's: `team`, a created-at window (`$gte`/`$lt`), custom labels and an order. + ### A person sends a PDF or Word document with a question `createResponse` takes `files`: one PDF or Word document (DOCX or DOC), as a data URI of up to 8 MiB or a URL of up to 20 MiB. The router reads it as text before the turn starts, and the agent answers from it on that turn and every later one: the file goes ahead of the conversation on every reply, so summarising a long chat never drops it. A conversation the same person carries on or forks reads it again; one opened by anybody else, a backend included, never does. In a conversation kept in Stream Chat the person's message shows the file. Refused before any turn starts, and so never billed: over 100 pages, or more than 120,000 characters of text together with the conversation's other files (413 `document_too_large`); a file with a password, damaged or with no text (400 `document_unreadable`); anything but a PDF or Word document (415); a file and a voice note together; a file in a thread, in a channel others write in, or in a speech-to-speech session. Under an organization's prompt injection policy the file is screened once, in pieces, and one that reads as an injection or cannot be screened is refused rather than read. A page that is only a picture, a chart and a picture in a Word file are read through the models knowledge reads them with (`ocr-figures`), billed as OCR: at most 10 such reads a file, and a file with more than 10 scanned pages is refused (413). A deployment with no OCR leaves them unread, and the model is told which pages those are (AI-3197). @@ -139,9 +148,11 @@ Every SDK takes the filter and the sort in place of its flat query fields, with ### Trial credit pays for usage but not phone numbers, and a sync that changed nothing costs nothing Credit has a `trial` kind, written once per organization by the gateway's `/v1/billing/trial`. `GET /v1/account` (and the usage service's account) gives `trial_micros`, the trial credit granted, and `funded`, true once credit other than trial was granted: a top-up or Stream's grant. Renting a phone number and submitting a 10DLC brand or campaign are refused with 402 `top_up_required` while the organization is not funded, so trial credit pays for models and usage only. Writing knowledge skips a document stored with the same text and passage count, so `getstream agents sync` with nothing changed in `knowledge/` writes nothing and is not refused for money (AI-3179). + ### Stream staff read an organization's sessions and turn latency `GET /v1/ops/orgs/{org_id}/activity` (ops key only) also answers `sessions` and `latency` for the same `since` window, over every shard. `sessions` holds how many the organization's apps opened, how many still run, how many ended for each `end_reason`, and the 20 opened last (id, app, agent, modality, opened, ended, end reason). `latency` holds one item per leg (`stt_latency`, `llm_ttft`, `tts_ttfb`, `roundtrip`): the turns counted and their p50, p95 and p99 in milliseconds, within 5% of the true ones. The fields the answer already had are unchanged (AI-2843). + ### An agent answers in the channel types its config names An agent config's `chat_channels` names Stream Chat channel types of the app's own Stream app, such as `[{"type": "callouts"}]` for a team's group channels. A message in a channel of a named type is answered there by the router's own session under that config, written as the Stream user whose id is the config's id. Who answers comes from the config alone, never from the channel's custom data, and the router's shared app answers no named type. A type is answered only when its grants pass the check an `agent` channel type is held to: members may write, and no user, guest or member may make, change or join a channel of it. An agent directory declares it in `agent.yaml` as `chat_channels`, which the Go SDK's sync sends. Left unset, an agent answers in `agent` channels alone, as before. @@ -159,6 +170,7 @@ In a channel of a named type, a connector call whose binding sets `require_appro ### A session's tools say where they came from `GET /v1/agents/sessions/{id}/tools` gives each tool `source`: `session` (declared by whoever opened the session), `worker` (hosted by a worker connected when the session opened), `connector` or `builtin`. It is kept with the session, so an ended session and each simulation conversation (by its `call_id`) answer it too; a session kept before this names none. A session's tools are fixed when it opens, so one opened while its worker was down simply has no `worker` tools, and this is how a test or a simulation shows it rather than leaving the model to say the tool is unavailable (AI-3131). + ### The operator hears when an organization's spend jumps With `ROUTER_LIMITS_SLACK_TOKEN` and `ROUTER_LIMITS_SLACK_CHANNEL` set, one router of the deployment checks every hour each organization's billable spend today (UTC) against its median day of the 7 before, read from `requests` on every shard. At 5x that median, and past a minimum spend, it posts once per organization and day to the channel the money limit notices go to, with today's spend, the median and a link to the staff page (`ROUTER_LIMITS_STAFF_URL`). An organization that spent on fewer than 4 of the 7 days is not checked. `cost_anomaly_notices` keeps each post, so another router or a restart does not post it again, and a post that failed is retried by a later check. No metric is added (AI-2835, #1899). @@ -194,6 +206,7 @@ With tracing on, each LLM model call has a `chat ` span with `gen_ai.oper ### An agent config names the models its llm falls back to `llm_fallbacks` on an agent config (`POST`, `PUT` and `PATCH /v1/agents/configs`) is an ordered list of up to five targets, each a provider/model, a capability shortcut or one of the customer's `custom/` models, that a conversation turn falls back to when `llm` fails it. `llm` stays the first choice. A response `llm` cannot start, a 5xx, a timeout, a 401, 402, 403, 404, 408 or 429 included, is asked of each next entry in turn; one `llm` says nothing in for as long as a reply may (`ROUTER_REPLY_HEDGE`, 1.2 s by default) is also asked of another entry, the first to start answering kept. A response that has started is never moved, however slowly it streams, and a run that outlasts its timeout still ends as a 504. A shortcut can never include a customer's own model, so this is how a `custom/` model is backed by one Stream serves, such as `xai/grok-4.7`. A target the deployment does not route, or one with no instance in the agent's region, is refused when the config is saved. Left out on a `PUT`, the stored list is kept; an empty list removes it. The flow controller and the subagent do not fall back along it. `POST /v1/agents/runs` now reports in `model_route` the model that answered, the fallback included, rather than `llm`. `agent.yaml` does not take it yet. + ### Agent teams: Chat's team isolation for agent configs and channels An app with the `agent_teams` setting on (set by Stream, off by default) keeps each team's agent configs to its own users, as Chat multi-tenancy does: an end user opens a config only if their Stream user shares its `team` tag (a config of no team is open only to users in no team), and is answered as for an unknown config otherwise. Backends are not limited. The agent channel and voice transcript channel a session makes are in its config's team. With the setting off nothing changes (AI-3059, #1885, #1887). @@ -205,6 +218,7 @@ An app with the `agent_teams` setting on (set by Stream, off by default) keeps e ### Go: a refusal says how long to wait `stream.RouterError.RetryAfter` is the wait the router's `Retry-After` asked for, as seconds or a date, on a 429 `user_limit_reached` or `concurrency_limited` and a 503 `spend_check_unavailable`, and zero when it sent none. The SDK still retries nothing itself (AI-1892). + ### A failed model call says what the provider answered The session socket's `model_call` frame and a call timeline's `model_calls` carry `error_message` beside `error_code` for a call that did not succeed: the start of the provider's error body, such as a self-hosted model's `502 failed to connect to local service`, redacted and capped as the requests table stores it. A successful call's frame is unchanged. The flow controller's and the agent's failure log lines name the `provider` and `model` (AI-2954). @@ -638,6 +652,7 @@ A tool function that needs the call it runs for (who asked, its id, `update`, `s ### A worker cannot host tools for an agent the app does not have `host_tools` on `/v1/dispatch` is answered `hosting_refused` with `there is no agent called ` when the app has no agent config of that name. Sessions are only opened by a config's name, so such tools were never offered to anything while the worker looked healthy. The Go, Python and JavaScript SDKs already stop the worker on a refusal, so a misspelled `api.Agent("")` now fails at start naming the agent; sync the agent before starting its worker. A router with no config store, or one that cannot read it at that moment, still takes the offer (AI-3132). + ### JavaScript: the SDK is @stream-io/stream-agents The JavaScript SDK is published as `@stream-io/stream-agents` from 0.4.0. Install it and import from `@stream-io/stream-agents`, `@stream-io/stream-agents/chat`, `/video` and `/node`. Nothing else in the API changed. @@ -645,6 +660,7 @@ The JavaScript SDK is published as `@stream-io/stream-agents` from 0.4.0. Instal ### JavaScript: an Agent opens its sessions on its synced config An `Agent` without `pipeline.config` now names its stored config by `name` when it opens a session (`join`, `chat`, `answer`, `reply`, `startCall`, `waitForCall`), so the session runs on the instructions, skills and knowledge `agent.sync()` stored. It used to name no config, so such a session ran on the router's defaults and the agent's instructions never reached it. An agent that was never synced under its name is now refused with a 404, `there is no agent called `, where it used to open a session: call `await agent.sync()` before opening one, or pass `pipeline.config` to name a stored config yourself. + ### Swift: VisionAgentsCore is StreamAgents The Swift SDK's product and module are `StreamAgents`, and its client is `StreamAgents`: `import StreamAgents`, then `StreamAgents(apiKey:)` or `StreamAgents(url:customerID:)` where it was `VisionAgents(...)`. Nothing else in the API changed. An app adding the package with SPM takes the `StreamAgents` product from the same repository URL. The package is `stream-agents`, so its test scheme is `stream-agents`. The empty `sdks/swift/ui` and `sdks/swift/rtc` directories are gone. @@ -660,6 +676,7 @@ The Dart packages are `stream_agents_core`, `stream_agents_chat` and `stream_age ### The Go SDK's helpers take a Backend, not the generated client `agents.DefineAgent`, `agents.DefineSkills`, `agents.IngestKnowledge`, `agents.SubscribeKnowledgeURLs` and `stream.NewPhone` take a `stream.Backend` where they took the generated `*acceleration.ClientWithResponses`, so using them no longer means building the generated client first. `stream.NewPhone` now returns an error too, for a backend that does not resolve. `agents.Harness.VM` is `agents.Harness.Sandbox`, the name every other SDK and the docs use. Pass `stream.Backend{}` for the environment's router, or `api.Backend()` for a client's. `stream.Backend.Client()` and `agents.Agent.Client()` still hand out the generated client, for operations the SDK has no method for yet; its operation methods carry no compatibility promise (AI-1892). + ### Rust: the string enums are no longer `Copy`, and a `match` on one needs an `Unlisted` arm Each of the Rust SDK's 133 string enums has a new `Unlisted(String)` variant that holds a value the SDK does not list, so none of them is `Copy` any more. Clone one where you copied it, and add an `Unlisted(_)` arm to a `match` on one that has no wildcard. The Python client has no such break (AI-2951). @@ -737,6 +754,7 @@ Every session that is not incognito keeps its conversation in Stream Chat, and ` ### A tool call's approval is no longer changed while its reply is being written to Chat Answering a held tool call's approval changed it in place on the reply, which the router could at the same moment be writing to the Chat channel or sending to the conversation's socket. The queued write and the update now each keep their own copy of the approval, so neither is encoded while it changes (AI-3198). + ### JavaScript: an approval shown from a conversation's history can be answered `session.decide` answered only a call its own socket was asked, and did nothing for one an approval card showed from the channel, after `sessions.resume` or a restart, so the card waited forever. `AgentChat.decide(toolCallId, { allowed, summary })` answers such a call from what the reply shows: its `ai_tool_call` step, and the request and turn in `support_runtime`. Allowed, a call of one of the session's tools runs on this device, once however often it is answered; declined, the model is told so. A call whose arguments were too long for the reply to keep is failed rather than left waiting, and `AgentChat.decide` rejects when no loaded message shows the call waiting, so the card can offer to try again. `session.decide` takes the same as an optional `call`, and without it still leaves a call it wasn't asked alone. @@ -748,6 +766,7 @@ The demo join link was logged with its API key and a one-hour user token in the ### A call placed for a session that ends while it rings is hung up Ending a session while the outbound call it placed still rings now hangs that call up at Twilio and Telnyx, and the call's outcome is `canceled`. A session ends when it is closed or deleted, when its start timeout passes, or when the router shuts down. At Twilio the call is canceled, or completed if it was just answered. Before, the phone kept ringing, and a person who picked up joined a call with no agent and was billed until they hung up. A call already answered is left as it is. So is a call the session hands to the agent's next session in the same call. Other vendors cannot hang up a placed call yet: the router logs that, and the call rings until the vendor gives up (#1987). + ### Call minutes are priced for Twilio in the US and Canada, and for Telnyx calls to Canada Twilio is priced at $0.018 a minute outbound and $0.0125 inbound in both countries: its local rate plus the SIP leg to the Stream trunk. A Telnyx call placed to Canada is priced at $0.022 a minute. A database gets these with the next release's start, which adds the items it has never priced. Telnyx calls arriving on a Canadian number, other countries and toll-free numbers are not priced yet (#1965). @@ -767,6 +786,7 @@ When the person a router-placed call rang answers, the router waits `phone.agent ### A custom connector cannot list oauth2_client_credentials `POST /v1/agents/connectors` answers 400 when `schemes` of a custom connector contains `oauth2_client_credentials`. That scheme reads its token URL from `endpoints.token`, which a custom connector has no way to set, so such a definition could be saved and never connected. Built-in connectors (Salesforce) are unchanged; a custom connector already stored keeps what it has (AI-3155). + ### A scheduled task refused for money is not retried, wherever it is refused A scheduled task whose session cannot open or be carried on, whose call the gate refuses, whose text is refused at its reservation, or whose turn the model refuses, in the session of a text channel, of the thread it was made in, or of the task, because the organization may not pay, is tried once and its run records the refusal's code, such as `insufficient_credits`. Before, only a text the gate refused was; the rest were retried and recorded `delivery_failed` or `reply_failed`. A refusal because the money could not be checked (`spend_check_unavailable`) is still tried again (AI-2941, AI-2942). @@ -774,6 +794,7 @@ A scheduled task whose session cannot open or be carried on, whose call the gate ### JavaScript: reliable configuration, shutdown and transport failures Agents respect an empty skill override and verify the remote sync fingerprint. Attaching a watcher no longer stops an existing conversation on failure. Closing a session after losing its socket stops it over HTTP, reports stop failures and ends unwatched sessions locally. Dispatch refuses ambiguous hosted tool names, cancels pending socket upgrades and settles pending tool holds on shutdown. Stream connections refuse tokens for a different signed-in user, and concurrent guest refreshes share one request. Safe requests retry truncated response bodies; text responses and event streams retain their formats. A `responses.create` retried after its answer was lost, which the router answers with no id, looks up the turn the first send started, so the response it returns names that turn. Socket receive queues are bounded. Folder loading propagates filesystem failures, preserves quoted hashes and rejects malformed durations. An ended voice session must be started or attached again before joining. + ### A search provider that fails answers 500, not 400 `POST /v1/search` answers a question its provider could not answer, or a session it could not open with a provider, with a 500 `internal_error` that says only `something went wrong`, instead of a 400 `invalid_request` carrying the provider's own error text. The router API 5xx monitor now counts a search outage. A request the router refuses (an unknown target, a term nothing serves), a model the customer's policies do not allow, and a refusal about money keep their 400, 402, 403 and 429 answers. The provider's error is in the access log under the request id. A search whose caller hung up before it was answered is a 400 that says so, not a 500. A request asking for a term the named or listed models do not take, a data policy no candidate meets, an input kind no candidate accepts, or more tags than a row carries once the policies' tags are added is marked as the caller's to change: on a stream socket it now ends with 1008 instead of 1011 (AI-3080). @@ -793,6 +814,7 @@ When `stream.Pipeline.Leave` (and so `Session.Close`) stops a session over HTTP ### Go: a client made on another client's Backend reads each answer once `client.New(api.Backend())`, which `agents` and dispatch workers do with the backend they are given, wrapped the first client's money watcher in a second one. A balance warning the first client handed to `WithBalanceWarning` was then logged as well. A client given no callback of its own (no options, or only `WithLogger`) on a backend another client already reads now shares that client's watcher, so an agent or worker built on `api.Backend()` reports what its requests cost to `api`'s callbacks, once (AI-1892). + ### After a voice failover the kept reply no longer waits for the lost voice to close A reply or check-in kept for the voice that replaces a lost one goes to that voice as soon as it is open. Before, it waited for the lost voice to close first, up to 2 s for ElevenLabs. The agent still waits for that close when it stops (AI-2974). @@ -828,6 +850,7 @@ The requests table, agent logs and socket error frames now redact a key sent as ### A session whose every reply was lost no longer counts as closed when the flow controller answered A flow controller or subagent request no longer stands in for a lost reply, so the session ends by whose fault the reply failed (AI-2939, #1811). + ### Photos an agent cannot see are refused before they are uploaded A question with photos and a `request_id`, on `createResponse` or the session socket's `respond`, sent to an agent with neither a vision skill nor a conversation model that accepts images, was refused only after its photos were uploaded and the person's message written, leaving a message in the channel that nothing would answer. It is now refused first, with the same 400 `not_configured`, and nothing is uploaded or written. A retry of a photo question already taken is still answered with its receipt, also after the session's model changed to one that cannot see, since it starts no turn. diff --git a/acceleration/README.md b/acceleration/README.md index e87c54722d..5a5cd15556 100644 --- a/acceleration/README.md +++ b/acceleration/README.md @@ -2156,7 +2156,11 @@ characters, time and cost), and `first_seen` and `last_seen` the first and last area. A hook counts under the app its signature or token was checked for, from client `api`: `phone_sms` for a phone or SMS vendor's, else `hooks_in`. A forward to an event destination counts once under `webhooks_out` from `api`, refused when given up. A socket refused in a frame -after its 101 counts as refused, with the code. +after its 101 counts as refused, with the code. Work the router starts itself also counts from +`api`, once each: a scheduled task's occurrence or run by hand under `scheduling` (refused when +missed or failed), a campaign dial under `phone_sms`, and a simulation run under `evals` +(refused when it errored). A signed MCP event counts under `hooks_in`, and a signed 10DLC report +on an app's campaign under `phone_sms`. `furthest_step` is the furthest of `dashboard_reads`, `agent_setup`, `workers`, `conversations`, `voice_video` and `phone_sms` (`activity.Steps`), in that order, with a request answered rather than refused, or `steady_use` once conversations or model calls diff --git a/acceleration/cmd/router/main.go b/acceleration/cmd/router/main.go index 65c63c3915..c035297cc0 100644 --- a/acceleration/cmd/router/main.go +++ b/acceleration/cmd/router/main.go @@ -1238,9 +1238,10 @@ func run(settings config.Config, logger *slog.Logger) error { LLM: streams.LLM, // Speech is what an audio simulation needs and a text one does not, so it is // passed where it exists rather than required. - TTS: streams.TTS, - STT: streams.STT, - Logger: logger, + TTS: streams.TTS, + STT: streams.STT, + Activity: activityRecord, + Logger: logger, }) if err != nil { return err @@ -1624,6 +1625,7 @@ func run(settings config.Config, logger *slog.Logger) error { var scheduler *scheduling.Scheduler if pgStore != nil && sessions != nil { scheduler = scheduling.NewScheduler(pgStore, server.DeliverScheduledTask, logger) + scheduler.SetActivity(activityRecord) // Before Start: a session the first delivery opens takes the kick then. server.SetSchedulerKick(scheduler.Kick) scheduler.Start(ctx) @@ -1638,6 +1640,7 @@ func run(settings config.Config, logger *slog.Logger) error { Dialer: campaign.NewPhoneDialer(telephony, sessions, pgStore, logger), Transcripts: server, Classifier: campaign.NewClassifier(streams.LLM), + Activity: activityRecord, Logger: logger, }) if err != nil { diff --git a/acceleration/internal/activity/activity.go b/acceleration/internal/activity/activity.go index bafc29fd87..205767dc00 100644 --- a/acceleration/internal/activity/activity.go +++ b/acceleration/internal/activity/activity.go @@ -1,10 +1,11 @@ // Package activity counts the requests each app makes, by area of the API and by client, and // writes the sums to customer_activity (store.RecordActivity). // -// Every authenticated request is one, and so is a hook checked as an app's (NoteCustomer) and a -// forward to an app's own endpoint, so the counting is in memory and the writing is not on -// the request path: a minute's requests for one app, area, client and day are one row's -// update. Nothing here fails a request: counting it is not what the caller asked for. +// Every authenticated request is one, and so is a hook checked as an app's (NoteCustomer), and +// the work the router does for an app on its own: a forward to its endpoint, a scheduled task's +// occurrence, a campaign's dial and a simulation run. So the counting is in memory and the +// writing is not on the request path: a minute's requests for one app, area, client and day are +// one row's update. Nothing here fails a request: counting it is not what the caller asked for. package activity import ( @@ -39,9 +40,9 @@ const ( AreaOther = "other" ) -// ClientRouter is the client work the router starts itself is counted under: a forward to a -// customer's own endpoint. It is the API's, the client a caller that named none is counted -// under, so no row says a client sent what nobody sent. +// ClientRouter is the client work the router starts itself is counted under, such as a forward to +// a customer's own endpoint or a scheduled task's occurrence. It is the API's, the client a +// caller that named none is counted under, so no row says a client sent what nobody sent. const ClientRouter = store.AuditSourceAPI // Note is who a request no credential named was for: a hook a vendor or Stream signed carries @@ -189,9 +190,10 @@ func (r *Recorder) Close() { }) } -// Record counts one request. A request naming no app is nobody's and is not counted. +// Record counts one request. A request naming no app is nobody's and is not counted, and a nil +// Recorder, a router with no database, counts nothing. func (r *Recorder) Record(request Request) { - if request.AppID == "" { + if r == nil || request.AppID == "" { return } at := request.At.UTC() diff --git a/acceleration/internal/api/dlc_test.go b/acceleration/internal/api/dlc_test.go index 0cf61a8139..616b99982b 100644 --- a/acceleration/internal/api/dlc_test.go +++ b/acceleration/internal/api/dlc_test.go @@ -37,6 +37,10 @@ type campaignRegistry struct { assigned map[string]string } +// campaignRegistryFails is a campaign status no vendor sends: the fake answers a read of a +// campaign set to it with a 500. +const campaignRegistryFails = "REGISTRY_FAILS" + func (r *campaignRegistry) ServeHTTP(w http.ResponseWriter, request *http.Request) { r.mu.Lock() defer r.mu.Unlock() @@ -55,6 +59,10 @@ func (r *campaignRegistry) ServeHTTP(w http.ResponseWriter, request *http.Reques fmt.Fprintf(w, `{"campaignId":%q,"campaignStatus":"TCR_PENDING"}`, id) case request.Method == http.MethodGet && strings.HasPrefix(path, "/v2/10dlc/campaign/"): id := strings.TrimPrefix(path, "/v2/10dlc/campaign/") + if r.campaigns[id] == campaignRegistryFails { + w.WriteHeader(http.StatusInternalServerError) + return + } fmt.Fprintf(w, `{"campaignId":%q,"campaignStatus":%q}`, id, r.campaigns[id]) case request.Method == http.MethodPost && path == "/v2/10dlc/phone_number_campaigns": r.assigned[body["phoneNumber"]] = body["campaignId"] diff --git a/acceleration/internal/api/hook_activity_test.go b/acceleration/internal/api/hook_activity_test.go index 5c7dbf8615..d527cc6f2d 100644 --- a/acceleration/internal/api/hook_activity_test.go +++ b/acceleration/internal/api/hook_activity_test.go @@ -3,9 +3,17 @@ package api import ( + "bytes" "context" + "crypto/ed25519" + "crypto/hmac" + "crypto/rand" + "crypto/sha256" + "encoding/base64" "net/http" "net/url" + "strconv" + "strings" "time" "github.com/GetStream/Vision-Agents/acceleration/internal/activity" @@ -256,3 +264,128 @@ func (s *TelnyxHookSuite) TestACallWhoseTrunkCannotBeRenderedIsCountedRefused() s.Equal(int64(1), row.refused) s.Equal(internalError().Code, row.code) } + +// activityWritten waits until what the suite's activity record holds now is written: a row +// recorded after it is written in the same write, one transaction, or a later one. +func (s *RouterSuite) activityWritten() { + marker := "written-" + rand.Text() + s.activity.Record(activity.Request{AppID: s.customerID(), Area: marker, Client: store.AuditSourceAPI, At: time.Now()}) + s.Require().Eventually(func() bool { return s.counted(marker, store.AuditSourceAPI).requests == 1 }, + settleFor, 10*time.Millisecond, "the activity record was not written") +} + +// hooksAfter delivers, and returns the app's hooks_in row before and after it. +func (s *RouterSuite) hooksAfter(deliver func()) (before, after counted) { + s.activityWritten() + before = s.counted(activity.AreaHooksIn, store.AuditSourceAPI) + deliver() + s.activityWritten() + return before, s.counted(activity.AreaHooksIn, store.AuditSourceAPI) +} + +// Defect: an event the app's MCP server signed with the subscription's secret left out of the +// app's hooks, or one signed with another subscription's secret counted under the app. +func (s *ConnectionEventsSuite) TestASignedMCPEventIsCountedAsTheAppsHookAndAForgedOneIsNot() { + connection := s.connection() + binding := s.binding(connection, issueCreated) + binding["events"] = append(binding["events"].([]map[string]any), map[string]any{"event": "issue.closed"}) + s.subscribed(binding) + subs := s.atTheFake(connection) + s.Require().Len(subs, 2) + + before, after := s.hooksAfter(func() { + s.Require().Equal(http.StatusUnauthorized, s.provider.DeliverEvent(subs[0].URL, subs[1].Secret, subs[0].ID, + "evt_"+s.utils.uuid(), subs[0].Name, map[string]any{})) + s.Require().Equal(http.StatusAccepted, s.provider.DeliverEvent(subs[0].URL, subs[0].Secret, subs[0].ID, + "evt_"+s.utils.uuid(), subs[0].Name, map[string]any{})) + }) + + s.Equal(before.requests+1, after.requests) + s.Equal(before.refused, after.refused) +} + +// Defect: a signed delivery the router then turned down left out of the app's hooks, so the +// digest shows none of the app's failing MCP events. +func (s *ConnectionEventsSuite) TestASignedMCPEventTheRouterTurnsDownIsCountedRefused() { + connection := s.subscribed(s.binding(s.connection(), issueCreated)) + sub := s.atTheFake(connection)[0] + + before, after := s.hooksAfter(func() { + s.Require().Equal(http.StatusBadRequest, s.provider.DeliverEvent(sub.URL, sub.Secret, sub.ID, + "evt_"+s.utils.uuid(), "issue.closed", map[string]any{})) + }) + + s.Equal(before.requests+1, after.requests) + s.Equal(before.refused+1, after.refused) +} + +// Defect: a report the registrar signed about an app's campaign left out of the app's phone +// work, or one signed with another key counted. +func (s *DLCSuite) TestASignedReportOnACampaignIsCountedAsItsAppsPhoneWork() { + _, campaign := s.pendingWithTheVendor() + _, stranger, err := ed25519.GenerateKey(rand.Reader) + s.Require().NoError(err) + s.activityWritten() + before := s.counted(activity.AreaPhoneSMS, store.AuditSourceAPI) + + s.Require().Equal(http.StatusUnauthorized, s.report(`{"campaignId":"`+campaign+`"}`, stranger)) + s.Require().Equal(http.StatusNoContent, s.report(`{"campaignId":"`+campaign+`"}`, s.signing)) + s.activityWritten() + + after := s.counted(activity.AreaPhoneSMS, store.AuditSourceAPI) + s.Equal(before.requests+1, after.requests) + s.Equal(before.refused, after.refused) + s.Equal(int64(0), s.counted(activity.AreaHooksIn, store.AuditSourceAPI).requests, "a report is phone work") +} + +// Defect: a signed delivery that is not JSON, answered 400, left out of the app's hooks. +func (s *ConnectionEventsSuite) TestASignedDeliveryThatIsNotJSONIsCountedRefused() { + connection := s.subscribed(s.binding(s.connection(), issueCreated)) + sub := s.atTheFake(connection)[0] + + before, after := s.hooksAfter(func() { + s.Require().Equal(http.StatusBadRequest, s.signedRaw(sub.URL, sub.Secret, sub.ID, []byte("not json"))) + }) + + s.Equal(before.requests+1, after.requests) + s.Equal(before.refused+1, after.refused) +} + +// signedRaw posts body signed with secret, as the MCP server signs a delivery, and returns the +// status the router answered. +func (s *ConnectionEventsSuite) signedRaw(url, secret, subscriptionID string, body []byte) int { + key, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(secret, "whsec_")) + s.Require().NoError(err) + id := "msg_" + s.utils.uuid() + stamp := strconv.FormatInt(time.Now().Unix(), 10) + mac := hmac.New(sha256.New, key) + mac.Write([]byte(id + "." + stamp + ".")) + mac.Write(body) + request, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body)) + s.Require().NoError(err) + request.Header.Set("Content-Type", "application/json") + request.Header.Set("webhook-id", id) + request.Header.Set("webhook-timestamp", stamp) + request.Header.Set("webhook-signature", "v1,"+base64.StdEncoding.EncodeToString(mac.Sum(nil))) + request.Header.Set("X-MCP-Subscription-Id", subscriptionID) + response, err := http.DefaultClient.Do(request) + s.Require().NoError(err) + _ = response.Body.Close() + return response.StatusCode +} + +// Defect: a signed report on an app's campaign the router failed to act on (5xx) left out of +// the app's phone work, so the digest shows none of its failing 10DLC reports. +func (s *DLCSuite) TestASignedReportTheRouterFailsOnIsCountedRefused() { + _, campaign := s.pendingWithTheVendor() + s.registry.set(campaign, campaignRegistryFails) + s.activityWritten() + before := s.counted(activity.AreaPhoneSMS, store.AuditSourceAPI) + + s.Require().GreaterOrEqual(s.report(`{"campaignId":"`+campaign+`"}`, s.signing), http.StatusInternalServerError) + s.activityWritten() + + after := s.counted(activity.AreaPhoneSMS, store.AuditSourceAPI) + s.Equal(before.requests+1, after.requests) + s.Equal(before.refused+1, after.refused) +} diff --git a/acceleration/internal/api/scheduled_tasks.go b/acceleration/internal/api/scheduled_tasks.go index e18fb2fbb6..7972e9ef56 100644 --- a/acceleration/internal/api/scheduled_tasks.go +++ b/acceleration/internal/api/scheduled_tasks.go @@ -1,6 +1,7 @@ package api import ( + "cmp" "context" "errors" "net/http" @@ -10,6 +11,7 @@ import ( "github.com/danielgtaylor/huma/v2" + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" "github.com/GetStream/Vision-Agents/acceleration/internal/auth" "github.com/GetStream/Vision-Agents/acceleration/internal/routing" "github.com/GetStream/Vision-Agents/acceleration/internal/scheduling" @@ -622,6 +624,14 @@ func (s *Server) deliverManualRun(task store.ScheduledTask, run store.ScheduledT s.logger.Error("could not record a scheduled task run by hand", "task", task.ID, "run", run.ID, "error", err) } + // Counted here whether or not its run was written: no worker takes a run by hand up again. + request := activity.Request{ + AppID: task.CustomerID, Area: activity.AreaScheduling, Client: activity.ClientRouter, At: time.Now(), + } + if state != store.ScheduledTaskRunDelivered { + request.Refused, request.Code = true, cmp.Or(failure, state) + } + s.activity.Record(request) } func (s *Server) deleteScheduledTask(ctx context.Context, request *deleteScheduledTaskRequest) (*struct{}, error) { diff --git a/acceleration/internal/api/scheduled_tasks_test.go b/acceleration/internal/api/scheduled_tasks_test.go index aabefc58cb..e0ae3a6037 100644 --- a/acceleration/internal/api/scheduled_tasks_test.go +++ b/acceleration/internal/api/scheduled_tasks_test.go @@ -20,6 +20,7 @@ import ( "github.com/gorilla/websocket" "github.com/uptrace/bun" + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" "github.com/GetStream/Vision-Agents/acceleration/internal/auth" "github.com/GetStream/Vision-Agents/acceleration/internal/channels" "github.com/GetStream/Vision-Agents/acceleration/internal/conversation" @@ -3467,6 +3468,44 @@ func (s *ScheduledTasksSuite) TestARunByHandThatFailsListsTheReasonItFailedUnder s.Equal(scheduling.ReasonNoNumber, *page.Items[0].Error) } +// Defect: a run by hand that landed counted refused, or not at all past the request that set +// it going. +func (s *ScheduledTasksSuite) TestARunByHandThatLandsIsCountedDelivered() { + config := s.agentWithScheduling(nil) + created := s.createTask(s.data.createUser(), taskRequest(config)) + s.activityWritten() + before := s.counted(activity.AreaScheduling, activity.ClientRouter) + + s.Require().Equal(http.StatusAccepted, s.serverClient.do(http.MethodPost, "/v1/agents/scheduled-tasks/"+created.Id+"/run", nil, nil)) + + // The request that set it going, and the run. + s.Require().Eventually(func() bool { + return s.counted(activity.AreaScheduling, activity.ClientRouter).requests == before.requests+2 + }, settleFor, 10*time.Millisecond, "the run by hand was not counted") + s.Equal(before.refused, s.counted(activity.AreaScheduling, activity.ClientRouter).refused) +} + +// Defect: a run by hand that failed counted only as the request that set it going, so the +// app's scheduling work shows no refusal, or one without the reason its run failed under. +func (s *ScheduledTasksSuite) TestARunByHandThatFailsIsCountedRefusedUnderItsReason() { + config := s.agentWithScheduling(nil) + user := s.data.createUser() + opened := s.openScheduling(user, config, nil) + created := s.taskOn(user, config, opened.Id, []string{"phone"}) + s.activityWritten() + before := s.counted(activity.AreaScheduling, activity.ClientRouter) + + s.Require().Equal(http.StatusAccepted, s.serverClient.do(http.MethodPost, "/v1/agents/scheduled-tasks/"+created.Id+"/run", nil, nil)) + + s.Require().Eventually(func() bool { + return s.counted(activity.AreaScheduling, activity.ClientRouter).refused == before.refused+1 + }, settleFor, 10*time.Millisecond, "the failed run by hand was not counted refused") + after := s.counted(activity.AreaScheduling, activity.ClientRouter) + // The request that set it going, and the run. + s.Equal(before.requests+2, after.requests) + s.Equal(scheduling.ReasonNoNumber, after.code) +} + func (s *ScheduledTasksSuite) TestATaskForAnAgentWithoutANumberHasNoLineToRingFrom() { s.retryAtOnce() config := s.agentWithScheduling(nil) diff --git a/acceleration/internal/campaign/campaign.go b/acceleration/internal/campaign/campaign.go index d02af5c9c0..39c529cfba 100644 --- a/acceleration/internal/campaign/campaign.go +++ b/acceleration/internal/campaign/campaign.go @@ -23,6 +23,7 @@ import ( "github.com/google/uuid" + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" "github.com/GetStream/Vision-Agents/acceleration/internal/chatlog" "github.com/GetStream/Vision-Agents/acceleration/internal/dlc" "github.com/GetStream/Vision-Agents/acceleration/internal/phone" @@ -115,8 +116,10 @@ type Options struct { // Now is the clock. Nil is time.Now. Now func() time.Time // Tick is how often to look for work. Zero is 2 seconds. - Tick time.Duration - Logger *slog.Logger + Tick time.Duration + // Activity counts each dial under its campaign's app, as area phone_sms. Nil counts none. + Activity *activity.Recorder + Logger *slog.Logger } // Runner places the calls of every running campaign that this router claims. @@ -128,6 +131,7 @@ type Runner struct { owner string now func() time.Time every time.Duration + activity *activity.Recorder logger *slog.Logger // calls is what calls are placed under. Close cancels it only once the calls in flight @@ -179,6 +183,7 @@ func New(options Options) (*Runner, error) { owner: options.Owner, now: options.Now, every: cmp.Or(options.Tick, defaultTick), + activity: options.Activity, logger: cmp.Or(options.Logger, slog.Default()), calls: calls, cancelCalls: cancelCalls, @@ -448,6 +453,8 @@ func (r *Runner) ring(ctx context.Context, campaign store.Campaign, contact stor } } + r.countDial(campaign, dialed, err) + written, cancel := context.WithTimeout(context.WithoutCancel(ctx), writeTimeout) defer cancel() ended := r.now() @@ -462,6 +469,29 @@ func (r *Runner) ring(ctx context.Context, campaign store.Campaign, contact stor return dialed, nil } +// codeDialFailed is the code a dial that could not be tried, and gave no reason, is counted +// refused under. +const codeDialFailed = "dial_failed" + +// countDial counts a dial in the activity record, refused when it could not be tried or placed +// no call, with its reason. A dial a shutdown cut off rang nobody and leaves the contact to be +// rung again, so it is counted when that dial is. +func (r *Runner) countDial(campaign store.Campaign, dialed Dialed, err error) { + if dialed.Result == NotPlaced { + return + } + request := activity.Request{ + AppID: campaign.CustomerID, + Area: activity.AreaPhoneSMS, + Client: activity.ClientRouter, + At: r.now(), + } + if err != nil || dialed.Result != "" && dialed.Result != Voicemail { + request.Refused, request.Code = true, cmp.Or(dialed.Reason, string(dialed.Result), codeDialFailed) + } + r.activity.Record(request) +} + // takeOver settles the contacts whose router died mid-call, from the call it placed. None is // rung again. func (r *Runner) takeOver(ctx context.Context) { diff --git a/acceleration/internal/campaign/runner_activity_test.go b/acceleration/internal/campaign/runner_activity_test.go new file mode 100644 index 0000000000..1477f9657d --- /dev/null +++ b/acceleration/internal/campaign/runner_activity_test.go @@ -0,0 +1,138 @@ +//go:build integration + +package campaign + +import ( + "context" + "database/sql" + "errors" + "log/slog" + "time" + + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" +) + +// dialsCounted is the campaign's app's phone_sms row in the activity record. +type dialsCounted struct { + requests, refused int64 + code string +} + +// failingDialer could not try any dial. +type failingDialer struct{} + +func (failingDialer) Dial(context.Context, Dial) (Dialed, error) { + return Dialed{}, errors.New("the vendor did not answer") +} + +// counting is a runner over dialer, as newRunner makes, that counts in an activity record of its +// own, which writes only when the test calls the write it returns. +func (s *RunnerSuite) counting(dialer Dialer) (runner *Runner, write func()) { + record, err := activity.New(activity.Options{Store: s.store, Interval: time.Hour, Logger: slog.New(slog.DiscardHandler)}) + s.Require().NoError(err) + runner, err = New(Options{ + Store: s.store, Dialer: dialer, Transcripts: s.transcripts, Classifier: s.classifier, + Owner: "router-counting", Now: s.clock.now, Activity: record, Logger: slog.New(slog.DiscardHandler), + }) + s.Require().NoError(err) + s.T().Cleanup(runner.Close) + return runner, record.Close +} + +// counted is what the activity record holds for the suite's app under phone_sms and client api. +func (s *RunnerSuite) counted() dialsCounted { + var ( + row dialsCounted + code sql.NullString + ) + err := s.store.DB().QueryRowContext(s.ctx, ` +SELECT COALESCE(sum(requests), 0), COALESCE(sum(refused), 0), max(last_refusal_code) +FROM customer_activity WHERE app_id = ? AND area = ? AND client = ?`, + s.customerID, activity.AreaPhoneSMS, activity.ClientRouter).Scan(&row.requests, &row.refused, &code) + s.Require().NoError(err) + row.code = code.String + return row +} + +// Defect: a call nobody answered counted refused, or the second dial of a double dial, which +// rings the person again, left uncounted. +func (s *RunnerSuite) TestEachDialOfADoubleDialIsCountedAndARingNobodyAnsweredIsNotRefused() { + runner, write := s.counting(s.dialer) + s.launched(DefaultSequence("config-1"), 1, "UTC", "+15550000001") + + s.tick(runner) + write() + + s.Require().Len(s.dialer.dialed(), 2) + s.Equal(dialsCounted{requests: 2}, s.counted()) +} + +// Defect: a call an answering machine took, which the dial placed, counted refused. +func (s *RunnerSuite) TestADialAMachineAnsweredIsNotCountedRefused() { + s.dialer.results["+15550000001"] = []Dialed{{Result: Voicemail}} + runner, write := s.counting(s.dialer) + s.launched(s.single(), 1, "UTC", "+15550000001") + + s.tick(runner) + write() + + s.Equal(dialsCounted{requests: 1}, s.counted()) +} + +// Defect: a dial refused for money, which placed no call, counted as placed, or refused without +// the reason the campaign paused under. +func (s *RunnerSuite) TestADialRefusedForMoneyIsCountedRefusedUnderItsReason() { + s.dialer.results["+15550000001"] = []Dialed{{Result: Paused, Reason: "insufficient_credits"}} + runner, write := s.counting(s.dialer) + s.launched(s.single(), 1, "UTC", "+15550000001") + + s.tick(runner) + write() + + s.Equal(dialsCounted{requests: 1, refused: 1, code: "insufficient_credits"}, s.counted()) +} + +// Defect: a dial that placed no call and gave no reason counted refused with no code, so the +// digest cannot say why. +func (s *RunnerSuite) TestADialToANumberThatCannotBeDialedIsCountedRefusedUnderItsResult() { + s.dialer.results["+15550000001"] = []Dialed{{Result: InvalidNumber}} + runner, write := s.counting(s.dialer) + s.launched(s.single(), 1, "UTC", "+15550000001") + + s.tick(runner) + write() + + s.Equal(dialsCounted{requests: 1, refused: 1, code: string(InvalidNumber)}, s.counted()) +} + +// Defect: a dial that could not be tried, which may have rung nobody, counted as a call placed. +func (s *RunnerSuite) TestADialThatCouldNotBeTriedIsCountedRefused() { + runner, write := s.counting(failingDialer{}) + s.launched(s.single(), 1, "UTC", "+15550000001") + + s.tick(runner) + write() + + s.Equal(dialsCounted{requests: 1, refused: 1, code: codeDialFailed}, s.counted()) +} + +// Defect: a dial a stopping router never placed counted refused, so the contact, rung by +// another router later, reads as two dials, one of them refused. +func (s *RunnerSuite) TestADialTheRouterDidNotPlaceAsItStopsIsNotCounted() { + s.launched(s.single(), 1, "UTC", "+15550000001") + runner, write := s.counting(s.dialer) + stopped, stop := context.WithCancel(s.ctx) + stop() + s.Require().NoError(runner.Start(stopped)) + s.Require().Eventually(runner.stopping, 5*time.Second, 10*time.Millisecond) + claimed, err := s.store.ClaimDueContacts(s.ctx, "router-counting", s.clock.now(), lease, 10) + s.Require().NoError(err) + s.Require().Len(claimed, 1) + + dialed, err := runner.ring(s.ctx, claimed[0].Campaign, claimed[0].Contact, s.single().First(), 1) + s.Require().NoError(err) + write() + + s.Require().Equal(NotPlaced, dialed.Result) + s.Equal(dialsCounted{}, s.counted()) +} diff --git a/acceleration/internal/dlc/service.go b/acceleration/internal/dlc/service.go index 3332e66016..b33c191b97 100644 --- a/acceleration/internal/dlc/service.go +++ b/acceleration/internal/dlc/service.go @@ -8,6 +8,7 @@ import ( "net/http" "time" + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" "github.com/GetStream/Vision-Agents/acceleration/internal/routing" "github.com/GetStream/Vision-Agents/acceleration/internal/stack" "github.com/GetStream/Vision-Agents/acceleration/internal/store" @@ -217,7 +218,8 @@ func (s *Service) Review(ctx context.Context, id, decision, notes, reviewer stri } // Hook takes a vendor's report on a brand or a campaign, verified, and moves whichever use -// cases it is about. +// cases it is about. A report on a campaign is counted as its use case's app's phone work. One on +// a brand is not: the hook reads no app for it, and has every waiting use case registered. func (s *Service) Hook(ctx context.Context, header http.Header, body []byte) error { if s.registrar == nil { return fmt.Errorf("%w: no registrar is configured", ErrRefused) @@ -234,6 +236,7 @@ func (s *Service) Hook(ctx context.Context, header http.Header, body []byte) err if err != nil { return err } + activity.NoteCustomer(ctx, useCase.CustomerID, activity.AreaPhoneSMS) return s.refresh(ctx, useCase, body) } if brandID != "" { diff --git a/acceleration/internal/mcpevents/mcpevents.go b/acceleration/internal/mcpevents/mcpevents.go index 8553f559a4..65bb694e3b 100644 --- a/acceleration/internal/mcpevents/mcpevents.go +++ b/acceleration/internal/mcpevents/mcpevents.go @@ -22,6 +22,7 @@ import ( "sync" "time" + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" "github.com/GetStream/Vision-Agents/acceleration/internal/auth" "github.com/GetStream/Vision-Agents/acceleration/internal/connectors/core" "github.com/GetStream/Vision-Agents/acceleration/internal/options" @@ -593,7 +594,8 @@ func (s *Service) owners(configs []store.AgentConfig, connection store.Connector // Receive answers one delivery to a subscription's callback: the challenge a server checks the // callback with, or an event, which opens a conversation. A delivery not signed with the -// subscription's own secret is refused before its body is read. +// subscription's own secret is refused before its body is read, and counted for nobody: a signed +// one is the subscription's app's hook, whatever is answered. func (s *Service) Receive(ctx context.Context, token string, header http.Header, body []byte) Reply { sub, err := s.store.ConnectionEventSubscriptionByToken(ctx, token) if errors.Is(err, store.ErrNoConnectionEventSubscription) { @@ -610,6 +612,7 @@ func (s *Service) Receive(ctx context.Context, token string, header http.Header, if err := webhook.Verify(secret, header, body, time.Now()); err != nil { return Reply{Status: http.StatusUnauthorized, Body: failure("the delivery is not signed with the subscription's secret")} } + activity.NoteCustomer(ctx, sub.CustomerID, activity.AreaHooksIn) var delivered webhook.Event if err := json.Unmarshal(body, &delivered); err != nil { diff --git a/acceleration/internal/scheduling/scheduler.go b/acceleration/internal/scheduling/scheduler.go index fc1f8c8266..c33e39a928 100644 --- a/acceleration/internal/scheduling/scheduler.go +++ b/acceleration/internal/scheduling/scheduler.go @@ -1,6 +1,7 @@ package scheduling import ( + "cmp" "context" "errors" "fmt" @@ -8,6 +9,7 @@ import ( "sync" "time" + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" "github.com/GetStream/Vision-Agents/acceleration/internal/store" ) @@ -188,6 +190,8 @@ type Scheduler struct { store *store.Store deliver DeliverFunc logger *slog.Logger + // activity counts each occurrence settled under its task's app. Nil counts none. + activity *activity.Recorder now func() time.Time lease time.Duration @@ -225,6 +229,12 @@ func NewScheduler(db *store.Store, deliver DeliverFunc, logger *slog.Logger) *Sc } } +// SetActivity has each occurrence the worker settles counted in record, as area scheduling. Call +// it before Start. +func (s *Scheduler) SetActivity(record *activity.Recorder) { + s.activity = record +} + // Start runs the worker until ctx ends or Close: it looks at start, when Kick // wakes it, at the next task due, and at least once a lease. Once ctx ends it // claims nothing more, and the deliveries in flight run on until they finish @@ -601,8 +611,33 @@ func (s *Scheduler) finish(claim store.ClaimedScheduledTask, outcome store.Sched log := s.logger.With("task", claim.Task.ID, "run", claim.Run.ID, "customer", claim.Task.CustomerID) switch { case err != nil: + // The run is left running, and whoever takes it up again settles and counts it. A + // run this claim failed itself, its workers lost, is not running: the next claim + // finds it settled by somebody else, and it is never counted. log.Error("could not record a scheduled task run", "error", err) + return case !applied: log.Info("a scheduled task changed under its run; the schedule was left alone") } + s.count(claim, outcome) +} + +// count counts an occurrence settled under claim in the activity record, once and not per +// attempt: delivered, or refused with the reason its run failed under, or missed. A run settled +// before claim took it was counted by whoever settled it, unless the claim itself failed it, its +// workers having been lost. +func (s *Scheduler) count(claim store.ClaimedScheduledTask, outcome store.ScheduledTaskOutcome) { + if claim.AlreadySettled && !claim.WorkersLost { + return + } + request := activity.Request{ + AppID: claim.Task.CustomerID, + Area: activity.AreaScheduling, + Client: activity.ClientRouter, + At: s.now(), + } + if outcome.RunState != store.ScheduledTaskRunDelivered { + request.Refused, request.Code = true, cmp.Or(outcome.Error, outcome.RunState) + } + s.activity.Record(request) } diff --git a/acceleration/internal/scheduling/scheduler_activity_test.go b/acceleration/internal/scheduling/scheduler_activity_test.go new file mode 100644 index 0000000000..3b5bd8dbd8 --- /dev/null +++ b/acceleration/internal/scheduling/scheduler_activity_test.go @@ -0,0 +1,204 @@ +//go:build integration + +package scheduling + +import ( + "context" + "database/sql" + "errors" + "log/slog" + "time" + + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" + "github.com/GetStream/Vision-Agents/acceleration/internal/store" +) + +// occurrencesCounted is a task's app's scheduling row in the activity record. +type occurrencesCounted struct { + requests, refused int64 + code string +} + +// counting starts a scheduler over deliver, as started does, that counts in an activity record +// of its own, which writes only when the test calls the stop it returns. +func (s *SchedulerSuite) counting(deliver DeliverFunc, tune func(*Scheduler)) (stop func()) { + record, err := activity.New(activity.Options{Store: s.db, Interval: time.Hour, Logger: slog.New(slog.DiscardHandler)}) + s.Require().NoError(err) + scheduler := s.started(deliver, func(scheduler *Scheduler) { + scheduler.SetActivity(record) + if tune != nil { + tune(scheduler) + } + }) + // The scheduler first, so an occurrence it is settling is counted before the record writes. + return func() { + scheduler.Close() + record.Close() + } +} + +// counted is what the activity record holds for task's app under area scheduling and client api. +func (s *SchedulerSuite) counted(task store.ScheduledTask) occurrencesCounted { + var ( + row occurrencesCounted + code sql.NullString + ) + err := s.db.DB().QueryRowContext(s.ctx, ` +SELECT COALESCE(sum(requests), 0), COALESCE(sum(refused), 0), max(last_refusal_code) +FROM customer_activity WHERE app_id = ? AND area = ? AND client = ?`, + task.CustomerID, activity.AreaScheduling, activity.ClientRouter).Scan(&row.requests, &row.refused, &code) + s.Require().NoError(err) + row.code = code.String + return row +} + +// Defect: a delivered occurrence left out of its app's scheduling work, or counted refused. +func (s *SchedulerSuite) TestADeliveredOccurrenceIsCountedOnceUnderItsApp() { + task := s.task(time.Now().UTC().Add(-time.Hour).Truncate(time.Second), nil) + stop := s.counting((&deliverer{}).deliver, nil) + + s.Require().Equal(store.ScheduledTaskRunDelivered, s.settled(task)[0].State) + stop() + + s.Equal(occurrencesCounted{requests: 1}, s.counted(task)) +} + +// Defect: each attempt counted, so an occurrence tried three times reads as three, or a failed +// one counted as delivered, or refused without the reason its run failed under. +func (s *SchedulerSuite) TestAFailedOccurrenceIsCountedOnceRefusedUnderItsReason() { + task := s.task(time.Now().UTC().Add(-time.Hour).Truncate(time.Second), nil) + deliver := &deliverer{err: &Failure{Reason: ReasonNoNumber, Err: errors.New("no number to text")}, only: task.ID} + stop := s.counting(deliver.deliver, func(scheduler *Scheduler) { + scheduler.retries = []time.Duration{time.Millisecond, time.Millisecond} + }) + + s.Require().Equal(store.ScheduledTaskRunFailed, s.settled(task)[0].State) + stop() + + s.Require().Len(deliver.callsFor(task.ID), 3, "the first attempt and both retries") + s.Equal(occurrencesCounted{requests: 1, refused: 1, code: ReasonNoNumber}, s.counted(task)) +} + +// Defect: an occurrence too late to deliver counted as delivered, so a router that was down at +// 09:00 shows the reminder as sent. +func (s *SchedulerSuite) TestAMissedOccurrenceIsCountedRefusedAsMissed() { + deliver := &deliverer{} + task := s.task(time.Now().UTC().Add(-25*time.Hour).Truncate(time.Second), nil) + stop := s.counting(deliver.deliver, nil) + + s.Require().Equal(store.ScheduledTaskRunMissed, s.settled(task)[0].State) + stop() + + s.Require().Empty(deliver.callsFor(task.ID)) + s.Equal(occurrencesCounted{requests: 1, refused: 1, code: store.ScheduledTaskRunMissed}, s.counted(task)) +} + +// Defect: a quiet-hours hold counted as an occurrence, so a reminder held and then delivered +// reads as two, one of them refused. +func (s *SchedulerSuite) TestAHeldOccurrenceIsCountedOnceWhenItIsDelivered() { + task := s.task(time.Now().UTC().Add(-time.Minute).Truncate(time.Second), nil) + deliver := &holdDeliverer{hold: 300 * time.Millisecond, task: task.ID} + stop := s.counting(deliver.deliver, nil) + + s.Require().Eventually(func() bool { + return s.read(task).State == store.ScheduledTaskCompleted + }, 10*time.Second, 10*time.Millisecond) + stop() + + s.Require().Len(deliver.calls, 2, "held once, then delivered") + s.Equal(occurrencesCounted{requests: 1}, s.counted(task)) +} + +// Defect: a run settled before its advance was lost counted again by the worker that only +// advances the task, so the occurrence reads as two. +func (s *SchedulerSuite) TestAnOccurrenceSettledBeforeItsAdvanceIsNotCountedAgain() { + due := time.Now().UTC().Add(-time.Hour).Truncate(time.Second) + task := s.task(due, nil) + started := due.Add(-time.Minute) + run := store.ScheduledTaskRun{ + CustomerID: task.CustomerID, TaskID: task.ID, ScheduledFor: due, + Trigger: store.ScheduledTaskTriggerSchedule, Prompt: task.Instructions, + State: store.ScheduledTaskRunDelivered, DeliveredTo: "conversation", + Attempts: 1, StartedAt: &started, FinishedAt: &due, + } + s.Require().NoError(s.db.InsertScheduledTaskRun(s.ctx, &run)) + stop := s.counting((&deliverer{}).deliver, nil) + + s.Require().Eventually(func() bool { + return s.read(task).State == store.ScheduledTaskCompleted + }, 10*time.Second, 10*time.Millisecond) + stop() + + s.Equal(occurrencesCounted{}, s.counted(task)) +} + +// Defect: a run failed because each of its workers was lost skipped as settled already, though +// no worker that settled it was left to count it. +func (s *SchedulerSuite) TestAnOccurrenceWhoseWorkersKeepDyingIsCountedRefused() { + due := time.Now().UTC().Add(-time.Minute).Truncate(time.Second) + task := s.task(due, nil) + // Three workers took it up and died: the store's cap. + started := time.Now().Add(-time.Hour) + stale := store.ScheduledTaskRun{TaskID: task.ID, CustomerID: task.CustomerID, ScheduledFor: due, + Attempts: 3, StartedAt: &started} + s.Require().NoError(s.db.InsertScheduledTaskRun(s.ctx, &stale)) + stop := s.counting((&deliverer{}).deliver, nil) + + s.Require().Eventually(func() bool { + return s.read(task).State == store.ScheduledTaskFailed + }, 10*time.Second, 10*time.Millisecond) + stop() + + s.Equal(occurrencesCounted{requests: 1, refused: 1, code: ReasonFailed}, s.counted(task)) +} + +// Defect: an occurrence settled under a claim an edit fenced (the run recorded delivered, the +// schedule left alone) left uncounted. +func (s *SchedulerSuite) TestAnOccurrenceSettledUnderAFencedClaimIsCounted() { + task := s.task(time.Now().UTC().Add(-time.Hour).Truncate(time.Second), nil) + fence := func(ctx context.Context, claimed store.ClaimedScheduledTask) (string, string, error) { + if claimed.Task.ID == task.ID { + // An edit during the delivery takes the claim away. + _, err := s.db.DB().ExecContext(ctx, `UPDATE scheduled_tasks SET claim_id = 'fenced' WHERE id = ?`, task.ID) + s.Require().NoError(err) + } + return "conversation", "", nil + } + stop := s.counting(fence, nil) + + s.Require().Eventually(func() bool { + runs := s.runs(task) + return len(runs) == 1 && runs[0].State == store.ScheduledTaskRunDelivered + }, 10*time.Second, 10*time.Millisecond) + stop() + + s.Equal(occurrencesCounted{requests: 1}, s.counted(task)) +} + +// Defect: an occurrence whose run could not be written counted now, and again by the claim that +// takes it up and settles it. +func (s *SchedulerSuite) TestAnOccurrenceWhoseRunWasNotWrittenIsNotCounted() { + task := s.task(time.Now().UTC().Add(-time.Hour).Truncate(time.Second), nil) + delivered := make(chan struct{}, 1) + lose := func(ctx context.Context, claimed store.ClaimedScheduledTask) (string, string, error) { + if claimed.Task.ID == task.ID { + // The run row gone, so FinishScheduledTaskRun's read of it fails. + _, err := s.db.DB().ExecContext(ctx, `DELETE FROM scheduled_task_runs WHERE id = ?`, claimed.Run.ID) + s.Require().NoError(err) + delivered <- struct{}{} + } + return "conversation", "", nil + } + var scheduler *Scheduler + stop := s.counting(lose, func(started *Scheduler) { scheduler = started }) + + select { + case <-delivered: + case <-time.After(10 * time.Second): + s.FailNow("the occurrence was never delivered") + } + s.Require().Eventually(func() bool { return scheduler.Running() == 0 }, 10*time.Second, 10*time.Millisecond) + stop() + + s.Equal(occurrencesCounted{}, s.counted(task)) +} diff --git a/acceleration/internal/simulation/activity_test.go b/acceleration/internal/simulation/activity_test.go new file mode 100644 index 0000000000..b78889d3da --- /dev/null +++ b/acceleration/internal/simulation/activity_test.go @@ -0,0 +1,88 @@ +//go:build integration + +package simulation + +import ( + "context" + "crypto/rand" + "database/sql" + "log/slog" + "os" + "testing" + "time" + + "github.com/stretchr/testify/suite" + + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" + "github.com/GetStream/Vision-Agents/acceleration/internal/store" + "github.com/GetStream/Vision-Agents/acceleration/internal/testenv" +) + +// ActivitySuite covers how a run that ends is counted in the activity record. +type ActivitySuite struct { + suite.Suite + ctx context.Context + db *store.Store +} + +func TestActivitySuite(t *testing.T) { + suite.Run(t, new(ActivitySuite)) +} + +func (s *ActivitySuite) SetupSuite() { + dsn := os.Getenv("ROUTER_POSTGRES_DSN") + if dsn == "" { + s.T().Skip("ROUTER_POSTGRES_DSN must be set") + } + s.ctx = context.Background() + db, err := store.Open(testenv.Database(dsn, "simulation")) + s.Require().NoError(err) + s.Require().NoError(db.Migrate(s.ctx)) + s.db = db + s.T().Cleanup(func() { s.Require().NoError(db.Close()) }) +} + +// runsCounted is an app's evals row in the activity record. +type runsCounted struct { + requests, refused int64 + code string +} + +// settled settles a run of an app of its own that ended in state, with a runner that counts in +// an activity record of its own, and returns what the record wrote for the app. +func (s *ActivitySuite) settled(state string) runsCounted { + record, err := activity.New(activity.Options{Store: s.db, Interval: time.Hour, Logger: slog.New(slog.DiscardHandler)}) + s.Require().NoError(err) + runner := &Runner{store: s.db, activity: record, logger: slog.New(slog.DiscardHandler)} + run := store.SimulationRun{ID: "run-" + rand.Text(), CustomerID: "app-" + rand.Text(), State: state, Cases: 1} + + runner.settle(s.ctx, run) + record.Close() + + var ( + row runsCounted + code sql.NullString + ) + s.Require().NoError(s.db.DB().QueryRowContext(s.ctx, ` +SELECT COALESCE(sum(requests), 0), COALESCE(sum(refused), 0), max(last_refusal_code) +FROM customer_activity WHERE app_id = ? AND area = ? AND client = ?`, + run.CustomerID, activity.AreaEvals, activity.ClientRouter).Scan(&row.requests, &row.refused, &code)) + row.code = code.String + return row +} + +// Defect: a run the router could not get a ruling on counted as a run that worked, or left out. +func (s *ActivitySuite) TestARunThatErroredIsCountedRefused() { + s.Equal(runsCounted{requests: 1, refused: 1, code: store.SimulationErrored}, s.settled(store.SimulationErrored)) +} + +// Defect: a run the judge ruled against counted as the router refusing it: the agent failed the +// check, and the router did what it was asked. +func (s *ActivitySuite) TestARunTheJudgeFailedIsCountedAndNotRefused() { + s.Equal(runsCounted{requests: 1}, s.settled(store.SimulationFailed)) +} + +// Defect: a run stopped before it had every ruling counted refused. +func (s *ActivitySuite) TestARunThatWasCancelledIsCountedAndNotRefused() { + s.Equal(runsCounted{requests: 1}, s.settled(store.SimulationCancelled)) +} diff --git a/acceleration/internal/simulation/simulation.go b/acceleration/internal/simulation/simulation.go index eb9e45058e..7fe5dec86e 100644 --- a/acceleration/internal/simulation/simulation.go +++ b/acceleration/internal/simulation/simulation.go @@ -24,6 +24,7 @@ import ( "sync" "time" + "github.com/GetStream/Vision-Agents/acceleration/internal/activity" "github.com/GetStream/Vision-Agents/acceleration/internal/llmrouter" "github.com/GetStream/Vision-Agents/acceleration/internal/routing" "github.com/GetStream/Vision-Agents/acceleration/internal/session" @@ -70,9 +71,11 @@ type Options struct { LLM *llmrouter.Router // TTS and STT give the simulated caller a voice and ears. Absent means text // simulations only, and an audio one is refused rather than quietly held in writing. - TTS *ttsrouter.Router - STT *sttrouter.Router - Logger *slog.Logger + TTS *ttsrouter.Router + STT *sttrouter.Router + // Activity counts each run that ends under its app, as area evals. Nil counts none. + Activity *activity.Recorder + Logger *slog.Logger } // Runner works through the simulation runs that have been started. @@ -82,6 +85,7 @@ type Runner struct { llm *llmrouter.Router tts *ttsrouter.Router stt *sttrouter.Router + activity *activity.Recorder logger *slog.Logger mu sync.Mutex @@ -118,6 +122,7 @@ func New(options Options) (*Runner, error) { llm: options.LLM, tts: options.TTS, stt: options.STT, + activity: options.Activity, logger: options.Logger, running: map[string]context.CancelFunc{}, stopKeeping: stop, @@ -461,7 +466,9 @@ func (r *Runner) finish(ctx context.Context, kase store.SimulationCase) { } } -// settle records how the run as a whole came out. +// settle records how the run as a whole came out, and counts it in the activity record: refused +// when it errored, which is the router not getting a ruling on every conversation. A run the judge +// failed is still a ruling, and one cancelled was stopped before it had them all. func (r *Runner) settle(ctx context.Context, run store.SimulationRun) { written, cancel := context.WithTimeout(context.WithoutCancel(ctx), writeTimeout) defer cancel() @@ -469,6 +476,11 @@ func (r *Runner) settle(ctx context.Context, run store.SimulationRun) { if err := r.store.FinishSimulationRun(written, run); err != nil { r.logger.Error("could not record how a run ended", "run", run.ID, "error", err) } + request := activity.Request{AppID: run.CustomerID, Area: activity.AreaEvals, Client: activity.ClientRouter, At: time.Now()} + if run.State == store.SimulationErrored { + request.Refused, request.Code = true, store.SimulationErrored + } + r.activity.Record(request) } // keep renews the runs this router holds, stops the ones it was asked to stop from