Skip to content

[AI-3334] Count scheduled tasks, campaign dials, simulation runs, MCP events and 10DLC reports per customer - #2071

Merged
kanat merged 13 commits into
feature/ai-3330-router-usage-trackingfrom
w54/ai-3334-activity-b
Nov 18, 2026
Merged

kanat merged 13 commits into
feature/ai-3330-router-usage-trackingfrom
w54/ai-3334-activity-b

Conversation

@kanat

@kanat kanat commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

AI-3334, part B of AI-2380 (#2066): the work the router does for an app on its own now leaves a row in customer_activity, under the app the job holds (its organization is read from app_organizations by the existing upsert), with client api (activity.ClientRouter). Serves AC-1 of AI-3330.

scheduler.run ──deliverWithRetries (1..3 attempts)──▶ finish ──stored──▶ count ─┐
campaign.Runner.ring ──Dialer.Dial──▶ countDial ────────────────────────────────┤
simulation.Runner.run ──cases──▶ settle ────────────────────────────────────────┼─▶ activity.Recorder.Record
MCP server ──signed delivery──▶ mcpevents.Receive ─ NoteCustomer ─┐             │      (once a minute)
registrar ──signed report────▶ dlc.Hook ───────── NoteCustomer ─┴▶ withRequestLog ┘
Work Where it is counted Area Refused when (code)
Scheduled task occurrence scheduling.Scheduler.finish, once the run is stored scheduling failed (the run's reason, e.g. no_phone_number), missed (missed), schedule unreadable
Campaign dial campaign.Runner.ring, after the dial phone_sms the dial errored (dial_failed) or placed no call (its reason, else its result: insufficient_credits, invalid_number)
Simulation run simulation.Runner.settle evals the run errored (errored)
MCP connection event mcpevents.Receive, after webhook.Verify hooks_in any 4xx/5xx after the signature check
10DLC campaign report dlc.Hook, once the report names a use case phone_sms any 4xx/5xx after that

Decisions

  • Scheduled tasks are counted where the scheduler settles the occurrence, not in deliverChain/deliverOn. deliverWithRetries calls the delivery once per attempt, a hand-off runs deliverChain on the peer, and a thread delivery never reaches deliverChain. At finish every path counts once, as the forwarder counts a forward once and not per attempt.
  • Not counted until the occurrence settles: a quiet-hours hold, a retry left to its lease by a drain, a claim lost to an edit, a failed run write (the next claim settles and counts it). A run settled before its advance was lost is not counted again, except one the claim itself failed because its workers were lost.
  • A dial a shutdown cut off (not_placed) rang nobody and the contact is rung again, so it is not counted. A voicemail is a placed call.
  • A run the judge failed, or a cancelled one, is counted and not refused (see Questions).
  • A 10DLC brand report names no app the hook reads, so it is not counted.
  • Recorder.Record on a nil recorder does nothing, so the three runners need no nil check of their own.

Tests

Each test names the defect it catches.

  • scheduling: delivered once; failed after 3 attempts counts 1, refused under its reason; missed counts refused missed; a hold then delivery counts 1; a run settled before its advance counts 0; a workers-lost run counts refused.
  • campaign: both dials of a double dial count, not refused; voicemail not refused; money refusal under its reason; invalid number under its result; dialer error dial_failed; a dial a stopping router did not place counts 0.
  • simulation: errored refused; judge-failed and cancelled not refused.
  • api: a signed MCP event counts once and a forged one not; a signed MCP event the router turns down (400) counts refused; a signed 10DLC report counts as phone_sms (not hooks_in) and a forged one not.

23 mutants (every guard, code fallback and note placement), all killed; review round 1 added 4 more tests (fenced settle, unwritten run, not-JSON MCP delivery, failed 10DLC report). A task run by hand ("Run now") is counted under scheduling too, refused unless it delivered. Untested: the cmd/router/main.go wiring.

Questions

  • Simulations: should a run the judge failed, or one cancelled by its 30 min timeout, count as refused? Now only errored does.
  • 10DLC brand reports: count them under the brand's app (needs a lookup by vendor brand id)?

🤖 Generated with Claude Code

@kanat kanat left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed #2071 at 6c53e3e merged onto b9b22a4 (base unchanged). Build, vet (both tags) and gofmt clean. Integration suites pass: activity, scheduling, campaign, simulation, mcpevents, dlc, and api ScheduledTasks, ConnectionEvents, DLC, DLCBooking, Simulations, Activity. CI at review time: 51 pass, 3 pending, none failed.

[Should fix] R1.2 acceleration/internal/api/scheduled_tasks.go:621 (not in the diff): a run made by hand ("Run now", deliverManualRun) delivers through DeliverScheduledTask but is never counted. Only its 202 request is counted, so a failed manual delivery never counts as refused. The ticket put the count in deliverChain/deliverOn, which cover manual runs, and the PR body does not mention them. Shown by: the only callers of DeliverScheduledTask are the scheduler (cmd/router/main.go:1626) and deliverManualRun (scheduled_tasks.go:607), and none of scheduled_tasks.go, scheduled_delivery.go or scheduled_thread_delivery.go records activity. Fix: after FinishManualScheduledTaskRun, record scheduling with ClientRouter, refused under cmp.Or(failure, state) unless delivered, as Scheduler.count does. Add one test: a failed manual run counts refused under its reason.

[Nit] R1.4 acceleration/README.md:2154-2159 (not in the diff): the ops activity paragraph lists only hooks, forwards and socket refusals. Add one sentence: scheduled occurrences (scheduling), campaign dials (phone_sms) and simulation runs (evals) count from api, and so do MCP events (hooks_in) and 10DLC campaign reports (phone_sms).

Checked and sound: counting at Scheduler.finish counts retries, hand-offs and thread deliveries once. The peer end notes nobody. The organization is filled by the upsert's app_organizations fallback. activityRecord.Close runs after the scheduler, simulation and dialer closes. No customer-visible reply changes, no migration, no new log fields. Author questions: only errored counts as refused for simulations (upheld); 10DLC brand reports go to a ticket.

Verdict: request changes, for R1.1 (4 surviving mutants) and R1.2 (manual runs not counted).

case !applied:
log.Info("a scheduled task changed under its run; the schedule was left alone")
}
s.count(claim, outcome)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should fix] R1.1 Two guards around this call survive the PR's tests. I checked each in a scratch copy: the test below passes on head and fails under its mutant.

  • MA: case !applied: gets a return, so an occurrence settled under a fenced claim (run written delivered, schedule left alone) is not counted. Test TestAnOccurrenceSettledUnderAFencedClaimIsCounted: the deliverer runs UPDATE scheduled_tasks SET claim_id = 'fenced' WHERE id = ? and returns delivered. Wait until the run is delivered, stop, and expect {requests: 1}.
  • MB: the return after a failed FinishScheduledTaskRun is dropped, so the occurrence is counted now and again by the claim that takes it up. Test TestAnOccurrenceWhoseRunWasNotWrittenIsNotCounted: the deliverer runs DELETE FROM scheduled_task_runs WHERE id = ? (the run) and returns delivered. Wait for Running() == 0, stop, and expect {}. This also covers the line the PR lists as untested.

if err := webhook.Verify(secret, header, body, time.Now()); err != nil {
return Reply{Status: http.StatusUnauthorized, Body: failure("the delivery is not signed with the subscription's secret")}
}
activity.NoteCustomer(ctx, sub.CustomerID, activity.AreaHooksIn)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should fix] R1.1 MG: moving this note below the the delivery is not JSON 400 survives every suite. Test (ConnectionEventsSuite) TestASignedDeliveryThatIsNotJSONIsCountedRefused: post not json signed with the subscription's secret (Standard Webhooks headers), expect 400, and expect hooks_in +1 request and +1 refused. It passes on head and fails under the mutant.

if err != nil {
return err
}
activity.NoteCustomer(ctx, useCase.CustomerID, activity.AreaPhoneSMS)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Should fix] R1.1 MH: noting only after s.refresh succeeds survives every suite, though the body says any 4xx/5xx after this point counts as refused. Test (DLCSuite) TestASignedReportTheRouterFailsOnIsCountedRefused: the fake registry answers 500 to GET /v2/10dlc/campaign/{id} for a campaign set to a failing status. Send a signed report on it, expect a 5xx, and expect phone_sms +1 request and +1 refused. It passes on head and fails under the mutant.

log := s.logger.With("task", claim.Task.ID, "run", claim.Run.ID, "customer", claim.Task.CustomerID)
switch {
case err != nil:
// The run is left running, and whoever takes it up again settles and counts it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nit] R1.3 This is not true for a WorkersLost claim. The claim's own transaction has already committed the run as failed. If this write then fails, the next claim sees the run settled (AlreadySettled, WorkersLost false), and count skips it, so the occurrence is never counted. Say so in the comment, or ticket it.

kanat and others added 5 commits November 18, 2026 13:55
… and a failed 10DLC report

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s never counted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…activity counts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kanat kanat left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review, round 2: GO

Reviewed 897daf2 merged onto the current base 625d45f (#2073 and #2075 in): the merge is clean, no conflicts, including campaign.go. Build, vet (both tags) and the touched suites pass: scheduling, mcpevents, dlc, campaign, and api ScheduledTasks, DLC, ConnectionEvents, Activity.

Prior findings

  • R1.1 fixed (efdaa0d). The four tests each fail under their mutant: fenced claim returning without a count, no return after a failed run write, the MCP note moved below the not-JSON 400, the 10DLC note only after a successful refresh.
  • R1.2 fixed (42729c5). deliverManualRun now records after the run write, and Record is nil-safe. Four mutants all fail a test: no Record, not refused, code = state only (failed vs no_phone_number), always refused.
  • R1.3 fixed (f30cdb0). The comment matches store/scheduled_tasks.go:602 and 619-630.
  • R1.4 fixed (efbcb39). Each README claim matches its call site.
  • Q1.1, Q1.2 and the scheduled-occurrence deviation: rulings from round 1 stand.

New findings

  • [Nit] The PR body is stale: it says the return after a failed run write is untested (it is now) and lists no run-by-hand row. Edit the body; no code change.

8 mutations checked, 0 survived. gh pr checks showed no integration jobs yet when I looked; the orchestrator waits for CI.

@kanat
kanat marked this pull request as ready for review November 18, 2026 13:25
@kanat
kanat merged commit 7f87f43 into feature/ai-3330-router-usage-tracking Nov 18, 2026
@kanat
kanat deleted the w54/ai-3334-activity-b branch November 18, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant