Skip to content

feat(blocks,create-blocks-app)!: remove the telemetry app task - #1633

Closed
VarshithaPamisetty wants to merge 1 commit into
mainfrom
fix/remove-telemetry-task
Closed

VarshithaPamisetty wants to merge 1 commit into
mainfrom
fix/remove-telemetry-task

Conversation

@VarshithaPamisetty

@VarshithaPamisetty VarshithaPamisetty commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

#1606 removes the telemetry command, the blocks-telemetry binary and AWS_BLOCKS_DISABLE_TELEMETRY, so Blocks telemetry follows CDK's opt-out only (CDK_DISABLE_CLI_TELEMETRY, cdk cli-telemetry --disable, or cli-telemetry: false in the context of cdk.json or ~/.cdk.json).

#1610 wired a telemetry package script into every template that has a scripts block. Once #1606 merges, a newly scaffolded app would ship npm run telemetry pointing at a command that no longer exists, and npm run migrate would keep adding it to apps being upgraded.

This removes the task.

Merge order: depends on #1606

This must merge after #1606. One assertion fails on today's main and passes once #1606 lands: help.test.ts's lists every runnable task once under its package script name compares root help against a hard-coded runnable list, and help.ts builds root help from the command registry, which still carries telemetry until #1606 removes it from registry.ts.

For whoever merges: hold this PR until #1606 is in, then confirm help.test.ts goes green — or land the two behind a single merge so main is never red. The diff itself needs no change for this.

Mutation-checked both directions, so the assertion still guards the registry/task-list contract:

What changed — 22 sites

Area Site
Templates the "telemetry" script in 13 of 14 create-blocks-app templates. amplify is an overlay (blocksTemplateOverlayOnly) and defines no scripts, so it never had it
Scaffold create-blocks-app/src/index.ts — the task list the Amplify integration writes into a scaffolded package.json
Migrate blocks/src/cli/commands/migrate.ts USER_TASKS — npm run migrate stops adding the script
Docs create-blocks-app/README.md — the example scaffolded package.json no longer lists the task (AGENTS.md rule 11)
Example apps native/kotlin/example/typescript and native/swift/Demo/typescript-demo, the two committed apps carrying the script
Tests create-blocks-app/src/cli.test.ts (scaffolded task map), blocks/src/cli/migrate.test.ts (rewritten-scripts assertions), blocks/src/cli/help.test.ts (runnable list)

packages/blocks/README.md is deliberately untouched — #1606 already removes its npm run telemetry and AWS_BLOCKS_DISABLE_TELEMETRY references (0 occurrences of each at 6cbdc1069).

packages/blocks/docs/README.md is not a 23rd site: it is untracked build output (git ls-files does not know it), produced by scripts/gen-block-docs.mjs, whose line 49 copyMarkdown(blocksDir, outDir) mirrors every root-level *.md of packages/blocks into packages/blocks/docs/ verbatim. So it is downstream of packages/blocks/README.md, i.e. of #1606, and no generator reads create-blocks-app/README.md at all. (scripts/sync-catalog.mjs only maintains the catalog table between the BEGIN:block-catalog / END:block-catalog markers inside packages/blocks/README.md; it does not produce docs/.)

Confirmed by simulating the merged state rather than waiting for it: with #1606's packages/blocks/README.md applied on top of this branch, node scripts/gen-block-docs.mjs (exit 0, 21 block docs) leaves 0 matches for npm run telemetry, "telemetry": or -- telemetry anywhere under packages/blocks/docs/. Branch state restored afterwards; tree clean at 46a2082e0.

Incidental fix forced by the lint scope

npm run lint is biome lint --changed --since=origin/main. Editing create-blocks-app/src/index.ts brings the whole file into scope, which exposes a pre-existing lint/suspicious/noAssignInExpressions error at line 469, unrelated to this change. It fails the run, so it is fixed here:

-  const hostScripts = pkg.scripts ??= {};
+  pkg.scripts ??= {};
+  const hostScripts = pkg.scripts;

No changeset: behaviour is identical, verified rather than assumed. The emitted dist/index.js differs only in that statement split, and both forms were run against scripts absent, undefined, null and populated — same resulting pkg, same bound value, and hostScripts aliases pkg.scripts in both, since ??= evaluates to the final value of its target.

Changeset

.changeset/remove-telemetry-task.md, minor for @aws-blocks/blocks and @aws-blocks/create-blocks-app — the two changed published packages, which .changeset/config.json also keeps in a fixed group. Breaking, and this repo is pre-1.0, so breaking goes in the minor channel, matching #1610's and #1606's own changesets. The two native/ example apps are absent from the root workspaces array, so changesets does not track them.

npx changeset status --since=origin/main → exit 0, bumps both packages minor.

Verification

Check Command Result
Build npm run build exit 0
Lint npm run lint exit 0
Undeclared deps npm run lint:deps exit 0, 627 files
Lockfile git diff HEAD origin/main --stat -- package-lock.json 0 lines
Templates e2e npm run test:templates exit 0 — 13 templates, 104 tests, 104 pass, 0 fail
blocks CLI node --test dist/cli/*.test.js 342 tests, 341 pass, 1 fail — the #1606 dependency above
create-blocks-app npm test 419 tests, 418 pass, 1 fail — pre-existing

Scaffold and migrate were exercised end to end: scaffolded the default template, deleted six tasks, ran migrate --force --output json → exit 0, re-added exactly docs docs:search docs:api db whoami migrate, no telemetry, zero occurrences in the resulting package.json. test:templates also runs migrate --force and --dry-run against a real installed app.

Known failures not addressed

  • create-blocks-app cli.test.ts refuses to scaffold into an npm workspace that does not list the app's aws-blocks/ folder — pre-existing, A/B verified: fails identically with these changes stashed.
  • packages/blocks conditional-exports.test.js — ENOENT packages/bb-data/package.json; bb-data is gone from the tree while the root package.json still lists it. Broken on main already.

The templates e2e was run with each scaffolded app's blocks.dev.port moved off the default, because this machine has 3001 occupied. No env var controls it: the port is read from the scaffolded app's package.json#blocks.dev.port by both npm run dev and the template's test/e2e.test.ts.

Migrate leaves existing apps alone, by contract

npm run migrate only stops adding the task; it does not strip a telemetry script from an app already scaffolded with one. That is not a new choice, it is what migrate's documented contract already requires:

  • blocks/src/cli/commands/migrate.ts header: a package.json script "is rewritten only when its value is exactly the command a template generated", and "Every other script stays as it is". Only a generated wrapper file is deleted, once nothing references it; a customized one "migrate never deletes".
  • blocks/src/cli/registry.ts for migrate: generated commands "are rewritten; missing framework tasks are added without overwriting custom scripts".

The changeset states the resulting behaviour: "An existing app keeps the script it was scaffolded with; npm run migrate does not remove it."

#1606 removes the `telemetry` command, the `blocks-telemetry` binary and
`AWS_BLOCKS_DISABLE_TELEMETRY`, so telemetry follows CDK's opt-out only.
#1610 landed a `telemetry` package script in every template that has one,
which would leave a newly scaffolded app running a command that is gone.

Remove the task everywhere it is written, documented or asserted:

- the `"telemetry"` script in 13 of the 14 create-blocks-app templates
  (`amplify` is an overlay and defines no `scripts`, so it never had it)
- the task list the Amplify integration writes into a scaffolded
  `package.json` (`create-blocks-app/src/index.ts`)
- `USER_TASKS` in `blocks/src/cli/commands/migrate.ts`, so `npm run migrate`
  stops adding the script
- the example scaffolded `package.json` in the create-blocks-app README
- the two committed example apps that carry the script, `native/kotlin`'s
  TypeScript example and `native/swift`'s TypeScript demo

Tests follow: the scaffolded task map in `create-blocks-app/src/cli.test.ts`,
migrate's rewritten-scripts assertions in `blocks/src/cli/migrate.test.ts`,
and the runnable-task list in `blocks/src/cli/help.test.ts`.

Migrate only stops adding the script. An app already scaffolded with it keeps
it; removing a stale script from an existing app is deliberately not in scope.

Also splits one statement in `create-blocks-app/src/index.ts`, from
`const hostScripts = pkg.scripts ??= {}` to the assignment and the binding on
separate lines. `npm run lint` is `biome lint --changed --since=origin/main`,
so editing this file brings it into scope and its pre-existing
`lint/suspicious/noAssignInExpressions` error fails the run. Behaviour is
unchanged: `??=` evaluates to the final value of its target, so `hostScripts`
binds the same object either way, for an absent, undefined, null or populated
`scripts`. No changeset, as nothing observable changes.

Merge order: this depends on #1606 and must merge after it. `help.test.ts`
compares root help against that runnable list, and root help is built from
the command registry, which still carries `telemetry` until #1606 removes it.
So that one assertion fails on today's main and passes once #1606 lands.
@changeset-bot

changeset-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 46a2082

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@aws-blocks/create-blocks-app Minor
@aws-blocks/blocks Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@VarshithaPamisetty

Copy link
Copy Markdown
Contributor Author

Folded into #1606. Every change from this PR now lives in that one commit, so the removal of the telemetry command and the removal of the task that invoked it land together rather than leaving either branch red on its own.

Verified before closing: 21 of the 22 files here are byte-identical in #1606, and the 22nd (packages/create-blocks-app/src/cli.test.ts) differs only because #1606 also edits that file — the change from here is present (the telemetry entry is gone from the task map). A repo-wide sweep for the task finds no remaining reference.

Keeping the branch fix/remove-telemetry-task for now rather than deleting it.

@github-actions

Copy link
Copy Markdown
Contributor
📖 Glossary & notes — scoring, indicators, per-metric thresholds (click to expand)
  • N = 1 — one rep per cell, so a small delta may be model variance, not a real change; re-run for certainty.
  • Indicators (change vs baseline, per metric): ✨ noticeably improved · ✅ OK / no significant change (within noise band) · ⚠️ slight regression (beyond 1× threshold) · ❌ serious regression (beyond 2× threshold) · 🆕 new — no baseline value yet (current value shown, tagged (new)) · — nothing to show this run · 🗑️ cell gone since the baseline. Each cell shows <indicator> <value> (<Δ vs main>) inline.
  • Columns: Tests (pass/denom) · Judge (overall 0-10 score with signed delta; per-dimension breakdown lives in the judge artifact JSON) · Cost · Turns (agent cycles) · Score · Stop reason.
  • Threshold definitions (per metric, DELTA_THRESHOLDS in overview.mjs):
    • composite/score: ±5 points — a change ≤5 pts is ✅; >5 improving is ✨; >5 worsening is ⚠️; >10 worsening is ❌.
    • judge (overall): ±0.3 — a shift ≤0.3 is ✅; >0.3 improving is ✨; >0.3 worsening is ⚠️; >0.6 worsening is ❌.
    • tests: ±1 pass — a ≤1 pass change is ✅; >1 improving is ✨; >1 worsening is ⚠️; >2 worsening is ❌.
    • cost: ±10% of baseline (floor $0.02) — within 10% is ✅; >10% cheaper is ✨; >10% more expensive is ⚠️; >20% more expensive is ❌.
    • turns: ±3 cycles — a ≤3 turn change is ✅; >3 fewer is ✨; >3 more is ⚠️; >6 more is ❌.
  • Directions: higher is better for tests, judge, and score; lower is better for cost and turns.
  • Composite (0-100) = round(60·test_rate + 4·judge·min(1, 4·test_rate), 1) — 60% objective pass-rate + 40% judge, the judge term gated below a 25% pass-rate.
  • Cost = builder token spend at Bedrock Claude Opus 4.8 rates ($5 in / $25 out per 1M tokens; BUILDER_PRICING in scoring.mjs).
  • SCORE = composite ÷ cost — composite points per dollar (higher = better). Cost, not raw token volume, is the denominator, so tokens can't dominate; a broken (composite 0) cell scores 0 no matter how cheap. Flip SCORE_PER_DOLLAR in scoring.mjs for cost-per-point (lower = better).
  • Baseline = the most recent main-branch bench (bench/runs/latest-main.json), NOT the PR base commit — the PR always diffs against the current state of main.
  • Excluded from the mean: harness_error cells (infra failures) and gradeable cells that ran no tests. agent_fail (agent produced no app in budget) and dead_server (built app never served / crashed) are BOTH included, as composite 0.
  • Run artifacts — per-cell source + full agent traces · 🕒 Last run: 2026-11-18T18:34:44Z · run #44361063451

Results — PR vs main baseline

Each metric shows its current value colored by the change vs the latest main baseline e1e4844. ✨ improved · ✅ within noise · ⚠️ slight regression · ❌ serious regression · 🆕 no baseline yet (value shown, tagged (new)).

  • Mean composite 95.8/100 across 10 scored cell(s) — ✅ -0.9 vs main · judge mean 8.96/10.
  • Verdicts: 10 pass · 0 partial · 0 fail · 0 harness_error.
  • Totals: cost $158.7 · tokens 30.9M in / 177.9K out · 421 turns.
  • Biggest gains: auth-notes/demo (+3.2), kb-chat-agent/demo (+2.4), file-gallery/bare (+0.8).
  • Biggest drops: cognito-profile/auth (-4.8), observability-api/backend (-3.2), async-word-counter/bare (-2.4).
  • Config: baseline e1e4844.
  • ✅ Composite band 🟢 across 10 scored cell(s). Threshold 65 — pass.
Task Template Tests Judge Cost Turns Score Stop reason
async-word-counter bare ✅ 13/13 (0) ⚠️ 8.8 (-0.6) ✅ $4.19 (-$0.23) ✅ 20 (-3) ✅ 22.7 (+0.6) endTurn
auth-notes demo ✅ 11/11 (0) ✨ 9 (+0.8) ❌ $19.85 (+$4.03) ✅ 45 (+3) ✅ 4.8 (-1.1) endTurn
cognito-profile auth ✅ 10/10 (0) ❌ 8 (-1.2) ✨ $21.09 (-$8.67) ✨ 58 (-16) ✅ 4.4 (+1.1) endTurn
collab-presence-board default ✅ 11/11 (0) ⚠️ 8.8 (-0.6) ✅ $3.17 (+$0.28) ✅ 16 (+2) ✅ 30 (-3.7) endTurn
email-digest demo ✅ 12/12 (0) ✅ 9.4 (0) ❌ $6.51 (+$1.58) ❌ 26 (+8) ✅ 15 (-4.8) endTurn
file-gallery bare ✅ 12/12 (0) ✅ 9.6 (+0.2) ✨ $2.44 (-$0.4) ✅ 12 (-1) ✨ 40.4 (+6) endTurn
kb-chat-agent demo ✅ 9/9 (0) ✨ 9.6 (+0.6) ⚠️ $29.69 (+$4.65) ❌ 63 (+11) ✅ 3.3 (-0.5) endTurn
observability-api backend ✅ 18/18 (0) ❌ 8.8 (-0.8) ✨ $9.01 (-$2.35) ✅ 48 (+1) ✅ 10.6 (+1.9) endTurn
oidc-dsql-notes react ✅ 11/11 (0) ⚠️ 8.8 (-0.4) ❌ $51.32 (+$13.32) ❌ 95 (+18) ✅ 1.9 (-0.6) endTurn
sql-kb-catalog nextjs ✅ 9/9 (0) ✅ 8.8 (-0.2) ❌ $11.43 (+$3.2) ✅ 38 (+2) ✅ 8.3 (-3.4) endTurn

Merge verdict

✅ This bench run does not block the PR.

  • ✅ Absolute floor — mean composite 95.8 >= floor 65.
  • ✅ Regression vs main — mean composite delta -0.9 pts vs main is within the 10-pt limit.

The Δ vs base ball above is a fixed ±5-pt display band; the regression gate uses BENCH_MAX_REGRESSION. The ball is cosmetic — the gate threshold is what blocks.

This branch was previously deployed

2 inactive deployments
e2e — 46a2082e Deployed Nov 18, 2026 by VarshithaPamisetty via E2E Teardown Verify #7152
agent-bench — 46a2082e Deployed Nov 18, 2026 by VarshithaPamisetty via bench / summary #6230
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant