Skip to content

Auto-run rowCount backfill during make dev#95

Open
urugondavishnu wants to merge 2 commits into
tinyfish-io:mainfrom
urugondavishnu:auto-backfill-row-counts-in-dev
Open

Auto-run rowCount backfill during make dev#95
urugondavishnu wants to merge 2 commits into
tinyfish-io:mainfrom
urugondavishnu:auto-backfill-row-counts-in-dev

Conversation

@urugondavishnu
Copy link
Copy Markdown
Contributor

Summary

Follow-up to #89. The rowCount migration is idempotent but currently has to be run manually — contributors who pulled main after #89 still see curated seed datasets capped at "5 rows" on their local dashboards, because seed datasets never receive writes so the self-heal in the row mutations never fires for them.

Hooks npx convex run datasets:backfillRowCounts into make dev right after convex-push, so a fresh clone reaches a fully-migrated state with zero manual steps.

Uses the env-var-only invocation shape (matching seed-public-datasets) instead of explicit --admin-key flags, because the admin key contains a | that gets re-parsed by Windows cmd when passed via the CLI.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 26, 2026

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: fc252e08-a395-4bd4-ab98-e84f9f5cae74

📥 Commits

Reviewing files that changed from the base of the PR and between a796bab and a8c102d.

📒 Files selected for processing (1)
  • makefiles/Makefile
🚧 Files skipped from review as they are similar to previous changes (1)
  • makefiles/Makefile

📝 Walkthrough

Walkthrough

Adds a new phony Makefile target backfill-row-counts that verifies the root .env and runs npx convex run datasets:backfillRowCounts via frontend/../scripts/with-root-env.mjs. The dev target now invokes $(MAKE) backfill-row-counts after convex-push, causing the Convex dataset rowCount backfill to run automatically during development startup.

Sequence Diagram(s)

sequenceDiagram
  participant Developer
  participant Makefile
  participant with_root_env_mjs as with-root-env.mjs
  participant Convex
  Developer->>Makefile: run `make dev`
  Makefile->>Makefile: run `convex-push`
  Makefile->>with_root_env_mjs: invoke `backfill-row-counts`
  with_root_env_mjs->>Convex: `npx convex run datasets:backfillRowCounts` (supplies CONVEX_SELF_HOSTED_ADMIN_KEY)
Loading

Possibly related PRs

  • tinyfish-io/bigset#109: Overlaps Makefile dev workflow changes around convex-push and environment/Convex validation.
  • tinyfish-io/bigset#19: Earlier modifications to makefiles/Makefile affecting the convex-push flow that this change extends.
  • tinyfish-io/bigset#89: Adds the backfillRowCounts mutation and rowCount migration which this Makefile target invokes.

Suggested reviewers

  • giaphutran12
  • hwennnn
  • MMeteorL
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and accurately summarizes the main change: automatically running the rowCount backfill during the make dev workflow.
Description check ✅ Passed The description provides clear context about the follow-up to #89, explains the idempotent migration, the rationale for auto-running it, and technical implementation details including the Windows cmd parsing workaround.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Collaborator

@giaphutran12 giaphutran12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding for launch safety. This branch is merge-conflicting and stale against latest main. The intended Makefile change is small, but the branch needs to be rebased or merged with current main so the diff does not conflict with the newer dev setup changes, then checks need to rerun.

PR tinyfish-io#89 added a denormalized `rowCount` field on the dataset doc plus a
one-shot `datasets:backfillRowCounts` migration for existing data.
Contributors who pull main and run `make dev` get the new schema via
the existing convex-push step, but curated seed datasets never receive
writes, so the self-heal in the row mutations never fires for them —
those cards keep showing the (capped at 5) preview-length fallback
until someone remembers to run the backfill manually.

Hooks the migration into `make dev` after convex-push. The mutation is
idempotent (re-runs report `patched: 0, alreadyCorrect: N`) so there's
no cost to running it every dev start.

Uses the same env-var-only invocation shape as seed-public-datasets to
avoid the Windows cmd issue where the `|` in the admin key gets parsed
as a shell pipe when passed via --admin-key on the command line.
@urugondavishnu urugondavishnu force-pushed the auto-backfill-row-counts-in-dev branch from fbe4964 to a796bab Compare June 1, 2026 06:11
@urugondavishnu
Copy link
Copy Markdown
Contributor Author

urugondavishnu commented Jun 1, 2026

Rebased onto current main (head 3222058). The conflicts were entirely from #109's Makefile restructure (self-healing admin key, install-deps, the BigSet is ready! block).

Resolution:

Verification:

  • Live-ran node ../scripts/with-root-env.mjs npx convex run datasets:backfillRowCounts against the dev Convex; got { alreadyCorrect: 29, patched: 0, total: 29 }. Idempotent on a fully migrated DB as designed.
  • git diff main is now +16/-1 in makefiles/Makefile only.

Single commit, force-pushed with --force-with-lease. Ready for re-review. @giaphutran12

@MMeteorL MMeteorL self-requested a review June 2, 2026 02:17
Copy link
Copy Markdown
Collaborator

@MMeteorL MMeteorL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants