Skip to content

docs(catalog): a home for a project's gitignored data (HF trial) + template prompt #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
R0SEWT merged 2 commits into from
Sep 27, 2026
Merged

docs(catalog): a home for a project's gitignored data (HF trial) + template prompt #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aadf63e
docs(catalog): research a home for gitignored project data; ask for i…
R0SEWT Sep 25, 2026
8dc3f37
docs: bump plugin to 0.2.0 and tidy the data-storage prompt
R0SEWT Sep 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "project-kit",
"version": "0.1.0",
"version": "0.2.0",
"description": "Bootstrap a new project with Rody's standard setup (beads, uv, CLAUDE.md/AGENTS.md, CI) and research new tooling to evolve the kit",
"keywords": [
"scaffold",
Expand Down
44 changes: 44 additions & 0 deletions catalog/setup-options.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,50 @@ Format per entry:

<!-- entries below -->

## 2026-09-24 — a home for a project's gitignored data

Need: every data project gitignores `data/{bronze,silver,gold}`, but the kit never says where that
data lives or how to rebuild it. Constraint: CLI-driven, no secrets in the repo, private data stays
private, and the profile's differentiator (own data, Peruvian regional domain) can be published when
it is ours to publish.

- **Candidate**: Hugging Face dataset repos (https://huggingface.co/docs/hub/datasets)
- What it does: versioned (git + Xet) dataset hosting with a card, a viewer and one-line download
(`hf download`, `datasets.load_dataset`); public or private.
- Fit: strong for **publishing own data** — a card, a license and a download line are the visibility
the profile lacks. Weak as working storage: deleting a file frees no quota until history is squashed.
Free accounts get 100 GB private; public storage is best-effort, and HF expects large public
datasets to carry a card and be reusable by others.
- Maintenance: `huggingface_hub` 2.0.0 (2026-09-24), very active.
- **Candidate**: Hugging Face Storage Buckets (https://huggingface.co/docs/hub/storage-buckets)
- What it does: S3-like, non-versioned, mutable storage on the Hub (`hf sync`, `hf buckets cp`, or an
S3-compatible gateway that rclone and DVC can use).
- Fit: good for private working data (bronze/silver, checkpoints, rolling backups): deleting frees
quota. Shares the 100 GB private tier. **A bucket is created public unless `--private` is passed**:
create it private and check its visibility before uploading.
- Maintenance: same library and CLI as above.
- **Candidate**: DVC (https://dvc.org)
- What it does: pointer files in git, content in a remote (S3, GCS, SSH, Google Drive; an HF bucket
through its S3 gateway).
- Fit: a second versioning system next to git, for data the stack mostly regenerates. Worth it only
when exact dataset versions must travel with the code.
- Maintenance: 3.67.1 (2026-03-31), ★15.9k, active.
- **Candidate**: git-annex with rclone (`rclone gitannex`)
- What it does: large files tracked by git-annex, content in any rclone remote.
- Fit: DVC's niche with a steeper learning curve, and it overlaps with the HF path. Not needed.
- Maintenance: rclone v1.75.1 (2026-09-04).
- **Current practice** (kept): remote silver on an own server via `$SILVER` (tesis_redes), or data
regenerable from source (infelix).
- **Verdict**: **trial** — no project in the stack uses HF yet. In the next data project: an HF dataset
repo (card + license) for data we publish and a **private** HF Storage Bucket for working data; DVC
or git-annex only if a project needs dataset versions pinned to commits. Never publish third-party or
personal data (course material, class recordings, PII). Revisit after one project ships.
- **Wiring**: not a default. `templates/CLAUDE.md.tmpl` gains a "Where it lives" fill prompt in Data

@R0SEWT R0SEWT Sep 26, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Convention break: a trial verdict gets wired into the default template. The catalog format says "Wiring (if adopt)", research-setup says "If adopt, offer wiring", and the SQLite precedent says trial means "per-project only, never in the default template".

The rule in CLAUDE.md is: "templates/ ... are kept faithful to real repos (tesis_redes, exp_tesis, infelix) — reuse, don't invent." The new template bullet recommends HF buckets and dataset repos, which this entry admits "no project in the stack uses ... yet". So every new data project gets an unproven option in its default CLAUDE.md. Keep the prompt to the proven options (regenerate / $SILVER) until HF is adopted.

Automated review (Claude Code, AI-generated). Validate before acting.

@R0SEWT R0SEWT Sep 27, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept on purpose (Rody's call). 8dc3f37 records it in the entry as a deliberate exception: the prompt asks the question and installs no HF tooling. (Claude Opus 5.5, on behalf of Rody)

Conventions listing these options next to `$SILVER` and regenerate-from-source, so every new data
project states where its gitignored data lives and how to rebuild it.
A deliberate exception to wiring only adopted tools: the prompt asks the question and lists the
options, but installs no HF tooling. Ships in plugin 0.2.0.

## 2026-06-02 — an MCP for SQLite

Need: an MCP server to let Claude query a local SQLite DB. Constraint: must fit the local data
Expand Down
4 changes: 4 additions & 0 deletions templates/CLAUDE.md.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@
<!-- Adjust/remove if this is not a data project. Defaults reflect the medallion + polars/duckdb stack. -->

- **Layers**: raw → `data/bronze/`, cleaned → `data/silver/`, analytic → `data/gold/`. All gitignored.
- **Where it lives**: <!-- fill: gitignored data still needs a home — say where each layer lives and how

@R0SEWT R0SEWT Sep 26, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes a template shipped by the plugin but doesn't bump .claude-plugin/plugin.json version (still 0.1.0). /project-kit:scaffold runs $CLAUDE_PLUGIN_ROOT/scripts/scaffold.sh, which reads templates from the cached project-kit/0.1.0/ copy.

After merge, projects scaffolded through the plugin still get the old CLAUDE.md with no "Where it lives" prompt, because the cache is keyed on the unchanged version. The PR's claimed effect only shows up when running the script straight from the checkout. PR #8's own contract requires a version bump in the same PR.

Automated review (Claude Code, AI-generated). Validate before acting.

@R0SEWT R0SEWT Sep 27, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8dc3f37: plugin bumped to 0.2.0. (Claude Opus 5.5, on behalf of Rody)

@R0SEWT R0SEWT Sep 26, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new prompt repeats the $SILVER option that the existing Remote silver bullet (L39) already covers. It also points to catalog/setup-options.md in project-kit, a path that doesn't exist in the scaffolded repo.

The rendered CLAUDE.md now describes remote silver twice in one section, so the two can drift, and the bare relative path can't be resolved from the consumer repo. Fold "Where it lives" into the Remote silver bullet, or drop the $SILVER item, and link the catalog by URL.

Automated review (Claude Code, AI-generated). Validate before acting.

@R0SEWT R0SEWT Sep 27, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8dc3f37: it now says "remote silver (see below)" and points to the catalog by URL. (Claude Opus 5.5, on behalf of Rody)

to rebuild it: regenerable from source (the command) · remote silver (see below) · a private HF
Storage Bucket for working data · an HF dataset repo (card + license) for data you publish.
Never publish third-party or personal data. See https://github.com/R0SEWT/project-kit/blob/main/catalog/setup-options.md -->
- **Polars over pandas** everywhere; only drop to numpy when a library requires it.
- **DuckDB for SQL joins** across parquet; Polars for local transforms.
- **Remote silver**: reference the remote root via `$SILVER` in SQL when pulling shared data.
Expand Down
Loading