From aadf63e1d7322bbb4f6500c039939851afcb43f6 Mon Sep 17 00:00:00 2001 From: Rody Vilchez <102562850+R0SEWT@users.noreply.github.com> Date: Thu, 24 Sep 2026 23:41:09 -0500 Subject: [PATCH 1/2] docs(catalog): research a home for gitignored project data; ask for it in the CLAUDE.md template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every data project gitignores data/{bronze,silver,gold}, but the kit never said where that data lives or how to rebuild it. Evaluated HF dataset repos, HF Storage Buckets, DVC and git-annex + rclone against the current practice ($SILVER on an own server, regenerable silver). Verdict: trial — no project in the stack uses HF yet. Not wired as a default: the template gains a "Where it lives" fill prompt listing the options, so each new data project states where its data lives and never publishes third-party or personal data. Co-Authored-By: Claude Opus 5.5 (1M context) --- catalog/setup-options.md | 42 ++++++++++++++++++++++++++++++++++++++++ templates/CLAUDE.md.tmpl | 4 ++++ 2 files changed, 46 insertions(+) diff --git a/catalog/setup-options.md b/catalog/setup-options.md index a687f6b..d765fc3 100644 --- a/catalog/setup-options.md +++ b/catalog/setup-options.md @@ -20,6 +20,48 @@ Format per entry: +## 2026-09-24 — a home for a project's gitignored data + +Need: every data project gitignores `data/{bronze,silver,gold}`, but the kit never says where that +data lives or how to rebuild it. Constraint: CLI-driven, no secrets in the repo, private data stays +private, and the profile's differentiator (own data, Peruvian regional domain) can be published when +it is ours to publish. + +- **Candidate**: Hugging Face dataset repos (https://huggingface.co/docs/hub/datasets) + - What it does: versioned (git + Xet) dataset hosting with a card, a viewer and one-line download + (`hf download`, `datasets.load_dataset`); public or private. + - Fit: strong for **publishing own data** — a card, a license and a download line are the visibility + the profile lacks. Weak as working storage: deleting a file frees no quota until history is squashed. + Free accounts get 100 GB private; public storage is best-effort, and HF expects large public + datasets to carry a card and be reusable by others. + - Maintenance: `huggingface_hub` 2.0.0 (2026-09-24), very active. +- **Candidate**: Hugging Face Storage Buckets (https://huggingface.co/docs/hub/storage-buckets) + - What it does: S3-like, non-versioned, mutable storage on the Hub (`hf sync`, `hf buckets cp`, or an + S3-compatible gateway that rclone and DVC can use). + - Fit: good for private working data (bronze/silver, checkpoints, rolling backups): deleting frees + quota. Shares the 100 GB private tier. **A bucket is created public unless `--private` is passed**: + create it private and check its visibility before uploading. + - Maintenance: same library and CLI as above. +- **Candidate**: DVC (https://dvc.org) + - What it does: pointer files in git, content in a remote (S3, GCS, SSH, Google Drive; an HF bucket + through its S3 gateway). + - Fit: a second versioning system next to git, for data the stack mostly regenerates. Worth it only + when exact dataset versions must travel with the code. + - Maintenance: 3.67.1 (2026-03-31), ★15.9k, active. +- **Candidate**: git-annex with rclone (`rclone gitannex`) + - What it does: large files tracked by git-annex, content in any rclone remote. + - Fit: DVC's niche with a steeper learning curve, and it overlaps with the HF path. Not needed. + - Maintenance: rclone v1.75.1 (2026-09-04). +- **Current practice** (kept): remote silver on an own server via `$SILVER` (tesis_redes), or data + regenerable from source (infelix). +- **Verdict**: **trial** — no project in the stack uses HF yet. In the next data project: an HF dataset + repo (card + license) for data we publish and a **private** HF Storage Bucket for working data; DVC + or git-annex only if a project needs dataset versions pinned to commits. Never publish third-party or + personal data (course material, class recordings, PII). Revisit after one project ships. +- **Wiring**: not a default. `templates/CLAUDE.md.tmpl` gains a "Where it lives" fill prompt in Data + Conventions listing these options next to `$SILVER` and regenerate-from-source, so every new data + project states where its gitignored data lives and how to rebuild it. + ## 2026-06-02 — an MCP for SQLite Need: an MCP server to let Claude query a local SQLite DB. Constraint: must fit the local data diff --git a/templates/CLAUDE.md.tmpl b/templates/CLAUDE.md.tmpl index 8682ec2..848cb63 100644 --- a/templates/CLAUDE.md.tmpl +++ b/templates/CLAUDE.md.tmpl @@ -30,6 +30,10 @@ - **Layers**: raw → `data/bronze/`, cleaned → `data/silver/`, analytic → `data/gold/`. All gitignored. +- **Where it lives**: - **Polars over pandas** everywhere; only drop to numpy when a library requires it. - **DuckDB for SQL joins** across parquet; Polars for local transforms. - **Remote silver**: reference the remote root via `$SILVER` in SQL when pulling shared data. From 8dc3f372e66e00d5fb532105640b94702ea56b8a Mon Sep 17 00:00:00 2001 From: Rody Vilchez <102562850+R0SEWT@users.noreply.github.com> Date: Sat, 26 Sep 2026 23:11:19 -0500 Subject: [PATCH 2/2] docs: bump plugin to 0.2.0 and tidy the data-storage prompt The template change must reach the plugin cache, which is keyed on the plugin version. Drop the duplicate $SILVER mention and point to the catalog by URL, since the scaffolded repo has no catalog/. Co-Authored-By: Claude Opus 5.5 --- .claude-plugin/plugin.json | 2 +- catalog/setup-options.md | 2 ++ templates/CLAUDE.md.tmpl | 4 ++-- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 2780d80..1d657c0 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "project-kit", - "version": "0.1.0", + "version": "0.2.0", "description": "Bootstrap a new project with Rody's standard setup (beads, uv, CLAUDE.md/AGENTS.md, CI) and research new tooling to evolve the kit", "keywords": [ "scaffold", diff --git a/catalog/setup-options.md b/catalog/setup-options.md index d765fc3..e0e5b44 100644 --- a/catalog/setup-options.md +++ b/catalog/setup-options.md @@ -61,6 +61,8 @@ it is ours to publish. - **Wiring**: not a default. `templates/CLAUDE.md.tmpl` gains a "Where it lives" fill prompt in Data Conventions listing these options next to `$SILVER` and regenerate-from-source, so every new data project states where its gitignored data lives and how to rebuild it. + A deliberate exception to wiring only adopted tools: the prompt asks the question and lists the + options, but installs no HF tooling. Ships in plugin 0.2.0. ## 2026-06-02 — an MCP for SQLite diff --git a/templates/CLAUDE.md.tmpl b/templates/CLAUDE.md.tmpl index 848cb63..b986845 100644 --- a/templates/CLAUDE.md.tmpl +++ b/templates/CLAUDE.md.tmpl @@ -31,9 +31,9 @@ - **Layers**: raw → `data/bronze/`, cleaned → `data/silver/`, analytic → `data/gold/`. All gitignored. - **Where it lives**: + Never publish third-party or personal data. See https://github.com/R0SEWT/project-kit/blob/main/catalog/setup-options.md --> - **Polars over pandas** everywhere; only drop to numpy when a library requires it. - **DuckDB for SQL joins** across parquet; Polars for local transforms. - **Remote silver**: reference the remote root via `$SILVER` in SQL when pulling shared data.