Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
fbf543d
feat(evidence-capture): add generalized evidence capture skill and re…
tmchow Apr 9, 2026
3d75fa0
refactor(git-commit-push-pr): restructure prose-heavy sections into b…
tmchow Apr 9, 2026
3d2825b
fix(evidence-capture): add commit/push to upload fallback and add tests
tmchow Apr 9, 2026
17c02ec
fix(evidence-capture): bash 3.2 compat and test ffmpeg probe
tmchow Apr 9, 2026
7495bef
fix(evidence-capture): validate frame inputs before checking tools
tmchow Apr 9, 2026
1e9bdea
feat(git-commit-push-pr): add narrative framing step for PR descriptions
tmchow Apr 10, 2026
11fc967
fix(evidence-capture): fix frame reduction for small sets and curl er…
tmchow Apr 10, 2026
895f0a2
refactor(evidence-capture): replace bash script with Python pipeline
tmchow Apr 10, 2026
7f98cdc
docs: clarify scratch space guidance — .context/ vs OS temp
tmchow Apr 10, 2026
f129c2a
refactor(demo-reel): rename evidence-capture to demo-reel
tmchow Apr 10, 2026
100e5ab
fix(ce-demo-reel): Angular dep key, curl timeout, and rename to ce-de…
tmchow Apr 10, 2026
ea26692
fix(ce-demo-reel): reduce upload timeout from 120s to 30s
tmchow Apr 10, 2026
52bc955
fix(ce-demo-reel): clean up silicon rendering defaults
tmchow Apr 10, 2026
b6fbf14
fix(ce-demo-reel): increase silicon vertical padding to 40px
tmchow Apr 10, 2026
b425cdf
fix(ce-demo-reel): timeout handling, fail-fast normalization, safe te…
tmchow Apr 10, 2026
d094c01
feat(git-commit-push-pr): add writing voice defaults to prevent AI slop
tmchow Apr 10, 2026
03123a7
fix(git-commit-push-pr): ban double-hyphen as em dash substitute
tmchow Apr 10, 2026
cc72464
fix(ce-demo-reel): check ffmpeg return codes for palette and GIF encode
tmchow Apr 10, 2026
1d033d9
fix(git-commit-push-pr): tighten ce-demo-reel integration
tmchow Apr 10, 2026
564afee
fix(ce-demo-reel): pass repo root to detect, fix Remix package name
tmchow Apr 10, 2026
558e92a
fix(ce-demo-reel): make project detection a signal, not a gate
tmchow Apr 10, 2026
942dc65
chore: remove completed refactor plan
tmchow Apr 10, 2026
b53e832
chore: remove solution docs for removed feature-video approach
tmchow Apr 10, 2026
54a738a
docs(solutions): document bash vs Python for pipeline scripts
tmchow Apr 10, 2026
a47a44b
refactor(ce-demo-reel): return URL and description, not markdown
tmchow Apr 10, 2026
a9a12ed
fix(ce-demo-reel): use full Go module paths for web framework detection
tmchow Apr 10, 2026
0964cf4
fix(ce-demo-reel): add missing web framework deps to detection
tmchow Apr 10, 2026
d83f5af
fix(ce-demo-reel): remove net/http from go.mod detection
tmchow Apr 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ bun run release:validate # check plugin/marketplace consistency
- **Release versioning:** Releases are prepared by release automation, not normal feature PRs. The repo now has multiple release components (`cli`, `compound-engineering`, `coding-tutor`, `marketplace`). GitHub release PRs and GitHub Releases are the canonical release-notes surface for new releases; root `CHANGELOG.md` is only a pointer to that history. Use conventional titles such as `feat:` and `fix:` so release automation can classify change intent, but do not hand-bump release-owned versions or hand-author release notes in routine PRs.
- **Linked versions (cli + compound-engineering):** The `linked-versions` release-please plugin keeps `cli` and `compound-engineering` at the same version. This is intentional -- it simplifies version tracking across the CLI and the plugin it ships. A consequence is that a release with only plugin changes will still bump the CLI version (and vice versa). The CLI changelog may also include commits that `exclude-paths` would normally filter, because `linked-versions` overrides exclusion logic when forcing a synced bump. This is a known upstream release-please limitation, not a misconfiguration. Do not flag linked-version bumps as unnecessary.
- **Output Paths:** Keep OpenCode output at `opencode.json` and `.opencode/{agents,skills,plugins}`. For OpenCode, command go to `~/.config/opencode/commands/<name>.md`; `opencode.json` is deep-merged (never overwritten wholesale).
- **Scratch Space:** When authoring or editing skills and agents that need repo-local scratch space, instruct them to use `.context/` for ephemeral collaboration artifacts. Namespace compound-engineering workflow state under `.context/compound-engineering/<workflow-or-skill-name>/`, add a per-run subdirectory when concurrent runs are plausible, and clean scratch artifacts up after successful completion unless the user asked to inspect them or another agent still needs them. Durable outputs like plans, specs, learnings, and docs do not belong in `.context/`.
- **Scratch Space:** Two options depending on what the files are for:
- **Workflow state** (`.context/`): Files that other skills or agents in the same session may need to read — plans in progress, gate files, inter-skill handoff artifacts. Namespace under `.context/compound-engineering/<workflow-or-skill-name>/`, add a per-run subdirectory when concurrent runs are plausible, and clean up after successful completion unless the user asked to inspect them or another agent still needs them.
- **Throwaway artifacts** (`mktemp -d`): Files consumed once and discarded — captured screenshots, stitched GIFs, intermediate build outputs, recordings. Use OS temp (`mktemp -d -t <prefix>-XXXXXX`) so they live outside the repo tree entirely. No `.gitignore` needed, no risk of accidental commits, OS handles cleanup.
- **Rule of thumb:** If another skill might read it, `.context/`. If it gets uploaded/consumed and thrown away, OS temp. Durable outputs like plans, specs, learnings, and docs belong in neither — they go in `docs/`.
- **Character encoding:**
- **Identifiers** (file names, agent names, command names): ASCII only -- converters and regex patterns depend on it.
- **Markdown tables:** Use pipe-delimited (`| col | col |`), never box-drawing characters.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
---
title: "Prefer Python over bash for multi-step pipeline scripts"
date: 2026-04-09
category: best-practices
module: "skill scripting / ce-demo-reel"
problem_type: best_practice
component: tooling
severity: medium
applies_when:
- Script orchestrates 2+ external CLI tools (ffmpeg, curl, silicon, vhs)
- Script needs retry logic or graceful degradation on tool failure
- Script will run on macOS where bash 3.2 is the default
- Script needs to be tested from a non-shell test runner (Bun, Jest, pytest)
- Script has conditional failure paths where some errors should be caught and others should abort
tags:
- bash-vs-python
- pipeline-scripts
- skill-scripting
- set-e-footguns
- error-handling
- ce-demo-reel
---

# Prefer Python over bash for multi-step pipeline scripts

## Context

When building the `ce-demo-reel` skill, the initial implementation used a bash script (`capture-evidence.sh`) to orchestrate ffmpeg stitching, frame normalization, and catbox.moe upload. Over 4 review rounds, the script hit 4 distinct bug classes that are inherent to bash's execution model rather than simple coding mistakes.

## Guidance

Use Python for agent pipeline scripts that chain multiple CLI tools with error handling. Bash `set -euo pipefail` works for simple sequential scripts but becomes a footgun when you need controlled failure paths.

**Python subprocess model (explicit error handling):**
```python
result = subprocess.run(
["curl", "-s", "-F", f"fileToUpload=@{file_path}", url],
capture_output=True, text=True, timeout=30, check=False
)
if result.returncode != 0:
# Retry logic runs normally
attempts += 1
continue
```

**Python timeout handling (explicit catch):**
```python
try:
result = subprocess.run(cmd, timeout=60)
except subprocess.TimeoutExpired:
# Controlled failure, not a crash
return subprocess.CompletedProcess(cmd, returncode=1, stdout="", stderr="Timed out")
```

**Bash equivalent (the footgun):**
```bash
set -euo pipefail

# Exits the entire script before retry logic runs
url=$(curl -s -F "fileToUpload=@${file}" "$endpoint")
# Never reaches here on curl failure

# Workaround: || true on every line that might fail
url=$(curl -s -F "fileToUpload=@${file}" "$endpoint") || true
# Works but fragile and easy to forget
```

## Why This Matters

Agent pipeline scripts run in environments the skill author does not control: different macOS versions (bash 3.2 vs 5.x), CI containers, worktrees. Each bash portability issue requires a non-obvious workaround that reviewers must catch. Python's subprocess model makes error handling explicit and testable rather than implicit and version-dependent.

The 4 bugs found were not unusual. They are the predictable consequence of using bash for scripts that exceed its sweet spot.

## When to Apply

Use Python when:
- The script orchestrates 2+ external CLI tools
- The script needs retry logic or graceful degradation on tool failure
- The script will run on macOS where bash 3.2 is the default
- The script needs to be tested from a non-shell test runner
- The script has more than ~3 subcommands

Bash is still the right choice when:
- Simple sequential scripts with no error recovery (set -e is fine)
- One-liner wrappers around a single tool
- Scripts using only POSIX features with no array manipulation
- Git hooks and CI steps where the only failure mode is "abort the pipeline"

## Examples

**Before (bash, 4 bugs across 4 review rounds):**

| Bug | Cause | Workaround needed |
|---|---|---|
| `url=$(curl ...)` exits on network failure | `set -e` + command substitution | `\|\| true` on every line |
| `${array[-1]}` fails | Bash 3.2 lacks negative indexing | `${array[${#array[@]}-1]}` |
| Frame reduction keeps all frames for n=3,4 | Integer math: `step=(n-1)/2` with min 1 | Minimum step of 2 |
| `command -v ffmpeg` in Bun tests | `command` is a shell builtin, not spawnable | Use `which` instead |

**After (Python, all 4 bug classes eliminated):**

```python
# Negative indexing just works
last = frames[-1]

# Timeout handling is explicit
try:
result = subprocess.run(cmd, timeout=30)
except subprocess.TimeoutExpired:
return None

# Tool detection is a regular function
if not shutil.which("ffmpeg"):
sys.exit("ffmpeg not found")

# Math is straightforward
step = max(2, (len(frames) - 1) // 2)
```

## Related

- `docs/solutions/skill-design/script-first-skill-architecture.md`: covers when to use scripts vs agent logic (complementary: that doc answers "should a script do this?", this doc answers "which language?")
- `docs/solutions/agent-friendly-cli-principles.md`: CLI design from the consumer side (overlaps on exit code and stderr patterns)

This file was deleted.

Loading