Auto-Clipper

Turn a long YouTube video into upload-ready 9:16 vertical Shorts — AI clip selection, smart reframing, animated captions, background music, watermarking, and direct YouTube publishing. Runs entirely on your own machine; no web service, no data leaves your PC except the API calls you configure.

You can drive it three ways:

• CLI — one command, fully automatic, resumable.
• Web editor — a local React UI with step-by-step control and instant preview.
• Claude — talk to Claude Desktop or claude.ai, which drives the pipeline itself via an MCP server (~18 tools) and a bundled Skill that does creator research and SEO metadata for you.

---

How it works

flowchart LR
    A["1. Transcript\n(YouTube captions)"] --> B["2. AI Clip Selection\n(Gemini 2.5 Flash)"]
    B --> C["3. Download\n(yt-dlp, up to 1080p)"]
    C --> D["4. Reframe to 9:16\n(YOLO11 + EMA camera)"]
    D --> E["5. Captions\n(Groq Whisper large-v3)"]
    E --> F["6. Render\n(FFmpeg NVENC)"]
    F --> G["Optional: Upload\n(YouTube OAuth)"]

Loading

Every step is cached in a per-video state.json, so an interrupted or re-run job skips whatever's already done.

---

Features

Clip selection

• Gemini 2.5 Flash scores and picks the best moments from the transcript (hook, viral signals, natural start/end points, self-contained context) — or skip Gemini and hand the pipeline your own timestamps (set_custom_clips via Claude, or type them into the web editor).
• Auto-generated per-platform metadata: YouTube title/description/tags, Instagram caption, TikTok caption.

Reframing (long → 9:16)

• YOLO11n person detection drives a lock-and-snap camera: static by default, only moves when a detection confirms real movement — no jittery micro-adjustments.
• SOLO mode for one speaker, SPLIT mode (stacked halves, divider line) when two people are far enough apart on screen.
• Scene-cut detection resets tracking instantly on hard cuts instead of drifting toward the wrong subject.

Captions

• Word-level timestamps from Groq Whisper (large-v3), burned in as animated ASS subtitles.
• Five styles: pop (scale + color pop per word), karaoke (classic fill), classic (plain), magnify (grow while spoken), oneword (one word at a time, large).
• Optional AI-picked emoji per caption line (Claude picks these directly when driving via MCP; a Groq/Llama call does it in the web/CLI path).
• Built-in Hinglish support: a rewrite_captions tool exists specifically to romanize Hindi (Whisper often returns Devanagari) into Latin-script Hinglish so captions never render in a broken script.

Polish

• Background music: a categorized library (hype / motivational / chill / funny) under music/, auto-picked or manually assigned per clip, volume-mixed under the voice.
• Watermark overlay: image or text, any of 4 corner positions, adjustable opacity/scale.
• Horizontal mirror/flip, applied before captions are burned in so on-screen text never reads backwards.
• Fast 15-second NVENC preview render before committing to the full-length final render.

Publishing

• Direct YouTube upload via OAuth2 (resumable chunked upload).
• Batch scheduling: clips auto-publish one after another, spaced a random 15–30 min apart, using a persistent global cursor so a new batch always queues after whatever the last batch scheduled — even across separate runs.
• Fetch-merge-update editing of already-uploaded videos (only the fields you pass are changed).

Everything is resumable — state.json per video tracks transcript, clip list, source video path, and each clip's status (pending → reframing → reframed → captioning → captioned → previewed → rendering → finalized).

---

Repository layout

clipper/
├── main.py                  # CLI entry point (the 6-step pipeline)
├── app.py                   # FastAPI backend — one-shot + phase endpoints + MCP mount + YouTube OAuth
├── mcp_server.py            # MCP tool definitions/handlers, mounted at POST /mcp in app.py
├── config.py                # Loads .env → API keys
├── pipeline/
│   ├── transcript.py        # YouTube caption fetch
│   ├── analyzer.py          # Gemini clip selection + metadata generation
│   ├── downloader.py        # yt-dlp download
│   ├── reframe.py           # YOLO11 + EMA camera → 9:16 crop (SOLO/SPLIT)
│   ├── captions.py          # Groq Whisper + ASS subtitle generation
│   ├── renderer.py          # FFmpeg final/preview render (captions + BGM + watermark + mirror)
│   ├── music.py             # Background-music library lookup
│   ├── state.py             # video_id-keyed state.json read/write helpers
│   └── uploader.py          # YouTube OAuth + resumable upload + scheduling cursor
├── routers/
│   └── upload.py            # /api/upload/* — batch YouTube upload/schedule endpoints
├── skills/
│   └── clipping-youtube-shorts/   # Claude Skill: pipeline call order, research phase, SEO rules
├── frontend/                # React + Vite editor UI
│   └── src/pages/
│       ├── Dashboard.jsx    # Video list
│       └── VideoPage.jsx    # 6-step wizard: Analyze → Download → Reframe → Captions → Export → Upload
├── music/                   # BGM library (hype/motivational/chill/funny subfolders)
├── watermarks/              # Drop your own watermark PNGs here
├── output/                  # Generated per-video: state.json, clips, temp files (gitignored)
├── yolo11n.pt                # YOLO11 nano weights (checked into the repo — no separate download needed)
└── requirements.txt

---

Prerequisites

Requirement	Notes
Python 3.11	Developed/tested on 3.11; other 3.10+ versions likely work but aren't verified.
ffmpeg	Must be on PATH. On Windows, the code also checks a WinGet fallback install path automatically.
NVIDIA GPU with NVENC	Required, not optional. Every FFmpeg encode call in this codebase is hardcoded to h264_nvenc — there is no CPU/libx264 fallback path. Without an NVIDIA GPU (with current drivers), the cut/reframe/render steps will fail.
CUDA (via PyTorch)	Used for YOLO inference; this part does fall back to CPU automatically if unavailable, but will be much slower.
Node.js	Only needed if you're running the web editor's frontend.
Gemini API key	Required — clip selection + metadata generation.
Groq API key	Required — Whisper transcription + emoji picks.

---

Setup

cd clipper
pip install -r requirements.txt
cp .env.example .env

Edit .env and fill in:

GEMINI_API_KEY=...     # required — Google AI Studio: https://aistudio.google.com/apikey
GROQ_API_KEY=...       # required — https://console.groq.com/keys
ANTHROPIC_API_KEY=...  # currently unused by the code — safe to leave blank
API_KEY=               # optional — only needed if exposing the backend publicly (see below)

yolo11n.pt is already committed to the repo, so no separate model download step is needed.

If you'll be using the web editor, also install the frontend deps:

cd frontend
npm install

If you'll be using YouTube upload, see YouTube upload setup below — it's not required for clipping/captioning, only for the publish step.

---

Usage

CLI — one-shot, fully automatic

cd clipper
python main.py "https://www.youtube.com/watch?v=VIDEO_ID" --clips 5 --duration 60

Common flags:

Flag	Default	Purpose
--clips N	5	Number of clips to produce
--duration S	60	Target clip length in seconds
--output DIR	./output	Output directory
--force	off	Discard cached state.json, rerun everything
--no-captions	off	Skip transcription + caption burn-in
--style	pop	pop | karaoke | classic | magnify | oneword
--font	Montserrat ExtraBold	Caption font
--size	72	Caption font size
--color	&H0000FFFF	Highlight color — web hex (#FF0000) or raw ASS format
--y-pos	1650	Vertical caption position (0–1920)
--emojis	off	Enable AI-generated emoji overlays

Output lands in output/<video_id>/clip_01.mp4, clip_02.mp4, etc.

Web editor — step-by-step with preview

# terminal 1 — backend, from clipper/
uvicorn app:app --host 127.0.0.1 --port 8000 --reload

# terminal 2 — frontend, from clipper/frontend/
npm run dev

Open the Vite dev server URL (default http://localhost:5173). The wizard walks through: Analyze (transcript + clip picks + metadata) → Download → Reframe (editable clip timestamps) → Captions (live style/color/font controls) → Export (preview render, then final render with per-clip BGM/mirror overrides) → Upload (YouTube).

Claude Desktop / claude.ai — conversational

Run the backend (uvicorn app:app ... as above). The MCP server is mounted at POST /mcp on the same FastAPI app — no separate process to start.

• Claude Desktop: point an MCP connector at http://127.0.0.1:8000/mcp.
• claude.ai (remote connector): expose the backend publicly (e.g. via ngrok http 8000), set API_KEY in .env to a strong random string first, and point the connector at your public /mcp URL. The backend implements a minimal OAuth handshake (/authorize, /token) that claude.ai's connector flow expects.

Once connected, just paste a YouTube URL and ask for shorts — the bundled Skill (skills/clipping-youtube-shorts/) tells Claude the required tool call order, makes it research the creator/topic for SEO before picking clips, enforces the Latin-alphabet/Hinglish caption rule, and defaults every render to mirrored output. Claude drives the same 18 MCP tools (fetch_transcript, analyze_clips, set_custom_clips, trigger_pipeline, get_video_status, set_clip_music, rerender_clip, upload_clip, schedule_uploads, etc.) that back the CLI and web UI.

---

YouTube upload setup

Only needed for the publish/schedule features — clipping and captioning work without it.

1. Go to Google Cloud Console → APIs & Services → Credentials.
2. Create an OAuth 2.0 Client ID (type: Web application), with an authorized redirect URI of http://localhost:8000/api/youtube/oauth-callback.
3. Download the JSON and save it as clipper/client_secrets.json.
4. Start the backend, then either:
   • click "Connect YouTube Account" in the web editor's Upload tab, or
   • call GET /api/youtube/auth-url, visit the URL, approve, and the callback will store clipper/token.json automatically.

Both client_secrets.json and token.json are gitignored — never commit them, they contain live OAuth credentials.

---

Configuration reference

Environment variables (.env, see .env.example):

Variable	Required	Purpose
GEMINI_API_KEY	Yes	Clip selection + metadata (Gemini 2.5 Flash)
GROQ_API_KEY	Yes	Whisper transcription + emoji picks
ANTHROPIC_API_KEY	No	Present in .env.example; not currently referenced by any code path
API_KEY	No	Bearer token gate for the backend — only relevant when exposing it beyond localhost (e.g. for claude.ai)

Caption styles (--style / CaptionSettings.style): pop, karaoke, classic, magnify, oneword — see Features above for what each looks like.

Reframe tuning constants (pipeline/reframe.py, edit directly if you want to retune): CROP_W / CROP_W_SPLIT (crop widths), DETECT_EVERY (YOLO sampling rate), EMA_ALPHA (camera smoothing), SCENE_CUT_THRESH, SPLIT_MIN_SEPARATION, SPLIT_OVERLAP_THRESHOLD.

---

Troubleshooting

• ffmpeg not found — install FFmpeg and ensure it's on PATH. On Windows the code also checks the WinGet install location as a fallback.
• FFmpeg exits immediately / NVENC errors — you need an NVIDIA GPU with a current driver; there's no CPU encode fallback in this codebase (see Prerequisites above).
• yt-dlp HTTP 403 on download — YouTube changes frequently break older yt-dlp versions. Run pip install -U yt-dlp and retry.
• "No transcripts available" — the source video has no YouTube captions (manual or auto-generated) in any language; this pipeline depends on YouTube's own captions, it does not transcribe the full source video.
• Resuming a broken run — just re-run the same command/URL with the same --clips/--duration; it'll pick up from state.json. Use --force to discard the cache and start over.