Video → Shorts

Turn a long talking-head video into vertical shorts (9:16, 25-45 s, captions in the picture). The skill transcribes the source, scores the best moments, plans an edit for the moment you pick, and renders it with scripts/build-short.py.

The deliverable is a finished mp4, ready to upload by hand, in a few variants (speed and caption style) so the owner can choose. No YouTube account, API key or upload CLI needed.

v2 changed the approach. v1 made a trimmed, cropped copy of the source with an SRT burned in. A professional reviewer called that flat. v2 edits for retention: hook first, cuts to each speaker and to reactions, captions from word timings, a held ending and an end card. Read references/retention-edit.md before you plan a cut. For two people on one wide shot, also read references/two-speaker.md.

When to use

• "Make shorts/reels/clips out of this podcast / interview / talk"
• "Find the best moments in this video and cut vertical clips"
• "Make this short more lively / add captions / add an end card"
• Any long-form talking footage that needs to become short vertical content

Best for static one- or two-person talking footage. Fast-cut montages, music videos or footage with a lot of on-screen graphics need a human editor. Say so and stop.

One-time setup (cross-platform)

You need ffmpeg with libass (captions and text), Whisper with word timestamps (faster-whisper or stable-ts), Python 3, and optionally yt-dlp (only if the source is a URL). The renderer also needs a display font and a mono font as files (defaults: Anton and IBM Plex Mono Medium, both OFL, from Google Fonts).

ffmpeg — must include libass

Check first: ffmpeg -version | grep -o libass. If it prints libass, you're done.

# macOS only — libass-enabled static ffmpeg
mkdir -p ~/.local/bin
curl -L https://evermeet.cx/ffmpeg/getrelease/ffmpeg/zip -o /tmp/ff.zip
unzip -o /tmp/ff.zip -d ~/.local/bin/
chmod +x ~/.local/bin/ffmpeg
~/.local/bin/ffmpeg -version | grep -E "libass|libfreetype"   # both must appear

build-short.py uses $FFMPEG if set, then ~/.local/bin/ffmpeg, then ffmpeg on the PATH. It draws all text with libass (ass= filter), not drawtext, so a build without drawtext works.

Whisper — transcription with word timestamps

Models download on first run (English base ≈ 140 MB, medium.en ≈ 1.5 GB, large-v3 ≈ 3 GB). Use a venv. If faster-whisper on CUDA fails with libcublas.so.12 not found, use the CPU settings above; they are fast enough for a clip.

yt-dlp (only if source is a URL)

pip install -U yt-dlp. For unlisted or age-gated videos add --cookies-from-browser chrome.

Workflow

1. Get the source video

• Local file → use it directly. Use the highest-resolution original you have (a 4K source

gives room for punch-ins). If the footage is log (D-Log, S-Log, V-Log), find its LUT; the renderer applies it after the crop.

• URL → download:

  yt-dlp -f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]" "<URL>" -o "/tmp/source.mp4"

• Note the resolution and frame rate (`ffprobe -v error -select_streams v -show_entries

stream=width,height,r_frame_rate -of csv=p=0 source.mp4`). You need both for the crops and the spec.

2. Transcribe the whole source, with word timings

The captions, the cut points and the speaker labels all come from word timings, so get them once for the whole source:

from faster_whisper import WhisperModel
import json
m = WhisperModel("medium.en", device="cuda", compute_type="float16")   # or cpu/int8
segs, _ = m.transcribe("source.mp4", word_timestamps=True, vad_filter=True,
                       condition_on_previous_text=False)
json.dump({"segments": [{"start": s.start, "end": s.end, "text": s.text,
           "words": [{"start": w.start, "end": w.end, "word": w.word} for w in s.words]}
           for s in segs]}, open("words.json", "w"))

stable-ts writes the same shape with result.save_as_json("words.json"). Also write an SRT for reading (mlx_whisper … --output-format srt, or build it from the segments).

Quick clean-up: Whisper mangles proper nouns and invents text over intro music. Do not edit words.json. Record fixes in the edit list (replace for a misheard word, patch for missing words), so the timings stay true.

3. Score the moments ← the core value

Read the full transcript and score each candidate with the rubric in references/retention-edit.md §1: hook 0.30, standalone 0.25, emotion 0.20, value 0.15, payoff 0.10. Skip anything under 60. Present the top 5-8 as a table:

Never use chapter markers as cut points. A chapter marks where a topic starts, preamble included. Anchor in and out to the words where the line begins and ends.

If the user gave a topic or a quote, find that moment instead (or as well).

4. Frame each person — never guess the crop

Crop width = source height × 9 ÷ 16, rounded to an even number. Every crop number must be even (libx264 with yuv420p rejects odd values).

1. Pull a frame from the middle of the moment and find each person's side of the frame.
2. For each person, render 3-5 candidate offsets within their side as one strip, at 4-6

times across the moment (people lean). Show the strip and let the user pick.

1. Give each person a name in the spec: C and C+ (punch-in) for the guest, A and A+

for the host. Anchor the punch-in to the bottom of the base crop so the face grows without moving up.

Details and the strip command: references/two-speaker.md § Framing.

5. Find who speaks when (two or more people)

A frame shows who is on screen, not who is talking. Take attribution from the audio: with two lavalier mics on L and R, each speaker is 2-7 dB louder on their own channel. Label every word. Then transcribe each channel on its own around any overlap, to recover short replies that the mixed transcript dropped. Method and code: references/two-speaker.md.

6. Plan the edit

Write the plan as <slug>.edl.json (schema in the header of scripts/build-short.py). Follow references/retention-edit.md:

1. Hook first. If the best line comes later, move it to the front as a cold open and

cut it from its original place.

1. A cut to every speaker, one-word interjections included.
2. Reaction shots of 1.5-2 s on the listener, only where the frames show a real smile

or laugh. The audio stays with the speaker.

1. Punch-ins at sentence boundaries, and over any jump where you trimmed a pause.
2. Tighten: trim pauses over 0.7 s to 0.3-0.5 s; cut repeats and false starts at word

gaps.

1. Hold the end 0.7-1.0 s after the payoff before the fade. Check the last frames for

a blink or an open mouth.

1. Cut in quiet. Each in and out point goes in a gap between words, at the quietest

point, not at the transcript's word boundary. Measure the RMS level on a 16 kHz mono copy and move each point to the minimum inside the gap.

A minimal spec:

{
  "slug": "past-life-in-china",
  "raw": "source.mp4", "words": "words.json", "fonts": "fonts/",
  "crops": {"C": [200, 0, 1216, 2160], "C+": [240, 360, 1012, 1800],
            "A": [2200, 0, 1216, 2160]},
  "shots": [[1240.45, 1244.95, "C+"], [1205.25, 1209.95, "C"],
            [1220.70, 1222.45, "A"], [1250.90, 1253.80, "C"]],
  "replace": {"infinity": "affinity"},
  "lower_third": {"name": "GUEST NAME", "sub": "Job title · Company", "at": 4.7, "dur": 3.6},
  "endcard": {"logo": "logo.png", "cta": "SUBSCRIBE", "url": "example.com", "dur": 2.8},
  "variants": {"hl-100": {"speed": 1.0, "captions": "highlight"},
               "hl-115": {"speed": 1.15, "captions": "highlight"},
               "hl-125": {"speed": 1.25, "captions": "highlight"},
               "phrase-115": {"speed": 1.15, "captions": "phrase"},
               "word-115": {"speed": 1.15, "captions": "word"}}
}

Add "audio" when the sound comes from a separate master on the same timeline, and "lut" for log footage. Put the lower third where it does not cover the face in that shot (it sits at the bottom left). Keep it off the hook.

7. Render

python3 skills/video-to-shorts/scripts/build-short.py past-life-in-china.edl.json          # all variants
python3 skills/video-to-shorts/scripts/build-short.py past-life-in-china.edl.json hl-115   # one

The script cuts each shot from the source (crop, scale to 1080×1920, LUT), joins them, measures the loudness, and makes the end card. For each variant it then does one pass: speed-up with atempo, captions and lower third through libass, gain to -14 LUFS with a limiter, and a fade to black. Then it appends the end card. It writes an SRT per variant beside the .ass in the work folder.

Shots are cached by frame range and crop. After you change captions, the lower third or the variants, a re-run costs one pass per variant. Delete endcard.mov to rebuild the end card.

8. Verify, then let the owner pick

• Pull frames at the hook, at the lower third, at a caption with a highlight, and on the

end card (ffmpeg -ss N -i out.mp4 -frames:v 1 check_N.png). Look for text on a face, text that overflows, and a cut in the middle of a word.

• Check the loudness: ffmpeg -i out.mp4 -af ebur128=peak=true -f null - should report

about -14 LUFS and a true peak under -1 dBFS.

• Watch the whole short once, as a stranger would (references/retention-edit.md § Final

check).

• Send the variants. Ask someone who has not heard the speaker before to choose the

speed.

Output conventions

For each short, keep:

• <slug>.edl.json — the edit list; the short can be rebuilt from it and the source
• <slug>-<variant>.mp4 — the renders; the chosen one is the deliverable
• <workdir>/<variant>.srt — the captions as a separate track, if the platform wants one
• <slug>.md — notes: source, score, what each shot is and why

slug = kebab-case summary of the moment, e.g. past-life-in-china.

Common mistakes