Video Analyzer Skill

Converts a video file into a complete, structured report (Markdown and JSON) that captures everything an AI or human needs to understand the video without watching it. Includes audio transcription (Whisper), visual frame analysis (Claude Vision), and optional scene clustering to aid frame selection.

Token cost estimates (choose your mode):

• Character-counting pass only (5–8 frames): ~8–12k vision tokens
• Full frame-by-frame timeline (up to 30 frames): ~40k+ vision tokens

Step 0 — Find the video file

ls /mnt/user-data/uploads/

If no video is present, ask the user to upload via the 📎 icon.

Step 1 — Install dependencies & run pre-processing

pip3 install openai-whisper opencv-python-headless numpy Pillow \
    --break-system-packages -q 2>/dev/null || true

python3 /home/claude/video-analyzer/scripts/process_video.py \
    "<VIDEO_PATH>" \
    --fps 0.5 \
    --max-frames 30 \
    --output-dir /tmp/video_work \
    --model base

Frames are resized so the long edge is at most 1568 px (aspect ratio preserved). This keeps vision token cost predictable at roughly ~1,400 tokens/frame.

Parameter guide:

Step 2 — (Optional) Run visual scene clustering

This step groups frames by color/composition similarity to help you pick representative frames for Claude Vision. It is not a character detector — it uses HSV histograms, not face recognition. Skip this step if you prefer to select frames manually.

python3 /home/claude/video-analyzer/scripts/cluster_frames.py \
    --frames-dir /tmp/video_work/frames \
    --output /tmp/video_work/clusters.json \
    --manifest /tmp/video_work/manifest.json \
    --similarity-threshold 0.72

The output clusters.json contains visual_cluster_count (a rough grouping estimate) and a representative_frame path per cluster. Use these paths to choose the 5–8 frames for the character-counting pass below.

Tuning `--similarity-threshold`:

• Too many clusters (>8 for a simple video): raise to 0.80
• Too few clusters (everything merged): lower to 0.60

Step 3 — Read metadata, transcript, and cluster data

cat /tmp/video_work/metadata.json
cat /tmp/video_work/transcript.txt
cat /tmp/video_work/clusters.json   # omit if Step 2 was skipped
cat /tmp/video_work/manifest.json

Step 4 — Identify characters with Claude Vision (character-counting pass)

Pick 5–8 well-distributed frames from the manifest (or use the representative_frame paths from clusters.json). Inspect each with the view tool.

view: /tmp/video_work/frames/frame_0001.jpg
view: /tmp/video_work/frames/frame_0009.jpg
…

For each frame, note every distinct person or character visible: appearance, clothing, setting. Then synthesize across all frames:

• How many distinct characters appear in total?
• Give each a short label and description (e.g. "Character 1 — Gioggia: blonde,

white suit, office setting").

This count and identification come from your visual judgment, not from the clustering script. The skill targets AI-generated and stylized video where geometric face detectors are unreliable; a vision model reasoning about "distinct characters" is more appropriate.

Record your findings:

Character 1: <name/label> — <description>
Character 2: <name/label> — <description>
…

Step 5 — Full visual timeline (optional, ~40k+ tokens)

If the user wants a frame-by-frame timeline, sweep through all frames in the manifest. Group near-identical consecutive frames and write a 2–4 sentence description per distinct scene:

[00:00:00] <who is present, setting, action, any on-screen text>
[00:00:05] <…>
…

Skip this step for a summary-only report to keep token cost low.

Step 6 — Build the JSON report

python3 /home/claude/video-analyzer/scripts/build_json_report.py \
    --work-dir /tmp/video_work \
    --output /mnt/user-data/outputs/<basename>_report.json \
    --visual-timeline '<JSON_ARRAY>' \
    --character-names '<JSON_OBJECT>' \
    --summary "<SUMMARY_TEXT>" \
    --key-observations '<JSON_ARRAY>'

Example values:

--visual-timeline '[{"timestamp":"00:00:00","description":"Donna bionda..."}]'
--character-names '{"1":"Gioggia","2":"Antoni","3":"Marina","4":"Anziano"}'
--summary "Video di satira politica AI-generated..."
--key-observations '["Prodotto con SHOWRUNNER.XYZ","4 personaggi distinti"]'

--character-names is the output of your Claude Vision identification pass (Step 4). If you skipped Step 4 (summary-only), pass {}.

Step 7 — Build the Markdown report

Save to /mnt/user-data/outputs/<basename>_report.md using this template:

# 📹 Video Report: <filename>
> Auto-generato dalla skill video-analyzer.

## 1. Metadata
| Proprietà | Valore |
|-----------|--------|
| Filename | `...` |
| Durata | `HH:MM:SS` |
| Risoluzione | `WxH` |
| Video codec | `...` |
| Audio codec | `...` |
| Dimensione | `N MB` |
| FPS | `N` |
| Bitrate | `N kbps` |

## 2. Personaggi Identificati
*(N personaggi distinti identificati da Claude Vision)*

| ID | Nome/Label | Descrizione |
|----|-----------|-------------|
| 1  | ...        | ...         |

## 3. Trascrizione Audio
*(Timestamped — [HH:MM:SS] testo)*

<trascrizione>

## 4. Timeline Visiva
### [HH:MM:SS] — Scena N
<descrizione 2-4 frasi>

## 5. Sommario AI
<150–300 parole>

## 6. Osservazioni Chiave
- ...

Step 8 — Present both files

present_files([
    "/mnt/user-data/outputs/<basename>_report.md",
    "/mnt/user-data/outputs/<basename>_report.json"
])

Tips & edge cases

• No audio: set transcript to "No audio track detected."
• Screen recordings: pay attention to UI, on-screen text, window titles
• Fast-cut video: --fps 1.0; very long (>30 min): --fps 0.1 --max-frames 60
• Foreign audio: Whisper is multilingual, no configuration needed
• AI-generated / animated video: face detectors won't work; rely entirely on

Claude Vision for character identification — this is the expected use case

• Skip clustering entirely: just pick frames evenly spaced from the manifest

for the character-counting pass