wjs-syncing-multicam

Compute a single time offset for each multi-source recording of the same event using audio cross-correlation, and emit a .sync.json sidecar next to each original. Originals are never modified, copied, or re-encoded. Downstream tools use -itsoffset to apply the offset at consume time.

Setup & commands

The implementation lives in the open-source `polysync` pip package (<https://pypi.org/project/polysync/> · <https://github.com/jianshuo/polysync>) — this skill no longer ships its own scripts. Ensure it's installed, then drive it via its CLI:

python3 -m pip install -U polysync      # needs ffmpeg/ffprobe on PATH

polysync sync   REFERENCE SOURCE        # align SOURCE to REFERENCE, write sidecars
polysync sync   REFERENCE SOURCE --partial   # source covers only part of the session
polysync verify REFERENCE SOURCE SOURCE.sync.json   # independent residual check

Run one polysync sync per non-reference angle (reference first, same reference each time). The sections below document the algorithm, the sidecar schema, and the gotchas baked into the package — read them to interpret output and choose flags.

Design principle — sidecar over re-encode

Earlier versions of this skill produced *_synced.MOV files by trimming + re-encoding to bake the offset into the file. We removed that:

• Disk — a 75-min 4K shoot from 3 cameras is 60+ GB. Re-encoded synced copies double that for no information gain.
• Quality — every re-encode is lossy. The originals are the source of truth; sidecars are reversible metadata.
• Speed — _synced.MOV generation took 10+ min per file on Apple Silicon; sidecar emission takes seconds.
• Composability — any downstream tool (polysync edit, NLE import, ffmpeg one-liners) reads the sidecar and applies the offset itself. No tool-specific file format lock-in.

When NOT to use

• Single-camera footage — nothing to sync to. For splitting one source into clips, use video-segmentation.
• Sources already aligned in an NLE timeline — don't fight the editor.
• For the auto-edit / cut / PiP rendering step that comes AFTER sync, use wjs-editing-multicam (consumes these sidecars).

Why envelope-based, not raw waveform

Raw PCM cross-correlation gives weak peaks and false matches when the two mics have different gain / room response — i.e., almost always with a secondary cam. The log-energy envelope captures dialogue and music dynamics, which both mics hear regardless of frequency response. Don't skip the envelope step — it's the entire reason this skill is robust at low SNR.

Algorithm

1. Extract mono PCM at 8 kHz, 16-bit from each input. The audio stream is auto-selected by loudness (loudest_audio_stream): probe each 0:a:N over a 60 s mid-file window and pick the highest mean volume. Multi-track pro cameras break a naive 0:a:0 — Sony FX6 MXF clips carry 4 mono PCM tracks and routinely leave a:0 / a:1 dead (~-90 dB) with the room mic on a:2 / a:3; correlating the silent track fails to sync. Single-stream inputs (most MP4 cams) short-circuit to a:0.
2. Log-energy envelope at 100 Hz (10 ms hop, 50 ms window). High-pass with a 2nd-order Butterworth, 0.05 Hz cutoff, filtfilt — removes slow drift and gain offsets.
3. FFT cross-correlate envelopes end-to-end → coarse offset (~10 ms).
4. Refine at sample level with a 60 s probe from B near the coarse-aligned position in A, ±2 s search window, parabolic peak interpolation.
5. Multi-probe drift check — repeat step 4 every ~3 min. Linear fit delta(t) = slope·t + intercept reveals real clock drift (5–50 ppm typical). Use the midpoint-canonical offset (slope · midpoint + intercept) so residual error is symmetric around zero.
6. Compute overlap window in the reference timeline: overlap = [max(0, delta), min(ref_dur, delta + src_dur)].
7. Emit `.sync.json` sidecar next to each non-reference input. No file is copied, trimmed, or re-encoded. The reference input gets a sidecar too (with delta_seconds: 0) so downstream code can treat all inputs uniformly.

polysync sync is the implementation. It emits only the .sync.json sidecar — no _synced.MOV, no re-encode.

Sidecar schema (<input>.sync.json)

One sidecar per original input, written next to it. Pure JSON, no comments in-file — the field reference below is canonical.

{
  "_about": "Sync metadata for cam_b.MOV. Apply via ffmpeg -itsoffset. See wjs-syncing-multicam SKILL.md for full schema.",
  "schema_version": 1,
  "source": "cam_b.MOV",
  "reference": "cam_a.MOV",
  "delta_seconds": 12.345,
  "drift_slope": 1.8e-5,
  "overlap_in_reference": [12.345, 4512.180],
  "overlap_in_source":    [0.000,   4499.835],
  "verification": {
    "median_residual_ms": 4.2,
    "residual_spread_ms": 11.8,
    "probe_count": 24
  }
}

Field reference

How downstream consumes the sidecar

-itsoffset is per-input in ffmpeg and applies BEFORE -i. Always read the source's delta_seconds from the sidecar:

# Play cam_b aligned to cam_a's timeline
ffmpeg -itsoffset $(jq -r .delta_seconds cam_b.MOV.sync.json) -i cam_b.MOV \
       -i cam_a.MOV \
       -filter_complex "[0:v][1:v]hstack" out.mp4

# Trim to mutual overlap window (read from cam_b.MOV.sync.json)
ffmpeg -ss <overlap_in_source[0]> -i cam_b.MOV -t <overlap_dur> ...

For wjs-editing-multicam, polysync edit ingests every <input>.sync.json automatically; you don't compose these flags by hand.

Partial-coverage clips — polysync sync --partial

Common case — main cams cover 75 min, a Riverside / phone / lavalier recorder only covers the middle 30 min. Run polysync sync REF.MOV NEW.mp4 --partial:

1. Cross-correlates the new input against the reference (same envelope algorithm as full-overlap mode).
2. Finds where the new clip's t=0 sits in the reference timeline (delta_seconds may be large, e.g. 1842.5).
3. Writes ONLY the source sidecar — no black padding, no audio padding, no re-encode. overlap_in_reference tells consumers exactly when this input has coverage; outside that window, fall back to the main cams.

The --partial flag changes only the failure philosophy: it degrades gracefully (median delta on few probes, coarse delta if none) instead of failing on <3 good probes, and skips the reference sidecar (the reference is assumed to already belong to an established sync set). Everything else is identical to the default mode.

When to skip drift correction

For camera-cut editing (the common case), ±25 ms residual across an hour is below human perception — pass drift_slope: 0.0 and use only the midpoint delta_seconds.

For sync-sound / lip-sync at long durations (>30 min and verification.residual_spread_ms > 40), downstream applies atempo = 1 + drift_slope to the source. Source files are still not modified — the atempo filter runs at consume time.

Verification (always run)

polysync verify REF.MOV SRC.MOV SRC.sync.json re-extracts audio from BOTH originals natively (loudest stream, no ffmpeg offset) and runs multi-probe correlation again. It applies the sidecar's delta_seconds (and, with --apply-drift, the drift slope) as index arithmetic in numpy — a probe at reference time bs is drawn from the source at local time bs - delta, then sought near reference index bs; the peak offset is the residual. Writes results back into the sidecar's verification field.

Pass criteria — median_residual_ms < 15 and residual_spread_ms < 1 frame at delivery fps. Fail = retry with drift correction enabled.

A spread-only fail with a near-zero median is usually noise, not desync. Far-field mics on a wide / B-roll camera (high reverb, low SNR, ncoef ~0.2) produce a few outlier probes that blow up the spread while the median stays at a few ms. For camera-cut editing that is aligned — the median is the truth; the spread gate is conservative. Only chase it for long-form lip-sync.

Common pitfalls

• Raw waveform cross-correlation gives false peaks under low SNR. Always envelope first — this is not a tunable, it's the entire premise.
• `-itsoffset` semantics differ for audio vs video — for sync-correctness it must be the FIRST flag for that input. ffmpeg -i src -itsoffset X is wrong; ffmpeg -itsoffset X -i src is right.
• `-itsoffset` is a NO-OP when muxing to headerless raw PCM (`-f s16le`). There are no container timestamps to carry the offset, so ffmpeg silently drops it and inserts NO leading silence. Any analysis that extracts to raw PCM (like polysync verify) must apply the offset by index arithmetic in numpy, never via -itsoffset on the extraction. Symptom of getting this wrong: verification residuals scatter by hundreds of ms with ncoef ~0 even though sync itself was perfect.
• Naive `0:a:0` extraction silently syncs against a dead track. Multi-track cameras (Sony FX6 MXF: a:0/a:1 often -90 dB, mic on a:2) need loudness-based stream selection. polysync sync and polysync verify both pick the SAME loudest track automatically — if you ever reimplement, keep that invariant or residuals are meaningless.
• Sidecar paths must be relative to the sidecar file's directory, not the working directory of the consuming process. Resolve source / reference against Path(sidecar).parent.
• Don't bake `drift_slope` into the sidecar's `delta_seconds`. They're separate fields for a reason — naive consumers can ignore drift, sync-sound consumers can apply it. Mixing them loses information.