movie_sync

Sync two releases/cuts of the same movie (different language dub, different master, different container delay) by measuring content-to-content offset and muxing a synced output.

Pipeline: pick a window → extract lossless clips → measure audio/video lag → mux delayed audio → verify A/B.

file_a ─┐
        ├─► extract.py ─► extracted/{stem}_{video,audio,proxy} ─► sync_audio_test.py (AST) ─┐
file_b ─┘                                                                                  ├─► lag (seconds) ─► apply_sync.py ─► output/{stem}_synced.mkv ─► verify_sync.py
                                                                                  sync_video_test.py (VST) ─┘

Lag convention (used everywhere, feeds apply_sync --lag directly): lag > 0 means B is delayed vs A (B starts later). lag < 0 means B is early. All estimators (sync_audio.correlate, VST methods) share this sign.

Requirements

• Python 3.11+ (tomllib from stdlib; 3.14 also works)
• ffmpeg + ffprobe on PATH (extraction, proxy, mux, all probing)
• mpv for verify_sync.py video playback (IPC-controlled, --no-audio)
• Python deps (installed via make install): torch, torchaudio, soundfile, scipy, numpy, pyqtgraph, PyQt6
• Optional: sounddevice (audio playback in AST/verify), openai-whisper (only for rename_audio.py), python-mpv

No GPU is required. torch.cuda (ROCm or CUDA) is used when available, otherwise everything falls back to NumPy CPU.

Install

make install        # auto-detect: ROCm > CUDA > CPU, create .venv, install deps
make install-cpu    # force CPU torch
make install-rocm   # force ROCm torch (e.g. RX 7900 XTX)
make install-cuda   # force CUDA torch

Activate manually with source .venv/bin/activate. Python entry points also work via make <target> without activating.

Configuration

All paths/settings live in config.toml (repo root). Copy the shipped template and set your two sources:

[extract]
file_a = "/path/to/movie_a.mkv"
file_b = "/path/to/movie_b.mp4"
duration = 30       # analysis window, seconds (the ONLY duration needed)
start = 0.0         # start offset, seconds (ffmpeg -ss before -i)
out_dir = "extracted"
target_fps = "24000/1001"
target_fps_eps = 0.02
proxy = true        # build {stem}_proxy.mkv (192px gray FFV1) for VST
proxy_width = 192

[audio]
sample_rate = 48000
channels = 1
normalize = true    # two-pass EBU R128 loudnorm
target_lufs = -14
target_tp = -1.0
target_lra = 11
# stream = null     # null = prompt, int = index, "skip" = no audio
volume = 70

[sync]
method = "phat"     # "phat" (GCC-PHAT, robust) or "standard"
use_gpu = "auto"    # "auto" | "cpu" | "cuda" (ROCm+CUDA both via torch.cuda)
out_dir = "output"

[video]
grid = "32x18"
bars = true         # auto-crop static letterbox/pillarbox before scaling
max_lag = 0.0       # search half-window, seconds (0 = full range)
prefer_proxy = true
decode_width = 192

Use generic local filenames (movie_a, movie_b) — never commit real paths or copyrighted titles.

Workflow

1. Pick a window (optional GUI)

make picker
# .venv/bin/python picker.py --config config.toml [--a … --b …]

Shows RMS waveforms + thumbnails for file_a, draggable start/duration region, audio-stream selector. Save to config.toml writes back [extract] start/duration (preserves other sections).

2. Extract analysis clips

make extract          # normal: FFV1 video + PCM WAV + gray proxy
make extract-loud     # same + agate transient gating (explosions/dialogue only)
make proxy            # rebuild {stem}_proxy.mkv from local masters only (no source access)

Per input, extract.py writes into extracted/:

• {stem}_video.mkv — FFV1 lossless intra-only, yuv444p, CFR-normalized to target_fps only if probed fps differs (avoids needless re-timing)
• {stem}_audio.wav — pcm_f32le, 48 kHz mono, two-pass loudnorm (skipped with fallback on silent/-inf segments)
• {stem}_proxy.mkv — 192 px gray8 FFV1 from the local master, for fast VST decode

Stems: differing filenames → bare stem; identical filenames → parent dirs are walked up until they diverge. Multi-audio files prompt for 0:a:N (or set [audio] stream); the choice is saved to .stream_state (gitignored) for later verify_sync --mode full mapping.

FPS note: if both inputs mismatch the target and differ from each other (e.g. PAL 25 vs NTSC film 23.976), both are normalized to the target and a warning suggests compensating audio tempo as well.

3. Measure lag

Audio (primary):

make ast
# LC_NUMERIC=C PYTHONPATH=. .venv/bin/python sync_audio_test.py --config config.toml

PyQtGraph GUI: overlay + per-track waveforms + GCC-PHAT xcorr, slider / seconds / samples lag controls, Auto peak, Swap A/B, Export JSON, stereo split playback (A left / B+lag right), Apply sync (background ffmpeg with frame progress bar) and Verify (short) launcher. make ast auto-runs make extract first if the expected WAVs are missing.

Video (cross-check / lab):

make vst
# .venv/bin/python sync_video_test.py --config config.toml
# headless: .venv/bin/python sync_video_test.py --no-gui [--audio-lag 0.042] [--export report.json]

Reads only extracted/ (proxy preferred, master fallback — never the originals). One decode + bar-crop + grid, then one batched per-pixel FFT. Methods: M1 global-luma, M2 3×3 grid-diff, M3 pixel-diff-sum (main), M4 median (logo-robust), M5 PHAT, M6 scene-cut spikes, M7 raw (no diff). GUI shows overlaid xcorr curves, per-pixel heatmap, 3×3 zone energy, frame scrub A|B, method table with median agreement, chunk-fit drift check (slope ≈ 0 confirms pure offset, no speed mismatch), JSON export.

Library use:

from sync_audio import load_wav, correlate
a, sr = load_wav("extracted/movie_a_audio.wav")
b, sr = load_wav("extracted/movie_b_audio.wav")
lag_samples, peak, corr = correlate(a, b, sr=48000, method="phat", device="auto")

4. Mux synced output

make sync LAG=0.042
# .venv/bin/python apply_sync.py --config config.toml --lag 0.042 [--a … --b … --out … --progress]

apply_sync.py builds: ffmpeg -i A -i B -filter_complex … -map 0:v -map 0:a -map [dN] -c:v copy -c:a:<A> copy -c:a:<B> <same-codec-or-aac> out.mkv.

• lag ≥ 0: adelay (silence pad at B start); lag < 0: atrim+asetpts cut.
• All A audio streams are copied; all B audio streams are re-encoded with their original codec where possible and appended after A's streams.
• Single undetermined-language B stream is tagged ger; existing language/title/forced flags are preserved. --progress streams frame=/progress=end on stdout for the AST background worker.

5. Verify

make verify LAG=0.042
# .venv/bin/python verify_sync.py --config config.toml --lag 0.042 --mode short
# --mode short: extracted 30 s parts (video + wav_a + wav_b)
# --mode full:  synced output MKV (audio streams re-extracted from it)

mpv window (video, IPC seek/pause) + PyQtGraph overlay/dual waveforms with playhead, timeline seek synced to mpv, A/B show toggles, per-track volume, stereo or solo playback. Full mode resolves output stream indices as A: sel_a, B: n_a_streams + sel_b using .stream_state.

Utility: audio language tagging

.venv/bin/python rename_audio.py movie.mkv            # dry run
.venv/bin/python rename_audio.py movie.mkv --apply [--stream 0,2] [--model base] [--segment 60] [--start 0]

Probes audio streams, runs Whisper language detection (confidence + top-5), then rewrites MKV language/title metadata with stream copy (no re-encode).

Files

File	Purpose
config.toml	Single source of truth: inputs, window, audio, sync, video settings
Makefile	install*, extract, extract-loud, proxy, picker, ast, vst, sync LAG=…, verify LAG=…, clean
extract.py	Lossless clip extraction (FFV1 + PCM + proxy), loudnorm, stream picker, start validation
picker.py	Qt picker GUI: waveforms, thumbnails, start/duration → config.toml
sync_audio.py	Core lib: load_wav, correlate (GCC-PHAT torch/CPU), lag_to_seconds, device_info
sync_audio_test.py	AST GUI: waveforms + xcorr + tuning + playback + background apply + verify launcher
sync_video_test.py	VST lab: 7 video methods, heatmap, scrub, chunk-fit, headless --no-gui reports
apply_sync.py	Mux A video + A audio + delayed B audio with --lag, --progress protocol
verify_sync.py	mpv + dual-waveform verify player (short preview / full synced MKV)
rename_audio.py	Whisper language detect + MKV metadata rewrite (dry-run by default)
waveform_cache.py	cache/waveforms/*.npy disk cache for loaded WAVs (mtime-keyed)
extracted/	Generated clips (*_video.mkv, *_audio.wav, *_proxy.mkv); contents gitignored
cache/	Waveform + decoded-video caches; gitignored
output/	Default apply_sync destination ({stem}_synced.mkv); gitignored via cache/-style flow, create as needed

Generated/ignored: extracted/*.{mkv,wav,mp3}, cache/, .stream_state, .venv/, __pycache__/, output/ (create per sync.out_dir).

Notes

• Content-to-content sync: container start_time/edit-list mux delay is ignored; alignment is achieved by padding silence on mux.
• Analysis codecs are deliberately CPU-only (FFV1/PCM have no GPU implementation; VAAPI/AMF are lossy delivery codecs, wrong for measurement).
• PAL-speedup cases (25 fps vs 23.976) need audio tempo compensation beyond frame-rate normalization — extract.py warns when it detects this.
• LC_NUMERIC=C is set for Qt/ffmpeg decimal parsing in ast/vst/verify.