anime_dub_sync/readme.md

anime_dub_sync

Transfer a dubbed audio track (e.g. German) from a low-quality source file onto a high-quality video file, with automatic A/V offset detection and verification.

Only use this with sources you legally own. All paths and titles in this repo are generic placeholders — do not commit real media filenames, absolute local paths, or release-group names.

Requirements

System tools (must be on PATH):

• ffmpeg + ffprobe
• mkvmerge (from MKVToolNix)

Python (>= 3.10):

• opencv-python, scipy, numpy, soundfile, tqdn
• see requirements.txt

Install:

pip install -r requirements.txt

Repository layout

main.py                  # batch + single-episode orchestration
utils.py                 # subprocess helper
audio/
  __init__.py
  audio_utils.py         # ffmpeg extraction, concat, shifting, mux helpers
  audio_cross_correlation.py  # intro/outro audio offset measurement
video/
  __init__.py
  video_utils.py         # ffprobe wrapper
  scene_change_cross_correlation.py  # primary video offset detection
  temporal_cross_correlation.py      # alternative/legacy motion-signal method
mkv/
  __init__.py
  chapter.py             # chapter parsing, intro/outro candidate search
  intro_outro.py         # final sync verification
  check_intro_outro.py   # CLI wrapper for verification

How it works

1. Video offset detection (video/)

video/scene_change_cross_correlation.py:70 — corelate_videos(vid1, vid2, fps):

1. get_scene_change_signal decodes each video with OpenCV at 160x90 grayscale, computes mean(diff^2) per frame. Cuts produce large spikes.
2. resample_signal resamples both signals to a common FPS with scipy.signal.resample.
3. Both signals are z-normalized, then scipy.signal.correlate(mode='full') finds the lag with peak correlation.
4. Lag in samples / FPS = offset in seconds. Positive means vid1 starts after vid2.

video/video_utils.py:14 — probe_video(path) returns (fps, duration, codec, start_time) via ffprobe -show_streams -show_format.

video/temporal_cross_correlation.py:52 — find_temporal_offset is a similar motion-signal correlator. It is exported but not used by main.py; the active path is corelate_videos.

2. Audio splicing (audio/)

audio/audio_utils.py:

• extract_audio_segment_file(video, start, end, out) — extracts 0:a:0 with ffmpeg to stereo AAC .mka.
• extract_audio_segment_tmp(file, (start, duration) | start, end, stream) — extracts a stream to a temp 44.1 kHz PCM WAV for analysis.
• get_audio_duration(file) — parses DURATION tag via ffprobe.
• stick_together_audio(left, right, out) — concatenates with the ffmpeg concat demuxer (-c copy, no re-encode).
• add_silence(file, offset) — adelay (offset > 0, silence at start) or apad (offset < 0, silence at end), re-encoded to Opus.
• replace_with_shifted_mka(mkv, mka, audioline, offset) — physically shifts with adelay / atrim+asetpts, re-encodes to Opus 160k, replaces the track with mkvmerge, tags it ger / German (Opus Stereo).
• get_audio_streams, get_track_info, shift_audio — stream enumeration and in-place shift helpers.

audio/audio_cross_correlation.py:39 — get_audio_offset(file1, file2, time): decodes to mono 44.1 kHz WAV, mean-normalizes, scipy.signal.correlate, returns lag / sample-rate in seconds.

3. Intro/outro verification (mkv/)

mkv/chapter.py:12 — get_chapters(file) lists (start, duration) via ffprobe -show_chapters.

mkv/chapter.py:35 — get_intro_outro(file, length=90.0) keeps chapters with duration within [length/2, length*2]. Expects exactly 2 (intro + outro).

mkv/intro_outro.py:8 — check_intro_outro_sync(file, length, uncertainty, max_offset):

1. Trims uncertainty (default 10%) off each intro/outro edge.
2. Extracts first and last audio streams for intro and outro.
3. Audio-correlates intro pair and outro pair.
4. Returns (True, mean_offset, last_stream) if |mean(outro_offsets) - mean(intro_offsets)| < max_offset (default 0.1 s), else (False, None, None).

4. Orchestration (main.py)

main.py:18 — mux_audio_into_output() runs mkvmerge to add the synced .mka as a German track.

main.py:37 — process_one(German_Video, Quality_Video, output_dir, offset=None) runs 9 tqdm steps:

1. Probe video — probe both inputs.
2. Compute offset — corelate_videos, or use supplied offset.
3. Extract pre beginning — head of quality audio before dub starts (skipped if < 0.05 s).
4. Extract german audio — main dubbed segment.
5. Extract post end — tail of quality audio (skipped if < 0.5 s, appended only if > MIN_END_DUR = 2.5 s).
6. Combine audio — concat head + dub + tail; falls back to add_silence.
7. Mux — mkvmerge into output_dir/<quality_stem>.mkv.
8. Cleanup — removes temp _original_audio*.mka.
9. Final check — check_intro_outro_sync; sub-frame residual is fixed with replace_with_shifted_mka if |offset| > 1/fps. Failures are renamed to [FAILED]_*.mkv, chapterless files to [NO CHAPTER]_*.mkv.

Usage

Single episode (output dir as 3rd arg, else ../output_dir):

python main.py <dubbed_source.mkv> <high_quality_source.mkv> [output_dir]
python main.py dubbed_ep01.mkv quality_ep01.mkv ./output_dir

With explicit offset in seconds (bypass video correlation), pass it as 4th argument handling inside process_one — currently CLI uses auto-detect; edit or call the function directly for manual offsets.

Batch: edit main() in main.py:225 (output_dir, dubbed_dir, quality_dir, episode range), then:

python main.py

Verify an existing file without re-muxing:

python mkv/check_intro_outro.py ./output_dir/episode.mkv

Configuration

• main.py:16 MIN_END_DUR = 2.5 — minimum tail length to append.
• main.py:14 DEBUG = False — enable ffmpeg segment logging.
• mkv/intro_outro.py:8 length=90.0, uncertainty=0.1, max_offset=0.1 — expected OP/ED length and tolerances.

Notes and limitations

• Needs chapter marks to run the final check; files without them are kept with a [NO CHAPTER]_ prefix and return False.
• Scene-change correlation assumes both files are the same episode cut. Different cuts, missing scenes, or very dark/flat video degrade the peak.
• utils.py:4 run(cmd) prints combined stdout on failure, then re-raises CalledProcessError.
• Do not commit .mkv / .mka media or absolute filesystem paths. Keep output_dir/ and temp audio out of git.