Recreating carykh's automatic lip-syncing tool

Introduction

Carrykh, a popular YouTube channel with significantly more subscribers than the author, recently developed a tool to automate lip-sync animation for his avatar. Inspired by this, the author created a program called Magic Video Tool to automatically align slides of a slideshow to a transcript using the program Gentle. This article explains the features of the tool, the process of creating videos with it, and the various components involved in producing videos without traditional video editing software.

In this tool, the annotated transcript, voiceover recording, and slideshow with precise timing are inputted to generate a video. The transcript is cleaned, sent to Gentle for analysis, and a timetable of words and phonemes is produced. Frames for the video are created based on annotations in the transcript, including commands for slides, B-roll footage, and mood changes. Mouth shapes and movements are synchronized with the speech using ffmpeg and image processing. Soundtrack creation involves specifying tags for background music and sound effects, all done without traditional video editing software.

Keywords

Automated lip-sync, Magic Video Tool, Gentle program, Annotated transcript, Frame creation, Mouth synchronization, Soundtrack creation, Video production automation

FAQ

1. What inspired the creation of the Magic Video Tool? Carrykh's automatic lip-sync tool inspired the author to develop the Magic Video Tool for automating video production processes.

2. How does the Magic Video Tool synchronize slides with transcripts? The tool uses annotations in the transcript to signal when to transition to the next slide, add B-roll footage, or change the avatar's mood during video production.

3. What role does Gentle play in the video production process? Gentle is used to analyze the voiceover recording and transcript to provide timings for words and phonemes, aiding in the creation of synchronized animations for the avatar.