I Spent 150h+ Making My Own Anime With AI

Introduction

After weeks of research and over 150 hours of production work, I finally arrived at my dream project: creating my own anime. This creative journey began long before I even thought about my unique style and story; I set out to learn the fundamentals of classical animation.

Rotoscoping: The Key Technique

To kickstart my project, I quickly discovered that rotoscoping could provide the foundation I needed. This technique is now possible to replicate using AI image-to-image tools, primarily utilizing Stable Diffusion for its fine-tuning capabilities. With Stable Diffusion, I could apply various styles derived from a model to a given image. I even experimented with transforming a low-quality snapshot of myself into a striking anime style. However, I soon encountered a significant challenge inherent to this technology.

Understanding Image Diffusion

So how does image diffusion work? In simple terms, diffusion models reference a vast array of visual elements, which can be quite specific (like a single anime model) or more generalized. The process involves taking an image, applying noise to it, and allowing the AI to recreate what the image should look like, based on both its model and your specific descriptions (known as prompts). A delicate balance exists between the noise level and the retention of the original image — too much noise leads away from realism and toward stylistic representation while too little keeps the image too close to its source.

The Challenge of Consistency

One major obstacle in creating an animation is maintaining consistency across frames. In an anime, characters may move while the background remains static; however, processing input videos into an image sequence, adding noise to each frame, results in slight variations which pose a challenge for consistency. Hence, finding a method to ensure that the character looks the same in different frames is critical to achieving a visually pleasing outcome.

Story Development and Concept

I also needed a story to animate. Inspiration struck as I thought of "The Matrix." I decided to adapt the iconic red pill and blue pill scene with a twist. Instead of Neo, I envisioned myself as the character who would consume both pills, revealing that I had created The Matrix. While I was excited about my concept, I soon realized that the intricacies of animation posed their own set of problems.

Filming in Thailand

Living in Thailand with limited resources made filming challenging. I decided to use my apartment as the backdrop and film against a neutral wardrobe background. With just a pancake lens and a single light source for the shoot, I got creative to make the most of my available space. Having to act as both characters in the scene also pushed my limits as an aspiring animator.

Problem-Solving and Experimentation

The process revealed to be a test of persistence and adaptability. My initial shoots were quick, at times rushed, but I remained determined to solve the issues of consistency. I stumbled across a video by Corridor Crew that inspired a new workflow, allowing me to train my own models for character consistency. Utilizing Google Colab, I trained several specialized models with a focus on my character’s appearance.

Advanced Techniques for Animation

Using a tool called ControlNet, I learned that I could improve consistency by analyzing frames using various neural network structures. Although there were challenges with skin tones, I adapted and improved my prompts until I achieved satisfactory results. Post-processing in DaVinci Resolve played a vital role in enhancing visual quality, reducing flicker, and improving overall aesthetics.

Voiceovers and Final Assembly

The final stages involved adding voiceovers for my characters, which I initially found uncomfortable. However, using tools like Metavoice and Adobe's audio enhancer, I was able to improve the voice quality. With all elements in place, I harmonized my animation efforts in Final Cut, combining multiple backgrounds and effects to construct my final 60-second anime.

After 150 hours of dedicated effort, I created something that fulfilled my vision as an amateur animator, blending storytelling with cutting-edge AI technology.

Keyword

• Anime
• AI
• Rotoscoping
• Image Diffusion
• Consistency
• The Matrix
• Voiceovers
• Post-processing
• Google Colab
• ControlNet

FAQ

Q: How did you create your anime?
A: I used AI tools, specifically Stable Diffusion and ControlNet, to generate animated sequences, focusing on consistent character portrayal and integrating various animation techniques.

Q: What challenges did you face during the process?
A: I encountered obstacles with maintaining consistency in character appearance, achieving realistic mouth movements, and dealing with limited filming conditions.

Q: How long did it take to complete the project?
A: The entire project took over 150 hours to develop, from initial research to final production.

Q: What inspired your anime's story?
A: I drew inspiration from "The Matrix," reimagining the iconic pill scene with myself as the protagonist.

Q: What software did you use for editing?
A: I used DaVinci Resolve for post-processing and Final Cut for assembling the final animation.