Next level AI Filmmaking: Turn your storyboard sketches into MOVIES!

Introduction

Lady Gwendaline, clad in shining armor that reflected the scorching desert sun, trudged through the sand towards the crumbling castle. This has to be one of the most fun workflows I've ever created. You can basically just take any sketch and have the AI turn it into a final cinematic image for free, and just look at these results! Isn't that cool? It also works with multiple characters in the image, and you can also combine it with my free character model sheet workflow for consistent characters and styles. It's an amazing and fun way to create pre-visualization for movie productions, precise concept art, or to generate panels for a graphic novel.

Introduction

For a full demonstration of the workflow's potential, I combined it with a bunch of other new AI tools to create a full AI short film about the endless fight between gnomes and knights. Make sure to watch to the end for the full movie and process.

Creating Cinematic Images

The easiest way to use this workflow is to just create a black and white sketch of your shot. I'm using Microsoft Paint with my mouse to prove that you don't need expensive software or even a drawing tablet. Let's say we want to create an establishing shot for our movie: our knight is standing in the center of the frame, overlooking a vast desert with a castle ruin on a distant mountain. Now, I can use this image to guide my composition and add a prompt for the style.

You can easily try out variations of this shot. Change the time of day, or perhaps set the movie in a lush flower field rather than a desert wasteland. When creating a movie or graphic novel, multiple characters in the same image are often necessary. Here, for example, we have our knight talking to a gnome. Stable Diffusion mixes both characters when a prompt is added to the full image, but this can be fixed by using different colors.

Drawing a scene in black, one character in red, and another in green allows the workflow to create masks for each character. You can then create individual prompts for these regions. This allows for adding character reference images for consistency across shots, and a reference image for the whole style of the image to ensure coherence.

Setting Up the Workflow

To set up this workflow, start by installing ComfyUI, an insanely powerful interface for stable diffusion. Follow my free step-by-step guide. Download ComfyUI from the official GitHub page, extract the folder using WinRAR, and install it. You also need to install Git if you haven't already. Download the standalone version and follow the installation steps. Download the Comfy Manager from GitHub, place it inside your ComfyUI directory, and install it.

Next, download the storyboarding simple workflow and import it into ComfyUI. Install missing custom nodes via the manager. For models, go to civitai.com and save the required models in the ComfyUI directory under respective folders. Refresh the settings to ensure all models are loaded.

Generating Images

Let's quickly create another shot using Paint. Set the scene dimensions, draw your desired composition, save it as a PNG, and drag and drop it into ComfyUI. Adjust the prompt to refine the image.

Multiple characters in a shot can be managed by using different colors to segment them. Make necessary adjustments using character reference images for consistency. Problems with prompt fighting can be solved by using neutral or emotion-specific images generated from my Consistent Character Model Sheet Workflow.

Advanced Workflow and Tools

The advanced version of this workflow includes additional features like style reference, an upscaler, and face detailers. Style reference can adjust color schemes and overall styles based on input images. Upscalers add more detail to the final image. Face detailers can fix character faces and change emotions, important for dialogue sequences.

To test the feasibility of creating a short film, I used this workflow and additional AI tools such as Luma Labs' Dream Machine, Eleven Labs for voice-over and sound effects, and Yudd for music. The results were impressive, making the entire process enjoyable and efficient.

Below is the story of Lady Gwendaline, which I created using the workflow:

Short Film: Lady Gwendaline and the Mischievous Gnomes

Lady Gwendaline, clad in shining armor that reflected the scorching desert sun, trudged through the sand towards the crumbling castle. Her mission was to vanquish the mischievous gnomes once and for all. For years, these pint-sized pranksters had terrorized her village with their antics: sand in the bread, cacti in the houses, and camels mysteriously painted purple. Enough was enough.

As she reached the castle gates, Gwendoline drew her sword and bellowed, "Come out, you diminutive devils! Your reign of ridiculousness ends now!" To her surprise, the gates creaked open, revealing a single gnome in a tiny three-piece suit. He cleared his throat and spoke in a squeaky voice, "Good day, madam. I'm Finn Wicket, the gnome's legal representative. How may I assist you?"

Gwendoline blinked, lowering her sword. "I... I'm here to stop your tricks." Finn Wicket nodded sagely. "Ah yes, well before we proceed, I must inform you that we gnomes have recently unionized. Any grievances must be filed through the proper channels. If you'll just fill these out in triplicate, we can begin the arbitration process."

Gwendoline stared at the forms, then at the gnome. "Fine," she grumbled, reaching for the pen. "But this had better not be another prank." As soon as her fingers touched the pen, it exploded, showering her in a fountain of vibrant purple ink. The legal representative burst into high-pitched laughter. "Oh, it's always another prank with us, my dear."

The castle erupted with the sound of hundreds of gnomes laughing uproariously. Gwendoline, still stunned and now very purple, couldn't help but crack a smile. She realized that sometimes the best way to deal with silly goings-on was to learn to laugh along, even if it meant looking like a walking grape.

Conclusion

Storyboarding is a crucial step in filmmaking as it allows for experimentation with compositions and camera angles. This workflow helps to convey ideas more precisely than ever before. Share your creations on our community Discord or tag me in your work. A huge thank you to my Patreon supporters who make the testing and development of these workflows possible.

Keywords

• AI filmmaking
• storyboard sketches
• ComfyUI
• stable diffusion
• pre-visualization
• character consistency
• Luma Labs' Dream Machine
• Eleven Labs
• Yudd
• Lady Gwendaline
• gnomes

FAQ

Q: Do I need expensive software to use this workflow?

A: No, you can use free tools like Microsoft Paint and ComfyUI to create amazing results.

Q: Can this workflow manage multiple characters in a single image?

A: Yes, by using different colors to segment characters, you can create individual prompts for each character.

Q: How do I maintain character consistency across shots?

A: Use character reference images and the Consistent Character Model Sheet Workflow to ensure consistency.

Q: What additional features are in the advanced workflow?

A: The advanced workflow includes style reference, an upscaler, and face detailers.

Q: What additional AI tools can enhance the filmmaking process?

A: Luma Labs' Dream Machine, Eleven Labs for voice-over and sound effects, and Yudd for music can greatly enhance your project.

Q: Is it possible to create a full short film using this workflow?

A: Yes, I successfully created a short film about Lady Gwendaline and the mischievous gnomes using this workflow and additional AI tools.