I Tested Adobe's New AI Video Model and Here's What Happened

Film & Animation

Introduction

Adobe recently unveiled its new Firefly video model, which promises to revolutionize the way we generate video clips using text prompts and images. As someone who has early access to this groundbreaking tool, I’m excited to share my experience and what I've learned so far.

Earlier this year, I created a short video exploring the introduction of generative AI in Adobe Premiere Pro. Features like object removal, object edition, clip extensions, and generating b-roll were all teased. At the recent IBC conference, I had the chance to speak with the Adobe Firefly video team, who revealed that they are working on incorporating generative AI functionalities into the Premiere Pro beta by the end of this year.

What is Generative Extend?

According to Adobe's blog, the generative extend feature allows users to fill in gaps in footage, smooth out transitions, and hold shots longer for precise editing. This will significantly enhance the editing process, particularly for those who often deal with incomplete footage or require supplementary clips.

Use Cases for Adobe Firefly Video Model

Adobe Firefly video is designed for specific applications rather than creating an entire film. Here are some notable use cases highlighted by Adobe:

  1. Generating Missing B-roll: The model can create supplementary shots for your projects. For instance, generating an aerial shot of a desert or a point-of-view shot of a magnifying glass focusing on a dandelion.

  2. Atmospheric Elements: Use Firefly to add effects, such as overlays like flames, while recognizing that users still need expertise in tools like After Effects for advanced integration.

  3. Ideation for Styles of Animation: The tool can create various animation styles, including detailed line art animations and unique text effects, like water splashes spelling out words.

  4. Adding Motion to Still Images: Users can upload images and instruct the model to add movement, helping to visualize ideas creatively.

Mock Commercial Experiment

To better grasp the capabilities of the Adobe Firefly video model, I decided to create a mock commercial for a fictional tennis ball company. The idea was to illustrate how the model could ideate visual shots rather than actually produce a polished video.

Inside the Firefly video generation interface, I experimented with various camera angles and motions. By prompting the model with detailed descriptions, like "a swimming pool full of hundreds of tennis balls," I generated unique shots in under two minutes each.

Throughout the process, I generated a variety of shots, including:

  • A Boeing 747 painted with tennis balls.
  • A highway filled with tennis balls instead of cars.
  • A rain scene with tiny tennis balls falling from the sky.

Despite some inaccuracies, I was able to compile a selection of clips that could serve as a 50-second commercial.

Music and Sound Design

After generating visuals, I turned to Lisa Musicbed for an upbeat track to accompany the commercial. Using their AI feature, I found "How Much is Much," which perfectly complemented the playful theme. To fit the video's length, I creatively edited the track using Premiere Pro’s remix tool and added sound effects for believability.

Conclusion

This whole process took about five hours, allowing me to translate my ideas into visuals effectively using Adobe Firefly video. While Firefly’s generative AI isn't perfect and still requires human intervention for final edits, it opens new avenues for ideation and creativity in video production.

Adobe Firefly is still in its early access phase, so there's a waitlist for those eager to try it out. If you're interested in leveraging generative AI for your video projects, I highly recommend signing up to explore the future of video editing.

Keywords

  • Adobe Firefly
  • Generative AI
  • Video Model
  • Premiere Pro
  • B-roll
  • Animation Styles
  • Musicbed
  • Commercial Production

FAQ

What is Adobe Firefly? Adobe Firefly is a generative AI video model that allows users to create video clips using text prompts and images.

What functionalities will be available in Premiere Pro? Generative AI features such as object removal, b-roll generation, and clip extensions are expected to be included by the end of this year.

Can Firefly generate an entire film? No, Adobe Firefly is designed to generate specific clips or ideas rather than complete films. It excels in ideation and filling gaps in existing footage.

How long does each clip generation take? Each generation typically takes less than two minutes to complete.

Is there a waitlist for Adobe Firefly? Yes, interested users can join a waitlist to gain access to the Firefly video model.