AI Explained - How do face swaps work?

Sure, let’s follow these steps:

Step 1: Write the Article in Markdown Syntax

Introduction

Face swaps are bizarrely funny, weirdly fascinating, and even a bit creepy, but they're incredibly popular on the web. So, how do face swaps actually work?

Hi, I'm Alvin, a computer science PhD student and lecturer at UC Berkeley. In this article, I'll explain how face swaps work, starting with the basic version and then adding advanced features to make professional, meme-worthy face swaps.

Basic Face Swap

Step 1: Detect Faces

First, detect all the faces in the image and draw boxes around each face.

Step 2: Switch Boxes

Next, switch the contents of these boxes between faces. However, the boxes may not match in size, so you'll need to resize both boxes to match the original sizes. And there you have it—a basic face swap.

Recap

• Detect all faces
• Swap pixels for those faces
• Resize faces to fit

Advanced Face Swap

To upgrade our basic face swap, we'll add three advanced techniques:

Hack 1: Use Key Points Instead of Boxes

Key points relate to notable parts of the face, such as the nose bridge, eyebrow area, and dimples.

• Step 1: Detect key points on the face instead of just boxing them.
• Step 2: Warp the face using these key points.
  • To visualize this, consider the key points around a person's left eye. These points form a triangle. In face swapping, we warp the triangle from one face to fit the corresponding triangle on another.
  • This warp handles resizing automatically, so no additional resizing is required.

Recap

• Detect key points.
• Warp triangles formed by these key points onto the corresponding triangles on the other face.

Hack 2: Smooth Colors Along the Edges

Edges can be visible after a face swap. To address this, we smooth the colors at these edges using a technique called Gradient Domain Fusion.

Consider a small 5x5 pixel image:

• If we see an edge (denoted as red), it means there’s a significant difference between pixel values at the edge.
• By setting this difference to zero while keeping other differences the same, we effectively remove the perceived edge.

Recap

• Detect key points.
• Warp key points.
• Smooth the edges using Gradient Domain Fusion.

Hack 3: Recolor the New Face

Sometimes, the swapped face stands out because its color doesn't match the rest of the image. To fix this, we match the average color intensities.

• Compare the original face and the new pasted face.
• Calculate their average numeric representations.
• Adjust the new face’s values to match the average intensity of the original face.

Recap

• Detect key points.
• Warp key points.
• Smooth edges.
• Match color averages.

And that's it! This three-step process—detecting key points, warping and smoothing edges, and matching color averages—produces the face swaps you've seen all over the internet.

Here are some examples:

• Before and after the face swap
• Focus on details like eyeglasses, before and after

To learn more and build your own face-swapping tool, check out my computer vision course on Skillshare (link in the description).

Thanks for reading!

Step 2: Extract Keywords

Keywords

• Face Swap
• Key Points
• Warping
• Gradient Domain Fusion
• Recoloring
• UC Berkeley
• AI
• Computer Vision
• Image Processing
• Skillshare Course

Step 3: Generate FAQs

FAQ

Q: What is the first step in performing a basic face swap? A: The first step is to detect faces and draw boxes around each face.

Q: How do you handle different-sized face boxes in a basic face swap? A: Resize both boxes to match the original sizes before swapping the contents.

Q: What are key points in the context of face swaps? A: Key points correspond to meaningful parts of the face like the nose bridge, area above the eyebrow, and dimples.

Q: How does warping work in an advanced face swap? A: Warping the face involves using triangles formed by key points and mapping these triangles from one face to another.

Q: What technique is used to smooth the edges in face swaps? A: Gradient Domain Fusion is used to smooth the colors along the edges.

Q: How do you match the color of the swapped face with the rest of the image? A: By adjusting the new face’s color values to match the average intensity of the original face.

Q: Where can I learn more about building a face-swapping tool? A: You can learn more from the computer vision course on Skillshare linked in the description of the article.

This concludes the three steps.