Today will be learning about Frame Interpolation in AnimateDiff. Frame Interpolation is an amazing technique that can transform your photos into smooth animations or your video footages into smooth, high FPS videos. I have always enjoyed making GIFs from my photos by aligning and merging them together in Photoshop and After Effects, but they never looked satisfying. They were always coarse and jerky, lacking the smoothness and cohesiveness of a video. I dreamed of being able to fill in the gaps between the frames and make them more expressive and artistic.

As an art photographer, I felt that would changed how we create videos for social media as photographers who just want to share art and not How-To-Videos. That’s exactly what Frame Interpolation does. It uses advanced algorithms to interpolate the missing frames and create a fluid motion with all the details preserved.

Create Smooth Videos with Frame Interpolation

In this blog, I will share with you everything I know about Frame Interpolation, with a special focus on how to use the AnimateDiff extension in Automatic1111. This extension allows you to create stunning animations from your photos with just a few clicks. However, before using this extension, you will need to install Deforum first, Deforum is an extension for Automatic1111’s web ui that adds more features and controls to AnimateDiff.

One of these features is frame interpolation, which uses an external tool called RIFE to generate the extra frames. Deforum provides the interface and the workflow to use RIFE with AnimateDiff. Without Deforum, you would have to manually run RIFE on the output of AnimateDiff, which would be more complicated and time-consuming.

I will also show you an awesome AI app that I have been using to produce these interpolated frames. This app uses deep learning to generate realistic and high-quality frames from your photos. You can use this app to create amazing animations that look like paintings come to life.

---

To change the speed and smoothness of your AnimateDiff video, you need to know how these three settings work: Number of Frames, FPS, and Interp X.

Number of Frames:	This is the number of frames that AnimateDiff will generate for each output. The higher the number, the longer the animation, but also the more VRAM and time required.
FPS (Frames Per Second):	This is the frame rate of the output video, or how many frames are displayed per second. The higher the FPS, the smoother the motion, but also the faster the playback speed.
Interp X (Frame Interpolation):	This is the parameter that determines how many extra frames are added between each original frame by frame interpolation. The higher the InterpX, the smoother the motion, but also the slower the playback speed.

The formula for calculating the output duration and frame rate with frame interpolation is:

Output Duration:	(Number of Frames – 1) / FPS
Output Frame Rate:	FPS * (InterpX + 1)

So, the settings you can use are:
Number of Frames	16
FPS	8
Interp X	2

For this tutorial, I’ll be using the “Ether Real Mix” model by gamerdan69 on Civitai.com. It’s a personal favorite of mine. This model requires the ‘kl-f8-anime2 VAE‘. and you can download both the VAE and Model below:

Prompt:
beautiful woman, dancers, dancing in puddles, splashes, dramatic clouds, fluid, dynamic, perfect shades, masterpiece, best quality, high-resolution

Negative Prompt:
NSFW, (worst quality, low quality:1.3)

Sampling Method: DPM++ 2m Karras	Sampling Method: DPM++ 2m Karras
Width height: 512×768	CFG Scale: 9
Clip Skip: 2	Seed: Don’t Matter.

How to Smooth Your Animation with Deforum and AnimateDiff:

• Install Deforum.

  (You Need this extension for frame interpolation to work)

• Click ‘Enable AnimateDiff’.

• Click on FILM to smooth your animation.

• Change Interp X to add more or less frames.

  a. More frames = smoother but slower animation.

  b. Less frames = faster but choppier animation.

Use a video or images as inputs for AnimateDiff:

• Use ControlNet to add different information types.

  (depth, edge, normal, pose) to your inputs.

• You can use images, gifs, or videos as inputs for ControlNet.

• Use multi-controlnet to mix different ControlNet models for one generation.

• Experiment with different Sampling Methods and adjust them as you like.

• Be careful with using too many prompts, especially negative ones, as they may ruin your animation.

1. Generating the Initial Video:

• You want a 4-second video at 8 frames per second (FPS).

• So you need 4seconds x 8FPS = 32 frames.

• Number of Frames: 32

• FPS: 8

• Click ‘Enable AnimateDiff’.

• Click ‘Generate’.

However, this initial video appears janky and lacks smoothness.

2. Adding Frame Interpolation (FILM):

• Click on FILM and use Interp X to smooth your video.

• Interp X is 10 by default.

  It adds 10 extra frames for each original frame.

• After generating the video, you will have 311 frames instead of 32.

  You get about 9.7 interpolated frames for each original frame.

• But the video is very slow and choppy at 8 FPS.

  You need to increase the FPS to make it faster and smoother.

3. Adjusting FPS for Smoother Playback:

• To remedy the slow motion effect,

  change the video’s frame rate to 24 FPS.

• The video shortens to 12 seconds,

  but appears much smoother due to the higher frame rate, still preserving the 311 interpolated frames created by enabling FILM.

4. Boosting Frame Rate for Real-Time Speed:

• Now increase the FPS to 70.

  You maintain the 311 interpolated frames while bringing the video back to its original 4-second duration.

• This allows for a comparison between the original video without FILM and the new one with FILM,

  highlighting the significantly smoother playback achieved with a higher frame rate.

Based on what we did in step 4, the equation is: Frames = FPS x Duration x (Interp X + 1)

• Frames is the number of frames in the video

• FPS is the frames per second of the video

• Duration is the length of the video in seconds

• Interp X is the number of interpolated frames added between each original frame.

This equation tells you how frame interpolation and FPS work together. Frame interpolation makes more frames for your video, so it looks smoother. FPS makes your video play faster or slower. If you want a smooth and fast video, you need to change both Interp X and FPS. Interp X is how many frames you add for each original frame. FPS is how many frames you show in one second.

You can use FILM to turn on frame interpolation and change Interp X. You can also change FPS to make your video faster or slower. Different software or tools may have different settings, so you may need to try different values to get the best result.

Frame interpolation adds more frames between the original ones. This makes the video longer and smoother. But if you keep the same FPS, the video will play slower and look choppy. To make the video faster and smoother, you need to increase the FPS. This will shorten the video and use all the interpolated frames. But if you increase the FPS too much, the video will look choppy again. This is because you don’t have enough interpolated frames for the high FPS. So you need to balance the frame interpolation and the FPS to get a smooth video.

I hope this helps you understand better. Here is a summary of what I explained:

• Frame interpolation = more frames = longer and smoother video

• Same FPS = slower and choppier video

• Higher FPS = faster and smoother video

• Too high FPS = not enough frames = choppier video

• Balance frame interpolation and FPS = smooth video

AnimateDiff is unsuitable for interpolating old videos. It is difficult to use video sources and multiple ControlNets to reproduce the original. I attempted many times, but I could not interpolate the original image. It only generated a new image that resembled, but did not match, the original. You also need to use prompts and negative prompts. But how can you determine what to use if you are unaware of the video source? And there is no denoising parameter to assist you.

Therefore, I think this is an ineffective method. If you have succeeded, please demonstrate it to me. This method is appropriate for creating new creative videos, not interpolating old videos.

I prefer using Flow Frames. It is simple and does real interpolation. It is quick and effortless. It performs better than AnimateDiff with ControlNet. It takes longer and produces lower quality results. The results are impressive!

Download Flowframes for Interpolation

An application for effortless frame interpolation with Flowframes is available for download. You can access it by clicking the button below and support the developer.

How to Use Flowframes

Flowframes is very easy to use and is self explanatory, so this will be a short guide on how to get started.

Here are the steps to use Flowframes:

• Choose the Interpolation AI.

  You can use the default RIFE-CUDA/Pytorch if you have an Nvidia GPU.

• Choose the Input Video.

  You can use a folder of images or a video file.

• Choose the Output Directory.

  This is where your files will be saved.

• Choose the Output FPS and Speed.

  You can use the multiplier to change the FPS. For example, 32 x 5 = 160 FPS. You can also change the Speed to Normal or Slow Mo.

• Choose the AI Model.

  This is the model that will interpolate your frames. You can use the recommended one.

• Choose the Output Mode and Encoding Settings.

  This is how your video will be saved and encoded. You can choose the format and quality.

• Click on Interpolate and wait.

  It will be very fast.

• Enjoy your smooth video!

---