AnimateDiff Animation Control: How to Prompt Travel in Automatic111
Learn how to Prompt Travel! Prompt Travel in Automatic1111 might not be immediately evident, as it requires a certain formula to implement in your Text-to-Image Prompt. New users may find it challenging to grasp right away. To aid in this process, I’ve developed a guide that provides step-by-step instructions on how to use Prompt Travel in Automatic1111.
As the application improves, so will this guide. At this stage, it isn’t very easy to get right and requires trial and error, testing it with different prompts and different models and different frames and timing. I’ve spent hours upon days, and I find that the simpler the prompt, the easier the results, but in this guide, we will do both. Keep it simple and make our lives a bit harder by not keeping it simple. Leg’s begin.
Table of Contents
Learn How to Prompt Travel – How it Works in Automatic1111
Prompt Travel in AnimateDiff allows you to control the change of your animations over time, enabling dynamic animations with shifting characteristics and actions. Frame numbers are linked to the quantity of generated frames, and the highlighted portion denotes the precise initiation of prompt interpolation and changes. This transition involves a gradual tapering, ensuring that prompts introduced at the specified frame number initiate earlier and extend further, creating a smooth transition between each interpolation.
When you install AnimateDiff in A1111, the Prompt Traveling features may not be immediately visible. Instead, it operates through the Prompt box, where you input the required prompt interpolations. This process begins with a Head Prompt, proceeds with Prompt Interpolation, and concludes with a Tail Prompt. These are done through separate lines in the prompts. While the inclusion of both the Head and Tail Prompts is optional, they contribute significantly to a more comprehensive scene description.
In the following section, we will dissect the structure of prompts into four essential components:
| 1. The Head | 2. Prompt Interpolation | 3. Tail Prompt | 4. Frame Number |
1. Head Prompt
| The Head Prompt serves as the starting point for your animation, allowing you to establish the initial state and style. In this section, you can provide one or more initial prompts to set the stage for your animation. For instance, a Head Prompt could include descriptors like “cute blonde woman with braids in a dark theme red room,” influencing the animation’s starting appearance. It’s basically writing a prompt for an image but tailored for animation. |
2. Prompt Interpolation
| (Second and Third Lines): The Prompt Interpolation section is at the backbone of Prompt Travel, designed to structure your animation’s transformation over time. Structured as “frame number: prompt,” these lines define when and how your animation evolves. The frame number denotes the animation’s position, starting at 0 for the first frame. Here, you can specify different prompts for various frames, facilitating precise control over your animation’s progression. Notably, prompts may include weights such as “(x:1.3)” to influence the animation’s characteristics at specific frames. |
3. Tail Prompt
| (Last Line, Optional): Much like the Head Prompt, the Tail Prompt offers the opportunity to influence your animation’s content and style, particularly as it nears its conclusion. This section enables you to guide the final moments of your animation with one or more prompts, creating an ending. By tactically combining the Head Prompt, Prompt Interpolation, and Tail Prompt within the AnimateDiff framework, you can create dynamic animations that evolve over time, conveying intricate emotions, actions, and narratives. |
4. Frame Number
| The Frame Number corresponds to the number of frames in the animation rather than the frames per second (FPS). For example, if you have a 32-frame animation, you can specify the Frame Number from 1 to 32. You can experiment by going beyond this range to test your boundaries. To understand the relationship between FPS and the Number of Frames, it’s important to note that FPS multiplied by the video’s length equals the total Number of Frames. For instance, if you’re creating a 24 FPS video lasting 4 seconds, you’ll need 96 frames (24 FPS x 4 seconds = 96 frames). In this case, your Frame Number will range from 1 to 96 frames. Your prompt animation can be divided into these individual time frames per line. To keep it straightforward, let’s work with 8 FPS for a 4-second video, resulting in 32 frames. Thus, our Frame Number will span from 1 to 32. |
| Frame Number Definition: The Frame Number indicates the count of frames in an animation, not to be confused with frames per second (FPS). For instance, if your animation contains 32 frames, you can specify a Frame Number from 1 to 32. |
| FPS and Number of Frames Relationship: -The relationship between FPS and the Number of Frames can be understood as follows: -FPS x Length of Video = Total Number of Frames. –For example: If you’re working on a 24 FPS video that lasts 4 seconds: -24 FPS x 4 seconds = 96 frames. -In this case, your Frame Number should vary from 1 to 96. |
The default settings include a Number of Frames set to 0, resulting in an FPS of 8. When left unchanged, this configuration generates 16 frames, each lasting 2 seconds. I found that using 8 FPS and setting the Number of frames to 16 gives you errors.
| FPS | Seconds | Number of Frames |
|---|---|---|
| 8 (Default) | 2 Seconds | 0 will create 16 Frames |
| 8 | 4 Seconds | 32 Frames |
| 16 | 4 Seconds | 64 Frames |
| 24 | 4 Seconds | 96 Frames |
Animation Tapering: It’s important to note that the animation tapers:
if you specify ‘eyes closed’ at Frame 10, the animation’s peak corresponding to that event will be around Frame 10, in theory. However, the action will start before and extend after Frame 10, creating a smooth transition.
If you specify an action like ‘eyes closed’ at Frame 10,
the peak of that animation occurs around Frame 10 in theory.
The action initiates before Frame 10 and extends after Frame 10,
to create a smooth and seamless transition between frames.
Avoid placing your final interpolation at the very end of the frame sequence,
as it may lead to errors. Instead, make sure that your last frame interpolation is positioned at least 1 frame before the last frame. For example, if your maximum frame count is set to 16, your final frame interpolation should be at frame 15 or lower.
Prompt Travel Example in Automatic1111
Here’s a practical example of Prompt Travel in action, demonstrating how to create a dynamic animation:
| Prompt: |
|---|
| cute blonde woman with braids 0: (closed eyes:1.3) (closed mouth:1.3) 5: bun hair 15: red hair 30: crying (tears:1.3) fox ears |
| Negative Prompt: |
|---|
| render, cartoon, cgi, illustration, painting, drawing, (hands:1.3) |
Frame 0:
The animation begins with the character having closed eyes and a closed mouth, intensified with a factor of 1.3.
Frame 5:
As the animation progresses, the character is prompted to “bun hair” with a factor of 1.3.
Frame 15:
At this point, the character’s hair transitions to red hair
Frame 30:
A sudden twist in the animation prompts the character to cry with an emphasis on tears.
Frame 30 and beyond:
As the animation concludes with a tail prompt, I added fox ears.
There is a fair amount of trial and error involved in achieving the ideal transitions, as seen below. It didn’t produce precisely what I wanted, but upon closer inspection, you’ll notice the changes when examining it frame by frame.
Getting Started: Creating an Animated GIF
Generate an Image Using these Settings
| Model: Download epiCPhotoGasm |
| SD VAE: Automatic |
| Clip Skip: 2 |
| Prompt: super cute blond woman in a dark theme |
| Negative Prompt: render, cartoon, cgi, render, illustration, painting, drawing |
| Sampling Method: DPM++ 2M Karras |
| Width and Height: 512×512 |
| CFG Scale: 5 |
| Seed: 763652634 |
Generate an Image Using Settings above:
Activate AnimateDiff:
- Enable AnimateDiff by selecting your Motion Module: mm_sd_v15_v2,
- then click the ‘Enable AnimateDiff’ button.
- Then click the ‘Enable AnimateDiff’ button.
- Now let’s begin with these basic settings:
1. Number of frames: 16
2. FPS: 8 - Click the “Generate” button
This will provide you, your initial AnimateDiff GIF which will be 2 seconds long and without the Prompt Travel.
Activate Prompt Travel
To enable Prompt Travel, you should include Prompt Interpolation lines in your prompt as demonstrated below.
The first line represents the Head Prompt, while the following three lines are the Prompt Interpolation.
I intentionally omitted the tail prompt for future experimentation.
If configured correctly, a message will appear in your command prompt stating:
” AnimateDiff – INFO – You are using prompt travel. “Prompt: super cute blond woman in a dark theme
0: open eyes
10: close eyes
14: smileNegative Prompt: render, cartoon, cgi, render, illustration, painting, drawing Click the “Generate” button, and you will receive these 16 images.
Challenges: Insights from a 2-Second Animation
For this experiment, I kept it simple with a short prompt and a 2-second clip containing 14 frames. I kicked things off with frame 0, featuring open eyes. Then, at frame 10, I had the eyes close, wrapping it up within those 14 frames.
When you’re dealing with such a short clip, like just 2 seconds, it’s important to think about how the animation should flow together. If you place your prompt interpolations too close, you might not get the results you want or even end up with some strange artifacts. Also, it’s important to note that not every word in your prompt will work as expected. Before you dive in, it’s a good idea to make sure the fine-tuned model is trained for the particular looks and actions you want to depict.
I tried to add a prompt interpolation on the very last frame (in my case, frame 16), but that led to some errors. To make it work, I had to use frame 15. However, this change affected some of the other frame interpolations because it was too close. So, it’s all about experimenting with the prompt interpolations and the words you use.
I didn’t end with a tail prompt in my case because it was causing issues and acted more like a main prompt. If you look at the three frames where I added the prompt interpolations, you’ll notice that the action occurs both before and after the indicated frames. This is because the animation needs some leeway to make it look smooth and natural, so that’s why you need frames both before and after the interpolations.
| Prompt Travel OFF | Prompt Travel ON |
|---|---|
| super cute blond woman in a dark theme | super cute blond woman in a dark theme 0: open eyes 10: close eyes 14: smile |
Advanced Prompt Traveling
Let’s take things to the next level and increase the number of frames, allowing us to incorporate additional prompt interpolations for a more action-packed creation.
We’ll start by using the same prompt travel approach as before, but this time, we’re going to boost the number of frames to 32. Right away, you’ll notice a significant improvement in the animation’s natural flow.
You’ll notice that if you stick with the same Frame Interpolation settings, she does infact begin with a smile at frame 0. However, even though the “closed eyes” prompt was scheduled for frame 10, you’ll notice it peeking around frames 8 or 9 and then reopening again, with parts of it visible by frame 10.
Typically, you might expect this to be the peak, but it’s not. This behavior is influenced by how the diffusion process operates. At frame 14, you’ll see her smiling and maintaining that expression until the end, as that was our last Prompt Interpolation
Now, let’s introduce another Prompt Interpolation and see how it affects a later frame.
| Prompt: |
|---|
| super cute blond woman in a dark theme 0: open eyes 10: close eyes 14: smile 25: angry face |
| Negative Prompt: |
|---|
| render, cartoon, cgi, render, illustration, painting, drawing |
By incorporating the “Angry Face” Frame Interpolation at frame 25 for this animation, you’ll observe that the earlier frames display her with an angry expression from the start. This is attributed to the tapering that transpires at the animation’s conclusion.
You can address this by employing various methods, such as modifying the Closed Loop or eliminating either the head or tail sections in the Reverse section.
Play Around with ‘Closed Loop’
The Closed Loop is initially set to R-P, indicating “Reduce loop” This option should be chosen when your intention is to minimize the number of closed loop contexts within your animation. Essentially, this prevents the animation from appearing repetitive.
When you activate this option, the transition from the end back to the beginning will make her appear less angry, although some residual effects from the tapering might still persist. If you wish to avoid this residual effect, you can opt for “N: No loop,” which makes sure that the animation will not cycle back to the initial frame.
What Happens When You Add a Tail Prompt?
| Prompt: |
|---|
| super cute blond woman in a dark theme 0: open eyes 10: close eyes 14: smile 25: angry face a fat woman |
| Negative Prompt: |
|---|
| render, cartoon, cgi, render, illustration, painting, drawing |
To conclude this guide on Prompt Travel for AnimateDiff, I introduced a Tail Prompt: “a fat woman.”
So, what exactly did this Tail Prompt achieve? It made her boobs bigger. I encourage you to experiment with Tail Prompts to discover their potential effects. I hope this guide has provided you with a better understanding of how to utilize AnimateDiff’s Prompt Travel feature in Automatic1111.
Behold! A two headed woman! This phenomenon, referred to as “twinning,” occurs when attempting to generate images larger than the standard training sizes, typically 512×512 and 512×768.
Discover how Stable Video Diffusion, powered by AI, is set to revolutionize the film industry. Create stunning videos with just a few clicks.
Leave a Reply