---

How to write prompts for AnimateDiff

A prompt is a string of text that describes the content and style of the image or animation that you want to generate. For example, if you want to generate an animation of a cat chasing a mouse, you can write something like this:

a cute cat (best quality, masterpiece) running after a mouse (funny, cartoonish)

The prompt can contain keywords, phrases, modifiers, and special tokens that influence the generation process. Here are some general guidelines for writing prompts for AnimateDiff:

• Use descriptive and specific words that capture the essence of what you want to generate.

  For example, instead of writing a dog, you can write a golden retriever (fluffy, adorable).

• Use parentheses ( ) to add emphasis or modifiers to a word or phrase.

  For example, (best quality, masterpiece) will increase the attention and quality of the generation, while (worst quality, low quality) will do the opposite.

• Use brackets [ ] to add scheduling or alternation to a word or phrase.

  Scheduling is a technique that allows you to change the prompt or the control image at different steps of the generation process. This can create more dynamic and diverse animations, as well as fix some issues with flickering or splitting scenes. For example, [dog|cat] will alternate between dog and cat in each frame, while [dog:10] will ignore dog for the first 10 steps.

• Use commas , to separate different words or phrases in the prompt.

  For example, a dog, a cat, a mouse will generate an image or animation that contains all three animals.

• Use vertical bars | to create prompt matrix or X/Y plot in the web UI.

  For example, dog|cat\mouse|bird will create a 2×2 matrix of images or animations with different combinations of animals.

• The backslash \ is used to separate the rows of the matrix, while the vertical bar | is used to separate the columns.

  You can use this feature to explore different variations or combinations of your prompts. You can also use parentheses, brackets, commas, and colons with backslashes and vertical bars to add more modifiers or effects to your prompts.

• Parentheses ( ):

  Use parentheses ( ) to add emphasis or modifiers to a word or phrase. The more parentheses you use, the more emphasis you add. You can also specify a numerical weight for attention by using the syntax (word:weight). For example, (word:1.5) increases attention to the word by a factor of 1.5, while (word:0.25) decreases attention by a factor of 4 (1 / 0.25).

• Double parentheses (( )):

  Use double parentheses (( )) to add even more emphasis or modifiers to a word or phrase. This is equivalent to using two single parentheses with the same weight. For example, ((very important)) is the same as (very important:1.21), which increases attention to the phrase by a factor of 1.21 (= 1.1 * 1.1).

• Numerical weight:

  Use numerical weight to adjust the level of attention or emphasis for a word or phrase. The weight can be any positive number, but it is recommended to use values between 0 and 2 for best results. The higher the weight, the more attention the word or phrase will receive. The lower the weight, the less attention it will receive. For example, (dog:2) will give twice as much attention to dog as (dog:1), while (dog:0.5) will give half as much attention.

How to avoid common problems with AnimateDiff

Sometimes, you may encounter some problems or issues when using AnimateDiff, such as flickering, splitting, or changing scenes in the output animation. Here are some tips and solutions for avoiding or fixing these problems:

• Keep your prompts below 75 tokens when generating locally.

  Prompts over 75 tokens will either exhibit no motion, or will be split into two different “scenes”.

• Use negative prompts to prevent unwanted elements or effects in the output animation.

  For example: a person (best quality) talking (lipsync) (no background noise, no movement)

• Use Euler ancestral or DPM2 ancestral samplers for smoother and more stable animations.

  These samplers are more suitable for AnimateDiff than the default Euler sampler.

• Use ControlNet v2v to animate a video or a sequence of frames using another video or a sequence of frames as the control source.

  This means that you can transfer the motion and style from one video to another, creating interesting and creative results.

• Use Pad prompt/negative prompt to be the same length option:

  In Automatic1111 settings if your prompt is changing scenes halfway through even after keeping it below 75 tokens. This option improves performance when prompt and negative prompt have different lengths by padding them with spaces until they are equal.

How Tokens Work in Stable Diffusion

Tokens are units of text or image that are processed by the Stable Diffusion model to generate outputs. Each token represents a specific feature or attribute of your prompt, such as color, font, size, or position. You can use tokens to create more complex and diverse prompts that better match your needs and preferences.

For example, if you want to generate an image of a cat, you can write a cat as your prompt, which is composed of two tokens: a and cat . The model will use these tokens to create an image that matches your prompt.

However, there are some limitations and tips that you need to know about tokens in Stable Diffusion. Here are some of them:

• The maximum number of tokens that you can use in your prompt is 75.

  If you use more than 75 tokens, the model will either ignore the extra tokens, or split your prompt into two different scenes. This may affect the quality and coherence of your output. This is something you don’t want to happen when generating videos with AnimateDiff.

• You can use as many tokens as you want in your prompt:

  But keep in mind that each token adds some overhead to the generation process and may affect the quality or speed of your output.

I hope this helps you understand how tokens work in Stable Diffusion when writing prompts better. [Source]

What is Pad prompt/negative prompt to be the same length?

‘Pad prompt/negative prompt to be the same length’ is an option in Automatic1111 settings that pads the prompt and negative prompt with spaces until they have the same length. This improves performance when prompt and negative prompt have different lengths by preventing the model from changing seeds during generation.

A seed is a random number that influences the generation process. If the seed changes during generation, it may cause the output animation to change scenes abruptly or unexpectedly. This may happen when the prompt and negative prompt have different lengths, because the model uses their lengths as part of the seed calculation.

By padding the prompt and negative prompt with spaces until they have the same length, this option ensures that the seed remains constant throughout the generation process. This results in smoother and more consistent animations. [Source]

To use this option, you need to go to settings -> Optimizations -> make sure the “Pad prompt/negative prompt to be the same length” clicker is clicked.

Conclusion

I hope this blog post has helped you understand how to write prompts for AnimateDiff, and how to avoid some common problems that may occur. AnimateDiff is a powerful and fun feature that allows you to create amazing and realistic animations from text or image prompts. With some practice and creativity, you can generate stunning and impressive results.

Thank you for reading this blog post. If you have any questions or feedback, please feel free to leave a comment below.

To learn how to use Prompts and Punctuations in Automatic1111, you can check out my comprehensive guide below.