AnimateDiff introduces a framework designed to take your still images or text prompts and infuse them with animation, thanks to AnimateDiff motion models. At its root, AnimateDiff incorporates a motion modeling module into the base text-to-image model, enabling it to grasp the intricacies of realistic motion dynamics, much like those seen in video clips.

Getting to Know the AnimateDiff Motion Models

Let’s learn more about the various AnimateDiff motion models, each offering its own set of features and capabilities.

---

Understanding Motion Models

Motion models are mathematical models that describe the movement of an object over time. They are used in a variety of fields, including robotics, computer vision, and autonomous vehicles. A motion model can be used to simulate the ideal motion of a target. It describes how a target or object moves with respect to time and is usually expressed as an equation of motion governing the transition of target states, such as position and velocity.

In the context of AnimateDiff, motion models are used to capture realistic motion dynamics in video clips 2. They allow Stable Diffusion text-to-image models to create animated outputs, ranging from anime to realistic photographs.

• mm_sd_v14.ckpt:

  This is the pioneering motion model from the AnimateDiff creators, embodying 417 million parameters and occupying 1.6 GB of storage space. Its animations are known for their smoothness, albeit with the occasional presence of artifacts and noise.

• mm_sd_v15.ckpt:

  Building upon its predecessor, mm_sd_v15.ckpt seeks to reduce artifacts and noise, delivering more realistic and diverse animations. It retains the same number of parameters and storage space as mm_sd_v14.ckpt, making it a valuable choice for animation creation.

• mm_sd_v15_v2.ckpt:

  The latest pinnacle of the motion module journey, boasting 453 million parameters and demanding 1.7 GB of storage space. Trained on larger resolution and batch size data, it shines with noticeable quality improvements, rendering high-quality animations replete with fine details and natural motion.

Understanding Motion LoRa’s

AnimateDiff, Motion LoRAs is designed to give you greater control over the motion in your animated images. The term “LoRAs” stands for Low-Rank Adaptations, which essentially means a way to fine-tune AnimateDiff’s motion module with minimal parameter adjustments.

The exciting part is that Motion LoRAs grant you the ability to steer the camera movement in eight distinct directions. You can pan left, pan right, pan up, pan down, zoom in, zoom out, rotate clockwise, and rotate counter-clockwise. But what makes it even more fun is the possibility of combining different LoRAs to create more intricate and engaging animations. For instance, merging pan left with zoom in can give birth to that classic dolly zoom effect, adding a cinematic touch to your creations.

Motion LoRAs exclusively functions with the AnimateDiff v2 mm_sd_v15_v2.ckpt module.
This version happens to be the latest and most advanced iteration of the motion module, equipped with various enhancements and refinements.

Below, I’ll describe what each Motion LoRa models do:

v2_lora_ZoomIn.ckpt:	This Motion LoRA allows you to zoom in on your animated image, making it appear larger and closer to the camera.
v2_lora_ZoomOut.ckpt	This Motion LoRA allows you to zoom out on your animated image, making it appear smaller and farther from the camera.
v2_lora_ZoomOut.ckpt:	This Motion LoRA allows you to pan left on your animated image, making it move horizontally to the left side of the frame.
v2_lora_PanRight.ckpt:	This Motion LoRA allows you to pan right on your animated image, making it move horizontally to the right side of the frame.
v2_lora_TiltUp.ckpt:	This Motion LoRA allows you to tilt up on your animated image, making it move vertically to the top of the frame.
v2_lora_TiltDown.ckpt:	This Motion LoRA allows you to tilt down on your animated image, making it move vertically to the bottom of the frame.
v2_lora_RollingClockwise.ckpt:	This Motion LoRA allows you to rotate your animated image clockwise, making it spin around its center in a circular motion.
v2_lora_RollingAnticlockwise.ckpt:	This Motion LoRA allows you to rotate your animated image anticlockwise, making it spin around its center in a circular motion.

To utilize Motion LoRAs, you employ trigger words and add the LoRa Offset string of codes available in the LoRa tab in Automatic1111. Simply swapping them within the motion modules won’t yield the desired results; it will only generate a series of images. While it may seem intuitive to switch models to Motion LoRAs, the process doesn’t function in that manner. To set Motion LoRAs into action, you must input the appropriate trigger words and prompts.

Motion LoRa Trigger Words

Motion LoRa	LoRa Offsets	Trigger Words	Prompt Examples:
[v2_lora_ZoomIn]	​<lora:v2_lora_ZoomIn:1>	Zoom In	​<lora:v2_lora_ZoomIn:1>Zoom In
[v2_lora_ZoomOut]	<lora:v2_lora_ZoomOut:1>	Zoom Out	<lora:v2_lora_ZoomOut:1>Zoom Out
[v2_lora_TiltUp]	<lora:v2_lora_TiltUp:1>	Pan Up	<lora:v2_lora_TiltUp:1>Pan Up
[v2_lora_TiltDown]	<lora:v2_lora_TiltDown:1>	Pan Down	<lora:v2_lora_TiltDown:1>Pan Down
[v2_lora_PanRight]	<lora:v2_lora_PanRight:1>	Pan Right	<lora:v2_lora_PanRight:1>Pan Right
[v2_lora_PanLeft]	<lora:v2_lora_PanLeft:1>	Pan Left	<lora:v2_lora_PanLeft:1>Pan Left
[v2_lora_RollClockwise]	<lora:v2_lora_RollingClockwise:1>	Roll Clockwise	<lora:v2_lora_RollingClockwise:1>Roll Clockwise
[v2_lora_RollAntiClockwise]	<lora:v2_lora_RollingAnticlockwise:1>	Roll Anticlockwise	<lora:v2_lora_RollingAnticlockwise:1>Roll Anticlockwise

How to use the Motion LoRa’s

Download the AnimateDiff Motion Modules Here:

AnimateDiff’s Secret Power: Action-Prompted Animations

While toying around with these camera zooms, I stumbled upon something pretty cool. With AnimateDiff, you can prompt actions. I found this out when I tried to trigger the zoom keywords. Instead of typing the trigger words, I used the motion LoRa file name, and she did something different.

Instead of the camera zooming in, she was moving towards the camera! I had a revelation and started prompting action words like “Running,” and voila! She started running. This is pretty impressive! I’ll be writing a guide on prompting actions with AnimateDiff in the blog below.

---

AnimateDiff’s Secret Power: Action-Prompted Animations (Coming)

---