AI Generated Animations Just Changed Forever | Dream Machine

Summary

In this video, we explore Luma Labs' Dream Machine, a tool for creating impressive AI-generated animations. The tutorial covers an overview of Dream Machine, how to access and use it, and a detailed workflow for generating animations. The process includes creating consistent characters with tools like MidJourney and ChatGPT, using Dream Machine for animation, and tools like 11 Labs for voiceovers and Sync Labs for lip-syncing. The video emphasizes the potential and future of AI tools in creative projects and encourages viewers to share their own creations.

Takeaways

• ✨ Luma Labs recently released Dream Machine, a tool for creating impressive animations using AI.
• 🎥 The video showcases a clip created with Dream Machine and breaks down the process for replicating it.
• 🔧 Dream Machine offers text-to-video and image-to-video capabilities, with impressive results, especially when combining AI tools.
• 🆓 Dream Machine has a free tier with 30 generations per month, but paid plans offer more usage and faster processing times.
• 📈 To get the best results, generate images using tools like MidJourney or DALL-E and then animate them with Dream Machine.
• 🎤 Voiceovers for the project were created using 11 Labs, a powerful AI voice generation tool.
• 🗣️ Lip syncing was enhanced using Sync Labs, though Pabs is a free alternative with manageable watermarks.
• 🎵 Music for the project was generated with Yudo, an AI tool that creates realistic instrumental and vocal tracks.
• 🖌️ Consistent character generation was achieved using ChatGPT and image generation tools like MidJourney and DALL-E.
• 💡 The overall process included image generation, animation with Dream Machine, voiceover creation, lip syncing, and final video editing.

Q & A

What is the main purpose of the video?

The main purpose of the video is to demonstrate how to use Luma Labs' Dream Machine for creating animations, providing a step-by-step tutorial on generating images, animating them, and integrating voiceovers and music.

What are the two main parts of the tutorial in the video?

The tutorial is split into two parts: an overview of Dream Machine, including how to access and use it, and a workflow demonstration showing how to create a demo animation, including generating images, doing lip sync, and adding music.

What are the tools mentioned for generating images and animations?

The tools mentioned include Luma Labs' Dream Machine for animations, MidJourney or DALL-E for generating images, and Microsoft's Copilot for free image generation.

What is the process for creating consistent characters using ChatGPT?

To create consistent characters using ChatGPT, you use the 'consistent character GPT' to generate prompts for images, which are then pasted into Microsoft Copilot to generate the images for free. This process ensures that the character remains consistent across different scenes.

How does the Dream Machine handle image prompts and what is its strength?

Dream Machine can generate animations from text prompts and enhance prompts to suit the tool better. Its strength lies in the image-to-video feature, where pre-generated images from tools like MidJourney can be animated with specified actions and movements.

What tools are suggested for voiceover generation and lip syncing?

11 Labs is suggested for generating voiceovers, while Syn Labs and Pabbs are recommended for lip syncing the voiceovers to the animated characters.

What tool does the video creator use for adding music to the animation?

The video creator uses Yudo to generate music and vocals for the animation. This tool allows the user to describe the type of music and lyrics they want, which Yudo then generates.

What are the benefits of using AI tools in animation as highlighted in the video?

AI tools lower the barrier to entry for creating animations, allowing creators with limited resources or skills to bring their stories to life. These tools are expected to improve over time, further enhancing their capabilities.

What are some limitations of the AI tools mentioned in the video?

Some limitations include the occasional weirdness in generated images, slow generation times on free plans due to high demand, and artifacts in lip syncing that may require manual adjustments in video editing.

How does the video creator suggest dealing with the limitations of the free tiers of these AI tools?

The video creator suggests using multiple accounts to take advantage of free tiers, generating images during low-traffic times for faster results, and using tools like ChatGPT and Microsoft Copilot to circumvent the limitations of paid plans.

Outlines

00:00

🚀 Introduction to Dream Machine and AI Animation Tools

Luma Labs has recently released Dream Machine, producing remarkable outputs. This video will explore its use for animations. The presenter shows a test clip and then breaks down its creation for viewers to replicate in their projects. The clip narrates the story of Max, who dreams of working at Stark Corporations and building a rocket to the sun.

05:02

🎬 Overview of Dream Machine and Its Features

The video discusses the impressive animations generated by Dream Machine, despite some AI imperfections. It outlines the video structure: an overview of Dream Machine, access methods, usage, and a detailed workflow for generating a demo. It mentions Dream Machine's free tier, the speed limitations on free plans, and the benefit of generating animations during low-traffic times.

10:02

🎨 Generating Animations with Dream Machine

The tutorial demonstrates Dream Machine's text-to-video and image-to-video capabilities. It showcases examples, explaining the enhanced prompt feature and the impressive results. The presenter advises using image generation tools like Mid Journey to create initial images and then animate them in Dream Machine for better results. Examples of using prompts to generate consistent animations are provided.

15:03

📚 Consistent Character Creation and Workflow Overview

The presenter explains how to create consistent characters for storytelling using Mid Journey and ChatGPT for free image generation. The tutorial covers generating images in different scenarios and orientations, and then animating them in Dream Machine. For voiceovers, the presenter recommends 11 Labs, detailing its free tier and benefits. Tools for lip-syncing, such as Sync Labs and Pabs, are discussed, showing how to achieve accurate lip movements.

🎵 AI-Generated Music and Final Assembly

The tutorial explains using Yudo for generating realistic AI music, highlighting the process of selecting genres and prompts. The presenter demonstrates assembling the final video using a video editor like Premier Pro or CapCut, integrating images, animations, voiceovers, and music. The video concludes with reflections on the potential of AI tools to lower barriers for creators, encouraging viewers to share their AI-generated projects.

Keywords

💡Dream Machine

Dream Machine is a new tool released by Luma Labs for generating animations. In the video, the speaker explores its capabilities, demonstrating how it can be used to create impressive animations from text or images. The Dream Machine's ability to generate realistic animations is highlighted as a key feature.

💡Animations

Animations refer to the moving visual representations created by the Dream Machine. The video showcases various examples of animations, including an alarm clock and different characters, emphasizing how these animations can be used in professional projects.

💡Text to Video

Text to Video is a feature of the Dream Machine that allows users to generate video content by inputting textual descriptions. This feature is illustrated in the video with examples such as 'an extreme close-up footage of a young sailor woman during a rainstorm,' demonstrating the tool's ability to create detailed and convincing animations from text.

💡Image to Video

Image to Video is another feature of the Dream Machine where users can upload an image and animate it. The speaker explains that this feature is particularly effective for achieving high-quality results, as it allows for the use of pre-generated images from tools like DALL-E or MidJourney, which are then animated by Dream Machine.

💡AI-generated Music

AI-generated Music refers to the music created using artificial intelligence tools. In the video, the speaker mentions using tools like Yudo to generate music, which is incorporated into the final animated video. This highlights the integration of various AI tools to enhance the overall production quality.

💡Voiceovers

Voiceovers are audio recordings used to provide narration or dialogue for the animations. The video mentions the use of 11 Labs for generating voiceovers, which are then synced with the animated characters to create a cohesive narrative.

💡Lip Syncing

Lip Syncing is the process of matching animated characters' mouth movements with the voiceover audio. The video explains how tools like Sync Labs and Pabs can be used to achieve accurate lip syncing, ensuring that the characters' speech appears realistic.

💡Prompt

A Prompt in this context is a textual description used to guide AI tools in generating images or animations. The speaker demonstrates how specific prompts can be used to create desired visuals, such as 'an animated scene featuring a close-up of a four-master.'

💡Image Generation

Image Generation refers to the creation of visual content using AI tools like DALL-E or MidJourney. The video discusses how these images can be generated with specific prompts and then animated using Dream Machine to create dynamic visuals.

💡Character Consistency

Character Consistency involves ensuring that a character's appearance remains uniform across different scenes. The video emphasizes the importance of this for storytelling and explains how tools like MidJourney and ChatGPT can help generate consistent character images.

Highlights

Luma Labs' Dream Machine produces impressive AI-generated animations, suitable for professional projects.

The tutorial demonstrates how to use Dream Machine for creating animations, including lip sync and music generation.

Dream Machine offers a free tier with 30 generations per month, with potential for slow performance during peak times.

Generating images using tools like MidJourney or ChatGPT can enhance the quality of Dream Machine animations.

Using the exact prompt from image generation tools in Dream Machine improves animation stability and reduces artifacts.

Combining image generation with precise prompts results in the best animation outcomes in Dream Machine.

Voiceovers can be generated using tools like 11 Labs, which offers a generous free tier.

For lip syncing, tools like Sync Labs and Pabs can be used to align mouth movements with audio, despite some limitations.

Yudo AI tool is recommended for generating realistic music and integrating it into animations.

The video editor CapCut can be used to combine all elements (animations, voiceovers, lip sync, music) into a final project.

The tutorial emphasizes the potential of AI tools to lower barriers for creators with limited resources or skills.

Despite AI advancements, artistic skills remain crucial in creating compelling content.

AI tools like Dream Machine are expected to improve significantly in the future, enhancing content creation possibilities.

Using consistent characters across scenes helps maintain coherence in animated stories.

Tagging the creator on social media platforms with projects inspired by the tutorial encourages sharing and community building.

Transcripts

Luma Labs recently released dream

machine and some of the outputs coming

out of this tools are really really

amazing and in this video we're going to

be exploring how we can use this for

animations I'm going to show you a

little clip I put together just to test

out the tool and then I'm going to come

back and break down exactly how I

created that so that you can create the

same in your project let's watch the

clip

together I am sure going to miss this

place but it's time for me to head out

to fulfill my dreams in Aurora we're

dreams come

alive wa Stark

corporations I'm going to work there

someday welcome to

Aurora fresh vegetables no

way hi I am Max and someday I am going

to build a rocket to take us all to the

sun they can laugh all they want

but I think I got what it takes and it's

all here in my book all I have to do is

finish school and get a job at stack

corporations yeah yeah yeah where have I

heard that before I guess I do say that

a lot I don't have it I

[Music]

promise they are after my

notebook where are we going just that

operation you said that was impossible

that's when I thought you were pray

easy I can't believe I have waited my

whole life for this

one that don't let achal cruises take it

away

from wow this is my city and Nobody

messes with my

[Music]

friends so everything you watched

including the music was generated using

one AI tool or another and I don't know

about you but I was very impressed by

some of the animations I was getting out

of dream machine of course it's not

perfect there are tell tals here that

you can tell that it's AI but I'm sure

if I spent a little bit more time with

the tool I can fix some of those issues

here and there but some of this like

this opening shot right here of the

alarm clock I could use that as is in a

professional project and a couple of all

the other shots too some of them are

actually pretty pretty pretty good this

video is going to be split into two

parts the first is just going to be an

overview of dream machine how to get

access and how to use it and in the

second part of it I'm going to show you

the workflow that I use to generate that

demo so doing your lip sync generating

your your images including generating

the music using AI I'm going to show you

how to put all of that together it's

going to be on a high level because if I

go into all the details then this video

is going to get too long but I'm going

to post a longer version of that in the

member section of this channel so if

you're a member and you joined as a

member you can go and watch that it's

going to be a long video and we're going

to go step by step everything that I did

if you want to watch that longer version

you can check it out on the member

section of this channel but let's get

straight into the tutorial to access the

tool simply head over to lumabs

the good thing is that they actually

have a free tier where you get about 30

Generations per month for free and this

limitation is per account so this might

not be a bad time for your mom to get an

account if you know you know but you do

burn through those credits pretty

quickly especially when you're doing

your first project experimenting so I

had to move to one of the paid plans to

finish that project that I was doing cuz

I generated almost 100 generations to

get the right shots another thing is

that if you're on the free plan

sometimes the generations are very slow

because the tool is very popular right

now so there's a long wait time in the

queue one thing I noticed is that about

12 midnight eastern time it seems like

the traffic is really low at that time

so that's a good time to have your

Generations done they move very fast so

I have dream machine open here and the

UI is very straightforward you can do

text to video and you can also do image

to video so let's try one of the example

prompts they have here which is an

extreme close-up footage of a young

sailor woman with a concerned expression

during a rain storm I'm going to leave

enhanced prompt on what this does is

that it tries to modify your prompt to

make it more suited for the tool and

let's hit generate and there you go

there are the results quite impressive

looks like some old theme um SD footage

but looks quite convincing if you didn't

think twice about it you might not have

thought that this was air generated so

this is quite impressive let's also now

try some animation related prompt so

right here I said an animated scene

featuring a close-up of a for master and

you can see that looks really really

good so let's try another example this

this one was a Pixy style animation of a

girl playing in the playground and this

one was a little bit me it's not that

great um so I'm going to show you the

trick towards getting this pixel style

animations to look good and that's where

we come to the image to video in my

opinion this is where the tool really

really shines where you generate your

image in a separate image generation

tool like dly or mid journey and bring

it on here to have it animated for

you so I have here one of the images I

generated for Mid Journey for my project

so I'm going to go back to dream machine

Ma and click on the little image icon

right here on the top and then upload

this image and this time around I'm not

going to type any prompt I'm actually

going to give it the creative directive

and just click on generate and see what

it does with it and for this one this

actually came out pretty good most of

the times you get a lot of weirdness if

you give it the complete creative

directive so here's an example where I

didn't have a prompt you can see there's

a weird looking hand coming out of there

and that's the same shot here where I

gave it some prompt for what to do

that's a more usable shot here's another

example

with no prompts you can see a guy moths

a hand comes out very weird but now when

I give it some prompts as to what

exactly to do that's a much better

usable shot so in my opinion the image

plus describing what you want it to do

gets you the best results from my many

examples that I've tried and sometimes

what I do is also take the exact prompt

that I use in the image generator so

like the prompt I used in mid Journey or

Del and paste it here and here's an

example of that where I attached this

image and I gave it the exact same

prompt that I used in mid Journey which

is you know a close-up of a camera

animated style character skating down

and Ali and you can see this shot right

here was very stable there was not

weirdness going on it really adhere

properly to this font and then what I do

is that I then add some SLE animations

that I would want to happen in addition

to that like in this case I added that

the boy is blinking his eyes and if you

look through the end there you can see

his eyes blinking a little bit so he did

adhir to that prompt and it's quite good

with that so in this example I did the

same thing but I added that it's windy

and the wind is moving his hair and as

you can see it's doing the hair movement

and the physics of that hair movement is

really phenomenal the way it's getting

that hair movement it's really blown

away that blew me away so it does a very

good job at that so in my opinion in

general if you want to get the best of

this generate your image from an image

generation tool bring it on here use the

exact same prompt or similar prompt that

you use in the image generator on this

and then add additional kind of

animations that you would like to see

the character doing moving their hands

moving their heads and things like that

have gotten the best result using those

two combinations if you're enjoying this

video so far kindly give it a like And

subscribe for more tutorials like this

one all right let's move on to the

second half of this video where I show

you the workflow that I used to generate

the little trailer we did in the intro

so my step was this create a consistent

looking character and then generate all

the images that I needed to use for the

project I then took that to dream

machine to animate them and then I used

11 labs to generate all my voice overs

used another tool to do the lip syncing

to fix any lip syncing issues and then

took all of that that together put it

together in a video editing tool put in

sound effect music and all of that and

the music was generated using a tool

called udio and I'm going to show you

all of that all right so let's get to

the first step which is generating your

character and generating it consistently

because I was telling a coherent story

it was important for me to have a

character that kind of looks like Max

across all sins I said kind of because

if you look at some sins there it's not

exactly the same person but you get the

point that yeah this is the character

we're talking about I used mid Journey

for that uh but since mid Journey not

free to use I'm going to show you how to

do this using chat GPT for free so both

if you're on the paid plan and the freed

plan I'll show you a trick that you can

use using chat GPT to generate your

images consistently for free all right

so I have chat GPT opened here and you

can create an account completely free to

use Char PT and as of two weeks ago gpts

which are like apps in Char PT are now

free to use which is what we're going to

be using to generate a consistent

character right here if you click on

explore gpts type and search for one

that is called consist consistent

character GPT fast and high quality

click on that and then start a chat now

if you have a paid account with chpt you

can go ahead and just follow all the

prompts and create your characters with

no issues but if you do not have a paid

plan with chpt paste this additional

extra prompt and I'll probably put that

in the description and the prompt is

this before we get started some

additional instructions for you which is

whenever you want to generate an image

do not generate it but just give me the

prompt that I can copy and paste in Del

we're then going to use Microsoft

co-pilot every time we need to generate

an image to dump that there to generate

it for free and let me show you how that

works so I'm going to give you that

instructions and you can uh copy that

and paste it so first it wants to know

what my character should be so I'm going

to say a male character all right so it

wants a name for the character so I'll

just call him Max like I did in my intro

and things like that so I have a prompt

that I already kept here so I'm just

going to dump that prompt and you can

pause this video to see what the prompt

is but the most important thing is that

if you want your character to look like

uh that 3D and animation style character

just add the word pixel so if you put

the word pixel in any image generator

most of the times they'll give you a

character looking like that 3D animation

kind of characters so I'm going to hit

enter so as you can see instead of

generating the image it gave me the

prompt so I'll copy that if you are on a

paid plan and you didn't type that

instruction on the top it will actually

generate an image of Max but since I

want to show you how to do this for free

I'm going to copy that prompt that it

gave I'm going to head over to Microsoft

copilot that's Coop pilot. microsoft.com

create an account for free and simply

paste that prompt here and hit enter all

right so you can see we have an image

here of Max very similar to what you saw

there in the trailer so I'm going to go

back to D now and just keep prompting so

if you want marks in different scenarios

you can just keep asking it to do that

and it's going to generate for you um a

prompt for that so in this case I'll say

create an image of Max in a rusty

Warehouse with metallic equipment around

Max is doing an experiment on a table so

it's a random prompt so I'm going to

click on that all right so I'm going to

copy that prompt I'm going to head over

to Copilot and I'm going to put that

here and let's hit enter and let's see

what that looks like so as you can see

that looks very much still like our

character and the results from this is

actually pretty good uh almost on par

with what I was getting from mid journey

and if you wanted to change the

orientation of this for a movie because

this is all square just click on the

little button right here and switch from

Square to landscape and that's going to

expand the image so that it looks a

little bit more like what you could use

in a video for Youtube and things like

that so this was the exact step I took

and I did the same for every single

scene that I wanted in that trailer so I

went ahead and kept prompting prompting

several times until I got it looking

right I think I created over 100 images

and eventually picked like 15 of them so

now after I did all of this I took these

images and put them into dream machine

to animate them using the way that I

showed you in the beginning of this

video for all my voiceovers I like to

use 11 Labs I still find it to be one of

the best voiceover tools out there

generated by AI so that's what I used

for the trailer and I have an entire

video on 11 Labs so I'm not going to go

into the details on how to use it I'll

put a link to that video down in the

description and the good thing is that

11 Labs also gives you a free tier you

can use it for free uh might in that

video a few people were pointing out the

11 Labs is not free they do have a

generous freit here gives you about

10,000 tokens each month and again this

limit is per account so this might be

another instance where your mom wants to

open an account you decide but yeah you

can check that video out I'll put a link

to that in the description all right

next after the voiceovers have been

generated the next step is to do all the

lip syncing now dream machine would try

to do some lip movement if it discovers

that the mouth is open or maybe if you

prompt it to but obviously that's not

going to be in sync with your actual

audio so to do that there is a tool I

use it's called sync Labs now the tool

is not free to use it is kind of free

but on the free version you get this

huge Watermark so if you're okay with

having that watermark on your final

video then it's fine you can go ahead

and actually use this for free but there

is another tool called uh pabs that also

has water marks but these water marks

are a little bit more manageable that

you can use for free it's not as great

as syn Labs but it also does decent job

I'm going to show you the two and you

decide which one you want to use so I

have syn labs open here and I'm just

going to click on a new project and what

you simply need to do is upload your

video so I have the video uploaded and

then next I'm going to upload the audio

so I have an audio right here and then

you can choose from the different models

I like this

1.6.1 but 1.6.0 also works pretty fine

just depends on how high a resolution of

an email a video you want either of them

works pretty fine and then you simply

just hit generate so this is before you

said that was impossible

you said that was impossible and now

this is after when we actually have the

lip sync working you said that was

impossible so you see that's a lot

better and with this weird artifact it

does that sometimes when it's a silence

at the end but this you can cut out when

you're editing this on your video editor

all right and on pabs once you go to p.

add create an account you just upload

your image your video sorry right here

and then click on the lip sync option

and then right here you can upload

whatever audio file you have and then

you say attach and continue where are we

going that's how that sound it's not bad

but you can see there's a lot of

flickering and artifacts on the mouth so

it's not necessarily 100% great but it's

also usable and the water mark they have

is just this little water mark right

here which for me is a little bit more

usable if you want to use this for free

and once you pay the watermark goes away

also all right finally let's talk about

the music so I wanted to keep everything

in the AI teamed space so I used a tool

called yudo which generates really

amazing music that sounds very realistic

so I have yudo opened right right here

and uh you can pretty much describe the

kind of music you want here and put the

lyrics and it will generate both the

music instrumental and the vocals for

the song So What I typically like to do

is that sometimes I don't know how best

to prompt this I come here under this

cover and I try to find something that

has been created in the same genre that

I'm looking for and then that way I can

just copy that prompt to get something

similar so all right so for example

let's say I wanted to do a hip-hop song

I'm going to click on hip-hop and just

just check out a few of these examples

that they've done to see the one that's

very close to what I want so for example

uh let's try out this one

[Music]

here out in the

pl Where Stars don't play nice two

Captain SP sacrifice what I would simply

do in a case like this if I like this

one you could if it's shorter than 30

seconds you can remix it or if it's

longer than 30 seconds then you can

simply just copy the same hip hop style

that they have here so you just copy

that get over here to S paste that in

and you can have it autogenerate random

lyrics for you or you can come here and

type whatever lyrics you have for your

songs and you can do that and generate

it and you get a full blown music for

that so that's how to use this tool I

can cover this tool in more details in

another video if you're interested in

that let me know in the comments and I

can cover that so the final step in all

of this is just to use a video editor to

put all of this together I do normally

use premier Pro but you could use a free

tool like cop cut to just simply uh join

all the clips together put your music

put the lip synced version of all your

videos and then add uh sound effects and

overall I was really pleased with the

outcome now the tool is not perfect but

one thing you have to remember is that

this is the worst this tool is going to

be at I can imagine version two version

three up to version five like two three

years from now how well this tool is

going to get so I really think that this

is breaking the barrier to entry for a

lot of people who have great stor they

want to tell but they don't have enough

resources or maybe the right skills to

put those things to life this is really

breaking that barrier to entry and I

think we're going to see a lot more

creators coming out creating really

awesome content as this tools mature and

don't get me wrong there's still

Artistry that goes into this cuz a lot

of people think that AI is coming and

Artistry is dying of like I had to

create over 200 images from my journey

just to get like 15 of those even on

dream machine itself I iterated back and

forth and there's so much going to like

the music the mood and there's still

artistry that goes into it it's just

that now the tools are much easier to

use and the barrier to entry to learn

these tools are lower so I'm really

excited for this and excited to see

where this technology evolves if you

ever create anything using the concept

I've taught you today you can tag me on

Instagram or Twitter I'm going to put my

handles down in the description I like

to see what you've created and to also

share it with others for them to see

thank you for stopping by I'm going to

catch you in the next one keep learning