AI Video Is About To Explode!

Summary

The video discusses several AI video developments including Google's new Lumiere video model which generates videos in a more temporally coherent way, speculation that fully AI-generated 30-60 minute films may be available within months, a sponsored segment on Meshi which is a text-to-3D generator that can create models which can then be animated, some interesting hints that Midjourney may be moving into more 3D capabilities possibly based on the Media Molecule game creation tool Dreams, and coverage of Sim Francisco, an AI simulation of an entire city populated by AI agents with wants, needs and lifespans aimed at developing artificial general intelligence.

Takeaways

• 👨‍💻 Google announced a new video AI model called Lumiere that uses a spacetime diffusion architecture for improved coherence
• 🎥 YouTube's Matt Wolf predicts 30-60 minute AI-generated films that are coherent and enjoyable will be available within the next 2 months
• 🖥️ Meshi is a free AI text-to-3D model generator that can help you get started with creating 3D assets even if you're not an expert
• 🤯 Nick Beller created an awesome mixed reality animation by sculpting a 3D character in VR and compositing it into video footage
• 👀 Ex-Media Molecule co-founder Alex Evans is now working at Midjourney as a principal research engineer, possibly on 3D capabilities
• 🔮 Midjourney is building hardware focused on managing tens of thousands of virtual 3D spaces
• 🚀 AI simulation company Anthropic created an AI city called S. Francisco populated by agents with wants, needs and lifespans
• 🕹️ The AI agents in S. Francisco relax by playing old school Nintendo games
• 🧠 Anthropic's goal with S. Francisco is to achieve artificial general intelligence
• 📺 The Simpsons predicted AI cities would be created according to the video narrator

Q & A

What new video model did Google recently announce?

Google announced Lumiere, a new video generation model that uses a spacetime diffusion model to generate videos all at once instead of frame by frame.

What are some of the capabilities of Lumiere?

Lumiere can generate videos from text prompts with coherent motions like walk cycles. It can also stylize videos, do video in-painting and out-painting, and generate videos from images.

What did YouTube's Matt Wolf recently predict about AI-generated films?

Matt Wolf predicted that 30-60 minute AI-generated films that are coherent and enjoyable will be available within the next 2 months.

What does the tool Meshy allow you to do?

Meshy is an AI text-to-3D model generator that allows you to generate 3D models and textures from text prompts across different art styles.

How can you animate models made in Meshy?

You can download Meshy models and bring them into Adobe Mixamo to auto-rig and animate them.

Who is now working at Midjourney and what might they be doing?

Alex Evans, co-founder of Media Molecule, is now at Midjourney. He may be working on 3D capabilities for Midjourney.

What is Sim Francisco?

Sim Francisco is an AI-generated city populated with AI agents that have wants, needs, and lifespans. It's an experiment in artificial general intelligence.

What graphical style is Sim Francisco designed in?

Sim Francisco uses a graphical style similar to the South Park animation style.

What are some interactive elements in Sim Francisco?

The AI agents in Sim Francisco can fall in love, play NES games, and have lifespans where they die.

What is the ultimate goal of Sim Francisco?

The goal of Sim Francisco is to achieve artificial general intelligence by simulating agents in a virtual city.

Outlines

00:00

🎥 Google's New Lumiere Video Model Generates High-Quality Videos

Google has announced Lumiere, a new video generation model that uses a Spacetime diffusion model to generate high-quality, coherent videos all at once instead of frame by frame. It performs well at text-to-video, image-to-video, stylized video generation, in-painting, and out-painting.

05:00

🎞️ AI-Generated 30-60 Minute Films Coming in the Next 2 Months

YouTube's Matt Wolf has seen AI-generated 30-60 minute coherent and enjoyable films that will be publicly available in the next 2 months. He cannot share details due to an NDA but says the tech will be available for everyone soon.

10:00

📹 Meshi - An AI Text-to-3D Model Generator That's Easy to Use

Meshi is a free AI text-to-3D model generator with 200 monthly credits. It offers various art styles and can generate 3D models from text prompts quickly. The models can be downloaded and used in other 3D software. A coupon code for 20% off is provided.

15:01

🌆 AI Simulates a Dynamic City (San Francisco) With Virtual Agents

The Simulation has created an AI-generated city called San Francisco populated by virtual agents with wants, needs and lifespans. It looks visually similar to South Park but focuses on simulating urban dynamics and interactions to study artificial general intelligence.

Keywords

💡video generation

Video generation refers to AI systems that can generate synthetic video content from text prompts. This is a key theme in the video, as it discusses several new video generation models from Google like Video Poet and Lumiere. The narrator analyzes their capabilities in generating coherent videos like the astronaut walking on Mars. Video generation models aim to create novel video content automatically using AI.

💡diffusion models

Diffusion models are a type of generative AI technique used for image and video synthesis. Lumiere uses a spacetime diffusion model which generates the entire video at once rather than frame-by-frame. This results in greater temporal coherence. The narrator explains how this spacetime architecture differs from other models that can result in 'temporal breaks'.

💡feature-length films

The narrator discusses the potential for AI to create full-length 30-60 minute films this year that are coherent and enjoyable to watch. This is based on comments from YouTube creator Matt Wolf and Marvel director Joe Russo. It reflects growing capabilities of AI creativity and storytelling.

💡3D generation

The video looks at AI systems that can generate 3D models like Meshi. The narrator recommends learning 3D principles as AI creativity moves more into 3D. Examples given include generating 3D characters and objects and bringing them into animation software.

💡virtual worlds

The video references 'virtual 3D spaces' that Midjourney is building, hinting at more interactive and immersive experiences. The narrator speculates this could be similar to software like Dreams that allows exploring imaginative 3D worlds.

💡simulation

Simulation refers to using AI agents in simulated environments to study emergence and human behavior. The video discusses 'Sim Francisco', a virtual city populated with AI agents living life. This aims to achieve general intelligence through simulation.

💡text-to-video

Text-to-video is the task of generating video from text descriptions. The narrator shows examples of text-to-video from Lumiere like 'astronaut on planet Mars' and 'woman walking through autumn forest'. This tests the coherence of generated video.

💡text-to-3D

Similarly, text-to-3D involves generating 3D models and objects from text prompts. The sponsor Meshi provides text-to-3D capabilities for generating characters, vehicles etc. The video shows examples like 'spaceship' and 'baby dragon'.

💡video stylization

Video stylization is the technique of transferring artistic styles onto video content. Lumiere can stylize input video into different rendering styles like painting, sketch, stained glass etc. as demonstrated through examples in the video.

💡deepfakes

Though not directly referenced, many AI video generation techniques can enable manipulative deepfakes. The narrator positively highlights creative potentials but cautions on misuse remain.

Highlights

Google has a new video model called Lumiere.

Lumiere uses a spacetime diffusion model to generate video all at once.

Matt Wolf predicts 30-60 minute coherent AI films will be available in the next 2 months.

Meshi is an AI text-to-3D model generator that is easy to use for beginners.

Meshi works well for generating cartoon and animate styles but struggles with ultra-realistic models.

Generated Meshi models can be textured with AI and animated in Mixamo.

A Media Molecule co-founder is now working at Midjourney, possibly on 3D features.

Midjourney is building hardware focused on collecting data for 3D.

Simulation created an AI city called S Francisco with autonomous AI agents.

S Francisco agents have lifespans, fall in love, and play Nintendo games.

S Francisco's visual style resembles previous Simulation South Park episode.

S Francisco runs continuously to achieve AGI like a hyper Sims game.

AR character sculpted in Dreams and composited onto video for cool effect.

Lumiere shows promise for coherent video stylization and image-to-video.

Upcoming full AI films predicted within Joe Russo's 2 year timeline.

Transcripts

hey everyone well we've got some pretty

big Bombshells in the world of AI video

namely Google has yet another new video

model and 30 to 60 Minute consistent and

enjoyable AI films coming soon we'll

talk plus we're going to take a look at

a tool that is going to be really

helpful if you are terrible at 3D like I

am I mean obviously if you're good at 3D

I think that this tool will be helpful

as well and I've got an interesting mid

Journey dive well maybe less of a dive

and more of a splunking into an area

that I don't think anybody else has

covered yet and just might show the

direction that mid Journey's future is

headed in all that plus an AI city of

the future that is already here all

right let's dive in kicking off Google

have announced Lumiere a new video

generation model now you might be

thinking to yourself didn't Google just

do this like a month ago and yes yes

they did about 3 weeks ago Google

dropped video poet a zero shot video

generator does anyone know what is

happening at Google right now does

anyone at Google know what's happening

at Google right now well anyhow as of

today we have Lumiere which does

something kind of different than

previous video generators namely This Is

A Spacetime diffusion model so we are

going to rip a hole into the SpaceTime

Continuum and see what makes Lumiere

different but first let's take a look at

sort of like the standard features uh

text to video for example it does a

really great job with walk Cycles like

this is uh astronaut on planet Mars he

doesn't look like he's like Moon walking

or sliding all over the planet handheld

shot of a woman walking through an

Autumn forest and an adorable puppy that

isn't morphing into something horrific

two other shots that I thought were

pretty exceptional were the Jack Russell

Terrier on a snowboard they actually

called out GoPro shot in this and it

definitely does have that characteristic

GoPro fisheye look the Lamborghini is

also really nice there are some

incoherencies in terms of the road and

maybe the physics of the car turning but

the fact that it's actually holding the

model of the car together and not like

morphing into a fire truck is impressive

image to video also looks looks very

good uh the characteristics and the

smile on the girl with the Pearl Earring

looks very naturalistic and I mean those

are five fingers on Sir Isaac Newton as

he's waving hello so funny enough they

also animated the famous flag over eima

shot uh which Google poet also did but

on the video poet side we kind of ended

up with like this weird gopher kind of

appearing out of nowhere it also has

this stylized generation which is

something that I don't think I've seen

before this wherein you can give the

model a reference image and then it will

generate videos in the style of that

reference image for example here we have

like this vector image and if you prompt

a bare twirling with delight you get

this and a cute bunny nibbling on a

carrot so it's definitely taking the

elements of that you know Vector

illustrat kind of style and applying it

over to the video model we've also got

video stylization kind of that gen one

thing where you know you can take an

input video and change it into a variety

of styles at first I thought that it was

actually completely segmenting out the

back ground but actually in further

reflection it does look like it does

change out the backgrounds for each one

of these like in the made of flowers we

have kind of like the Eiffel Tower in a

bouquet look back there made of flowers

for some reason is also super disturbing

to me you would think like made of

flowers it's that's nice and pleasant

but it actually comes off as like like

one step away from the corep virus in

The Last of Us it also does video in

painting and out painting and it looks

like it does that pretty well

considering you've got like this balloon

here with the mask on this side and it's

completely making up you know these

other balloons and the remainder of the

sky and Horizon the pizza example is

actually really impressive as well

considering that it has to generate not

only the top half of that pizza but the

hand Crossing into the Mast area and

dropping basil onto it um yeah that's

pretty cool so how is Lumiere different

from other video models and remember

this is my caveman brain reading the big

words in the paper so this is going to

be really simplified basically it all

comes down to this archit iture called

SpaceTime unit which allows for the

video to be created all at once as

opposed to I guess other models which

begin with an input frame and then have

an output frame and then generate key

frames between those the problem with

other video models as you can see in

this example is like you have your input

frame here and your output frame here

and then if you look down here there's

kind of this this temporal break in the

chain right here whereas with the

SpaceTime unet um you know we have

unbroken chain also because the video is

generated all at once in SpaceTime as

opposed to you know creating individual

frames from an in and an out it frees

the model up for a lot of other tasks

like video stylization in andout

painting and image to video so all of

those will be a lot more temporally

coherent as well so it all looks really

cool and super promising and of course

that leads to the question do we get to

play with it the answer is I don't know

I don't know if they release another

video model in like 3 weeks I'm just

hoping that they name it something like

on the nose like Kafka moving on are we

going to be seeing 32 60 Minute long

completely AI generated films sometime

this year or even trimming that down

sometime in the next 2 months well in a

recent tweet by YouTube's own Matt wolf

uh congratulations Matt you have gone

from reporting on the news to becoming

the news anyways Matt says yes the Tweet

reads in my 2024 predictions video that

video will be linked below I made a

comment that I don't think AI is going

to be creating fulllength shows and

films this year 30 to 60 Minute stories

that were coherent and enjoyable won't

be available this year I was wrong the

next two months will be wild winky face

now Matt can't go into a lot of details

on this due to the fact that he signed a

non-disclosure agreement but he was able

to field a few questions for example

when Dan the Man asked if this was going

to be stock footage type mashups uh

something that we saw with in video but

no Matt says he's talking about fully

generated videos additionally this

technology will be available for

everyone to use it's not going to be

locked behind some like private

firewalled gate in some you know Studio

mogul's house and lastly in terms of a

time frame as to when we will all see

this Matt says that he thinks that next

month people will see what he saw now I

can't comment too much on Matt's post

namely because I may or may not have

signed a very similar non-disclosure

agreement uh that said now would be a

good time to hit the Subscribe button

what I can say is that I keep thinking

about a news story involving Avengers

director Joe Russo in which he said that

he fully expected full AI movies within

2 years that story came out in April of

2023 so you know Tick Tock moving on if

you're not great with 3D or just

downright suck at it like I do uh I

think now is a really good time to start

learning some very basic fundamentals

about it because I do think that a lot

of creative AI is going to be moving in

that direction as the year prog

progresses I'll have some news on that

in just a little bit now I don't think

that you need like a 70-hour tutorial on

blender unless you're interested in it

in which case go for it you can start

with something a little bit simpler

which brings us to Meshi who are

sponsoring today's video so Mesi is an

AI text2 3D model generator it is free

with 200 credits per month so that's

perfect if you're just getting started

although it does have paid tiers

obviously as you ramp up and I do have a

coupon code for you as well we'll get

that in just a minute Mesi is very easy

to use you just come down to this like

text to 3D button here um you know you

can describe your object here put in a

style this would be sort of the

descriptors that you want for your

object and then negative prompts as well

below that you can choose from a number

of different art styles ranging from

realistic uh voxel which is kind of that

Minecraft um you know Lego block look um

all the way down to like realistic

handdrawn and cartoon line art uh I

quickly generated a up a spaceship and

yeah there you go a 3D spaceship now I

didn't give it too many details uh the

style was just highly detailed sci-fi

Unreal Engine and I did not give it any

negative prompts you do also have

texture options over here for color or

PBR uh PBR as I learned has to do with

sort of reflectivity I will say that

Meshi really excels when you're

generating things like props or if

you're aiming for more of a like cartoon

or animated style when you aim for the

realistic stuff it gets a little bit on

the wonky side we'll take look at that

in one second but first let's generate

us a cute Dragon so with the very simple

prompt cute baby dragon you end up with

four kind of like low res versions of

dragon options like we could go in and

do a lot of other stuff in the prompt to

call out certain colors or whatnot um

but once you find one that you like if

you hit this refine button you will end

up with a refined version so you can see

definitely the model has improved on a

lot of details so circling back to the

the realistic side or at least quote

unquote realistic in this case I did try

generating up a superhero full body pose

in a comic book style and we did get our

guy although you will see that there are

some issues uh going on in the face

there um that's that's fine that's if

that's where the technology is right now

that's where it is if you are somebody

that is good with 3D I'm sure that that

would be a super easy fix but for

someone like me I mean honestly the

simplest solution is just to start

generating Up characters that are either

wearing helmets or Mas like this kind of

Warhammer inspired Space Marine so if

you do end up with a model that you like

you just don't like the overall kind of

look or Vibe of it you can always

download it uh and just download it as

an fbx uh and then kick back over to the

main menu where you can use this AI

texturing module um you simply come and

hit new project uh we'll describe this

as robot uh and then upload your fbx

file here once that's in you see that

our model is here but textureless uh and

then you can come through and prompt for

what textures you would like to see for

example here we take our untextured

model and run it with the prompt a

cyberpunk robot with black metal armor

uh and yeah we get this which looks

pretty cool now you might be thinking

well that's kind of cool but what can I

do with that it's just a character

that's just sort of standing there I am

not a 3D animator for that we're going

to download our model and take it over

to adobe's miimo so Adobe miimo is a

completely free uh Auto riger for 3D

characters it basically builds you know

the skeleton for 3D characters as well

as provides a number of animations for

it actually don't know why mixim Mo

isn't bigger than it is it seems to be

kind of one of those like lost forgotten

experiments of adobe they actually do

provide a number of different characters

here but what's really cool is that you

can upload your own characters so uh

we're going to take the zip file that we

downloaded of our cyber Punk Robot from

here you just basically Point these dots

you know chin wrists Etc and and once

you have everything lined up uh hit the

next button and in just a few minutes we

have our character completely rigged up

and we have total camera control over

everything it's really kind of a lot of

fun now you will notice that our

character is a little bit on the low res

side right now uh that's just a mixo

limitation to get those textures back

and into full res you would bring it

into something like blender um we're not

going to get into that cuz like I said

that is a rabbit hole that is 70 hours

deep so if you're just dipping your toes

into 3D like I am I think that Meshi

provides a really cool solution in that

you can generate up assets and then

bring them into a 3D software package

and start playing around with it if

you're on sort of a higher 3D level

Meshi does have a number of tutorials um

for you on incorporating it into blender

or Unity which they also do have a

plugin for yeah all of this is way above

my head though you can also apparently

generate your Textures in meshy up to 4K

so yeah that's that's pretty cool the

link to meshy is down below once again

they do do have a free tier but they

also have a pro and a Max tier as well

if you are interested in subscribing to

either of those if you use the coupon

code Theo t h EO uh you get 20% off my

thanks to meshy for sponsoring this

video Moving on but still kind of

staying within the realm of 3D I do love

a good magic trick and Nick Beller uh

posted this up which is just really

super cool check this out yeah yeah yeah

yeah that's really cool it's a 3D object

in front of of the TV that has that

background yeah it's really awesome so

to accomplish this Nick actually

sculpted the character in the

PlayStation software dreams and did so

in VR as well so uh yeah it's pretty

awesome it is a real shame that Sony

kind of gave up on dreams uh I do hope

that somebody ends up picking up the

technology especially as we're making

this push into to VR with things like

you know the Apple Vision Pro from there

Nick took the model out into AR software

I believe it was Luma Labs that he ended

up using um yeah and there you go and

finally using a still image export of

the environment on his TV kind of like

it's a duct tape version of Disney's the

volume and placing the AR character in

front of it we we end up with this which

is I don't it's so cool I just love

experiments like this speaking of Dreams

which was developed by a company called

media molecule which I do not believe

exists anymore uh one of the programmers

and co-founders of media molecule uh

Alex Evans is now working at Mid Journey

turns out you can learn a lot just by

stumbling around LinkedIn now I don't

necessarily know what work Alex is doing

at Mid Journey he is listed as a

principal research engineer and is doing

so remotely actually in the UK as well

but I don't think it's too much of a

stretch to think that he might be

working on some of the 3D aspects of mid

journey I do think that the general

consensus when it comes to all of this

is that you will be able to you know

prompt an image and then have camera

rotational tools and be able to sort of

move around in 3D space for your 2D

images but I mean honestly who knows

given the fact that Alex is working with

mid journey I mean who knows what this

is going to look like but it is

interesting to think that there is going

to be a little bit of that dreams DNA in

mid Journey also did you know that mid

Journey has a head of Hardware I mean I

had to go digging into that and as it

turns out yeah in an office hours that I

missed but luckily Nick St Pierre was at

um Mid journey is building Hardware in

office hours yesterday they mentioned a

newly formed Hardware team it's

currently focused on collecting data for

3D it's going to be an orb it's

described is a device that enables

anyone to organize and manage tens of

thousands of virtual 3D spaces I mean I

don't I still don't know what that means

but I don't know I kind of want one

rounding out we have the first AI City

uh you might remember a while back the

simulation created a fully AI generated

South Park episodes well they're back

this time with s Francisco props on that

name so s Francisco is populated by a

bunch of AI agents all of whom have been

prompted to have wants needs and desires

and interact and learn from one another

the agents have instructions to fall in

love and in fact the agents do have

lifespans so they actually die as well

additionally they relax by playing old

school Nintendo games up up down down

left right left right kids now I'll say

the overall graphical interface of s

Francisco is very much in line with that

South Park animation that they did as

you can see we're kind of scrolling

around through the city here zering in

on one of the characters and uh you know

we see her talking to and Alexa it very

much looks like the South Park style of

Animation so yes while the overall

visual presentation does look a lot like

something we've already seen uh I'm

actually a lot more interested in what's

happening under the hood because

apparently San Francisco is just running

all the time so it's kind of like a

hyperactive version of The Sims That

Never Ends the ultimate goal that the

simulation are trying to achieve with

Sim Francisco is Agi and I I mean I

don't know who knows maybe it'll work it

actually really reminds me of uh the old

Simpsons episode with Lisa's tooth if

there's one thing that we've learned if

the Simpsons did it it ends up coming

true on that note I thank you for

watching my name is Tim