Let's Explore Sora + New Runway Tools (Essential AI News)

Summary

今週のAI映画ニュースでは、RunwayとP Labsの新機能、Soraの深掘り、Adobeが発表した音楽編集を変革する新ツールなどが紹介されました。ShelbyがAI制作のプロセスやデモクラシー化されたストーリーテリングの将来像について話すために、TVアカデミーやUSC、Sun Valley映画祭に招待されました。また、グラスゴーでのユーモラスなWilly Wonka体験や、AIを利用した映像制作の新技術、音楽ツールの進化についても触れられています。この動画は、AIが映像制作、音楽編集、ストーリーテリングをどのように変えていくかについての最新情報を提供しています。

Takeaways

• 🎥 New motion brush feature in Runway allows for easy object selection
• 🎬 Pabs released auto lip sync to add synced lips to videos
• 🧠 Emo converts images to realistic animated faces based on audio
• 🌄 Sora seen as a 'world generator', great for VFX and custom animations
• 🎶 Adobe music tool creates dynamic songs and music loops from text
• 👩‍💻 AI design course enrollment open at Curious Refuge
• 🎞 Nicholas Newbert created impressive AI-generated fashion dystopia film
• 🌍 Earth home turkey farm video envisions Earth after human destruction
• 🎵 Justin Kaplan made an AI music video with fantastic water simulations
• 🧠 AI can now predict Alzheimer's 7 years in advance

Q & A

このスクリプトは何のニュースですか?

このスクリプトはAIフィルムニュースの週報です。RunwayとP Labsの新機能、SoraとAdobeの新しい音楽ツールなどのトピックをカバーしています。

Outlines

00:00

🎥 AI映画ニュースの週間概要

このパラグラフでは、AI映画制作に関する最新の動向として、RunwayとP Labsの新機能、Soraの詳細な解説、そしてAdobeが発表した革新的な音楽ツールについて語られています。また、ShelbyがAI制作プロセスと民主化されたストーリーテリングの未来について話すために、TVアカデミーやUSC、サンバレーフィルムフェスティバルに招待されたことが紹介されています。さらに、グラスゴーでのウィリー・ウォンカ体験の面白い失敗例や、Runwayのモーションブラシツールの新機能、P Labsによる自動リップシンク機能の追加など、様々なAI関連の進展が述べられています。特に注目されるのは、emoという新しいホワイトペーパーで、音声だけで画像をアニメーション化する技術が紹介されている点です。

05:00

🚀 Soraと新しいAIツールの紹介

このパラグラフでは、Soraのレンダリング時間とその他の注目すべき機能に焦点を当てています。Soraが世界を生成し、オブジェクトがどのように存在するかを理解する点で他のAIビデオツールと一線を画すと説明されています。特に、画像からビデオへの変換、モーションデザインへの応用、ビジュアルエフェクトの作成、ビデオの拡張や無限ループの作成、フッテージのスタイライズや異なるビデオの組み合わせなどの機能が紹介されています。また、Adobeが新しい音楽ツールを開発中であることや、LTX Studioという新しいAIストーリー作成プラットフォームについても触れられています。

10:01

🎶 Adobeの新音楽ツールとAIデザイン

Adobeが開発中の新しい音楽ツールについて詳しく説明しています。このツールは、シンプルな曲をダイナミックなスタイルに変換し、楽曲をカスタマイズする機能を提供します。また、テキストプロンプトからループを作成できる点も特徴です。このツールがAIワークフローを高品質なポストプロダクションパイプラインに変える可能性があると述べられています。さらに、プロジェクトエイオンのNvidia Inceptionプログラムへの参加、Google Geminiの再起動計画、およびGoogleの画像生成ツールに関する議論も含まれています。最後に、AIデザインに関するJulie Wheelandのポッドキャストエピソードと、AI広告コースおよびAI映画製作コースの開講情報が共有されています。

15:02

🌍 AIを活用したプロジェクトと映画

このパラグラフでは、AIを利用した著名なプロジェクトと映画について紹介しています。特に「not a dystopia」というプロジェクトや、MITのハッカソンで作成された「Earth home turkey farm」プロジェクト、Justin KaplanによるAI音楽ビデオコンセプトなどが取り上げられています。これらのプロジェクトは、AIの力を利用してユニークなビジュアル表現を実現している例として挙げられています。また、AIがアルツハイマー病を最大7年前に予測できるようになったというUCサンフランシスコの研究チームの成果も紹介されており、AIの医療分野での応用可能性にも言及されています。最後に、視聴者に感謝を述べ、YouTubeでの購読を促しています。

Keywords

💡Runway

Runway is an AI tool for creating visual media. The video discusses a new motion brush feature in Runway that allows for easier selection and animation of objects.

💡Pabs

Pabs is an AI tool that can automatically add lip sync to images and videos. The video demonstrates how Pabs can create realistic mouth movements.

💡Sora

Sora is an unreleased AI world building tool that can generate animated scenes and videos. The video speculates on Sora's capabilities.

💡Adobe music tool

Adobe announced a new AI-powered music tool that can remix songs and generate music loops from text prompts.

💡AI filmmaking course

The Curious Refuge platform offers an AI filmmaking course to teach how AI can assist with video production.

💡AI advertising course

Curious Refuge also offers an AI advertising course to teach how to apply AI in marketing and advertising.

💡AI art direction

The video praises Nicholas Newbert's skills in using AI tools to create unique and visually compelling media.

💡AI music videos

The video shares an AI-generated music video concept as an example of AI's potential for automatic music video creation.

💡MIT hackathon

An AI-generated travel video shown in the episode was created during the recent MIT hackathon event.

💡Alzheimer's detection

The video concludes by mentioning AI's ability to predict Alzheimer's disease earlier to improve treatment.

Highlights

Runway came out with a brand new feature inside of their motion brush tool that allows you to automatically select elements inside of your scene

Pabs came out with the ability to automatically add synced lips to your footage inside of Pabs

There's a new white paper about a tool called Emo that converts images into animated faces with just audio

Sora is actually creating a world and understanding how objects live, so it has capabilities like image-to-video, visual effects, extending videos, infinite loops, stylizing footage, and combining videos

We don't know when Sora will be available publicly or what the user experience will be like, but it will likely change creative video production forever

A new tool called LTX Studio seems interesting - it's an all-in-one platform for creating AI stories with storyboards, art direction, and animation

Adobe announced a new music tool that converts songs into dynamic styles, customizes songs, and creates loops from text prompts

Project Aeon, which creates automatic videos from articles, joined the Nvidia Inception program and was founded by a Curious Refuge student

Google plans to relaunch Google Imagen soon, but chatGPT is still much more popular based on Google Trends data

Our AI Advertising course enrollment is open and is likely to sell out again

Our AI Filmmaking course enrollment is at its biggest session ever

Nicholas Newbert created an interesting AI art project called Not a Dystopia with a cool aesthetic

Siki Wang created a funny travel video for Earth after humans destroyed it, using motion design and sound design well

Justin Kaplan made an AI music video concept with great consistent shots and fantastic water simulations

AI can now predict Alzheimer's 7 years early, allowing earlier treatment and better quality of life

Transcripts

we have some brand new features inside

of Runway and P Labs we'll take a deep

dive into Sora and Adobe just announced

a new music tool that is probably going

to change the way that we edit and

create songs Forever this is your AI

film news of the week first off I'd like

to thank the TV Academy for inviting

Shelby to come out and share the process

of creating AI Productions along with

our vision for creating a democratized

storytelling future we'd also like to

thank USC for inviting Shelby to come

out and speak about the ethics of AI

along with the Sun Valley film festival

in Idaho for having her come out and

talk about AI Productions Shelby has

really been on the speaker circuit

lately and that brings us to our first

story and of course this is the most

important story and I am talking about

the Willy Wonka experience in Glasgow

have you seen this it's absolutely

hilarious basically some people put

together a landing page with the help of

AI that basically talks about this Willy

Wonka experience and it has this

beautiful enchanted garden and an

imagination lab in a Twilight tunnel and

then when people showed up they got

this and it's basically a

warehouse with some really cheap props

and some tapestries on the wall and a

bouncy house and the very best part was

this Oompa

Loompa in tool new Runway came out with

a brand new feature inside of their

motion brush tool it basically allows

you to automatically select elements

inside of your scene it's really easy to

use so I'm inside of Runway Gen 2 all

you have to do is go to the motion brush

tool and make sure you have autod detect

area selected so basically now we can go

in and select different parts of our

frame and you can see it creates an

outline automatically which makes it

really easy for selecting certain app

aspects and the cool part is because

it's a multi-motion brush tool you can

select different brushes here and then

we can go in and select different parts

of our frame like this and go in and

adjust the movements and animation and

all of that now the thing that I should

point out and I didn't really realized

this at first when I started using this

tool is you don't actually have to

select every single Pixel of the part of

the frame that you're trying to select

if there's some edges here like with

this stuffed animal character that are

peeking through it's totally okay it

still will animate it the brush is

really an approximation and it doesn't

have to be perfect pabs also came out

with the ability to automatically add

synced lips to your footage inside of

pabs this works by uploading an image

and adding in lips and also adding in

video if you add in video and then sync

the lips you can get longer Generations

so I do recommend going with that route

it's really easy to use let me show you

how to use it to use the tool just just

go to pabs and then go to the image or

video button and select the video or

image that you want to add sync lips to

I have this animated shot that was

actually animated inside of P Labs with

a Runway image and I want to add lips

onto this character and go ahead and

click open so now all you have to do is

go to the lip sync and we will upload

our file I have a voice from 11 laps

let's take a listen listen to me

carefully I've made my decision tomorrow

we attack the enemy castle at dawn Okay

cool so we'll import that into P Labs

attach and continue and then click

generate okay the render took about a

minute let's check out our final result

listen to me carefully I've made my

decision tomorrow we attack the enemy

castle at

dawn okay so this is one of the best

Generations that I've seen using this

tool it does a great job with animated

styles now we also ran some experiments

that were more grounded in reality or at

an angle and the results were definitely

mixed so it's not a perfect tool but it

is incredibly easy to use you'll find a

link below this video to check out Pika

and we also have the assets that I just

used in case you want to download them

and play around with this tool there's

also a new white paper that came out

this last week called emo that basically

converts your images into animated faces

with just audio now we've seen tools

like this in the past this one seems

fairly realistic and the results are

very impressive so we have this image of

Audrey heern here and we add in some

audio and here is the

result never knew you were the someone

waiting for

me we were just kids when

we love so as you can see the results

are pretty realistic it has I blinks it

has the eyebrows that raise the cheeks

expand and I think that this underlying

technology is actually going to push

tools like the new Pika lip sync tool

even further So within about 3 to 6

months we're going to have user facing

lip sync tools that look absolutely

incredible this really reminds me of a

demo that we saw recently of flawless AI

which is really branding itself as a

very high-end industry tool for

converting films into other languages is

with lip sync it looks very realistic

you can check out the link to Flawless

below this video as well last week we

talked about Sora and how some of the

renders are taking a while for example a

one minute clip can take up to an hour

to render and some people mentioned that

those render times are actually not that

bad especially when you compare them to

a modern VFX Pipeline and we totally

agree so this week we wanted to talk

about some awesome features inside of

Sora that aren't getting as much

attention but we think our are really

big deals you'll find a link to a page

below that talks about Sora being a

world generator and that's what makes

this so unique compared to other AI

video tools it's actually creating a

world and understanding how objects live

so the first feature I want to talk

about is image to video so Sora like so

many video tools out there allow you to

take an input image and animate it so we

have this footage of this dog here he

blinks the camera is a little handheld

the parallaxing in the background looks

realistic it also looks very interesting

for motion design so we have this 2D

image of these monsters here and then it

animated the monsters and it looks

pretty darn good it would have taken

quite a while to rig all these

characters in a animation tool like

after effects it's also capable of doing

visual effects so we have this Cloud

image that says Sora and it animates

with particle physics now obviously you

may want to go in and clean up a few of

those things using some rotoscoping

techniques but it looks really good

there was also this example of this wave

inside of a museum and the Guy's surfing

and he does turn into two Surfers there

at the end but the water simulation is

really really impressive it also has the

ability to extend videos so you can make

videos longer so there's an example here

of this trolley car inside of San

Francisco basically they added new

intros to the beginning of each video

example here and it's really interesting

to imagine that in the future if you

needed a longer take or perhaps you

wanted to transition from some crazy

sci-fi world into reality you'll be able

to use tools like Sora to make that

happen it also has the ability to create

infinite Loops so there's this amazing

example on their website of someone

riding a bike and basically it Blended

the video together to where they

basically ride in a loop forever it also

has the ability to stylize footage so

think about a really advanced Runway gen

1 style experience we have this input

footage of a car driving through a

forest and then we can change it to a

jungle and here are the results they

look absolutely fantastic it also has

the ability to combine two different

videos together so there's this really

great example of a butterfly basically

flying underwater or through a Roman

Coliseum that gets converted into this

underwater scene there's this really

weird like lizard mixed with a bird

character and uh of course this lion

running after a Jeep is just absolutely

fantastic there's quite a few other

features that we could talk about here

now of course everyone's question is

when can we use Sora and the truth is we

don't know when we'll be able to use it

and we also don't know what that user

experience is going to be like but of

course Sora is likely going to change

the way that we do creative video

production forever and we we can't wait

to check out the tool and if anyone

wants access to Sora we found this video

on X that perfectly sums it up with some

lip sying from pabs thinking about

getting your hands on Sora forget it

it's not

available so there you go there was also

a new tool announced this week called

LTX Studio that seems really interesting

and is really branding itself as an

all-in-one platform for creating AI

Stories the tool is on weit list right

now so you don't actually have the

ability to go in and test it yet but

basically what we can tell from the

marketing is you will be able to create

storyboards with consistent characters

all in one place art direct those

storyboards and then animate them so

it's an all-in-one experience that

allows you to hypothetically create a

rough AI film all in one place it's a

really ingenious way of thinking about

the process of putting an AI video

together there are many different tools

for performing different task in an AI I

video production pipeline but having a

tool that consolidates a lot of those

tools together will likely save time and

be an interesting alternative Adobe also

announced that they are working on a

brand new music tool that allows you to

do a lot of really impressive things

first off the song allows you to convert

simple songs into Dynamic Styles and you

also have the ability to customize the

song so unlike tools like sunno where

you basically have the ability to create

a verse and then a chorus with the Adobe

music tool you can go in and customize

and say I want Chorus Verse then chorus

or customize your song further you also

have the ability to create Loops from

text prompts so in the video example

they had this

song okay that's not bad I was curious

because in the video they actually have

their prompt groovy Rock in there so I

basically went to suo typed in groovy

Rock and then clicked instrumental

version three and generated a song and

this is the song quality that I got from

[Music]

there it basically sounds like some

pretty basic stock music but not bad I

think this Adobe music tool is a bigger

deal than we realize because this will

really Elevate some of these AI

workflows from just prev visualization

and creating experimental Concepts into

legitimate post-production pip lines

that can be used on high-end projects

I'd also like to extend a huge

congratulations to project aeon for

joining the Nvidia Inception program it

was actually founded by a curious Refuge

student named Dan and it creates

automatic videos for Publishers to

create videos of their content so

articles or reviews into automatic

videos it's really an incredible tool

and we'll actually be doing a live

stream with Dan inside of our program on

March 7th so if you're student at

curious Refuge feel free to hop in and

you can ask Dan all the nerdy questions

about his incredible tool Google also

announced that it plans to relaunch

Google Gemini soon they actually lost 70

billion in Market evaluation on the news

that they had to take down their image

generator and to create a little context

for what was happening with Google's

image generator basically they went in

and added in mandatory University into

their photos but that doesn't make sense

when you prompt in certain things so for

example people typed in founding fathers

and there were founding fathers that

were African-American and didn't really

make any sense there were also people

that prompted in Vikings and popes so

this really does bring up a much larger

conversation obviously you want AI

models to not reflect the biases that

are inherent inside of our data but you

also want them to be factually accurate

in his historically helpful whenever you

are prompting in images so it's

definitely a back and forth and I'm

really surprised that Google missed the

mark so big with the launch of their

image tool and speaking of Google we

actually came across an interesting

conversation point this last week where

you know we really feel like chat gbt

and Google they kind of battle it out

but when you actually use Google Trends

to see what is being searched for more

it's not even close on its best day

Google Gemini is 11 times less popular

than chat GPT so chat GPT still seems to

be the language model of choice for the

world and in so many tests whether we're

talking about custom gpts or just simply

the knowledge base and its ability to be

more creative with less limits Chad GPT

is still our preferred language model

this last week we also dropped a podcast

episode with Julie Wheeland who goes by

Julie W design online she shared a lot

of good insights for how AI is shaping

graphic design she also shared some Pro

tips on working on AI projects

specifically how she gets such high

quality from her work and she also went

into a lot more detail about how she's

able to make more money as a graphic

designer using the power of artificial

intelligence and while we're talking

about AI design I want to point out that

enrollment is open for our AI

advertising course this is our very

first session ever and we will likely

sell out yet again we have people from

some of the largest Brands and ad

agencies inside of this program and we

can't wait for you to learn alongside

them we also have enrollment open for

our AI film making course if you want to

learn how to use AI to assist with every

aspect of the Film Production pipeline

you'll find a link below this video this

is already our biggest session ever so

congratulations to everyone who was able

to snag a spot and that brings us to our

AI films of the week I want to kick

things off with a project from Nicholas

Newbert Nicholas works at Runway and is

basically one of the best AI art

directors in the world he did this

really interesting project called not a

dystopia and it features some really

good curation and it kind of has this

kylo Ren meets High fashion Vibe which I

think looks very good so great job on

that Nicholas next up we have this

project called Earth home turkey farm

from siki Wang who actually created this

during the MIT hack on recently that uh

our instructor Dave Clark actually spoke

at and the project is really funny it's

kind of like a travel video for Earth

after humans have basically just

decimated the planet and it's a really

good use of motion design and sound

design to create a cinematic atmosphere

so I recommend checking it out you'll

find a link below this video and our

final film of the week comes from Justin

Kaplan who put together this AI music

video concept I think that it has some

really good consistent shots some of the

simulations especially with the water

look absolutely fantastic and I think it

really does showcase the potential of

using AI for music videos and finally a

research team from UC San Francisco

basically said that AI can now predict

Alzheimer's disease up to 7 years in

advance by diagnosing the disease

earlier they're better able to treat the

patients and increase their quality of

life thank you so much for watching this

episode of AI film news of course like

always you can subscribe over on Curious

Refuge to get AI film news directly to

your inbox every single week and if you

want to subscribe here on YouTube you

will be the first to get AI film news

tutorials and the latest AI content

thank you so much for being an awesome

member of the Curious Refuge Community

we'll see you in the next one