Let's Explore Sora + New Runway Tools (Essential AI News)
Summary
TLDR今週のAI映画ニュースでは、RunwayとP Labsの新機能、Soraの深掘り、Adobeが発表した音楽編集を変革する新ツールなどが紹介されました。ShelbyがAI制作のプロセスやデモクラシー化されたストーリーテリングの将来像について話すために、TVアカデミーやUSC、Sun Valley映画祭に招待されました。また、グラスゴーでのユーモラスなWilly Wonka体験や、AIを利用した映像制作の新技術、音楽ツールの進化についても触れられています。この動画は、AIが映像制作、音楽編集、ストーリーテリングをどのように変えていくかについての最新情報を提供しています。
Takeaways
- 🎥 New motion brush feature in Runway allows for easy object selection
- 🎬 Pabs released auto lip sync to add synced lips to videos
- 🧠 Emo converts images to realistic animated faces based on audio
- 🌄 Sora seen as a 'world generator', great for VFX and custom animations
- 🎶 Adobe music tool creates dynamic songs and music loops from text
- 👩💻 AI design course enrollment open at Curious Refuge
- 🎞 Nicholas Newbert created impressive AI-generated fashion dystopia film
- 🌍 Earth home turkey farm video envisions Earth after human destruction
- 🎵 Justin Kaplan made an AI music video with fantastic water simulations
- 🧠 AI can now predict Alzheimer's 7 years in advance
Q & A
このスクリプトは何のニュースですか?
-このスクリプトはAIフィルムニュースの週報です。RunwayとP Labsの新機能、SoraとAdobeの新しい音楽ツールなどのトピックをカバーしています。
Outlines
🎥 AI映画ニュースの週間概要
このパラグラフでは、AI映画制作に関する最新の動向として、RunwayとP Labsの新機能、Soraの詳細な解説、そしてAdobeが発表した革新的な音楽ツールについて語られています。また、ShelbyがAI制作プロセスと民主化されたストーリーテリングの未来について話すために、TVアカデミーやUSC、サンバレーフィルムフェスティバルに招待されたことが紹介されています。さらに、グラスゴーでのウィリー・ウォンカ体験の面白い失敗例や、Runwayのモーションブラシツールの新機能、P Labsによる自動リップシンク機能の追加など、様々なAI関連の進展が述べられています。特に注目されるのは、emoという新しいホワイトペーパーで、音声だけで画像をアニメーション化する技術が紹介されている点です。
🚀 Soraと新しいAIツールの紹介
このパラグラフでは、Soraのレンダリング時間とその他の注目すべき機能に焦点を当てています。Soraが世界を生成し、オブジェクトがどのように存在するかを理解する点で他のAIビデオツールと一線を画すと説明されています。特に、画像からビデオへの変換、モーションデザインへの応用、ビジュアルエフェクトの作成、ビデオの拡張や無限ループの作成、フッテージのスタイライズや異なるビデオの組み合わせなどの機能が紹介されています。また、Adobeが新しい音楽ツールを開発中であることや、LTX Studioという新しいAIストーリー作成プラットフォームについても触れられています。
🎶 Adobeの新音楽ツールとAIデザイン
Adobeが開発中の新しい音楽ツールについて詳しく説明しています。このツールは、シンプルな曲をダイナミックなスタイルに変換し、楽曲をカスタマイズする機能を提供します。また、テキストプロンプトからループを作成できる点も特徴です。このツールがAIワークフローを高品質なポストプロダクションパイプラインに変える可能性があると述べられています。さらに、プロジェクトエイオンのNvidia Inceptionプログラムへの参加、Google Geminiの再起動計画、およびGoogleの画像生成ツールに関する議論も含まれています。最後に、AIデザインに関するJulie Wheelandのポッドキャストエピソードと、AI広告コースおよびAI映画製作コースの開講情報が共有されています。
🌍 AIを活用したプロジェクトと映画
このパラグラフでは、AIを利用した著名なプロジェクトと映画について紹介しています。特に「not a dystopia」というプロジェクトや、MITのハッカソンで作成された「Earth home turkey farm」プロジェクト、Justin KaplanによるAI音楽ビデオコンセプトなどが取り上げられています。これらのプロジェクトは、AIの力を利用してユニークなビジュアル表現を実現している例として挙げられています。また、AIがアルツハイマー病を最大7年前に予測できるようになったというUCサンフランシスコの研究チームの成果も紹介されており、AIの医療分野での応用可能性にも言及されています。最後に、視聴者に感謝を述べ、YouTubeでの購読を促しています。
Mindmap
Keywords
💡Runway
💡Pabs
💡Sora
💡Adobe music tool
💡AI filmmaking course
💡AI advertising course
💡AI art direction
💡AI music videos
💡MIT hackathon
💡Alzheimer's detection
Highlights
Runway came out with a brand new feature inside of their motion brush tool that allows you to automatically select elements inside of your scene
Pabs came out with the ability to automatically add synced lips to your footage inside of Pabs
There's a new white paper about a tool called Emo that converts images into animated faces with just audio
Sora is actually creating a world and understanding how objects live, so it has capabilities like image-to-video, visual effects, extending videos, infinite loops, stylizing footage, and combining videos
We don't know when Sora will be available publicly or what the user experience will be like, but it will likely change creative video production forever
A new tool called LTX Studio seems interesting - it's an all-in-one platform for creating AI stories with storyboards, art direction, and animation
Adobe announced a new music tool that converts songs into dynamic styles, customizes songs, and creates loops from text prompts
Project Aeon, which creates automatic videos from articles, joined the Nvidia Inception program and was founded by a Curious Refuge student
Google plans to relaunch Google Imagen soon, but chatGPT is still much more popular based on Google Trends data
Our AI Advertising course enrollment is open and is likely to sell out again
Our AI Filmmaking course enrollment is at its biggest session ever
Nicholas Newbert created an interesting AI art project called Not a Dystopia with a cool aesthetic
Siki Wang created a funny travel video for Earth after humans destroyed it, using motion design and sound design well
Justin Kaplan made an AI music video concept with great consistent shots and fantastic water simulations
AI can now predict Alzheimer's 7 years early, allowing earlier treatment and better quality of life
Transcripts
we have some brand new features inside
of Runway and P Labs we'll take a deep
dive into Sora and Adobe just announced
a new music tool that is probably going
to change the way that we edit and
create songs Forever this is your AI
film news of the week first off I'd like
to thank the TV Academy for inviting
Shelby to come out and share the process
of creating AI Productions along with
our vision for creating a democratized
storytelling future we'd also like to
thank USC for inviting Shelby to come
out and speak about the ethics of AI
along with the Sun Valley film festival
in Idaho for having her come out and
talk about AI Productions Shelby has
really been on the speaker circuit
lately and that brings us to our first
story and of course this is the most
important story and I am talking about
the Willy Wonka experience in Glasgow
have you seen this it's absolutely
hilarious basically some people put
together a landing page with the help of
AI that basically talks about this Willy
Wonka experience and it has this
beautiful enchanted garden and an
imagination lab in a Twilight tunnel and
then when people showed up they got
this and it's basically a
warehouse with some really cheap props
and some tapestries on the wall and a
bouncy house and the very best part was
this Oompa
Loompa in tool new Runway came out with
a brand new feature inside of their
motion brush tool it basically allows
you to automatically select elements
inside of your scene it's really easy to
use so I'm inside of Runway Gen 2 all
you have to do is go to the motion brush
tool and make sure you have autod detect
area selected so basically now we can go
in and select different parts of our
frame and you can see it creates an
outline automatically which makes it
really easy for selecting certain app
aspects and the cool part is because
it's a multi-motion brush tool you can
select different brushes here and then
we can go in and select different parts
of our frame like this and go in and
adjust the movements and animation and
all of that now the thing that I should
point out and I didn't really realized
this at first when I started using this
tool is you don't actually have to
select every single Pixel of the part of
the frame that you're trying to select
if there's some edges here like with
this stuffed animal character that are
peeking through it's totally okay it
still will animate it the brush is
really an approximation and it doesn't
have to be perfect pabs also came out
with the ability to automatically add
synced lips to your footage inside of
pabs this works by uploading an image
and adding in lips and also adding in
video if you add in video and then sync
the lips you can get longer Generations
so I do recommend going with that route
it's really easy to use let me show you
how to use it to use the tool just just
go to pabs and then go to the image or
video button and select the video or
image that you want to add sync lips to
I have this animated shot that was
actually animated inside of P Labs with
a Runway image and I want to add lips
onto this character and go ahead and
click open so now all you have to do is
go to the lip sync and we will upload
our file I have a voice from 11 laps
let's take a listen listen to me
carefully I've made my decision tomorrow
we attack the enemy castle at dawn Okay
cool so we'll import that into P Labs
attach and continue and then click
generate okay the render took about a
minute let's check out our final result
listen to me carefully I've made my
decision tomorrow we attack the enemy
castle at
dawn okay so this is one of the best
Generations that I've seen using this
tool it does a great job with animated
styles now we also ran some experiments
that were more grounded in reality or at
an angle and the results were definitely
mixed so it's not a perfect tool but it
is incredibly easy to use you'll find a
link below this video to check out Pika
and we also have the assets that I just
used in case you want to download them
and play around with this tool there's
also a new white paper that came out
this last week called emo that basically
converts your images into animated faces
with just audio now we've seen tools
like this in the past this one seems
fairly realistic and the results are
very impressive so we have this image of
Audrey heern here and we add in some
audio and here is the
result never knew you were the someone
waiting for
me we were just kids when
we love so as you can see the results
are pretty realistic it has I blinks it
has the eyebrows that raise the cheeks
expand and I think that this underlying
technology is actually going to push
tools like the new Pika lip sync tool
even further So within about 3 to 6
months we're going to have user facing
lip sync tools that look absolutely
incredible this really reminds me of a
demo that we saw recently of flawless AI
which is really branding itself as a
very high-end industry tool for
converting films into other languages is
with lip sync it looks very realistic
you can check out the link to Flawless
below this video as well last week we
talked about Sora and how some of the
renders are taking a while for example a
one minute clip can take up to an hour
to render and some people mentioned that
those render times are actually not that
bad especially when you compare them to
a modern VFX Pipeline and we totally
agree so this week we wanted to talk
about some awesome features inside of
Sora that aren't getting as much
attention but we think our are really
big deals you'll find a link to a page
below that talks about Sora being a
world generator and that's what makes
this so unique compared to other AI
video tools it's actually creating a
world and understanding how objects live
so the first feature I want to talk
about is image to video so Sora like so
many video tools out there allow you to
take an input image and animate it so we
have this footage of this dog here he
blinks the camera is a little handheld
the parallaxing in the background looks
realistic it also looks very interesting
for motion design so we have this 2D
image of these monsters here and then it
animated the monsters and it looks
pretty darn good it would have taken
quite a while to rig all these
characters in a animation tool like
after effects it's also capable of doing
visual effects so we have this Cloud
image that says Sora and it animates
with particle physics now obviously you
may want to go in and clean up a few of
those things using some rotoscoping
techniques but it looks really good
there was also this example of this wave
inside of a museum and the Guy's surfing
and he does turn into two Surfers there
at the end but the water simulation is
really really impressive it also has the
ability to extend videos so you can make
videos longer so there's an example here
of this trolley car inside of San
Francisco basically they added new
intros to the beginning of each video
example here and it's really interesting
to imagine that in the future if you
needed a longer take or perhaps you
wanted to transition from some crazy
sci-fi world into reality you'll be able
to use tools like Sora to make that
happen it also has the ability to create
infinite Loops so there's this amazing
example on their website of someone
riding a bike and basically it Blended
the video together to where they
basically ride in a loop forever it also
has the ability to stylize footage so
think about a really advanced Runway gen
1 style experience we have this input
footage of a car driving through a
forest and then we can change it to a
jungle and here are the results they
look absolutely fantastic it also has
the ability to combine two different
videos together so there's this really
great example of a butterfly basically
flying underwater or through a Roman
Coliseum that gets converted into this
underwater scene there's this really
weird like lizard mixed with a bird
character and uh of course this lion
running after a Jeep is just absolutely
fantastic there's quite a few other
features that we could talk about here
now of course everyone's question is
when can we use Sora and the truth is we
don't know when we'll be able to use it
and we also don't know what that user
experience is going to be like but of
course Sora is likely going to change
the way that we do creative video
production forever and we we can't wait
to check out the tool and if anyone
wants access to Sora we found this video
on X that perfectly sums it up with some
lip sying from pabs thinking about
getting your hands on Sora forget it
it's not
available so there you go there was also
a new tool announced this week called
LTX Studio that seems really interesting
and is really branding itself as an
all-in-one platform for creating AI
Stories the tool is on weit list right
now so you don't actually have the
ability to go in and test it yet but
basically what we can tell from the
marketing is you will be able to create
storyboards with consistent characters
all in one place art direct those
storyboards and then animate them so
it's an all-in-one experience that
allows you to hypothetically create a
rough AI film all in one place it's a
really ingenious way of thinking about
the process of putting an AI video
together there are many different tools
for performing different task in an AI I
video production pipeline but having a
tool that consolidates a lot of those
tools together will likely save time and
be an interesting alternative Adobe also
announced that they are working on a
brand new music tool that allows you to
do a lot of really impressive things
first off the song allows you to convert
simple songs into Dynamic Styles and you
also have the ability to customize the
song so unlike tools like sunno where
you basically have the ability to create
a verse and then a chorus with the Adobe
music tool you can go in and customize
and say I want Chorus Verse then chorus
or customize your song further you also
have the ability to create Loops from
text prompts so in the video example
they had this
song okay that's not bad I was curious
because in the video they actually have
their prompt groovy Rock in there so I
basically went to suo typed in groovy
Rock and then clicked instrumental
version three and generated a song and
this is the song quality that I got from
[Music]
there it basically sounds like some
pretty basic stock music but not bad I
think this Adobe music tool is a bigger
deal than we realize because this will
really Elevate some of these AI
workflows from just prev visualization
and creating experimental Concepts into
legitimate post-production pip lines
that can be used on high-end projects
I'd also like to extend a huge
congratulations to project aeon for
joining the Nvidia Inception program it
was actually founded by a curious Refuge
student named Dan and it creates
automatic videos for Publishers to
create videos of their content so
articles or reviews into automatic
videos it's really an incredible tool
and we'll actually be doing a live
stream with Dan inside of our program on
March 7th so if you're student at
curious Refuge feel free to hop in and
you can ask Dan all the nerdy questions
about his incredible tool Google also
announced that it plans to relaunch
Google Gemini soon they actually lost 70
billion in Market evaluation on the news
that they had to take down their image
generator and to create a little context
for what was happening with Google's
image generator basically they went in
and added in mandatory University into
their photos but that doesn't make sense
when you prompt in certain things so for
example people typed in founding fathers
and there were founding fathers that
were African-American and didn't really
make any sense there were also people
that prompted in Vikings and popes so
this really does bring up a much larger
conversation obviously you want AI
models to not reflect the biases that
are inherent inside of our data but you
also want them to be factually accurate
in his historically helpful whenever you
are prompting in images so it's
definitely a back and forth and I'm
really surprised that Google missed the
mark so big with the launch of their
image tool and speaking of Google we
actually came across an interesting
conversation point this last week where
you know we really feel like chat gbt
and Google they kind of battle it out
but when you actually use Google Trends
to see what is being searched for more
it's not even close on its best day
Google Gemini is 11 times less popular
than chat GPT so chat GPT still seems to
be the language model of choice for the
world and in so many tests whether we're
talking about custom gpts or just simply
the knowledge base and its ability to be
more creative with less limits Chad GPT
is still our preferred language model
this last week we also dropped a podcast
episode with Julie Wheeland who goes by
Julie W design online she shared a lot
of good insights for how AI is shaping
graphic design she also shared some Pro
tips on working on AI projects
specifically how she gets such high
quality from her work and she also went
into a lot more detail about how she's
able to make more money as a graphic
designer using the power of artificial
intelligence and while we're talking
about AI design I want to point out that
enrollment is open for our AI
advertising course this is our very
first session ever and we will likely
sell out yet again we have people from
some of the largest Brands and ad
agencies inside of this program and we
can't wait for you to learn alongside
them we also have enrollment open for
our AI film making course if you want to
learn how to use AI to assist with every
aspect of the Film Production pipeline
you'll find a link below this video this
is already our biggest session ever so
congratulations to everyone who was able
to snag a spot and that brings us to our
AI films of the week I want to kick
things off with a project from Nicholas
Newbert Nicholas works at Runway and is
basically one of the best AI art
directors in the world he did this
really interesting project called not a
dystopia and it features some really
good curation and it kind of has this
kylo Ren meets High fashion Vibe which I
think looks very good so great job on
that Nicholas next up we have this
project called Earth home turkey farm
from siki Wang who actually created this
during the MIT hack on recently that uh
our instructor Dave Clark actually spoke
at and the project is really funny it's
kind of like a travel video for Earth
after humans have basically just
decimated the planet and it's a really
good use of motion design and sound
design to create a cinematic atmosphere
so I recommend checking it out you'll
find a link below this video and our
final film of the week comes from Justin
Kaplan who put together this AI music
video concept I think that it has some
really good consistent shots some of the
simulations especially with the water
look absolutely fantastic and I think it
really does showcase the potential of
using AI for music videos and finally a
research team from UC San Francisco
basically said that AI can now predict
Alzheimer's disease up to 7 years in
advance by diagnosing the disease
earlier they're better able to treat the
patients and increase their quality of
life thank you so much for watching this
episode of AI film news of course like
always you can subscribe over on Curious
Refuge to get AI film news directly to
your inbox every single week and if you
want to subscribe here on YouTube you
will be the first to get AI film news
tutorials and the latest AI content
thank you so much for being an awesome
member of the Curious Refuge Community
we'll see you in the next one
5.0 / 5 (0 votes)