Use AI & Daily to generate Automatic Video Highlights

Summary

John from Daily introduces an Automated Video Highlights tool that transforms a one-hour poker game into a TikTok-style summary using AI. The tool combines voice, video, and recording APIs to create contextually relevant, shareable content without manual editing. It uses speech-to-text AI to transcribe audio and game state data to identify key moments, then employs a cloud agent and GPT-4 to summarize the game into a JSON timeline. The VCS stitches the media into a dynamic, shareable video, showcasing the potential of Daily's AI toolkit for developers.

Takeaways

• 🃏 Automated Video Highlights is a new tool designed to create engaging, TikTok-style summaries of long events.
• 👥 The tool was tested during a virtual card game with Daily's engineering team, but not everyone could participate.
• 🎥 It combines voice, video, and recording APIs to produce contextually relevant content without manual editing.
• 💾 Individual video and audio files for each participant are stored in an S3 bucket for later use.
• 🔍 Speech-to-text AI transcribes audio tracks to create a textual record of the game, aiding in identifying key moments.
• 📊 Game state data and player text chats are also captured to provide additional context for the AI workflow.
• 🤖 A cloud-based AI, such as GPT-4, is used to summarize the game by highlighting interesting moments based on the provided data.
• 📝 The AI generates a JSON file outlining a timeline of events to guide the media compositing process.
• 📚 Daily's Video Component System (VCS) automatically stitches together the individual media files according to the timeline.
• 🎨 VCS allows for dynamic rendering and the addition of graphic overlays for visual engagement and branding.
• 🚀 The tool is part of Daily's AI toolkit, and the company is excited to see how developers will integrate it into their apps and products.
• 🔗 Interested parties can learn more by visiting daily.co/AI or contacting Daily directly.

Q & A

What is the purpose of the Automated Video Highlights tooling mentioned in the script?

The Automated Video Highlights tooling is designed to create short, engaging, TikTok-style summary reels from longer video content, focusing on moments of high stake drama, without requiring manual video editing.

How does the Automated Video Highlights tooling utilize Daily's APIs and AI technology?

The tooling combines Daily's voice, video, and recording APIs as part of an AI-powered, cloud-based workflow to generate contextually relevant, shareable content.

What is the duration of the poker game that the team played?

The poker game took around an hour.

Why is it impractical to share the full hour-long recording of the poker game?

Sharing the full recording is impractical because realistically, no one would sit through a start-to-finish screen recording of an hour-long game.

How does the Automated Video Highlights tooling handle the storage of individual participant data?

It leverages Daily's raw track recording mode to store individual video and audio files for each participant directly to an S3 bucket.

What role does speech-to-text AI play in the Automated Video Highlights process?

Speech-to-text AI is used to transcribe audio tracks for all players and the card dealer, providing a fully diarized textual record of what was said throughout the game.

What additional data is used to provide context for identifying key moments of action in the game?

The game state data and player text chats are used to provide extra context for identifying the key moments of action.

How does the cloud-based AI workflow for producing the final composite timeline work?

A custom cloud agent uses the S3 bucket's transcripts and timestamp data to converse with an LLM like GPT-4, which summarizes the game by highlighting interesting moments, resulting in a JSON file of events.

What is VCS and how does it contribute to the Automated Video Highlights process?

VCS, or Video Component System, is Daily's cloud-based compositor that automatically stitches together individual audio and video files into shareable content based on the timeline of events.

How does VCS enhance the final output of the Automated Video Highlights?

VCS dynamically renders relevant scenes based on the context, and allows for additional graphic overlays for visual engagement and branding.

What are the next steps or resources for developers interested in using Automated Video Highlights in their apps and products?

Developers can learn more by visiting daily.co/AI or reaching out to Daily for further information.

Outlines

00:00

🃏 Automated Poker Night Highlights

John, an engineer at Daily, introduces the Automated Video Highlights tool developed by Daily. The tool was used to condense a one-hour virtual poker game into a short, engaging TikTok-style summary. The process involves combining voice, video, and recording APIs with AI to create contextually relevant, shareable content without manual editing. The workflow includes storing individual video and audio files in an S3 bucket, transcribing audio tracks for all participants, and using game state data and player text chats to identify key moments. A cloud agent converses with an LLM, like GPT-4, to summarize the game, resulting in a JSON timeline of events that guides the media compositing process.

Keywords

💡Automated Video Highlights

This term refers to the technology that creates short, engaging video summaries from longer events or recordings. In the video's context, it is used to condense a one-hour poker game into a TikTok-style summary reel, capturing only the high-stakes drama moments. The script mentions that this tool does not require manual editing, emphasizing its efficiency and ease of use.

💡CloudPokerNight.com

This is the name of the virtual platform where the team at Daily played cards. It serves as the setting for the video script, illustrating the need for the Automated Video Highlights tool to share the experience with those who couldn't participate. It is a fictional example used to demonstrate the application of the technology.

💡Daily

Daily is the company mentioned in the script, which has developed the Automated Video Highlights tool. It represents the organization behind the innovation and is central to the video's narrative, showcasing their capabilities in creating AI-powered solutions for content sharing.

💡AI-powered workflow

This phrase describes the process driven by artificial intelligence to automate tasks. In the script, it is used to explain how Automated Video Highlights combines voice, video, and recording APIs to create contextually relevant content. It underscores the role of AI in streamlining the video creation process.

💡S3 bucket

An S3 bucket is a data storage virtual container available in Amazon Web Services (AWS). In the video script, it is used to store individual video and audio files for each participant, which are then used in the Automated Video Highlights process. It is a key component of the cloud-based infrastructure supporting the workflow.

💡Speech-to-text AI

This technology converts spoken language into written text. In the context of the video, it is used to transcribe audio tracks from the poker game, creating a textual record that aids in identifying key moments for the highlights. It exemplifies the integration of AI in content analysis and summarization.

💡Game state data

This refers to the information about the current status of a game, including scores, player positions, and other relevant details. In the script, it is mentioned as part of the data used to provide context for the AI when identifying key moments for the video summary, enhancing the accuracy of the highlights created.

💡LLM (Large Language Model)

LLM stands for Large Language Model, which is a type of AI that processes and generates human-like text based on input data. In the video, GPT-4, an example of an LLM, is used to summarize the game by highlighting interesting moments from the timeline of events, demonstrating the application of advanced AI in content creation.

💡JSON file

JSON stands for JavaScript Object Notation, a lightweight data-interchange format. In the script, the JSON file is the output that contains the timeline of events, indicating which media tracks to show during the compositing process. It is a crucial element in the automated video editing workflow.

💡Video Component System (VCS)

VCS is a system mentioned in the script that automates the process of stitching together video and audio files into a final product. It is used to dynamically render scenes based on the context provided by the JSON timeline, showcasing the automation capabilities of Daily's technology in creating shareable video content.

💡Graphic overlays

These are visual elements superimposed onto video content for added engagement or branding. In the video script, the possibility of applying graphic overlays during the rendering process with VCS is mentioned, highlighting the customization options available in the final output of the Automated Video Highlights.

Highlights

Introduction of Automated Video Highlights tooling at Daily.

Virtual card game experience shared with the team.

The need for a short, engaging summary reel of the game.

Utilization of Daily's voice, video, and recording APIs.

AI-powered, cloud-based workflow for creating video highlights.

No manual video editing required for contextually relevant content.

Storing individual video and audio files in an S3 bucket.

Transcription of audio tracks using speech-to-text AI.

Inclusion of game state data and player text chats for context.

Custom cloud agent conversing with an LLM like GPT-4.

Summarization of the game by highlighting interesting moments.

Creation of a JSON file for the timeline of events.

Automatic stitching of audio and video files with VCS.

Flexibility in rendering final output with VCS.

Application of graphic overlays for visual engagement.

Excitement for developers to use Automated Video Highlights in apps and products.

Invitation to learn more about Daily's AI toolkit.

Transcripts

Jon Taylor: Hey, my name is John, an engineer here at Daily.

Recently, we got together for a virtual game of cards over at CloudPokerNight.com.

We had a lot of fun, but unfortunately everyone in the team wasn't able to

make it at the time, and we'd really like to share that experience with them.

This is the perfect opportunity to use the new Automated Video Highlights tooling

that we've been working on at Daily.

Let's take a look.

Our game of poker took around an hour.

Realistically, no one is going to sit through a start to

finish screen recording of that.

Automated Video Highlights can be used to create a short, engaging, TikTok-style

summary reel, cutting it down to just the moments of high stake drama.

Automated Video Highlights work by combining Daily's voice, video,

and recording APIs as part of an AI-powered, cloud based workflow.

The result is contextually relevant, shareable content that doesn't require

any manual video editing at all.

Dig a little bit deeper as to how it works.

By leveraging Daily's raw track recording mode, we can store individual

video and audio files for each participant directly to an S3 bucket.

Audio tracks for all players, as well as the card dealer, are also

obtained and transcribed using speech-to-text AI, giving us a fully

diarized, textual record of what was individually said throughout our game.

We're also going to grab the game state data and the player text chats to

help provide that little bit of extra context for when it comes to identifying

the key moments of action later on.

Let's turn our attention to a cloud-based AI workflow for producing

our final composite timeline.

With our S3 bucket full of transcripts and timestamp data, we can use a custom

cloud agent to begin a conversation with an LLM, such as GPT-4 in this case.

With direction provided via a custom prompt and our game data, GPT-4

will attempt to summarize the game.

It cycles through our timeline of events and highlights any

moment that it deems interesting.

At the end of the process, we'll arrive at a single JSON file, our timeline of

events, which will tell us which media tracks to show at a certain point in

time during the compositing process.

All we need to do now is take our individual audio and video files

and stitch them together into that final piece of shareable content.

Now normally this would be a very manual and labor intensive process, but with

Daily's cloud-based compositor, VCS, Video Component System, we can do all of

this automatically in just mere minutes.

VCS steps through each moment in our timeline, obtaining the related

raw video and audio tracks from the S3 bucket and rendering out the

relevant scene dynamically based on the context of what's happening.

It might be that in one scene we show the players placing their bets, and in another

we show a participant winning or losing.

Since we're using VCS, we also have the flexibility for how

we render our final output.

We could apply some additional graphic overlays for that extra bit

of visual engagement and branding.

We're just getting started with the cool things that you can do with

automated video highlights and it's a great use case for Daily's AI toolkit.

We're really excited to see how developers use this as part of their apps and

products and if you'd like to know more, please head on over to daily.co/AI

or reach out to us at any time.