Generate STUNNING SET EXTENSIONS For Your Projects! [2D & 3D | FREE Blender + AI]

Summary

This video introduces two AI-driven workflows for creating seamless set extensions in film projects. The first workflow allows users to add any object to an image, similar to Photoshop’s tools but more advanced and free. The second workflow integrates 3D models into footage, considering lighting, colors, and style for a realistic result. The video provides a detailed step-by-step guide on how to install and use the required software, including setting up COM UI and managing models, and shows how to implement these workflows for both 2D and 3D set extensions in film scenes.

Takeaways

• 😀 Two free workflows are introduced for creating set extensions: one for selecting and adding items to images, and the other for integrating 3D models seamlessly.
• 💻 The first workflow functions similarly to Photoshop's generator field, allowing users to select areas of an image and add desired objects.
• 🌐 The second workflow allows for the input of 3D models, considering light direction, color, and style to blend them into footage.
• 🛠 Users need to install Com UI, a node-based interface for AI models, and follow a step-by-step guide to download models and tools.
• 🔧 The process includes tracking footage using After Effects, DaVinci Resolve, or Blender for camera tracking, followed by applying the set extension.
• 🌍 A demonstration of transforming a city image into a sci-fi scene with an overgrown spaceship ruin is shown as part of the workflow.
• 🎥 Users can generate high-quality image extensions, blending them with original footage by adjusting prompts, masks, and other settings.
• 🏠 A second 3D workflow is demonstrated for adding a 3D house to footage, using render passes in Blender to integrate AI-generated textures and models.
• 📈 The set extension workflows support scaling, depth passes, and line art for maintaining consistency in image resolution and style.
• 🤖 Users can easily swap textures or prompts to generate different variations of objects, such as turning a cozy farmhouse into a post-apocalyptic shed.

Q & A

What are the two workflows introduced in the video?

The first workflow allows you to select an area of your image and add anything you want, similar to Photoshop's generator field but better and free. The second workflow lets you input a 3D model, and the AI seamlessly integrates it, considering light direction, colors, and the general style of the original footage.

What are some key tools mentioned for tracking footage?

The video mentions tools like After Effects for camera tracking, DaVinci Resolve for point tracking, and Blender for full 3D tracking.

What is ComfyUI, and why is it important in the workflow?

ComfyUI is a node-based interface for Stable Diffusion and other AI models. It is crucial because the workflows introduced rely on ComfyUI to generate and integrate AI-powered set extensions or 3D elements seamlessly.

What models are required to run the workflows?

The models mentioned include the Wild Card Turbo checkpoint, two control net models, and the Ultra model. These are essential for generating images and blending them with the original footage.

How does the AI handle light direction and realism in the image generation process?

The AI understands the entire image, including elements like sunlight direction, casting correct shadows, and adjusting black values for distant parts of the image. This helps maintain realism and visual coherence.

How do you generate a set extension using a specific frame in ComfyUI?

First, export the desired frame, then select the area for the set extension in the mask editor. Input a prompt describing the extension (e.g., 'spaceship ruin'), and the AI generates an image that integrates seamlessly into the original footage.

What steps are taken to avoid visible seams between the original footage and the generated image?

The workflow uses masks to blur the seams, helping to blend the newly generated image with the original. Line art and reference control nets also help by extracting lines and using the original image as a reference for consistency.

How does the 3D set extension process differ from the 2D one?

For 3D set extensions, you need to track the 3D camera using software like Blender, model a simple geometry for the extension, and export render passes (e.g., depth and line art) that help the AI generate a 3D image, which is later integrated into the scene.

What is the purpose of render passes in the 3D workflow?

Render passes, such as the depth pass and line art pass, provide the AI with detailed geometry and spatial information. These passes help the AI accurately place and integrate the 3D extension into the original footage.

How can the generated textures be projected onto 3D models in Blender?

After generating the texture, you can create a projection shader in Blender, UV project the texture onto the 3D model, and match it to the original footage. This ensures that the texture aligns perfectly with the geometry and camera view.