Style Transfer Using ComfyUI - No Training Required!

The video script discusses a new style transfer feature in ComfyUI, which allows users to control the style of stable diffusion generations by simply providing an image. This visual style prompting is compared to other methods like IP adapter, style drop, style align, and DB Laura, with the ComfyUI version standing out for its impressive results. The video demonstrates how to use the feature with examples of cloud formations and a robot, showing how the style can be applied to different subjects. The script also covers the integration of this feature into ComfyUI and its compatibility with other nodes, such as the IPA adapter and control net. The presenter notes that while the feature works well, it is a work in progress and may change in the future. The video concludes with a brief overview of how to install and use the ComfyUI extension for this feature.

Takeaways

• 🎨 **Visual Style Prompting**: The user can control the style of stable diffusion generations by showing an image to the system, making it easier than using text prompts.
• 📈 **Comparison with Other Tools**: The script compares the effectiveness of the visual style prompting to other methods like IP adapter, style drop, style align, and DB Laura, with the clouds example standing out.
• 💻 **Accessible Testing**: For those without the necessary computing power, two Hugging Face spaces are provided for testing, with the option to run them locally.
• 🐶🌩️ **Initial Results**: The first example generates a dog made of clouds, which after a prompt correction, becomes a more accurate rodent made out of clouds.
• 🤖 **Control Net Integration**: The control net version of the tool uses the shape of another image via its depth map to guide the style, as demonstrated with a cloud and a robot example.
• 🧩 **Comfy UI Extension**: The tool can be integrated into Comfy UI, allowing users to incorporate it into their workflow, though it's noted as a work in progress.
• 📚 **Installation Process**: The installation is straightforward, similar to other Comfy UI extensions, and requires a restart to enable the new visual style prompting node.
• 📷 **Basic Image Captioning**: The script mentions using an AI model (BLIP) for basic image captioning to avoid retyping descriptions, with an option to toggle automatic captions.
• 🌈 **Style Customization**: Users can apply any style image they like to their generations, with examples showing a switch from a colorful paper cut art style to a darker, blue-toned style.
• 🤝 **Compatibility with Other Nodes**: The tool works well with other nodes, such as the IPA adapter, merging the style of the input image with the style of the style image.
• ☁️🐭 **Unexpected Results**: There are some discrepancies in the results when using different models, such as SD 1.5 versus SDXL, particularly with the cloud rodent example.
• 📹 **Further Instructions**: The script concludes with a mention of a follow-up video that explains how to install Comfy UI to use these workflows.

Q & A

What is the main feature of the Style Transfer Using ComfyUI?

The main feature is the ability to control the style of stable diffusion generations by showing it an image and instructing it to adopt that style, which is easier than using text prompts.

How does visual style prompting compare to other methods like IP adapter, style drop, style align, and DB Laura?

Visual style prompting appears to produce more realistic cloud formations and better fire and painting styles compared to the other methods mentioned.

What are the two Hugging Face spaces provided for testing the style transfer?

The two Hugging Face spaces provided for testing are called 'default' and 'control net'.

How does the control net version of the style transfer differ from the default?

The control net version is guided by the shape of another image via its depth map, allowing for a more controlled style transfer based on the reference image's structure.

Is it possible to integrate the style transfer feature into other workflows?

Yes, the style transfer feature can be integrated into the workflow of choice, and it is also available as an extension for ComfyUI.

What is the process for installing the style transfer extension for ComfyUI?

The installation process is similar to any other ComfyUI extension and can be done via git clone or the ComfyUI manager.

How does the style transfer feature handle prompts and style images?

The feature allows users to input a prompt and a style image, which it uses to generate a new image with the desired style applied.

Can the style transfer feature work with other nodes in the workflow?

Yes, the feature is compatible with other nodes, such as the IPA adapter and control net, to create a merged style that combines the input image and the style image.

What issues were encountered when using stable diffusion 1.5 instead of sdxl?

When using stable diffusion 1.5, the cloud rodents appeared colorful instead of white, which was not the case with sdxl. This suggests a difference in how the two models handle style transfer.

How does the style transfer feature handle automatic image captioning?

The feature uses an automatic image captioning system, such as BLIP, to generate captions for the images, allowing users to quickly cycle through and view style changes without manually entering descriptions.

What is the current status of the style transfer feature in ComfyUI?

As of the time the video was made, the style transfer feature is a work in progress and improvements are expected in the future.

How can users learn more about installing and using ComfyUI for style transfer?

Users can find more information on how to install and use ComfyUI for style transfer in the next video mentioned in the transcript.

Outlines

00:00

🌟 Visual Style Prompting in Stable Diffusion Generations

The video script introduces a method for controlling the visual style of stable diffusion generations by providing an example image. The process is compared to other techniques like IP adapter, style drop, style align, and DB, Laura. The speaker demonstrates using two Hugging Face spaces, 'default' and 'control net,' to test the method. The 'default' space generates images based on a cloud example, while the 'control net' version uses another image's depth map to guide the style. The speaker also discusses integrating this feature into a workflow using Comfy UI and provides a brief overview of the process, including the use of a 'style loader' for the reference image and the application of visual style prompting. The results are shown to be quite effective, with the generated images closely reflecting the style of the reference image.

05:00

🤔 Exploring Compatibility and Potential Issues

The script continues by examining how well the visual style prompting works with other nodes, such as the IP adapter. The speaker shows an example where the generated image successfully merges the features of a full-face portrait with the desired style. However, when testing with 'Cloud rodents,' the speaker notices a discrepancy in coloration between the stable diffusion 1.5 and the SDXL models. The colors appear more vibrant in the former, which is suspected to be due to differences between the two models. The speaker suggests that further investigation is needed to understand this difference fully. The video concludes with a mention of an upcoming video that will explain how to install Comfy UI to utilize these workflows.

Keywords

💡Style Transfer

Style transfer is a technique in machine learning where the style of one image is applied to another image while maintaining the content of the original image. In the video, style transfer is used to control the visual style of generated images through a process that doesn't require training, allowing users to input their own style reference images.

💡Stable Diffusion

Stable Diffusion is a type of generative model used for creating images from textual descriptions. It is mentioned in the context of generating images with specific styles without the need for traditional text prompts. The video discusses how style transfer can be applied to Stable Diffusion generations to achieve desired visual styles.

💡Visual Style Prompting

Visual style prompting is a method where an image's style is used as a guide for generating new images, instead of relying on textual descriptions. The video demonstrates how this approach can lead to more intuitive and easier control over the style of generated images, as opposed to crafting detailed text prompts.

💡Hugging Face Spaces

Hugging Face Spaces is a platform that allows users to share and use machine learning models without the need for significant computational resources. In the video, it is mentioned as a way for users without the necessary hardware to access and test the style transfer capabilities.

💡Control Net

Control Net is a tool or method mentioned in the video that guides the style transfer process by using the shape of another image via its depth map. It is used to influence the style and appearance of the generated images, ensuring they align with the desired style while maintaining the structural integrity of the content.

💡Comfy UI

Comfy UI is a user interface extension that simplifies the process of integrating various machine learning models into a workflow. The video discusses how Comfy UI can be used to incorporate style transfer into a user's existing workflow, making it easier to generate images with specific styles.

💡Depth Map

A depth map is an image that represents the distance of each point in the image from a certain point, usually the viewer's perspective. In the context of the video, the depth map from one image is used to guide the style transfer process, ensuring that the generated image has a similar structure to the reference image.

💡Stable Diffusion 1.5

Stable Diffusion 1.5 refers to a specific version of the Stable Diffusion model. The video compares the performance of style transfer using Stable Diffusion 1.5 with other versions, noting differences in how colors and styles are applied to the generated images.

💡SDXL

SDXL likely refers to a different or extended version of the Stable Diffusion model, used to generate higher quality or more detailed images. The video discusses the differences in style transfer results when using SDXL compared to Stable Diffusion 1.5, highlighting how certain visual aspects are handled differently.

💡IP Adapter

IP Adapter is a tool or method used in conjunction with style transfer to merge different visual elements, such as the full face and input image, with a specific style. The video shows how the IP Adapter can be used to create images that combine various features and styles effectively.

💡Workflow Integration

Workflow integration refers to the process of incorporating new tools or methods into an existing sequence of tasks. In the video, it is mentioned that the style transfer functionality can be integrated into a user's chosen workflow using Comfy UI, allowing for a seamless adoption of the new technology.

Highlights

ComfyUI allows users to control the style of stable diffusion generations by showing an image as a reference.

Visual style prompting is compared to other methods like IP adapter, style drop, style align, and DB Laura.

The cloud example demonstrates the effectiveness of the style transfer.

ComfyUI provides two Hugging Face spaces for users to test the style transfer without high computing power.

The default version of ComfyUI can be run locally for ease.

The control net version is guided by the shape of another image via its depth map.

ComfyUI can be integrated into the user's workflow of choice.

The tool is a work in progress and improvements are expected in the future.

Installation of ComfyUI extensions is straightforward via git clone or ComfyUI manager.

The visual style prompting node is a key feature of the ComfyUI extension.

The tool allows for automatic image captioning using the BLIP model.

Visual style prompted generations are noticeably different from the default render.

Users can apply any style image they like to their generation.

The tool works well with other nodes, such as the IPA adapter.

Control net works well with the tool, but some issues were observed with stable diffusion 1.5.

Using SDXL models instead of stable diffusion 1.5 resolved the issue with colorful cloud rodents.

A detailed guide on how to install ComfyUI and use these workflows is available.