AI art, explained
Summary
TLDRThe video script explores the evolution of AI-generated images, from the early experiments in 2015 to the present capabilities of models like DALL-E. It discusses the concept of 'prompt engineering' and the creative potential unlocked by these technologies, while also highlighting the ethical and societal implications, including biases in training data and the impact on professional artists.
Takeaways
- 🧠 The major development in AI research in 2015 was automated image captioning, where machine learning algorithms could label objects and describe them in natural language.
- 🔄 Researchers became curious about the reverse process, text-to-image generation, aiming to create novel scenes that didn't exist in reality.
- 🚀 Early attempts resulted in rudimentary images, but the 2016 paper showcased the potential for future advancements in this field.
- 🌐 The technology has advanced dramatically in a short time, with AI now capable of generating images from text prompts that were previously unimaginable.
- 🎨 AI-generated art, like the portrait sold for over $400,000 in 2018, required specific datasets and models to mimic styles, unlike the newer, more versatile models.
- 📈 The newer models are so large that they can't be trained on individual computers, but once trained, they can generate a wide range of images from a simple text prompt.
- 🤖 The process of communicating with deep learning models to generate images is known as 'prompt engineering', which involves refining the text prompts to get desired results.
- 🖼️ The models use a 'latent space' to generate images, a mathematical space with meaningful clusters representing different concepts, rather than copying from the training data.
- 🔮 The generative process involves starting with noise and arranging pixels into a coherent image through a process called diffusion, which adds an element of randomness.
- 🌐 The technology raises copyright and ethical questions, as it can replicate styles and generate images from biased datasets found on the internet.
- 🔑 This technology has the potential to change the way humans imagine, communicate, and work within their culture, with both positive and negative long-term consequences.
Q & A
What was the major development in AI research seven years ago that led to the concept of text-to-image generation?
-The major development was automated image captioning, where machine learning algorithms could label objects in images and put those labels into natural language descriptions.
What was the initial challenge faced by researchers when they attempted to create images from text descriptions?
-The initial challenge was to generate entirely novel scenes that didn't exist in the real world, rather than retrieving existing images like a search engine does.
Can you describe the first attempt to generate an image from the text prompt 'the red or green school bus'?
-The first attempt resulted in a 32 by 32 tiny image that was barely recognizable, appearing as a blob of something on top of something else.
What is 'prompt engineering' and why is it significant in the context of text-to-image AI?
-Prompt engineering is the craft of communicating with deep learning models by providing the right text prompts to generate desired images. It's significant because it allows users to refine their interaction with the machine, creating a dialog that guides the AI to produce specific outputs.
What is the significance of the 'latent space' in the context of deep learning models used for image generation?
-The latent space is a multidimensional mathematical space where the deep learning model organizes variables that represent different aspects of images. It allows the model to generate new images that have not been seen before, based on the navigation within this space using text prompts.
How does the generative process called 'diffusion' work in creating an image from a point in the latent space?
-Diffusion starts with noise and, over a series of iterations, arranges pixels into a composition that makes sense to humans. Due to randomness in the process, the same prompt will not always result in the exact same image.
What is the role of a large, diverse training dataset in training image-generating AI models?
-A large, diverse training dataset provides the AI model with a wide range of images and their text descriptions, which helps the model learn to associate concepts with visual patterns and generate new images from text prompts.
What ethical and copyright concerns arise with the use of AI-generated images, especially when mimicking the style of known artists?
-Ethical and copyright concerns include the use of artists' work in datasets without their consent and the potential for AI to mimic their style, which may infringe on their intellectual property rights and raise questions about originality and attribution.
How does the AI's latent space reflect societal biases and cultural representation?
-The latent space of AI models contains biases and cultural representations based on the data they were trained on, often reflecting stereotypes and underrepresentation of certain groups or cultures, as it mirrors the content and biases found on the internet.
What potential long-term impacts does the advancement in text-to-image AI have on creators and the way humans imagine and communicate?
-The advancement in text-to-image AI has the potential to revolutionize the way humans create, communicate, and interact with visual content. It may lead to new forms of artistic expression, challenges in copyright and originality, and changes in the job market for creators.
What is the significance of the name 'DALL-E' given to the AI model announced by OpenAI, and what does DALLE-2 promise?
-DALL-E is named after the famous artist Salvador Dalí, reflecting the model's ability to create images from text prompts. DALLE-2, its successor, promises more realistic results and seamless editing capabilities, though it has not been released to the public yet.
Outlines
🤖 Evolution of AI Image Generation
This paragraph discusses the inception of AI in image captioning and the subsequent challenge of generating images from text descriptions. It details the initial experiments with generating novel scenes that had never been seen before, such as a green school bus, and the progression of this technology from rudimentary 32x32 pixel images to highly advanced and realistic outputs within a year. The rapid development is highlighted through examples of AI-generated art sold at auction and the capabilities of modern models like DALL-E and Midjourney, which can create images from text prompts without the need for traditional artistic tools.
🎨 The Art of Prompt Engineering in AI Imagery
The second paragraph delves into the intricacies of 'prompt engineering,' where the input text prompts are carefully crafted to guide AI models in generating specific images. It explores the creative aspect of this process, comparing it to having a dialog with a machine to refine the desired output. The paragraph also touches on the technical side, explaining how AI models learn from vast datasets to create images not by copying existing ones but by navigating a 'latent space'—a multidimensional mathematical space where points represent potential images. The generative process of 'diffusion' is introduced, which transforms noise into coherent images, and the uniqueness of each AI-generated image due to the randomness in the process is emphasized.
🔮 Ethical and Creative Implications of AI Image Generation
The final paragraph addresses the broader implications of AI-generated imagery, including the ethical considerations surrounding the use of artists' styles without their explicit consent and the unresolved copyright issues. It raises concerns about the biases present in the training data and the potential for these biases to be reflected in the generated images. The paragraph also contemplates the impact of this technology on human imagination and communication, suggesting that it represents a significant shift in how we interact with our own culture. Finally, it invites viewers to consider the future of professional image creators in the face of advancing AI technologies and encourages further discussion on the topic.
Mindmap
Keywords
💡Automated Image Captioning
💡Text-to-Images
💡Deep Learning Models
💡DALL-E
💡Prompt Engineering
💡Latent Space
💡Diffusion
💡Midjourney
💡Copyright Questions
💡Bias in AI
💡Cultural Representation
Highlights
In 2015, AI research saw a major development in automated image captioning, where machine learning algorithms began to describe images in natural language.
Researchers were curious about reversing the process to generate images from text, aiming to create novel scenes not found in the real world.
The initial experiments in text-to-image generation resulted in simple, abstract images that were far from realistic.
AI-generated images have come a long way in a short time, with dramatic improvements observed within just one year.
The technology now allows for the creation of images from text prompts without the need for traditional art tools.
Open AI announced DALL-E, a model capable of creating images from a wide range of text captions, with an even more advanced version, DALLE-2, on the horizon.
Independent developers have built their own text-to-image generators using pre-trained models, making this technology accessible to the public.
Midjourney, a company with a Discord community, allows users to generate images from text in under a minute using bots.
The art of communicating with deep learning models to generate desired images has been termed 'prompt engineering'.
The process of image generation from text involves a mathematical 'latent space' where the model finds variables to represent different concepts.
Deep learning models learn to distinguish between images by identifying patterns and variables that humans may not recognize.
The generative process called 'diffusion' is used to translate points in latent space into actual images, starting with noise and arranging pixels into a coherent composition.
The technology raises copyright and ethical questions, as it can replicate an artist's style without using their images, just by mentioning their name in the prompt.
The latent space of these models may contain biases and stereotypes learned from the internet, reflecting societal prejudices.
The technology's impact extends beyond the immediate technical consequences, potentially changing the way humans imagine, communicate, and work with their own culture.
The video includes a bonus section with insights from creative professionals on the implications of text-to-image technology for those who make a living creating images.
Transcripts
Seven years ago, back in 2015,
one major development in AI research was automated image captioning.
Machine learning algorithms could already label objects in images,
and now they learned to put those labels into natural language descriptions.
And it made one group of researchers curious.
What if you flipped that process around?
We could do image to text.
Why not try doing text to images and see how it works?
It was a more difficult task.They didn’t want
to retrieve existing images the way google search does.
They wanted to generate entirely novel scenes that didn’t happen in the real world.
So they asked their computer model for something it would have never seen before.
Like all the school buses you've seen are yellow.
But if you write “the red or green school bus” would it actually try to generate something green?
And it did that.
It was a 32 by 32 tiny image.
And then all you could see is like a blob of something on top of something.
They tried some other prompts like “A herd of elephants flying in the blue skies”.
“A vintage photo of a cat.”
“A toilet seat sits open in the grass field.”
And “a bowl of bananas is on the table.”
Maybe not something to hang on your wall but the 2016 paper from those researchers
showed the potential for what might become possible in the future.
And uh... the future has arrived.
It is almost impossible to overstate how far the technology has come in just one year.
By leaps and bounds. Leaps and bounds.
Yeah, it's been quite dramatic.
I don’t know anyone who hasn’t immediately been like
“What is this? What is happening here?”
Could I say like watching waves crashing?
Party hat guy.
Seafoam dreams.
A coral reef. Cubism.
Caterpillar.
A dancing taco.
My prompt is Salvador Dali painting the skyline of New York City.
You may be thinking, wait AI-generated images aren’t new.
You probably heard about this generated portrait going for over $400,000 at auction back in 2018.
Or this installation of morphing portraits, which Sotheby’s sold the following year.
It was created by Mario Klingemann, who explained to me that that type of AI
art required him to collect a specific dataset of images and train his own model to mimic that data.
Let's say, Oh, I want to create landscapes, so I collect a lot of landscape images.
I want to create portraits, I trained on portraits.
But then the portrait model would not really be able to create landscapes.
Same with those hyper realistic fake faces that have been plaguing
linkedin and facebook – those come from a model that only knows how to make faces.
Generating a scene from any combination of words requires a different, newer, bigger approach.
Now we kind of have these huge models, which are so huge that
somebody like me actually cannot train them anymore on their own computer.
But once they are there, they are really kind of— they contain everything.
I mean, to a certain extent.
What this means is that we can now create images without having to actually
execute them with paint or cameras or pen tools or code.
The input is just a simple line of text.
I'll get to how this tech works later in the video
but to understand how we got here, we have to rewind to January 2021
When a major AI company called Open AI announced DALL-E – which they named after these guys.
They said it could create images from text captions for a wide range of concepts.
They recently announced DALLE-2, which promises more realistic results and seamless editing.
But they haven’t released either version to the public.
So over the past year, a community of independent, open-source developers
built text-to-image generators out of other pre-trained models that they did have access to.
And you can play with those online for free.
Some of those developers are now working for a company called Midjourney,
which created a Discord community with bots that turn your text into images in less than a minute.
Having basically no barrier to entry to this has made it like a whole new ballgame.
I've been up until like two or three in the morning.
Just really trying to change things, piece things together.
I've done about 7,000 images. It’s ridiculous.
MidJourney currently has a wait-list for subscriptions, but we got a chance to try it out.
"Go ahead and take a look."
“Oh wow. That is so cool”
“It has some work to do. I feel like it can be — it’s not dancing and it could be better.”
The craft of communicating with these deep learning
models has been dubbed “prompt engineering”.
What I love about prompting for me, it's kind of really
that has something like magic where you have to know the right words for that, for the spell.
You realize that you can refine the way you talk to the machine.
It becomes a kind of a dialog.
You can say like “octane render blender 3D”.
Made with Unreal Engine...
...certain types of film lenses and cameras...
...1950s, 1960s...
...dates are really good.
...lino cut or wood cut...
Coming up with funny pairings, like a Faberge Egg McMuffin.
A monochromatic infographic poster about typography depicting Chinese characters.
Some of the most striking images can come from prompting the model
to synthesize a long list of concepts.
It's kind of like it's having a very strange collaborator to bounce ideas off of and get
unpredictable ideas back.
I love that!
My prompt was "chasing seafoam dreams,"
which is a lyric from the Ted Leo and the Pharmacists' song "Biomusicology."
Can I use this as the album cover for my first album? "Absolutely."
Alright.
For an image generator to be able to respond to so many different prompts,
it needs a massive, diverse training dataset.
Like hundreds of millions of images scraped from the internet, along with their text descriptions.
Those captions come from things like the alt text that website owners upload with their images,
for accessibility and for search engines.
So that’s how the engineers get these giant datasets.
But then what do the models actually do with them?
We might assume that when we give them a text prompt,
like “a banana inside a snow globe from 1960."
They search through the training data to find related images and then copy
over some of those pixels. But that’s not what’s happening.
The new generated image doesn’t come from the training data,
it comes from the “latent space” of the deep learning model.
That’ll make sense in a minute, first let’s look at how the model learns.
If I gave you these images and told you to match them to these captions, you’d have no problem.
But what about now, this is what images look like to a
machine just pixel values for red green and blue.
You’d just have to make a guess, and that’s what the computer does too at first.
But then you could go through thousands of rounds of this
and never figure out how to get better at it.
Whereas a computer can eventually figure out a method that works- that’s what deep learning does.
In order to understand that this arrangement of pixels is a banana, and this arrangement
of pixels is a balloon, it looks for metrics that help separate these images in mathematical space.
So how about color? If we measure the amount of yellow in the image,
that would put the banana over here and the balloon over here in this one-dimensional space.
But then what if we run into this:
Now our yellowness metric isn’t very good at separating bananas from balloons.
We need a different variable.
Let’s add an axis for roundness.
Now we’ve got a two dimensional space with the round balloons up here and the banana down here.
But if we look at more data we may come across a banana that’s pretty round,
and a balloon that isn’t.
So maybe there’s some way to measure shininess.
Balloons usually have a shiny spot.
Now we have a three dimensional space.
And ideally, when we get a new image we can measure those 3 variables and see
whether it falls in the banana region or the balloon region of the space.
But what if we want our model to recognize,
not just bananas and balloons, but…all these other things.
Yellowness, roundness, and shininess don’t capture what’s distinct about these objects.
That’s what deep learning algorithms do as they go through all the training data.
They find variables that help improve their performance on the task and in the process,
they build out a mathematical space with way more than 3 dimensions.
We are incapable of picturing multidimensional space, but midjourney's model offered this and I like it.
So we’ll say this represents the latent space of the model. And It has more than 500 dimensions.
Those 500 axes represent variables that humans wouldn’t even recognize or have
names for but the result is that the space has meaningful clusters:
A region that captures the essence of banana-ness.
A region that represents the textures and colors of photos from the 1960s.
An area for snow and an area for globes and snowglobes somewhere in between.
Any point in this space can be thought of as the recipe for a possible image.
The text prompt is what navigates us to that location. But then there’s one more step.
Translating a point in that mathematical space into an actual image involves a
generative process called diffusion. It starts with just noise and then,
over a series of iterations, arranges pixels into a composition that makes sense to humans.
Because of some randomness in the process,
it will never return exactly the same image for the same prompt.
And if you enter the prompt into a different model designed by different
people and trained on different data, you’ll get a different result.
Because you’re in a different latent space.
No way. That is so cool. What the heck? The brush strokes, the color palette. That’s fascinating.
I wish I could like — I mean he’s dead, but go up to him and be like, "Look what I have!"
Oh that’s pretty cool. Probably the only Dali that I could afford anyways.”
The ability of deep learning to extract patterns from data means that you can copy an
artist’s style without copying their images, just by putting their name in the prompt.
James Gurney is an American illustrator who
became a popular reference for users of text to image models.
I asked him what kind of norms he would like to see as prompting becomes widespread.
I think it's only fair to people looking at this work
that they should know what the prompt was and also what software was used.
Also I think the artists should be allowed to opt in or opt out of having their work
that they worked so hard on by hand be used as a dataset for creating this other artwork.
James Gurney, I think he was a great example of being someone
who was open to it, started talking with the artists.
But I also heard of other artists who got actually extremely upset.
The copyright questions regarding the images that go into training the
models and the images that come out of them…are completely unresolved.
And those aren’t the only questions that this technology will provoke.
The latent space of these models contains some
dark corners that get scarier as outputs become photorealistic.
It also holds an untold number of associations that we wouldn’t
teach our children but that it learned from the internet.
If you ask an image of the CEO, it's like an old white guy.
If you ask for images of nurses, they're all like women.
We don’t know exactly what’s in the datasets used by OpenAI or Midjourney.
But we know the internet is biased toward the English language and western concepts,
with whole cultures not represented at all.
In one open-sourced dataset,
the word “asian” is represented first and foremost by an avalanche of porn.
It really is just sort of an infinitely complex mirror held up to our society and what we
deemed worthy enough to, you know, put on the internet in the first place and
how we think about what we do put up.
But what makes this technology so unique is that it enables any of
us to direct the machine to imagine what we want it to see.
Party hat guy, space invader, caterpillar, and a ramen bowl.
Prompting removes the obstacles between ideas and images, and eventually videos, animations,
and whole virtual worlds.
We are on a voyage here, that is it's a bigger deal than
than just like one decade or the immediate technical consequences.
It's a change in the way humans imagine, communicate, work with their own culture
And that will have long range, good and bad consequences that we
we are just by definition, not going to be capable of completely anticipating.
Over the course of researching this video I spoke to a bunch of creative people
who have played with these tools.
And I asked them what they think this all means for people who make a living making images.
The human artists and illustrators and designers and stock photographers out there.
And they had a lot of interesting things to say.
So I've compiled them into a bonus video.
Please check it out and add your own thoughts in the comments. Thank you for watching.
تصفح المزيد من مقاطع الفيديو ذات الصلة
Ultimate Guide to Generative AI for Businesses
AI vs Artists - The Biggest Art Heist in History
DALL·E 2, Stable Diffusion, Midjourney: How do AI art generators work, and should artists fear …
How artists and critics think AI art will transform the industry | Art Works
MIT 6.S191 (2018): Issues in Image Classification
Why Artists are Fed Up with AI Art.
5.0 / 5 (0 votes)