"Next Level Prompts?" - 10 mins into advanced prompting

Summary

The video script discusses the challenges and solutions associated with crafting effective prompts for AI, specifically highlighting the Guidance framework developed by Microsoft. The framework allows for precise control over the output structure and enables the creation of complex prompts with loops, conditionals, and predefined answers. The speaker demonstrates how to use Guidance with large language models like OpenAI's DaVinci and open-source alternatives. Additionally, the script covers using Guidance for advanced logic, such as if-conditions, and for generating structured responses in applications like email writing. The video also explores integrating AI with real-time chart generation via APIs like Quick Chart and creating images with Pollinations. The speaker recommends FlowGPT for prompt discovery and GPT Prompt Engineer for evaluating prompt quality. The script concludes by emphasizing the iterative nature of prompt engineering and the need for fine-tuning to achieve optimal results.

Takeaways

• 📝 **Guidance Framework**: An open-source tool introduced by Microsoft that allows for programming prompts to get specific output structures.
• 🌟 **Custom Output Control**: Guidance provides specific control over the final output structure and the ability to insert loops, conditions, and candidate answers within prompts.
• 🚀 **Getting Started with Guidance**: Start by installing necessary packages in Visual Studio Code and use Guidance with large language models like OpenAI's DaVinci or open-source alternatives.
• 🔗 **Defining Output Structure**: Create a prompt template with variables and special terms to define the structure of the generated text.
• ➡️ **Multiple Outputs**: Guidance enables the generation of multiple outputs with a single prompt, providing flexibility in content creation.
• 📋 **Restricting Model Outputs**: Predefine answers for the model to choose from, which is useful for maintaining a standard response format.
• 🔧 **Advanced Logic with If Conditions**: Implement if conditions to control the model's responses based on predefined criteria, enhancing the interaction flow.
• 🛠️ **Hiding Internal Logic**: Use hidden blocks to keep the internal logic out of the final output for a cleaner presentation.
• 📧 **Email and Customer Response Use Case**: Guidance can be used to generate emails or customer responses by leveraging its ability to restrict and control the model's outputs.
• 📈 **Real-time Chart Generation**: Integrate with services like Quick Chart to generate real-time charts from natural language queries using the model's data structuring capabilities.
• 🖼️ **Image Generation**: Utilize APIs like Pollinations to generate images directly from story ideas provided to the model.
• 🔍 **Prompt Engineering Community**: Leverage resources like FlowGPT and GPT Prompt Engineer for discovering better prompts and evaluating their effectiveness.

Q & A

What is the main challenge in creating prompts for AI?

The main challenge is to create prompts that deliver consistent results at scale and can control the output structure for specific needs.

What is the 'guidance' framework?

Guidance is an open-source framework introduced by Microsoft that allows programming of prompts to get specific output structures. It provides specific control over the final output and supports constructing prompts with loops, lists, and conditionals.

How can one get started with the guidance framework?

To get started with guidance, one can open Visual Studio Code, create a Jupyter notebook, install the necessary packages like ipywidgets and openai, and then import the guidance package.

What are some of the capabilities of the guidance framework?

Guidance allows defining specific output structures, restricting the actual output the large language model generates, setting up advanced logic like if conditions, and hiding internal logic from the final output.

How can the guidance framework be used to improve email responses?

The guidance framework can be used to restrict the responses generated by a large language model to best practices, and also to create conditional logic for different types of customer inquiries, such as scheduling a call for high-priority emails.

What is the role of 'flow GPT' in prompt engineering?

Flow GPT is a large prompt library and community that provides a wide range of collections for various purposes, which can be used as a starting point for creating specific prompts and learning from others' work.

What is the 'GPT prompt engineer' project?

GPT prompt engineer is a project that uses GPT to generate prompts and evaluate their quality. It can generate a set of prompts based on a given goal and then test them to determine the best performing prompt.

How can the 'prompts Royale' platform be used?

Prompts Royale is a frontend based on the GPT prompt engineer that allows users to get GPT to generate a prompt and do the testing. It shows evaluation results between different prompts and gives a score to indicate which one is better.

How can the guidance framework be used to generate real-time charts?

By using the guidance framework to transform natural language queries into data structures, it can generate real-time charts using services like Quick Chart, which is an open API for generating charts from URL-based chart data.

What is the 'Pollinations' service?

Pollinations is an open API that allows the creation of AI-generated images on the fly. It can be used to generate illustrations or images based on story ideas provided to the large language model.

What are some benefits of using the guidance framework?

The guidance framework is beneficial for its ability to control the output structure, restrict the language model's responses, incorporate conditional logic, and automate the generation of specific content types like emails, charts, and images.

What is the importance of iterating and fine-tuning prompts in the guidance framework?

Iterating and fine-tuning prompts is crucial to achieve the best results with the guidance framework, as it allows for the refinement of the output to meet specific needs and use cases more accurately.

Outlines

00:00

📝 Introduction to the Guidance Framework

This paragraph introduces the Guidance framework, an open-source tool developed by Microsoft for prompt engineering. It allows users to control the prompt to get specific output structures, such as general images and charts, using a single prompt. The framework provides specific control over the final output structure and supports features like for-each loops, defining candidate answers, and inserting if-conditions within prompts. The speaker demonstrates how to get started with Guidance by installing necessary packages in Visual Studio Code and using it with large language models like OpenAI's DaVinci. The paragraph also covers how to define a specific output structure using a prompt template, restrict the output to predefined answers, and set up advanced logic like if-conditions for use cases like email writing and customer responses.

05:02

🔍 Advanced Prompt Engineering Techniques

This paragraph delves into more advanced prompt engineering techniques using the Guidance framework. It discusses how to define a list of predefined priorities for generating email responses based on the importance of a customer inquiry. The speaker demonstrates how to insert specific messages in the email based on the priority level, such as scheduling a call for high-priority inquiries. The paragraph also covers how to use the framework to generate real-time charts using the Quick Chart API by converting natural language queries into data structures. Additionally, it explores using custom functions within prompts to generate images using the Pollinations API, allowing for the creation of stories and illustrations from a single prompt. The speaker emphasizes the iterative nature of prompt engineering and the need for fine-tuning and iterating prompts for optimal results.

10:03

🌐 Community and Open Source Resources for Prompt Engineering

The final paragraph highlights community and open-source resources that can accelerate the prompt engineering process. It mentions FlowGPT, a comprehensive prompt library and community that provides a wide range of prompts across various domains. The speaker recommends using FlowGPT as a discovery tool to find high-quality prompts created by others and gain insights into effective prompt construction. The paragraph also introduces the GPT Prompt Engineer, a tool that uses GPT to generate and evaluate the quality of prompts. While the speaker finds the default prompts generated by GPT to be suboptimal, they appreciate the evaluation framework for comparing different prompts. The speaker also mentions Prompts Royale, a frontend built on top of GPT Prompt Engineer that provides an interface for generating, testing, and evaluating prompts to determine the best one. The paragraph concludes by inviting viewers to share their prompt engineering tactics and frameworks and encouraging subscriptions for more AI project updates.

Keywords

💡Guidance Framework

Guidance is an open-source framework introduced by Microsoft, designed to allow users to program prompts for specific output structures. It is highlighted in the video as a tool that provides specific control over how the final output should look, enabling the construction of prompts with loops, conditionals, and predefined answers. It is used in the context of prompt engineering to achieve consistency and structure in AI-generated content.

💡Large Language Model

A large language model refers to a sophisticated AI system capable of understanding and generating human-like text based on given prompts. In the video, it is mentioned in conjunction with the Guidance framework to illustrate how such models can be directed to produce specific types of outputs, like structured responses or predefined answers.

💡Prompt Engineering

Prompt engineering is the process of carefully designing the inputs or 'prompts' given to AI language models to elicit desired responses. The video discusses the use of advanced frameworks like Guidance to improve prompt engineering, emphasizing the need for precision and control over the AI's output.

💡Visual Studio Code

Visual Studio Code is a popular source code editor developed by Microsoft. It is mentioned in the script as the environment where one can create a Jupyter notebook for using the Guidance framework, indicating its utility in setting up and managing development environments for AI applications.

💡Jupyter Notebook

A Jupyter Notebook is an open-source web application that allows creation and sharing of documents containing live code, equations, visualizations, and narrative text. In the context of the video, it is used as a tool for setting up and demonstrating the use of the Guidance framework with AI models.

💡Conditional Logic

Conditional logic refers to the process of executing different blocks of code or generating different outputs based on certain conditions. The video showcases how Guidance allows for the incorporation of if-conditions within prompts, enabling the AI to generate responses that depend on specific criteria or inputs.

💡OpenAI

OpenAI is a research and deployment company focusing on creating and utilizing AI technologies. The video discusses using OpenAI's text-based models like DaVinci for generating text with the help of the Guidance framework, highlighting the integration of OpenAI's models with prompt engineering techniques.

💡Predefined Answers

Predefined answers are a set of responses that are already determined and can be selected by the AI instead of creating new ones. The video explains how Guidance can be used to restrict the output of a large language model to a list of predefined answers, which is useful for applications like customer service responses.

💡FlowGPT

FlowGPT is described as a large prompt library and community for prompt engineering. It is mentioned as a resource for discovering and evaluating different prompts across various fields, such as marketing and programming. The video emphasizes its utility in providing a starting point for building specific prompts.

💡GPT Prompt Engineer

GPT Prompt Engineer is a tool that uses AI to generate and evaluate prompts based on a given goal. The video discusses its use in generating multiple prompts and then testing them to determine which performs best. It is portrayed as a valuable asset for evaluating the effectiveness of different prompt variations.

💡Quick Chart

Quick Chart is an open API that allows for the real-time generation of charts by passing chart data in a URL. The video demonstrates how the Guidance framework can be used with Quick Chart to generate charts from natural language queries, showcasing the integration of AI with data visualization tools.

💡Pollinations

Pollinations is an open API service for creating AI-generated images on the fly. The video script mentions using Pollinations to generate images based on story ideas provided to the AI, indicating its use in creating visual content alongside textual narratives.

Highlights

Guidance is an open-source framework introduced by Microsoft that allows for specific control over the output structure of AI-generated content.

The framework enables the construction of prompts with loops, conditional statements, and predefined answers for more predictable outputs.

Getting started with Guidance involves installing it in Visual Studio Code and using it with large language models like OpenAI's DaVinci.

Defining a prompt template with Guidance involves using special syntax to specify the structure and desired content generation.

Guidance allows for the restriction of the AI's output to predefined answers, which is useful for applications like email or customer responses.

Advanced logic, such as if conditions, can be implemented in prompts to create dynamic workflows based on the AI's classifications.

The framework can hide internal logic from the final output, allowing for clean and focused results.

Guidance can be used to create advanced prompts for generating emails with specific structures and priorities.

The framework can integrate with real-time chart generation services like Quick Chart, enabling the AI to create visual data representations.

Guidance supports triggering custom functions within prompts, allowing for complex data manipulation and output generation.

AI-generated images can be created on-the-fly using services like Pollinations, which can be integrated with Guidance for comprehensive content creation.

FlowGPT is a large prompt library and community resource that provides a wide range of prompt examples and can accelerate prompt engineering.

GPT Prompt Engineer is a tool that uses AI to generate and evaluate the quality of prompts, helping to identify the most effective ones.

Prompts Royal is a frontend application based on GPT Prompt Engineer that visually presents evaluation results, aiding in the selection of optimal prompts.

Iterative fine-tuning and prompt iteration are crucial for achieving the best results with Guidance, despite its powerful capabilities.

The Guidance framework, while powerful, may still require significant time investment for learning and mastering its functionalities.

The transcript provides example codes and resources in the description for readers to experiment with and learn from.

The speaker encourages viewers to share their own tactics and frameworks for prompt engineering in the comments section.

The content is part of an ongoing series on AI projects, inviting viewers to subscribe for more insights and updates.

Transcripts

getting the prompt right is hard

especially if you want the prompt to

deliver consistent results at scale but

what if you can control the prompt to

get specific output structure and get a

general image and the charts with just

one single prompt there are few Advanced

proper engineer Frameworks that I found

extremely useful but not enough people

using it and that's what I want to talk

about today the first one I want to talk

about is called guidance it is an open

source framework initially introduced by

Microsoft and got more than 12 000 stars

on GitHub fundamentally it is a

framework that allow you to program the

prompt to get specific output it gives

you very specific control of how the

final output structure should look like

and also allow you to construct The

Prompt with for each Loop Define list of

candidate answers and even insert if

conditions inside your prompts so this

allows you to configure the final output

very easily to get started you can open

Visual Studio code and then we can

create a jupyter notebook that we

install ipy and B so firstly we will

install the guide lens and open AI

package and we will import the guidance

and then to Guidance the large language

model with openai text DaVinci you can

also use GPD 3.5 gp4 or even open source

model like llama as well but they do

have specific syntax you need to follow

and the first thing we will try is

Define a specific output structure you

want so you can create something like

prompt template like how you do in

launching by creating variable with

guidance and inside here you will Define

The Prompt template where I would Define

one variable with this bubble curly

bracket and also another double curly

bracket with special term gen the part

that you actually want large model to

creating so the final output would look

something like this when Steve Jobs said

without guidance AI would do more harm

than good with this it allow you to get

a specific structure output and if you

want you can even change multiple output

together so I can add another one

counter argument and let me run this so

you can see here the large language

model actually generates this two parts

and without guidance framework it's

actually possible to do manually but it

will just require a a lot of manual work

and apart from defining the structure

you can also restrict the actual output

the large energy model going to generate

for example I can ask large knowledge

model to choose one of the predefined

answers instead of creating their own so

I can define a list of options here and

then create another guidance is the

following sentence offensive where I

will give an example and ask it to

select an answer from the three options

I give here I'll give you example is

pretty rude your taco tastes like

and try to run this so it will follow

that specific structure and give me the

answer is yes and here I didn't give it

any extra prompt about what the answer

should be just follow the list of

predefined answers I give and if I

change this to be something more polite

like your tacos taste can be improved

and try it again then it will answer no

so this is a really powerful way that if

we want to restrict down the answers the

large language model actually generates

in use case like writing email or

customer response where you still want

to use the reasoning part of the large

language model but you already got best

practice and the other part I want to

talk about is that you can also set up

Advanced logic like a if condition for

example in the same use case where we

asked large language model to classify

if the user response is root we can

actually create a workflow where if the

answer is rude then it will trigger

specific typo response if the answer is

not root then the system will generate a

response like normal the way we will do

that is we'll still use a slap and give

the response name root and this

basically means we will have a variable

called root that we can use in other

parts of the prop and then we can create

a if condition if root is yes which is

from the predefined condition we give

then a system will say please be polite

but and if root is no then a system will

generate answer so let's try this okay

with this user response your toggle

tests are shared the large large model

detail it is actually root so it

triggers this workflow where a system

will say please be polite but if I

change to something like your taco

tastes too salty it'll be showing the

answer is not rude then it actually

generate a proper response so this is a

way for you to achieve if condition

larger in your final output and these

are the value costs usage as well for

example I probably don't want this part

in the middle to be showing in the final

output because it's kind of internal

logic so I can actually hide this from

the final output by creating block to

wrap this part with attribution hidden

equal to true and if I run this again

then the part in the middle will be

hidden so this is how you can achieve if

condition and I think if condition is

probably one of the most powerful use

case for guidance those are just a few

functions that guidance have and they

are actually more things that you can

learn from their documentations but with

those basic building blocks you can

already create very Advanced props for

example for the use case when I want

light language model to generate emails

one problems I always face with is that

sometimes I do want to use the reasoning

power of large Range model but I want

response to be exactly like the best

practice that I come up with and I don't

want any creativities around them and on

the other side I also want to create

some if conditions there if this client

is very important then book a meeting

right away but if it's just general in

priority then just response like normal

so to achieve their use case I will

Define a list of predefined Priority

First from low to high and I'll create

one blocks at the beginning to let large

language model give a priorities and

then I will ask large language model to

generate the email response and in the

end I will hide those two logic blocks

and showing the email response is

generated and if the priority is high

priority then I want to insert this

specific message to schedule call with

customers right away with my calendar

link so that's why I don't want it to be

created because if you change the link

it won't work so let's try it assume a

customer complaint come to my inbox your

server is so terrible that I want to

refund this is kind of high stake

situation that I wanted to schedule call

right away so you can see here it

actually generate a message also insert

the predefined message with my calendar

link but if I change to be something

more soft like what features does

webflow have but Wix don't have then

this will show a different message and

won't trigger this specific calendar

link and on the other side I can also

use guidance framework to let large

knowledge model generate real-time hard

for me with service like quick chart and

if you don't know what a quick chart is

it's a open API where it allowed me to

generate charts in real time by passing

on the chart data in a URL if I can get

a URL looks something like this then it

will generate this chart on the Fly

which means if we can get large energy

model to turn a natural language query

into a data structure like this then we

can generate charts in real time and

here we will use another function from

guidance which is ability to trigger

custom functions inside the problem so

the way we will do that is I will give

the prompt a few short examples so that

it will learn if the input is this it

should generate Json data like this and

once it generates Json data I will use

this function pause chart link to put

together a URL and displaying markdown

so the way I would do that is I would

Define this function that will pass on

the Json file into a URL linking

markdown and then I will Define a few

short examples here and then I'll create

another Guidance with this prompt you

are a world-class data analyst you will

generate chart output based on the

natural language which in here I have

this 4 inch wrapper and I will run this

q a pair from the predefined list and

once they finish the free of sharp

prompts I will insert the actual user

query and I'll see large language model

to generate chart data and all this

logic will be hidden from the final

output because I wrap under this block

and the final output will be hello here

is the charter you want and the result

will be calling the customer function I

defined above with the variable chart

the large language model created which

is Json data and once I did that I can

try to use this function create a pie

chart showing the population of the word

by continent it will generate this

response and it is in markdown format so

if you paste this in a markdown preview

you will see this is results a

population breakdown by continent chart

that generated in real time and if you

want I also include a streamlined app

where you can visualize and test those

three dots better and with the same

shred of salt you can also use large

language model to generate image

straight away with service like

pollinations and if you don't know what

pollination is it's also another open

API they allow you to create an AI image

on the Fly for example I can do

image.pollination.ai prompt a cute girl

and this will generate this image right

away and this allows us to build a use

case like if I give a story idea it can

generate a whole story as well as a

proper illustration so I can do

something similar where I will have a

logic block to ask large language model

to generate story based on my story idea

and then have another block to generate

the image linking markdown based on the

story and in the end I'll put them

together and if I try this again it will

generate a proper story from a single

prompt and also generate proper

illustration as well so this is a

guidance framework I think it's very

powerful even though it is still a bit

buggy and the lack of documentation but

it is quite powerful once I get it I

have attached all those example codes I

show you here in the description below

so feel free to try it out even though

guidance is a very powerful framework

but at the end of the day it will be

still the iterative process you will

still need to spend a top time fine

tuning and iterating your prompts to get

the best results but there are a few

other community and open source projects

that I think can really speed up the

process here one of them is flow GPT so

flow GPT is one of the biggest prompt

library and probably one of the biggest

prompt engineering community as well I

always go up to flow GPT every time when

I try to build some specific prompts

because it's a wide range of collections

from marketings programming so for

example if I want to create a prompt for

SEO blocks I can just come here see what

are the prompts that Community has voted

most and I can click on that take a look

about the problems that other people

create so this provides me a much better

starting point and I can also get a

preview about what the readouts will be

this is where I learned a lot of tactics

here for example I actually learned

about permissions from The Prompt here

because someone actually built a full

text-based adventure with just one

single prop which is super impressive so

I definitely recommend to use this as a

prompt Discovery tool at the rare

beginning and on the outside there's

also another project called GPT prompt

engineer and the concept is pretty

simple it basically asks GPD to generate

prompt and also use GPT to evaluate and

test the quality of prompts so it refers

to use gpd4 to generate a set of prompts

based on the goal you give it for

example I can use GPT prompt engineer to

generate prompt for writing SEO blog

posts and it will use gpd4 to generate 4

or 10 different prompts and then it will

start using gbd4 as evaluation machine

to do the testing it was starting doing

20 or 30 rounds of testing across

prompts and figure out which one is the

best to be honest from my experience the

prompts generated by GPD is actually not

as good as the one set you will come up

by yourself so I won't rely purely on

gpe to generate bombs autonomously

however the part I found is most value

is actually this evaluation framework

because normally I will come up with two

or three different variations between

the prompts and I want to figure out

which one gonna performs best at scale

they're actually someone built a front

end based on this GPD prompt engineer

called prompts Royal where you can

basically get GPT to generate a prompt

and do the testing and once it finished

it will show you the evaluation results

between different prompts and in the end

give you a score so even though I don't

really like the default prompts that GPT

generated but I often use this as a way

to evaluate different prompts I

generated so I can manually paste in two

or three different prompts that I

created and use this platform to

evaluate which one is better so those

are a few learnings I want to share with

you about problem engineer if you know

more interesting tactics and Frameworks

please comment below and let me know if

you enjoyed this content please consider

give me subscribe I'll continue posting

interesting AI projects that I'm

building thank you and I see you next

time