LLMs: A Hackers Guide

Summary

Der Vortrag beschäftigt sich mit der Verwendung von KI und der Generierung von Dokumenten. Der Sprecher führt seine Erfahrungen mit verschiedenen AI-Projekten ein, wie dem Erstellen eines Kommunikations-Automations-Tools und eines visuellen RAG. Er betont die Bedeutung des iterativen Lernprozesses und der Verwendung verschiedener Modelle. Es werden auch Ressourcen und Werkzeuge für die Entwicklung mit KI vorgestellt, sowie Tipps zur Fehlerbehebung und zukünftige Entwicklungen in der KI-Branche.

Takeaways

• 🚀 Die Arbeitsweise mit künstlichen Intelligenzen (AI) und Generativen AI hat sich verändert und erfordert ein neues Denken.
• 💡 Die Verwendung von großen Sprachmodellen (LLMs) hat die Natursprachverarbeitung (NLP) stark verbessert und macht alle englischen Texte und Nachrichten zugänglich.
• 📈 Die Anwendung von AI bei kommerzieller Schifffahrt, einschließlich der Verwendung von Multi-Modal RAG (RAG für visuelle Informationen), hat die Beantwortung missionkritischer Fragen verbessert.
• 🔄 Der iterative Loop (CPLN) ist ein wichtiger Aspekt der AI-Arbeit, bei dem es darum geht, ständig neue Ansätze zu entwickeln und zu testen.
• 📚 Die Erstellung von Dokumenten kann durch die Verwendung von AI-Tools automatisiert werden, was die Wiederholung von Informationen in Meetings oder Dokumenten reduziert.
• 🎨 Die Verwendung von verschiedenen AI-Tools und -Methoden ist entscheidend, um komplexe Probleme zu lösen, und erfordert die Verwendung aller verfügbaren Modi und Strukturen.
• 🛠️ Die Verwendung strukturierter Eingaben und Ausgaben ist wichtig, um die Genauigkeit und Zuverlässigkeit von AI-Modellen zu verbessern.
• 🔄 Die ständige Iteration und Verbesserung von Prompts ist notwendig, um die besten Ergebnisse zu erzielen, und sollte einen großen Teil der Arbeitszeit einnehmen.
• 🔍 Die Verwendung von Debugging-Techniken ist entscheidend, um Probleme in AI-Systemen zu identifizieren und zu beheben, insbesondere durch die Identifizierung von Fehlerquellen und die Anwendung von Korrekturen.
• 📈 Die zukünftige Entwicklung von AI wird durch Hardware- und Speicheroptimierungen sowie Quantifizierung nochmals um ein Vielfaches schneller und kostengünstiger werden.
• 💬 Die Verwendung von AI in der Codierung und Programmierung wird zunehmen und wird in Zukunft einen großen Einfluss auf die Art und Weise haben, wie Entwickler arbeiten.

Q & A

Was ist das Hauptthema des Gesprächs in dem Transkript?

Das Hauptthema des Gesprächs ist die Arbeit mit künstlichen Intelligenz (AI), insbesondere mit Generativen AI und der Verwendung von Prompts. Es wird auch über die Anwendung von AI in verschiedenen Bereichen und die Herausforderungen bei der Entwicklung mit AI diskutiert.

Wie definiert der Sprecher die 'iterative Schleife' im Kontext der AI-Entwicklung?

Die 'iterative Schleife' bezieht sich auf den Prozess des kontinuierlichen Tests und Änderns von Prompts und Ansätzen bei der Arbeit mit AI-Modellen. Es geht darum, verschiedene Ansätze zu erkunden und zu optimieren, um bessere Ergebnisse zu erzielen.

Was ist der Hauptunterschied zwischen traditioneller Programmierung und der Arbeit mit AI-Modellen?

Die traditionelle Programmierung ist deterministisch, während die Arbeit mit AI-Modellen nicht deterministisch ist und eher auf dem Experimentieren und Finden neuer Ansätze basiert. In AI ist es wichtig, sich neue Denkmuster anzueignen und sich nicht an die traditionellen Vorgehensweisen zu halten.

Welche Bedeutung haben 'CPLN'-Prompts in der AI-Entwicklung?

CPLN steht für 'Chat, Prompt, Loop, Nest' und ist ein von dem Sprecher eingeführtes Muster für die Arbeit mit AI. Es betont die Bedeutung des Chattens mit Modellen, des Änderns von Prompts, des Hinzufügens von Daten und Testfällen sowie des unterteilens der Aufgaben in kleinere Unteraufgaben.

Welche Rolle spielt die Verwendung von Strukturen in der Eingabe und Ausgabe bei der Arbeit mit AI?

Die Verwendung von Strukturen in der Eingabe und Ausgabe erleichtert es, die Modelle besser zu steuern und zu lenken. Strukturierte Eingaben helfen, die Relevanz und Richtigkeit der Anfragen zu erhöhen, während strukturierte Ausgaben dazu beitragen, dass die Antworten der Modelle präziser und leichter zu interpretieren sind.

Welche Ressourcen empfiehlt der Sprecher für die weitere Ausbildung in der AI-Entwicklung?

Der Sprecher empfiehlt, sich auf der Suche nach Projekten und Ressourcen in der AI-Community zu orientieren. Er verweist auf seine eigenen Artikel sowie auf Artikel von anderen Experten, die er auf der Website bereitgestellt hat.

Was ist der Vorteil der Verwendung von verschiedenen AI-Modellen?

Jeder AI-Modell hat unterschiedliche Trainingsdaten und -methoden und hat daher auch unterschiedliche Stärken und Schwächen. Durch die Verwendung verschiedener Modelle kann man bessere Ergebnisse erzielen und sich an unterschiedliche Anwendungsfälle anpassen.

Wie kann man die Effektivität von AI-Modellen bei der Problemlösung verbessern?

Die Effektivität kann durch das Ändern der Eingabedaten, das Hinzufügen von Struktur zur Ausgabe, das Testen verschiedener Modelle und das kontinuierliche Experimentieren mit verschiedenen Ansätzen verbessert werden.

Welche Art von Fehlern können bei der Arbeit mit AI auftreten?

Fehler können auf der App-Ebene, bei der Datenverarbeitung oder bei der Anweisungsbefolgung auftreten. Jeder Fehlertyp erfordert eine andere Art der Problemlösung, um die Leistung der AI-Modelle zu verbessern.

Wie sieht die Zukunft der AI-Entwicklung aus Ansehen des Sprechers?

Die Zukunft der AI-Entwicklung wird von schnelleren Hardware, besseren Optimierungen und der Fähigkeit, komplexere Projekte mit weniger Ressourcen zu erstellen, geprägt sein. Der Sprecher erwähnt auch, dass die Kosten für die Nutzung von AI in naher Zukunft signifikant sinken werden.

Outlines

00:00

🌟 Einführung und Hintergrund

Der Sprecher führt in diesem Abschnitt ein und erläutert, dass er keine Vorbereitung für diese Session hatte. Er diskutiert häufig gestellte Fragen zu künstlichen Intelligenzen (KI), insbesondere zu Generativen KI und der Verwendung von Prompts. Er teilt mit, dass er ein Unternehmen namens Greywing in Singapur leitet, das sich auf kommerzielle Versande konzentriert und eine Menge an KI- und Datenarbeit durchführt. Der Sprecher beschreibt auch seine früheren Projekte, einschließlich eines Kommunikationsautomatisierungstools und eines eigenen Assistenten, sowie seine Arbeit mit multimodaler RAG-Technologie, um komplexe Informationen zu verarbeiten und kritische Fragen zu beantworten.

05:01

🔄 Iterativer Loop und Prompt-Entwicklung

In diesem Abschnitt betont der Sprecher die Bedeutung des iterativen Loops und der kontinuierlichen Verbesserung von Prompts. Er erklärt, dass die Arbeitsweise mit KI-Modellen neue Muster erfordert, die er als CPLN (Chat, Prompt, Loop, Nest) bezeichnet. Er plädiert für mehr Experimentation und das Finden neuer Ansätze, um Probleme zu lösen, und warnt davor, in der Entwicklung zu früh aufzuhören. Er schließt mit der Idee ab, dass manche Tools und Technologien besser geeignet sind, um bestimmte Arten von Aufgaben durchzuführen, und dass es wichtig ist, diese zu identifizieren und anzuwenden.

10:01

🛠️ Werkzeuge und Methoden für KI-Entwicklung

Der Sprecher konzentriert sich in diesem Abschnitt auf die Werkzeuge und Methoden, die bei der KI-Entwicklung eingesetzt werden. Er diskutiert die Verwendung von Strukturen in Ein- und Ausgabe, die Bedeutung von TypeScript und Zod für die Typspezifikation und wie SQL als Hilfsmittel zur besseren Modellierung von Suchanfragen verwendet werden kann. Er betont auch die Notwendigkeit, die Größe und Form von Daten zu ändern, um Probleme zu lösen, und gibt eine Warnung vor der Verwendung von Abstraktionen, die das Verständnis der KI-Modelle behindern können.

15:03

🚫 Dos und Don'ts in der KI-Entwicklung

In diesem Abschnitt werden einige wichtige 'Dos und Don'ts' in der KI-Entwicklung vorgestellt. Der Sprecher ermutigt, alle verfügbaren Modalitäten zu nutzen, einschließlich Sprache, Vision und Code, um die besten Ergebnisse zu erzielen. Er warnt vor der Verwendung zu viel Abstraktionen und empfiehlt, verschiedene Modelle zu verwenden, da sie sehr unterschiedlich sind. Er gibt auch Tipps zur Vermeidung von KI-Halluzinationen durch die Verwendung strukturierter Ausgaben und ermutigt, die Größenordnung von Ein- und Ausgabe zu berücksichtigen, um bessere Ergebnisse zu erzielen.

20:04

🔍 Debugging und Problemlösung in KI

Der Sprecher diskutiert in diesem Abschnitt die Herausforderungen des Debuggings in KI und wie man effektiv Probleme identifizieren und löst. Er empfiehlt, immer auf die Prompt-Ebene zurückzugehen, wenn nichts funktioniert, und bietet verschiedene Ansätze, um die Quellen von Problemen zu ermitteln. Er betont die Bedeutung von Struktur in der Eingabe und Ausgabe, um Probleme besser identifizieren und beheben zu können. Schließlich teilt er einige Tipps zur Klassifizierung von Fehlern und wie man sie effektiv beheben kann.

25:04

📈 Projektbeispiel und zukünftige Entwicklungen

In diesem letzten Abschnitt präsentiert der Sprecher ein Projektbeispiel, das von einem spezifischen Problem ausgeht und zeigt, wie man durch iteratives试验 und Hinzufügen von Struktur Schritt für Schritt zu einer vollständigen Lösung kommt. Er diskutiert auch die zukünftige Entwicklung der KI-Technologie, insbesondere in Bezug auf die Kosten und Geschwindigkeit, und ermutigt die Zuhörer, über die Möglichkeiten nachzudenken, die durch diese zukünftigen Veränderungen entstehen werden. Er schließt mit einem Hinweis auf verfügbare Ressourcen und bietet an, weiterhin bei Fragen zur Verfügung zu sein.

30:05

🙏 Dank und Abschluss

Zum Abschluss bedankt sich der Sprecher bei den Zuhörern für ihre Aufmerksamkeit und eröffnet die Floor für Fragen oder Diskussionen. Er betont seine Bereitschaft, weiterhin zu helfen und Diskussionen zu führen, und bittet um Fragen oder Feedback.

Keywords

💡AI

Künstliche Intelligenz (AI) bezieht sich auf das menschliche Denken und Handeln von Computern und anderen Maschinen. Im Video wird AI als zentrales Thema hervorgeschoben, da sie in vielen Bereichen eingesetzt wird, um Probleme zu lösen und Prozesse zu verbessern. Ein Beispiel ist die Verwendung von AI im kommerziellen Schifffahrtssektor, um komplexe Informationen zu verarbeiten und missionentscheidende Fragen zu beantworten.

💡GenAI

GenAI ist ein Akronym für Generative Artificial Intelligence und bezieht sich auf die Verwendung von AI, um neue, originalliche Inhalte zu erstellen. Im Video wird erwähnt, dass die Arbeit mit GenAI neue Denkmuster erfordert, da sich die Modelle nicht deterministisch verhalten und die Arbeit mit Prompts wie codierenartige Aktivitäten sind.

💡Prompting

Prompting bezieht sich auf die Technik, AI-Modelle durch die Bereitstellung von Anweisungen oder Stichworten zu lenken und zu steuern. Im Video wird betont, dass Prompting ein wichtiger Aspekt der Interaktion mit AI-Modellen ist und dass es erforderlich ist, sich mit verschiedenen Ansätzen auseinanderzusetzen, um optimale Ergebnisse zu erzielen.

💡Iterative Loop

Ein iterativer Loop oder Schleife bezieht sich auf den Prozess, bei dem man fortlaufend Änderungen an einem Projekt vornimmt, um es zu verbessern oder zu verfeinern. Im Video wird der iterative Loop als entscheidendes Werkzeug für die Arbeit mit AI-Modellen vorgestellt, da er es ermöglicht, verschiedene Ansätze zu testen und die besten Ergebnisse zu erzielen.

💡CPLN

CPLN ist ein Akronym, das im Video als Abkürzung für ein neues Muster der Interaktion mit AI-Modellen eingeführt wird. Es steht für Chat, Prompt, Loop, Nest und bezieht sich auf die verschiedenen Schritte, die man durchlaufen muss, um effektiv mit AI-Modellen zu arbeiten.

💡Playground

Der Begriff 'Playground' im Kontext des Videos bezieht sich auf eine Umgebung oder ein Tool, in dem man Experimente mit AI durchführen und verschiedene Ansätze ausprobieren kann, ohne sich um Produktionsumgebungen sorgen zu müssen. Es ermöglicht das Lernen und Finden von Lösungen in einem sicheren Umfeld.

💡Diarization

Diarization ist die Prozessierung von Audiodaten, um die Quellen der Stimme zu identifizieren und zu trennen. Im Video wird diarization als ein Problem genannt, das gelöst wurde, aber auch als Beispiel dafür, wie manche Probleme spezifisch für AI-Modelle sind und wie wichtig es ist, sich auf neue Ansätze zu konzentrieren.

💡Structured Data

Strukturierte Daten sind Daten, die in einem vordefinierten Format vorliegen und leicht von Computern verarbeitet werden können. Im Video wird betont, dass es wichtig ist, strukturierte Daten zu verwenden, um die Interaktion mit AI-Modellen zu verbessern und die Ergebnisse zu steuern.

💡Debugging

Debugging bezieht sich auf das Identifizieren und Beheben von Fehlern oder Problemen im Code oder in einem System. Im Video wird erwähnt, dass Debugging in der AI-Entwicklung eine neue Paradigme erfordert, da die Art und Weise, wie AI-Modelle funktionieren, komplex ist und viele Abläufe nicht deterministisch sind.

💡Latent Space

Der Latente Raum ist ein Begriff aus der Mathematik und der KI, der eine Abstraktionsstufe darstellt, in der bestimmte Eigenschaften von Daten oder Modellen liegen. Im Video wird erörtert, wie man im Latenten Raum 'surfen' kann, um Veränderungen an Prompts und Gesprächshistorien vorzunehmen und somit die Interaktion mit AI-Modellen zu verbessern.

💡TypeScript und Zod

TypeScript ist eine Sprache, die JavaScript erweitert und statisches Typings unterstützt, während Zod ein Schema-Validator für TypeScript ist. Im Video werden beide als Werkzeuge eingeführt, die dazu beitragen, die Interaktion mit AI-Modellen zu steuern und zu verbessern, indem sie helfen, klare und präzise Anweisungen für die Modelle zu formulieren.

💡NLP

NLP (Natural Language Processing) bezieht sich auf die Verarbeitung und Analyse von natürlicher Sprache durch Computer. Im Video wird NLP als ein grundlegender Aspekt der AI-Entwicklung hervorgehoben, da es ermöglicht, Dokumente zu klassifizieren, Informationen zu extrahieren und zu transformieren sowie verschiedene andere Aufgaben auszuführen.

💡HTTP

HTTP (Hypertext Transfer Protocol) ist ein Protokoll, das für den Austausch von Daten zwischen Computern im World Wide Web verwendet wird. Im Video wird HTTP erwähnt, um zu zeigen, wie verschiedene Technologien und Modalitäten wie Audio, Vision und Sprache in die Entwicklung von AI-Modellen integriert wurden.

Highlights

The speaker runs a company called Greywing in Singapore focused on commercial shipping and AI.

Greywing's initial launch was a communications automation tool utilizing AI.

The company also developed an assistant with multi-tool usage capabilities before OpenAI.

Significant work is done with multi-modal RAG (Visual RAG) for processing complex information.

The importance of output in shipping and the use of AI to handle critical data sets is emphasized.

The speaker's company has created a tool to generate documentation from meetings and conversations.

The iterative loop (CPLN) is introduced as a key pattern for working with AI.

The importance of chatting with AI models and iterating prompts is highlighted.

The concept of 'playgrounds' for testing AI tools and the value of retroactive editing of prompts is discussed.

The strategy of adding more data and test cases to loop and refine AI models is mentioned.

Nesting prompts by breaking them into smaller subsegments is recommended for clarity and simplicity.

The idea of using AI models in an iterative process from zero to 0.1 is introduced.

The importance of trying new approaches with AI, as these models are still very new, is emphasized.

The speaker advises using all modalities (speech, vision, text) when working with AI.

The use of structured input and output in AI interactions is recommended to reduce errors and hallucinations.

The potential cost reduction and speed increase in AI technology is discussed, suggesting a rethink of what can be built.

Debugging AI systems by examining prompts, model levels, and input transformations is advised.

The speaker shares personal experiences and learnings from working with AI in a commercial setting.

Transcripts

So this is just meant to be relatively casual,

I didn't really prep for this.

This is just a template and through to get a bunch of stuff.

But a lot of it is just a lot of questions

that I've been getting over the last couple of years,

just about AI and now about prompting and everything else.

Because a lot of the work that we now do with GenAI

kind of requires that you think about things differently.

So this is just a collection of all of those things

and as we go through them, we'll see.

Just tiny bit about me.

I run a company called Greywing here in Singapore.

We're focused on commercial shipping.

We do a ton of AI work.

We do a ton of data work.

When LLMs came up, we started looking at them

as NLP effectively on steroids.

Because it was a suddenly all of the English language

and people's messages, emails were all accessible.

So we started working there.

Out of that work came that initial launch,

which was a comms automation tool.

That's doing pretty well.

Another one was an assistant of our own.

Far before OpenAI had multi-tool usage or any of those things,

we were doing charts and a bunch of things.

Then came sort of RAG.

We do a ton of work with multi-modal RAG,

which is visual RAG.

So being able to process sort of complex information,

how to visually index it and sort of how to use that

to answer mission critical questions.

So that is, for example, a DRAM data sheet from Samsung.

So they were testing the product.

And in shipping, all of these data sets are really important

and the importance of the output is also very high.

So this is just to say,

like we've done quite a bit of work

across different parts of AI,

and this is just everything we've learned.

Also, I'll put up like little QR codes

just in lieu of links.

The slides are gonna be up later.

So if you see something and you're like,

oh, I wanna know a little bit more,

there's gonna be a QR code.

So this is, in the interest of hacking,

everything today is gonna be about going from,

let's say, zero to 0.1.

Right, where do you start?

How do you start?

So this is just an open source project

that I recently released.

I'll use that as an example later on,

how to go from like script to project to release.

This is effectively a tool to generate docs.

So we had a docs problem.

Everybody's got a docs problem.

But what we did have is a ton of meetings.

We'd have tons of meetings,

repeated meetings about the same things,

explaining the same things.

So now we have Whisper,

we have all of these things accessible.

Can we just make docs out of things

we've explained like 10 times before?

So this was launched like two days ago.

This has already had like, I don't know what,

50 or 60 projects already make docs using it.

Because it turns out, people, it's easy to talk.

It's very hard to write docs, right?

So that's just an example.

So the first kind of thing I wanna talk about really,

and this is, you know, it took me a long time to discover

and it made a huge difference,

is just the iterative loop, right?

And when I say iterative loop,

I mean what is your process to build stuff, right?

So long time ago, you know,

if you were coding that long ago,

we had write compile run, right?

We basically would write code,

it would take a long time to compile.

People still working on Xcode

would have that same problem today.

We'd run it, we'd go back, write, compile and run.

And then we had sort of interpretive languages come along

and then we now have like the REPL,

which is effectively the read eval print loop, right?

Which is, you write code, runs instantly,

you keep changing it, you keep changing it,

something happens.

And then, you know, a bunch of guys came along.

I don't fully agree with these guys,

but you know, they had a good point.

You know, you have test-driven development,

test build, test, that kind of thing.

But I think AI really,

because the way these models work is not deterministic

and prompting can feel kind of code-like

and working with prompts can feel kind of code-like,

but they're really the furthest thing.

So they really need new patterns.

So the pattern that, you know, me and my team

and a lot of other people that I know have fallen into

is, you know, what I'm calling CPLN.

So we'll see what that means.

So the first one's just chat, right?

Whatever your problem is, whatever you're trying to do,

just chat with models,

make more and more and more examples,

and just keep changing prompts,

keep changing what you're doing, right?

A lot of people, and myself included,

get into this habit because we have that habit from coding

and sort of building things of doing it once

and it sort of works.

And then forever, you're iterating on that particular prompt.

You make very small changes,

you keep fixing things and you keep fixing things.

But I think really where you should be spending

most of your time really is just changing

and finding new approaches to solve things

because that makes a huge, huge difference.

And that's unheard of in code, right?

You wouldn't write something with the intention

of rewriting it seven times before you got to the end, right?

You'd write something, something be broken,

you'd fix that broken thing,

you'd fix the next broken thing,

and then you'd just, you'd be done.

So the most important thing is just chat.

And I'm still surprised,

and I still talk to the people on my team, me included,

have this problem where we just don't play,

we don't chat enough, really, right?

Once we get to a system that sort of gets to like 40%,

we're like, okay, we're now going to production.

We're just, this is almost good.

And so that's a big problem, right?

The next one's just, you know,

take whatever you've learned, go to the playground.

There's a lot of tools and some of them are really good,

but in most cases, 90% of what you want

is still just in the playground.

Everyone's got a playground.

All you're really looking for is the ability

to retroactively edit prompts and conversation histories.

You know, some people call it surfing the latent space

and sort of make changes, right?

So this is where you'd spend maybe 20% of your time.

Once you've got that working, right,

let's say you've got one of it working,

the next in most cases,

and this is just examples from like Lamentis, is loop, right?

Add more data, add more test cases, a lot more.

See how solid your hypotheses were when you started, right?

And always reset if it doesn't work.

Once you're done with that, right, nest, right?

Once you're done with that,

you've got a general sense of the approach you want to take.

99.999% of the time,

and I almost dared a few people to do it,

and I so far haven't seen a single prompt

or a single approach where it couldn't be nested.

And by that, I mean effectively break the prompt,

break the work you're doing

into smaller and smaller and smaller subsegments.

We'll go into it later, right?

But if you're not gonna accept

like a 700-line code file as good,

you shouldn't accept a 100-line prompt as good, right?

Or a 50-line prompt as good.

It could always be made simpler.

It could always be broken down.

So really, you just wanna keep doing that, right?

And once you've gotten this far,

and luck would have it,

if you go to production and you've got users,

you've got subtasks now,

and they can go through the exact same loop, right?

You run into a problem,

you've got a new customer with new kind of data,

new problems, new things,

you wanna go back to the original loop.

So this is kind of where you wanna be spending

or where I found the best division of your time being, right?

This entire blue segment is just try new approaches, right?

Because these models, they've been around for about a year,

but they are so new,

and we are still finding new ways to use them,

that you might try something,

you might be the first person in the world to have tried it.

You might genuinely be the first person

to have thought of that particular way

of solving a problem with a model, right?

So I really can't emphasize that enough.

And you probably wanna spend about 20% of your time

tuning the prompts.

Almost everything usually is a prompt issue,

because I'm presuming that the people you work with

and the things you're building,

that you guys are good at coding.

If you're not, there's tons of ways to get better at it.

Computers are really good at coding.

So in most cases, it's your prompt, right?

If it's not your prompt, it's your input.

It's the data you're providing.

Ideally, see if you can change the size

and shape of that data.

And in most cases, that fixes your problem.

So a couple of do's and don'ts.

So the first one, and I still have this problem,

although it's wonderful to know that someone in this room

has solved my biggest problem,

which is diarization, which is awesome.

But really, just use all modalities, right?

I think a lot of people kind of forgot

that when we got to HTTP and in short order,

we also got audio, we got vision, right?

And we got speech to text

and all of these different modalities.

And even just the input modality of text,

you can transform it into so many things, right?

You can take text and transform that into code

to get a more structured representation.

You can get structured data.

You can do language transformations.

So use all of the tools that you've got, right?

So let's, yeah.

So speech, for example, here's where you'd use each one.

Speech is verbose.

If you've got anything dealing with users,

we love to talk, right?

This entire talk is probably gonna be,

I don't know, I'm hoping not, like about 8,000 words, right?

If you ask me to type out 8,000 words,

it would take me far longer.

I would be far less likely to do it.

And I'd probably tell you no, right?

If you present the users with a text box,

they'll give you five words.

If you ask them to just press a button and talk,

they'll give you 200 words, right?

And these models, the things that we work with,

they love context.

The more context you can provide, the better.

Vision is insanely useful, right?

There's a lot of relationships that you can capture

with a picture that you can't with text.

Like we know this, right?

It's a thousand words.

Anytime you write as a person,

you wanna put pictures in for the same reason, right?

So all of that can be captured.

And now we're getting smarter and smarter and smarter models

that can understand that information.

You can use it as really expensive OCR if you want to.

We do in some places.

But in a lot of cases, it's also far more dense, right?

Even if the diagram on the top right, top left,

that's an actual diagram that we use, by the way,

were to be represented in text,

that would be far more token heavy than that picture,

right, can encode.

Code is awesome for structure, both for input and output.

Like you almost always wanna be using structure

both on the input and the output, right?

Use structured output whenever possible.

Structure your input whenever possible, right?

Almost everything humans ever touch

usually has some structure, right?

Like when I talk, my talk has a structure.

When you write a paragraph, there's a topic sentence.

Everything humans ever do usually has structure.

And if you're leaving it out, if you're not extracting it,

it's a lot harder to control.

Yeah, this is just stuff that we use, right?

So we use TypeScript and Zod to build type specs,

and that makes it so much easier to steer these models.

We use SQL when we wanna express something as a search query.

Even if we never run that SQL, it helps the model think,

it helps the AI system sort of better guide these things.

Yeah, same thing here.

Use structured output as often as you can,

far easier to guide.

It's also far less prone to hallucinations

because you've got a type spec on the inside,

and structured output usually constrains the output

that's coming out of it, that you see far fewer generation.

Sorry, far fewer hallucinations with structured output.

And I can talk about that more if we have time at the end,

but it usually has to do with token probabilities

and the output set.

The same thing is, again, use as much as you can

because you got this massive model for free, kind of.

Commoditized down, and you got this massive model

that had two trillion, three trillion tokens thrown in

about human information into it.

Use that as much as you can, lean into it.

There's a lot of libraries, let's say projects

that I've either consulted or advised with,

where they're inventing their own DSLs,

they're inventing their own languages

to express what they want.

When ideally, if they expressed it as a superset

of something that existed, say TypeScript, Python,

English, Hindi, whatever's in there,

you'd get a lot more benefit out of that.

Cool, so this is a bunch of don'ts.

None of these are hard rules,

but they're general rules of thumb,

especially when you start out.

In AI, I mean, this is a meme at this point,

but we are still very, very early.

This is not very early days of development

or very early days of design.

If you wanted to get into design

and you wanted to be a good painter or a good designer,

you wouldn't use DALI.

You wouldn't add an abstraction between you and the thing.

You would learn how to paint

because you want that knowledge.

You actually want that harder knowledge

of how these things work, how they behave,

what the actual nature of these things are.

The more abstractions and toolkits and libraries

you put between yourself and the model

when you're developing, the less you learn.

Some of them, honestly, are really good,

but that's also a problem because they're really good

and they have this little circle of things

that they do really well.

And very quickly, if you're lucky, somewhat slower,

you'll wanna step out of it,

and then it's just a wasteland.

If you've ever built something with WordPress or Squarespace

and then just wanted to do one thing that it didn't do,

you know what I'm talking about.

That's impossible.

Everything will fight you.

So ideally, don't add abstractions.

I know it can be, especially people

with a coding background,

kind of sometimes I've seen want to distance themselves

from prompting, distance themselves

from that non-deterministic nature of these things.

Bad instinct.

Don't look away from it.

The next one is also, I know we've got credits to OpenAI,

but everyone wants to give you free money.

Everyone wants to give you free credits these days

if you're a provider.

It's too much investor money in this space.

Don't stick to one model.

They're all very different.

They were all kind of similar when they came out

because everyone was working with the same information set,

but things have diverged massively.

They're all practically different people.

It's almost like if you gave some work

to someone on your team and they couldn't do it,

you wouldn't go, oh, this is undoable.

You'd probably give it to someone else.

Same thing, work with different models.

They're all very, very different, differently trained.

There's even different personalities in there.

This one is kind of easy to keep track of.

Basically, have a general rule of thumb

that your outputs are not gonna be that much bigger

than your inputs in most cases.

Again, rule of thumb.

That's not gonna end up well.

If you're looking to generate, let's say,

20 paragraphs of an article from five words of input,

you're usually just gonna get very generic,

not so good input.

Not so good output.

So try and keep those ratios relatively the same

if you can.

Cool.

Some smaller FAQs,

because these questions get asked a lot.

So agents, a lot of people have asked me about agents.

The simple answer there is anything

with looping and termination is usually considered an agent.

So anytime you've got a system

and it basically loops on the same prompt

or some set of prompts,

and it basically has the ability to continue execution

and then decide when it wants to stop,

that's usually an agent.

This one is really helpful.

When you run into problems

or when you start working on a project,

or you're just looking for a project to work on,

it's useful to know what capabilities

just got added to the tool set, right, within AI.

These are four of the biggest ones.

The first one is just plain NLP.

If you've done NLP or anything close to it,

it just got way better.

We can classify documents,

we can classify information all sorts of ways,

we can label them,

and we can do all sorts of things with them

that previously NLP really couldn't do.

The second one's filtering and extraction, right?

So you can pull information out, right?

And the next one's sort of transformation.

So anytime you've got rags, summarization,

that's a transformation, right?

If you're doing code generation,

a lot of cases that's transformation.

If you're doing translation, that's transformation, right?

So oftentimes it's useful to look at your problem, right,

in an industry or your problems that are in front of you,

or you're just looking for ideas.

If you look for one of these four things,

if you look for one of these four classes,

it's an easier way to structure,

maybe that's where you wanna go,

instead of where to put things.

The final one, and I think some people are using it,

but I've seen that use case sort of go down for some reason,

is just general purpose generation, right?

You want it to write things no one's ever written before.

You want it to make things up.

So some resources.

I'm not gonna be talking about prompting,

not gonna be talking about rag.

These are just some articles.

These are my articles.

If you don't like me,

the top of it has people that I respect

that are far smarter than me,

so click the links and go there and read those.

Cool.

The next one, and this might be the final one,

is debugging, right?

I don't think I've heard that many people talk about it.

I mean, among people who work with AI,

this is a massive conversation, right?

How do you debug?

Because the sort of curse and sort of the benefit

that we got with modern AI things

is that it's very easy to build a demo.

It's very easy to get to something that sort of works,

but it's very hard to debug things when they go wrong.

That's almost, again, new paradigm.

So what is happening to you?

If nothing works, always go down to the prompt level,

and if you can't, then get rid of your abstractions

and work up from there.

Try a different model.

Try going up a level of intelligence and see if it fixes it.

That should tell you where your problems are.

Or try going down a level of intelligence

and see what happens.

The next one is transform the input.

In most cases, it's your input that's the issue.

Either it's too verbose, it's not the right transformation,

it's not structured the right way.

So any transformations you can do on the input

is gonna make a massive difference, right?

And finally, if you're not doing this already,

add more structure to the output, right?

More structure is gonna help you point out

where your problems are.

More structure is gonna tell you,

sort of expose some of the big issues there.

Okay, so this is, you know,

this doesn't usually happen to people.

This usually does, right?

Is it's kind of working, it's kind of working.

And I can spend another, you know, like two weeks on it

and it'll get a bit further down the line

of kind of working, but it's not working necessarily, right?

So again, I'm gonna go back to data.

In most cases, you wanna find out

what separates your offensive data,

which is where it doesn't work,

to the stuff that does work, right?

Try all sorts of transformations.

One of those is gonna point to some sort of difference

between the stuff that works and the stuff that doesn't,

right?

If you do, that's a prompt, right?

More validation is always gonna help.

And then we saw the classification before, right?

If you're trying to do more than one of those things

inside the same system, inside the same,

with the same model, usually separate it out, right?

And it makes a huge difference.

Finally, yeah, just classify your errors.

Most errors I've seen sort of fall into these three issues.

You've either got app level issues

in terms of how that data is being fed in and fed out

and how models are orchestrated once things get too large,

or you get factuality issues, right?

It's just making things up that don't exist

or it's just giving you information

that it really shouldn't

or pulling out the wrong information.

That's a factuality issue.

The third one's just instruction following.

Is it just not listening to the specific instructions

that you're giving it, right?

And this is at the model level,

but it happens at the meta level as well.

Even if you're working with, say, three models,

and 300 prompts, all of these things still apply.

Okay, so what do you do, right?

The first one is whatever you're doing, right?

Whatever you're doing as far as prompting

and working with Models Go,

you're almost always too verbose

because in most cases it's English,

and once we start adding things, they kind of work.

So you get to this sort of Pareto level of, you know,

it works, but it just doesn't.

It's almost how humans behave.

Cut them down.

There's usually space to cut them down.

Cut them again.

The lower your task level, the more you can do it.

The higher your task complexity per prompt,

or per task, or per function, the better, right?

The easier it is to debug,

the easier it is for you to have, you know,

things with defined blast radiuses,

where if something goes wrong,

you can swap it out and fix it.

Otherwise, you know, something goes wrong someday,

you're gonna have a problem.

So, how much time have we got left?

Okay, 10 minutes, perfect.

So this is just an example of that particular project

that I mentioned at the beginning, right?

So it started with just a specific issue, you know?

Like, you know, honestly, it wasn't even me.

It was Hiby, who's actually here,

who had a transcript for me, and she was like,

okay, can we make docs out of this, right?

Or I think it came partly from that.

So there was a lot of talking.

There was a lot of trying to figure out

what we can pull out,

what it understood out of the transcript.

You're trying to look for understanding.

You're trying to see if this can even be done.

You're just testing very high level hypothesis, right?

Some of the things I tested

were sort of trying to pull out structure directly.

Some of the other ones were trying to classify that data

before pulling out structure.

So you learn just a lot about what it is.

You figure out where you want to put the transcript,

whether chunking is a valid strategy,

all of that you can learn from just talking, right?

The next one is talk, but then start changing things, right?

Now you start adding steps.

Now you start adding structure.

You start getting information out.

And once you're done with that, you know,

the entire thing, and this actually worked,

was just this one script, right?

Really, I mean, you don't have to read that.

It's actually in the repo.

It's just this one script, right?

And really all it did was just loop, you know,

twice over everything,

and then break it down into sections

and use different models to write different things, right?

So there's one model, you know,

that's generating the structure.

There's another model that's actually doing

the long form writing.

And then the final one is just breaking it down

into smaller and smaller and smaller functions.

So if you look in the repo, still not that big, right?

But there's a lot more state management.

There's a lot more state management.

There's a lot of self healing.

There's a lot of correction.

All of that stuff can go in after,

like you've proven the thesis.

Cool, actually I'm ahead of time.

I didn't think I would be.

So the final thing, and I will say this,

is a lot of people I speak to

are still very concerned about cost, right?

I don't know how many of you guys watched

the NVIDIA keynote that happened like a couple of days ago,

but long story short, everything you're using now

is gonna get at least 10x, if not 50x cheaper

in very short order, right?

It's gonna get 10x, if not 50x faster in very short order.

So what would you build if you were building

for say six months from now, or what would you make

if you just presumed that today, right?

And it's a different way of working with these things.

You know, if something costs 10 bucks,

that's a different system than if it costs one cent, right?

If something takes an hour,

that's different from if it takes six minutes, right?

So I would say this is a valid presumption to make, right?

When you're building something,

is what more can you do if you just presume

that about the future, immediate future, right?

Because we still haven't even gotten

hardware level optimizations,

that's what NVIDIA is doing now.

You know, that's a 10x.

Memory level optimizations, again, still coming up.

That's a 10x.

Quantization, that's probably another 10x.

So all of these things are almost being done now,

and they're very comparatively easy engineering-wise.

It's just incremental optimization to get there.

Okay, that's everything.

Feel free to find me after, or just reach out on Twitter.

I'm happy to help.

Do you have some questions?

Oh, sure, yeah.

They're gonna think about models, and embedding models.

Okay, so long context is tough, right?

Because I might say something

where I don't know what I'm talking about.

That said, this has been my question as well.

The problem with context windows is

our algorithm for attention is quadratic.

What I mean by that, it scales exponentially.

To get twice as much context out of something,

you've gotta spend four times the amount of memory

and compute.

We still have that curse.

There's no way to, we still don't know

a good way to get around it, right?

So what that means, effectively,

is to get really long context windows, you have to cheat.

You effectively have to say,

okay, I'm gonna have something,

before I run the model, that's gonna kind of figure out

which part of the context to actually pay attention to.

So you don't actually get the full context window, right?

You kind of do, but if you take the full context window,

and you're trying to use every single token in it

to compute an answer, it's not gonna work.

So that is still very much a problem.

That could be solved, that's one of those open problems,

I think it's an open problem,

that could be solved tonight by someone

that's working somewhere, or 10 years from now.

We just don't know, right?

You've mentioned a bunch about transforming the input.

How do you go about doing that?

Do you use AI to transform the user input?

In most cases, yes, you're gonna be using AI

to transform it, but there's also just a ton

of structured stuff you can do, right?

Very easily, like most documents.

Let's say I've got a PDF, or I've got,

let's say these slides, or I've got one of my documents

is in Markdown, there's a ton of structure in there

you can just grep for, right?

Because I can very quickly figure out

what the sections are, I can very easily

separate by sentences, that's all stuff

that you can do today, right?

So even just knowing that that's got 300 sentences in it,

that's a transformation of the input that is valuable.

Super valuable, right?

Because we can make assumptions already, right?

If I give you a document that someone's written,

I can presume that the title is probably

the highest compressed information in there, right?

That is a good enough thing.

I can presume that the first section

will have some sort of intro of what the thing is.

Right, those are all transformations.

But yes, usually you use AI.

I had a question.

So how do you think about leaving the new,

the new feature?

Devon?

Yeah.

I actually haven't used Devon.

I just have not had the time,

but I've had people tell me that it's good.

Look, coding is gonna be where these models

make just a massive, massive difference, right?

I already use a cursor, which can understand

just a massive amount of context and sort of work.

It has been six months since I wrote any code

that wasn't at least partially AI generated.

So it's just gonna keep getting bigger and bigger and bigger.

That said, I will say the time that most devs that I know

and most companies that I know spend

is in business logic, maintenance,

and sort of really trying to transform customer input

to really massive systems with a ton of legacy code.

Like we're a long way away from that, right?

What I mean is it's getting easier and easier

for you to spin up a more and more

and more complex project from scratch, right?

But the massive dev work that sort of sits,

kind of sits past that, right?

That still hasn't been touched.

The efforts to do something there,

because that's where the money is in a lot of ways,

because that's where most enterprises are, right?

If you look at SAP or you look at most of these guys,

have not so far borne active fruit.

Like I know most of the companies in that space,

they're still having trouble getting it to work

with very large code bases, right?

Like let's say anything above like a 50 person company

that's existed for more than three years.

That code base, so far, AI hasn't been able to touch, right?

Yeah.

Okay, I had a question.

So I recently read a, not read,

I wouldn't say read the paper, I read the abstract, right?

So where it was like, I think from Amazon

or from somewhere like, or Netflix perhaps,

that getting cosine similarities between embeddings,

it's not really a good measure

for getting the meaning of things, right?

And this is a preface for my question.

And also when we do like vector searches

and just try to pull relevant information,

I don't know, it feels like it doesn't work.

I'm trying to figure out like what am I doing wrong?

How to do it better?

I watched Jerry Hill's talk like on from our index,

is it 18 minute talk or something?

It's a very nice talk, but it just kind of flew over.

So like, what's your recommendation?

I think the problem here is embeddings

are sort of fuzzy search on steroids.

If you're using them for anything more,

I think even today you have a problem, right?

Because the couple of things,

one, these are really tiny models comparatively, right?

Big brain, small brain, tiny brain,

these are really tiny models.

In most cases, they don't have a good understanding

of the underlying text.

That's why long context embeddings never made sense, right?

The longer the context, it just doesn't really make sense.

Not to mention in most cases,

that's a transformation of the input, right?

What Hibbe was saying,

that's a transformation of the input,

is you're transforming it,

but you're transforming it into a space

where it's a lot harder for you to work with it, right?

You're transforming it to a set of numbers.

And now the only thing you have is cosine similarity.

You can have a bias matrix,

you can push that math a little bit more,

but because that model is unknown to you,

the model's workings are unknown to you,

those are forever gonna be a bunch of numbers, right?

In some insanely high dimensional space.

So there's not a lot to do there, right?

What is becoming very possible now

that I see a lot of companies switching to

is just use the whole brain.

Use the LLM, right?

Like whatever you're using embeddings for,

you can use an LLM, right?

It's just more expensive.

Right, in most cases, you can use an LLM for that.

Like let's say you're using,

I'll give you the most brute force example of this.

Let's say using embeddings to take 100,000 items

and see which ones are similar

or which ones closest to your query.

You can take an LLM,

run it through every single one of those documents

and ask, hey, is this close?

Is this close?

Is this close?

And you'll get an answer, right?

That is not a good way to do it, do not do it this way.

But you see what I mean, right?

So they are kind of, you know,

you can substitute one for the other just a little bit.

I think embeddings have a place, right?

But they should always be the last step in your pipeline.

You should cut down the search space as much as possible

with structured search, transformations,

you say BM25, there's a bunch of stuff you can do, right?

You should never be searching your search space

with embeddings, right?

You should always be searching some reduced search space

where, you know, hey, last 20 things, you know?

And I know these are relevant because keywords,

I know these are relevant because location,

I know these are relevant because an LLM told me

after transformation, whatever.

Now I can embed, that's fine, right?

But if you embed at the beginning,

like in most cases, it just doesn't work at scale.

So it's more like to get the results and then sort it.

Is that where embeddings come in?

More like to get the results and yes, kind of to sort it,

but kind of also to identify like useful parts

of those results.

Let's say the results you got were pages,

but you want sentences, right?

You wanna know which part of it is, you know,

heat map wise, the most important.

You can use embeddings for that, right?

All right, thank you so much.

Yeah.

Thank you.

Thank you.