Building a Generative UI App With LangChain.js

Summary

このビデオでは、3部作のシリーズの第2部として、Langing Chainを使用して生成型UIアプリケーションの構築方法が解説されています。シリーズでは、生成型UIの概念やユースケース、そしてチャットボットのハイレベルアーキテクチャが紹介されています。さらに、PythonパワードのバックエンドとNext.jsフロントエンドを使用した実装が詳述され、TypeScript開発者向けにReactサーバーコンポーネントを活用したストリーミングUIコンポーネントの実装方法が解説されています。

Takeaways

• 😀 このビデオは、LangChainを使用して生成的UIアプリケーションを構築する三部作シリーズの第二部です。
• 😀 最初のビデオでは、生成的UIの基本概念、ユースケース、高レベルのアーキテクチャについて説明しました。
• 😀 次のビデオでは、Pythonをバックエンド、Next.jsをフロントエンドとして使用するチャットボットを取り上げます。
• 😀 TypeScriptの開発者でない場合は、次のビデオを待つか、今日のビデオの一部のトピックが重複するため視聴することをお勧めします。
• 😀 今日のビデオでは、ユーザー入力、画像、チャット履歴をLMに送信し、条件付きエッジを使用して応答をクライアントにストリーミングするチャットボットの内部構造について説明します。
• 😀 Reactサーバーコンポーネントを使用してUIコンポーネントをストリーミングし、サーバーからUIに戻すためのユーティリティサーバー.tsxファイルを実装します。
• 😀 LangChainのストリームイベントエンドポイントを使用して、すべてのイベントをクライアントにストリーミングします。
• 😀 チャットボットの構築に使用するLグラフエージェントを構築するためのLグラフのビデオへのリンクも提供されます。
• 😀 GitHubリポジトリの詳細を取得するためのツール、請求書の詳細を抽出するツール、天気情報を取得するツールを実装します。
• 😀 最後に、デモ用のアプリケーションを起動し、チャットボットが正常に動作することを確認します。

Q & A

このビデオは何について説明していますか？

このビデオは、Lang Chainを使用して生成的なUIアプリケーションを構築する方法について説明しています。

前のビデオでどのような概念がカバーされましたか？

前のビデオでは、生成的なUIとは何か、ユースケース、高レベルのアーキテクチャについて説明されました。

次のビデオで取り上げる予定の内容は何ですか？

次のビデオでは、PythonチャットボットのバックエンドとNext.jsフロントエンドについて取り上げる予定です。

このビデオではどのような技術を使用していますか？

このビデオでは、Reactサーバーコンポーネント、Lang Chain、AI SDKなどの技術を使用しています。

サーバーコンポーネントで行われる処理は何ですか？

サーバーコンポーネントでは、UIコンポーネントのストリーミングと実行可能なランダム処理が行われます。

ストリームイベントとは何ですか？

ストリームイベントは、Lang Chainアプリケーションからクライアントに中間ステップをストリーミングする方法です。

実行可能なUIを作成するための主要なファイルは何ですか？

utils/server.TSXファイルが、UIコンポーネントのストリーミングと実行可能なランダム処理のロジックを含んでいます。

GitHubリポジトリツールの役割は何ですか？

GitHubリポジトリツールは、GitHub APIを使用してリポジトリの詳細を取得し、UIに表示する役割を果たします。

チャット履歴の管理方法について説明してください。

チャット履歴は、ユーザーの入力や画像とともにLMに送信され、適切なツールを使用して応答が生成されます。

エージェントエグゼキューター関数の目的は何ですか？

エージェントエグゼキューター関数は、ステートグラフを作成し、ノード間のエッジを定義して、ツールの呼び出しや応答の生成を管理します。

Outlines

00:00

😀 シリーズ第2回目のビデオ紹介

この段落では、Braceが3部作のシリーズの第2回目のビデオを紹介しています。シリーズはジェネラティブUIアプリケーションの作成に焦点を当てており、Chainを使用しています。前回のビデオではジェネラティブUIの概念や使用事例、Chatbotのアーキテクチャが説明されています。また、明日公開される次のビデオではPythonパワードのバックエンドとNext.jsのフロントエンドについて触れる予定です。TypeScript開発者でない場合は、次のビデオを待つか、現在のビデオでもNext.jsのフロントエンドやコードのストリーミングに関連するトピックが重なっているため視聴を検討するべきです。

05:00

🛠️ 高レベルアーキテクチャのリフレッシュ

この段落では、ビデオの高レベルアーキテクチャのリフレッシュが行われています。Chatbotがどのように動作するかを説明し、ユーザーの入力やチャット履歴をLanguage Model（LM）に送信し、応答を生成するプロセスが解説されています。応答がテキストのみの場合は、クライアントにストリームバックされることが示されています。また、ツールが使用された場合は、ツールの実行とUIコンポーネントの更新が行われます。このプロセスはLangChainと呼ばれるツールを使用して構築されており、条件付きエッジやストリームイベントなどの概念が紹介されています。

10:03

📝 utils.server.tsx ファイルの実装

この段落では、utils.server.tsxファイルの実装について説明されています。このファイルは、UIコンポーネントのストリーミングとランナブルの呼び出しに関するロジックを含むReact Serverコンポーネントです。重要なインポートと型の定義、およびwithResolvers関数の作成が行われています。この関数は、プロミスとresolve/reject関数を返すために使用され、UIコンポーネントのストリーミングを完了する際に使用されます。

15:04

🌐 エンドポイントの公開とUIコンポーネントのストリーミング

この段落では、APIエンドポイントの公開方法とUIコンポーネントのストリーミングについて説明されています。exposeEndpoints関数を使用してアクションに適切なコンテキストを提供し、AIプロバイダーReactコンテキストを活用してクライアント側のファイルにコンテキストを提供します。また、createRunnableUI関数とstreamEvents関数が実装され、これらの関数はランナブルとUIコンポーネントのストリーミングを処理します。

20:05

🔗 ストリームイベントとランタイムUIの実装

この段落では、streamEvents関数とcreateRunnableUI関数の詳細な実装について説明されています。これらの関数は、ランナブルからUIコンポーネントをストリーミングバックし、ツールの呼び出しを処理します。streamEventsは、ランナブルからのイベントをストリームし、UIの更新やテキストのストリーミングを担当します。createRunnableUIは、UIコンポーネントを作成し、ランナブルをラップしてストリームイベントを生成します。

25:06

🛠️ ツールの実装とランタイムUIの使用

この段落では、ツールの実装方法とランタイムUIの使用について説明されています。GitHubのリポジトリ情報取得ツールの例を通じて、Zodを使用したスキーマの定義、ocitを使用したGitHub APIの呼び出し、createRunnableUIを使用したUIコンポーネントのストリーミングが実装されています。ツールはダイナミックに定義され、言語モデルから取得されたパラメータに基づいてAPIを呼び出し、UIコンポーネントを更新します。

30:08

📈 天気予報ツールとインボイスツールの紹介

この段落では、天気予報ツールとインボイスツールの概要が紹介されています。これらのツールは、言語モデルから得られたパラメータをもとにAPIを呼び出し、UIコンポーネントを更新します。天気予報ツールは、city、state、およびオプションのcountryを受け取り、weather.gov APIを利用して現在の天気情報を取得します。インボイスツールは、アップロードされた画像からインボイスの詳細を抽出し、UIコンポーネントを生成します。

35:08

🔄チャットUIの実装とデモ

最後の段落では、チャットUIの実装とデモが行われています。チャットUIはクライアント側のコンポーネントで、サーバー側のランタイムUIを呼び出します。チャット履歴、メッセージの送信、ファイルのアップロードなどが行えるインターフェースが提供されています。また、デモでは、天気予報、インボイスの詳細、GitHubリポジトリ情報の取得など、実装されたツールを使用して様々な機能がテストされています。

Keywords

💡Generative UI

Generative UI refers to user interface components that are created or modified dynamically by a system, typically using machine learning models. In the video, generative UI is the central theme, where the process of building such interfaces using LangChain is explored. The concept is vital as it allows for more responsive and adaptive user interfaces based on user interactions and inputs.

💡LangChain

LangChain is a framework mentioned in the video for building generative UI applications. It enables the creation of dynamic and interactive user interfaces by leveraging language models and other tools. The video series uses LangChain to construct a chatbot with sophisticated backend processes that enhance user experience by dynamically updating UI components.

💡Next.js

Next.js is a React framework used for building server-rendered React applications with support for static site generation. In the video, Next.js is employed to develop the front end of the generative UI application, highlighting its importance in handling the dynamic content rendered by the backend processes.

💡React Server Components

React Server Components are a feature of React that allows developers to render components on the server, which can then be streamed to the client. This concept is crucial in the video as it describes how UI components are generated and sent from the server to the client, ensuring efficient updates and interactions in the generative UI application.

💡AI SDK

AI SDK refers to a software development kit used to integrate artificial intelligence capabilities into applications. In the video, the AI SDK is used to manage the streaming of UI components from the server to the client, facilitating the integration of dynamic content generated by the backend processes.

💡Stream Events

Stream Events are mechanisms used to stream intermediate steps or data from a server to a client in real-time. In the context of the video, Stream Events are utilized to transmit UI components and data dynamically from the LangChain application back to the client, enabling real-time updates and interactions.

💡Runnable Lambda

Runnable Lambda is a function or component that can be executed to produce or modify UI components dynamically. The video explains how Runnable Lambda functions are created and used within LangChain to handle the streaming and updating of UI components, making the UI adaptive to user interactions.

💡Conditional Edge

A Conditional Edge is a concept from the LGraph used in LangChain to determine the next step in a process based on conditions. The video illustrates how Conditional Edges are used to decide whether to stream text responses directly to the client or to invoke additional tools for more complex processing, enhancing the chatbot's functionality.

💡Multimodal Inputs

Multimodal Inputs refer to inputs that involve multiple forms of data, such as text and images. In the video, the chatbot is designed to handle multimodal inputs, processing and responding to user queries that may include text and images, demonstrating the flexibility and advanced capabilities of the generative UI application.

💡Agent Executor

Agent Executor is a component in LangChain that manages the execution of tasks and processes within the application. The video details how the Agent Executor handles the flow of data and interactions, invoking appropriate tools and managing the state of the application to ensure seamless user experiences.

Highlights

シリーズの第2弾として、generative UIアプリケーションを構築するための3部作のビデオを紹介

前回のビデオではジェネラティブUIの基本概念やユースケース、チャットボットのアーキテクチャが説明されている

明日公開の次のビデオではPythonパワードのバックエンドとNext.jsフロントエンドを構築する

TypeScript開発者ではない場合は、Next.jsのビデオを見逃さないで推奨

ビデオではReactサーバーコンポーネントとLang Chainを使用してUIコンポーネントをストリーミングする方法が解説されている

Lang ChainとLang Graphを使用して、サーバーからUIへのイベントストリーミングを実装

stream eventsエンドポイントを活用して、Lang Chainアプリケーションからの中間ステップをクライアントにストリーミング

utils server.tsxファイルの解説で、Reactサーバーコンポーネントのロジックを含む

AI SDKのインポートとその使用方法、UIコンポーネントのストリーミングに活用方法が紹介

withResolvers関数の解説で、プロミスとresolve/reject関数の使い方と重要性が強調

exposeEndpoints関数を使用して、アクションに適切なコンテキストを提供する方法

createRunnableUI関数の解説で、ランナブルLambdaとUIコンポーネントのストリーミング方法が説明されている

streamRunnableUI関数の解説で、エージェントの呼び出しとUIコンポーネントのストリーミングを実装する方法

L-graphエージェントの作成と、チャットボットの内部アーキテクチャの解説

invoke modelノードとinvoke toolsノードの役割と実装方法が解説

GitHubツールの実装例で、Zodを使用したスキーマ定義とocit SDKの活用方法

ツールのロジックを実装し、APIのレスポンスをUIコンポーネントにストリーミングする

チャットJSXコンポーネントの解説で、クライアントサイドの実装とサーバーサイドの連携方法

デモアプリケーションの実演と、実際に構築されたチャットボットの機能紹介

Pythonビデオの告知と、今後の公開予定についての案内

Transcripts

what's up everyone it's brace and this

is the second video in a three-part

series on building generative UI

applications with laying chain if you

haven't seen the first video already you

should go back and watch that in that

video we cover some high Lev Concepts

such as what is generative UI we cover

use cases and then also we go over the

highle architecture of how we're going

to build this chatbot and then also in

the next video which releases tomorrow

the python chatbot uh which will be a

python powered back end and nextjs front

end that being said if you are not a

typescript developer you should probably

hold out for tomorrow's video or you can

still watch today's video because some

of the topics will overlapped with the

nextjs front end um and the aisk which

we will use to power some of the uh code

which sends the UI component from the

server or in this video's case the react

server component back to the UI so as a

quick quick refresher this is the highle

architecture of what the chap bot we're

going to be building today will look

like on the inside so it takes some

inputs user input any images some chat

history sends that to an LM the LM has

some tools bound to it then using that

response uh we have this conditional

Edge if you're familiar with L graph you

should know the what a conditional Edge

is if not I'm going to link a uh video

going over L graph which you should

watch because that is what we're going

to be using to build our L graph agent

in this video so the conditional Edge

says if there's only a plain text

response then stream those chunks back

to the client and then as this those

chunks come in from open AI or whatever

model provider we use we render them on

the client if a tool is used that gets

sent to our invoke tool section where we

first stream the initial component back

to the UI which could be some sort of

loading component or UI element that

tells the user hey we have process your

request we're using this tool um and

we're going to get back to you in a

second so it just allows for a quicker

time to First interaction we then can

execute some arbitrary tool function

which is just a generic typescript

JavaScript function so in our case it'll

typically be hitting an a an external

API and then once we get the response

back from that we update our UI

component stream with the final

component and close it uh you can call

update or append as many times you would

like to update or append UI components

um to the UI your user see

sees the way we're able to stream all

these events back to the client is via

the stream events endpoint in Lang chain

stream events essentially is a way to

stream intermediate steps from your

Langan chain application back to the

client or just send them in a in a

response object um since this is all

using Lane chain and Lane graph stream

events is able to access every single um

yield that a function might yield so in

our case we're going to be yielding UI

components we're going to be yielding

data and then once again yielding UI

components again or yielding text um in

stream events is able to just capture

all of those streams and then forward

them back to the client okay so the

first file we're going to want to

implement is our utils server. TSX oh

and if you want to follow along the link

to this GitHub um repo will be linked in

the description you can clone it and

then you can go through and read each of

the files as we as we code them um the

code will all be there in that refo but

this first server. TSX file this file is

what's going to contain all of our logic

around streaming UI components and

invoking our runnable so let's add our

Imports the first line you can see it's

server only the all of the streaming UI

components and invoking the runnable

will have will happen inside of a react

server component so we want to make sure

this file is only used on the server

next we're going to import react node

and is valid element from react we're

going to use uh the this for typing as

this is a type and then is valid element

we're going to use to make sure the

elements inside of our stream from our

runnable are UI elements before sending

them back via the aisk which leads us

into this next import the AIS SDK

Imports um we use the AI SDK under the

hood to handle streaming back UI

components because they do a lot of the

heavy lifting with react server

components next we have some imports

from runnables or sorry from Lang chain

core runnables uh this will we're going

to be using as a type and then also

runnable Lambda is what we're going to

use to wrap our one of our streaming UI

components back so we can um essentially

upsert the UI components into the stream

event next these are all going to be for

Ty types we have our stream event type

and then we have our AI provider which

is going to provide context we're going

to use this to wrap um our children so

that all of the UI components have the

proper context and then this we're also

going to use as a

type so the first function we're going

to implement is the with resolvers

function this function is going to

return um a promise and then a resolve

and a reject function we're going to use

these so that when you are streaming

back from the UI component or sorry from

when you're streaming the UI component

and any other values from our chain

we're able to properly resolve the final

promise and know when our UI is done um

streaming inside of this component we

have a resolve reject function a new

promise which then assigns the resolve

and the reject function to the resolve

and reject function from the promise and

then we return it we need to expect the

error here because typescript does not

think that these have been assigned

technically that true however the way

we're going to use this um we will

always use this promise

before calling these resolve and reject

functions so although typescript thinks

they're not used yet in practice we will

use them after calling our

promise next we're going to implement

our expose endpoints function this

function is what we're going to use to

return the proper actions and provide

context to these actions so they can

call our agent and then also so our UI

components have the proper context we're

using this AI

provider react context that we imported

from client. TSX which is is in which is

in the same UTS file if you're following

along and then as we see here we also

have this use context hook which uses

the use context from react and it's

going to provide context to our client

side uh files so they can properly

invoke the

agent here it's pretty simple returns a

new function AI which contains an AI

provider um react contacts or jsx

component which you saw in the other

file the next function we're going to

want to implement is the one which will

handle streaming back all the UI

components so handling this stream

handling this stream and then it's what

passes the UI components from our tool

call functions up into our stream events

call so we'll implement the stream

events function after this one but this

function we're going to implement will

handle we will

is essentially wrapping the aisk and a

runnable Lambda and is uh streaming and

yielding these UI components so that we

can access them inside of our stream

events we'll Implement that here it's

going to be called create runnable UI

takes in two args one is required the

second is optional the config argument

this is going to be used to we're going

to pass in our config config values here

and this is so these stream events um

when we invoke stream events it's going

to have access to this runable Lambda

function next we take in a an optional

initial value this gets passed to the

create streamable UI function so you see

right down here we have our runnable

Lambda this Lambda takes in a single

input of initial value which should

actually be

optional um this initial value is going

to get passed to the create streamable

UI function from the asdk and then we're

going to return this UI value this UI uh

function is going to have a value which

is the jsx element this is what we're

going to use to actually send back to

our client and it's going to contain the

stream which can update uh anytime we

call update aen error done um and this

is what we're going to render on the

client we then attach a config so that

this runnable Lambda always has the same

name stream UI Lambda and this is what

we'll use later on to identify this

runnable Lambda and extract the UI value

from it and then we're going to return

us invoking that function passing in the

proper config so that using the we pass

in the config so that stream events is

able to find this runnable Lambda inside

the St stream

event the next function we're going to

want to implement is the function which

we'll we'll use to invoke our agent this

is also going to call stream events and

extract any UI values we return from

here so when we invoke our agent it's

going to be using calling stream events

inside of this function so let's paste

that in and then let's walk through

exactly what this does

so it's called stream runnable

UI the first line we're creating a new

streamable UI from thek and then using

that with resolvers function we

implemented below uh the last event

which is our promise and then our

resolve function we then have an async

function which executes in here uh we

have some callbacks which contain a

string and then either the return type

of create streamable UI or create

streamable value we'll use those in a

second and then as you can see our first

input is a runnable

we're going to call stream events on

this runnable to stream back all the

events this runnable will be our

agent we iterate over each event and the

first thing we do is we check to see if

it's a UI value which was returned from

this Lambda so as we can see our run

name is here we're checking to see if

the stream event. name is that run name

and it's after that Lambda has finished

so on chain end should be the event if

it is that event and and runnable Lambda

we're then going to check and make sure

that the value of the output is a valid

UI element if it is then we're going to

append that UI element to our UI which

we created via the great streamable

UI next we're going to check to make

sure that it's a stream and it's not a

chain um this is what we're going to use

to extract any text values from the uh

from the our agent or our Ling graph

graph that will be this part of our

diagram where the LM returns just text

I'm going to send it right

back so if it

is text and type of chunk. text is

string we want to make sure we have not

already processed this this run run

event because your llm could in theory

invoke a language model twice and get

two sets of text streams back so we make

sure that we've not already processed it

if we haven't then as this comment says

the create streamable value sluse

streamable value is preferred as the

stream events are updated immedi in the

UI rather than being batched by react

via create streamable UI so if we're

just updating text we want to use this

create streamable value and not create

streamable UI then we're going to append

our create streamable UI with a generic

react or jsx function this could be you

know this is customizable for what you

want we have ai message text which is

going to be our text bubbles but you

should probably replace this with

whatever UI you want and here you say

you see using Create streamable value

which we mentioned there and then the

value of that is the value of our text

stream and we're using text stream so we

can bypass any sort of batching that

react does and instantly update our UI

with the stream as it comes

in next if the Run idea is true which it

is because we just said it we're going

to append the text from the stream event

chunk this will be the text language

while stream back and then we're also

going to be updating our last event

value of the stream event this happens

at the end of each stream event so we

know that the last event value will

always be the last stream event finally

when our stream events has resolved

we're going to resolve our promise which

we implemented in the with resolvers

function with the output of our stream

event this will typically be some sort

of string but it could also be um an

object with say our tool

call we're then going to iterate over

all of our callbacks and call done on

them to finish our callbacks and then

call UI Doone which is going to close

the UI stream between the server and the

client via the the aisk finally we're

going to return these values in the last

event which we will use in our client

when we Implement

that now that we've implemented our

stream runnable UI and create runnable

UI we can go ahead and Implement our

graph this is going to be this language

model graph with the Edge invoking tools

or sending back the response we're going

to go into the AI graph file and first

things first we're going to add our

Imports as you can see we're not adding

any serveron um text because since this

is only going to be used inside of our

serveron code here it'll already be

server only we're importing some prompt

templates um start and end uh variables

from L graph State graph which is what

we're going to use to create our lane

graph chat open AI you can obviously

replace this with any language model

which supports tool calling from the the

lane chain Library um our GitHub tools

which we'll Implement later or after

after this I guess we're going to find

our base message which we use for types

and then runable con config which we

will also use for types once we've added

our Imports we're going to want to add

our type for our L graph agent so we're

going to name it agent executor State

the first value is a is the input this

is going to be the input that the User

submitted right here next is chat

history once again the chat history from

their previous conversations and then we

have some optional values these are

optional because they're only going to

be popular ated later on in our graph so

result the plain text result in LM if no

tools used that's going to be this part

so if the LM does not invoke a tool and

only returns some text that's going to

populate this value next is the parse

tool result that was called if the LM

does call a tool we're going to parse

that and return it that will be pointing

to this conditional Edge and then

finally the result of a tool that will

be anything that this arbitrary function

returns um and we're going to uh

actually include the result of that

because we want to update that in our

chat history later on so the language

model knows that it did in fact um

invoke and complete any tool requests

that the user ask

for after that we can skip adding our

nodes for now because we're want to con

we're going to want to construct the uh

graph

first so we're going to create our agent

executor function this is going to

create a new state graph passing in our

state and then also creating these

channels um here we're going to add just

two nodes one for invoking the model and

one for for invoking the tools invoke

model will obviously be this part make

that a little bigger and then invoke

tools will be what happens when we pick

a tool and we want to invoke that tool

invoke model will then always call

invoke tools or return which is this

conditional Edge this conditional Edge

will just check to see if the tool is

used if it is used then this function

will then return invoke tools so invoke

tools is called if it's not used then

it's going to return end which is this

end variable and that indic Ates to Lane

graph that it should always finish and

that's what this is where it finishes

and sends the response back invoke tools

will also then always end so that once

the tool is done invoking it returns and

finishes the L graph graph and responds

back to the UI and obviously start is

always going to call invoke model

because that's the first thing we want

to do in all of our graphs we're then

going to compile it and return our graph

and then we're going to use this a

little bit later on when we're passing

it to our stream runnable UI

and this graph right here will be the

runnable that is in is invoked via

stream

events so if we go back to our graph

file the first node we're going to want

to implement is the invoke model as

that's going to be the first node which

is always called so if we paste that in

let make this a little bit bigger we can

see it takes in two values state which

is our agent executor State this is kind

of the magic behind Lang graph where

it'll always pass the state to every

single node even though we're not

returning the full State here so Lane

graph can recognize what values in your

state that you returned it just appends

that to the total State and then passes

the complete State through to each node

so we take in our state and then also

our config value this is what we're

going to pass to invoke so that out our

Lang Smith traces and the stream events

all have the same all contain the same

runs in the single trace so stream

events can get back all the

values first thing inside our functions

we Define our prompt you're a helpful

assistant you given a list of tools need

to determine which tool is best to

handle the user input or respond with

plain text we then have a message

placeholder for the chat history this is

where our chat history rle go so the

language model has access to all of our

history it's obviously going to be

optional because there will be no

history on the very first invocation and

then we have our human message with just

a plain input uh

argument next we Define our tools we'll

Define those after this function but

we're going to provide three Tools in

language model uh GitHub tool invoice to

web weather tool when we Define these

tools we'll talk about what each of them

do next we're going to Define our llm

we're not going to give it we're going

to give it a temperature of zero so it's

more predictable and not

as um creative I guess we're going to

use gbt 40 because that's their newest

fastest model which can give us text

back super quickly and also process our

images um and then we're going to bind

tools to this model so all of the the

model has access to all the tools we've

defined next we're going to use the line

chain expression language anguage to

pipe our prompt to our language model

and create a chain and then we're going

to invoke our chain passing in our input

from the user input and the chat history

and then also our config object once

this is finished invoking we're going to

check to see if any tool calls were on

the model or if the model uses any tool

calls and if they are we're going to

return this tool call Value which will

populate this

field if the model did not use any tool

calls we're just going to return the

content now that we've defined our first

node we're going to want to Define our

conditional Edge which will always be

called after this

node invoke tools or return this takes

in the state and it essentially says if

tool calls are defined then you want to

call the invoke tools node next and if

it's not defined but the result field is

defined then we're going to end because

that's just the string that was returned

and then this should never happen but if

for some reason neither of these are

defined um then it's going to throw an

error but we're never going to get this

so that should not cause an issue for us

finally we're going to Define our last

node invoke

tools this takes in the same input

arguments as the invoke model our state

and our config um and it's going to

first check to make sure there's a tool

call once again this should never happen

because it should only call invoke tools

if tool calls are defined but because of

typescript we need to add this here but

we should never see this error next

we're going to Define our tools map

which is going to be a map containing

key value pairs each each key is going

to be the name of the tool and then the

value is going to be the actual tool

using this map and our tool input we're

going to try and find the tool in there

once again this should never happen

because our language model especially if

you're using a um state-of-the-art

language model it should never pick a

tool which doesn't exist you know that

you didn't provide to it um but we have

this here once again for typescript once

we have our selected tool we're going to

then invoke that tool passing the

parameters that the tool called for that

the language will provided to us in our

config and then finally once we get a

result back this result will always be a

string but usually we're returning an

objects we're going to parse that result

and then return it in our tool result

value which will populate this

field now that we've done this we've

implemented our entire agent and we can

go and Implement our tools which we will

then provide to the agent these tools

are going to contain all the logic

around streaming back UI components and

hitting any external

apis so for our tools we have this tools

folder which contains some files uh get

a repo weather and invoice we're just

going to implement the get a repo tool

and then I'll quickly walk through the

other tools because they're all pretty

redundant and contain kind of the same

logic uh but for for our GitHub repo

tool we're going to want to add our

Imports first we're going to be using

Zod Zod is what we're going to use to

define the schema so language model

knows what parameters to pass or extract

from the input and then pass to our tool

um ocit which is the GitHub API wrapper

SDK which is is what we're going to use

to actually call the giab API create

runnable UI which we defined in our

server. TSX file which is going to wrap

that Lambda create a new streamable UI

and that's what we're going to use to

actually stream back these UI components

our tool from Lane chain and then our

pre-built components which we can

quickly look at we have a loading

component which just contains some

skeletons to show that we're loading and

then the actual component which contains

a card um this card is going to have a

link to the GitHub repo it's going to

show how many stars they have the repo

description and other things like that

which we'll get back from the GitHub

API the first thing we're going to want

to do is Define our schema our schema is

can to be the owner in the repo which is

the which are the fields that the giab

API requires in order to fetch details

about a repository um if the language

model sees that a user is submitted an

input with an owner and repo fields that

look like a get of repo it'll likely

call this tool and provide us the name

of the repository and the repository

owner next we're going to Define our fun

which we'll actually call the giab API

we're going to call it the giab revo

tool the input is going to be the type

of our Zod schema so z. infer type of

our schema and that will infer the type

here next we're going to make sure you

you have your GitHub API token in your

environment if it's not we're going to

throw an airror obviously if you're

going to use this tool you should set

that in the read me of this repo I add

instructions on how to get all the API

Keys you need for free for the different

tools um except for your language model

API key which will obviously cost money

uh when you invoke the language model

we're then going to instantiate our octo

kit SDK passing in our GitHub token and

that's going to return an instance of

the GitHub client then we can just call

our GitHub client calling the repos um

with a get request and that's going to

get us the information on this repo and

then we return the data from the um from

the API response which is the input so

we have the owner and the repo and then

the get

repo description how many stars they

have and the primary programming

language if there's an error we're just

going to return a string and then we'll

process this inside of our tool to

return either the GitHub final component

or an error component if an error was

occurred now we can Implement our tool

it's going to be a dynamic structured

tool with a name GitHub repo a

description tool to fetch details of

GitHub repository um we're going to pass

in our schema so language model knows

what fields to provide and then we have

our function inside this function we see

it takes two arguments input which

should be the schema we defined and then

config we're going to first create a new

runnable UI stream passing in

our initial value which is going to be

this loading component and this is going

to tell the user that as soon as the

language model picks this tool and this

function is invoked it's going to the

user is instantly going to get back

their

first loading component so they see that

we're working on something that's this

step here next we're going to hit our GI

of API with this function we defined

above passing in our input then if they

get of API return to string there's an

error and we're just going to return um

a P tag with the error message calling

stream DOD if the type was not a string

so this object here then we're going to

return our GitHub component which is

that jsx component which will actually

show all the data passing our data and

finally we're going to return the result

of all result of our tool you can also

call stream. update or append as as many

times you would like if you want to say

hit an API update your tool hit another

API add some more values to that UI

element you can really call update um

and append as many times you would like

to keep interacting or updating that

interactable UI component with the user

the nice thing about this as well is

since it takes in a react node our

GitHub component these don't have state

however they could be stateful they

could contain some button which hits an

API or makes another language model call

which then updates the UI again and

they're really just generic react

components so everything that you could

do before with react components and make

them super Dynamic and interactable you

can do that here as well because it's

any sort of react component you want and

you can pass props to it so they can

take in these different inputs and be

dynamic and customizable for the user so

now that we've imped these tools we can

quickly look at our invoice and weather

tool invoice is just a schema this is

because we're going to want the language

model to extract these fields from any

uploaded image and then it just creates

a own UI with this initial loading which

is kind of redundant because then it

instantly turns around and um updates it

with the final component but just to

show the same thing and then it Returns

the input and then for the weather

component or for the weather tool same

thing our weather schema is a city and

state and then an optional country which

which it defaults to USA and then it

hits a couple apis if you want this API

key I've added some instructions in the

REM on how to get it it's totally free

um it's going to get the uh longitude

and latitude from this API and then it's

going to use the weather.gov API which

is free passing in these values and then

it's going to extract the current

weather for your location the tool also

the same thing creates a stream passes

back that loading weather component then

it invokes our weather data function to

actually get the weather data from these

apis and finally it updates the weather

component with our um with the data that

the API returns

finally we're going to want to implement

our chat jsx component which is going to

be the chat you can interact with if you

want to follow along you should go to

pre component prebuilt chat this going

to be a client component because it

doesn't is not actually uh using

anything in the server instead it's

going to call our server component we're

going to have a state input which are

just some sh Shad CN components that

they built uh endpoints endpoints

context which we already defined

here or sorry we need to Define this

after this um our use actions which we

saw how that was defined that's going to

provide the agent act action for us and

then our context which is going to

provide context to our action and also

um render our UI

elements these are just some util

functions on converting files to base 64

we need to convert them to a Bas 64

string on the client because react

server components don't allow for any

arbitrary function fun to be passed over

the or sorry any arbitrary object to be

passed over the wire only specific

objects and obviously uh key value pairs

where they're both strings and we would

need to convert it to Bas 64 in the

server anyway so we just do it on the

client U because they don't allow for

file objects to get passed over the wire

so it's now string and we send it over

and then we use it to invoke our

language

model for our chat function we're going

to want to add our state variables first

our use actions hook providing our

endpoint cont text which is our agent

takes in our inputs which we also see

here this is how we're going to then

invoke our agent we then have a few

State variables Elements which is a list

of jsx elements these are going to be

the elements that the UI returns to be

rendered uh your chat history chat

history input and then a selected file

you've selected here we see we're

wrapping our elements with this local

context. provider um context jsx uh

element and that's going to provide

context to our react elements

and then we have a simple form as we

would see right here just for submitting

your inputs and any uh images you might

upload next we're going to want to

implement our on submit function which

is going to actually call our

agent so this on submit function does a

few things we're going to paste it in

and then walk through it so first it

makes a copy of our elements array so in

case this somehow gets updated later on

it's not going to mix and match them so

it's always going to be the same at the

beginning of the function it's then

going to convert your file to base 64

string format if you did upload one and

then it's going to use our action our

agent action to invoke our agent this is

going to return an element which is UI

and last event we saw that here it's

essentially calling this function which

returns our UI and last

event passes any necessary inputs and

then updates our element array with the

UI value from our return value from our

agent and then also so the human message

that the User submitted and then if they

uploaded a file the

file finally we saw in our server

function the first function we

implemented was was this with resolvers

we're going to use that here so element.

last event which once again is

the last event here from with from our

with resolvers

function it's then going to check and

see if it's an object this right here is

specific to this chatbot we've

implemented so you're obviously going to

want to update these to reflect your L

graph node but we have or nodes we have

invoke model and invoke tools as we saw

we defined in our graph so if it's an

object then we're going to want to see

if last event. invoke model. result is

true that would be this plain text

string if it is then we update the

history with that plain text and if it's

not true that means that tool was used

so we see invoke tool

and then we update our history um but

make the assistant message be this tool

result and then the result of our tools

and that's so the the assistant knows

it's it's successfully completed the

request we made in the past using the

result of our tool API um and that's so

you can say something like uh you know

what's the details on this GI up repo it

sends the response it didn't actually

give you any text there wouldn't be any

text in the chat history but since we

are creating this assistant message when

you send a followup it's able to see

your question and see that it was

submitted or resolved and in those it

should ignore any questions in your chat

history because it's already resolved

them here finally we clean up by setting

our elements State value and then

clearing any

inputs finally we can go and Implement

our agent wrapper which is going to uh

use this stream runnable UI function and

call our graph agent executor function

um and and that's what we're going to

use to invoke our agents let's go

Implement that now so if you're

following along you should go to app SL

agent and you're going to want to paste

in our Imports once again server only

because we're going from this client

component to our server so we want to

make sure it's only invoked on the

server we're then going to import our

agent executor from our graph which we

built a little while ago and our expose

endpoints and stream runal UI from our

server file which we invoked in the

first part finally we're going to want

to import our AI message and human

message uh Lane chain message types

we're going to use this when we're

constructing our chat history to give

the proper message types to our um chat

history and this will also when we look

at the Lang graph Trace you'll be able

to see human message AI message human

message AI message um in the proper

types uh so the language W knows what or

who said

what now we can implement this little

util function for converting our chat

history type see from chat roll and

content into a proper list of human

message and AI message so iterate over

if it was a human then return a human

message or is thiser AI return an AI

message and then by default we're

returning a human message this could

also be say chat message from linkchain

core and this would just be a generic

um we also need a roll so you could say

you

know roll um and this would just be a

generic chat message and it wouldn't be

specifically human or AI but in our case

this will probably never happen because

we know we're always adding this role

and assistant but we're adding it just

to make our switch case

happy next since we're doing image um

inputs we need this process file

function this is essentially going to

process any files you upload and convert

them to the proper message type for

passing multimodal inputs to our

language model so if file is defined

it's going to create a new human message

um passing into content which is a list

of Type image URL and then image URL

with our base

64 uh conversion of our file and this is

the right or this is the proper uh

template format for uploading multimodal

types to language models I'll add a link

in the description to our multimodel

how-to guide in the JS documentation if

you're interested in that and then

finally we return our input and our chat

history which matches as we saw the

input in chat history from our Lan graph

agent if you didn't upload a file then

we just need input in chat history and

don't need to do anything with image

prompt

templates now we need to implement our

agent function this is going to be the

function that when you

call where is it actions. agent um this

is going to be the function that's going

to that it's going to invoke so it's

going to take in as we see the same

inputs we were passing in here input

chat history and file

um use server so we make sure that this

is always executed on the server as in a

react server component we're then going

to this does not need to be a

sync um we're then going to process the

file to get the process inputs which is

input in chat history that's going to do

what we saw up here and then finally

it's going to return a new stream

runnable UI passing in our Asian

executor which is our Lang graph agent

and any inputs um we saw this already

but this is just going to call stream

events on our runable which we pass into

it which is our agent executor function

and the last thing we need to do is

actually create uh expose the context

for this agent so that our actions.

agent um and with our use actions hook

has the proper context available to

invoke

this so as we see here there's an error

because exposed context does not exist

yet but once we expose our context

passing in our agent function and Export

this the goes away and as we

see with our actions. aent it has access

to our agent function with the same

inputs we just defined here we see we're

passing expose uh endpoints this agent

function with the same inputs and this

just gives the proper context to um our

CL C client component so it knows that

it can invoke this agent now that we've

implemented all this we can go and

actually demo our application

restart

go now that we've implemented all this

we can go and start our Dev server and

actually check out our demo so we're

going to want to navigate to your

terminal go into the proper directory

and run yarn Dev or you can build it run

yarn start kind of the same once it's

running we're going to go to Local Host

3000 and we will see our website so

gener VII with L chain we see our chat

bot and our inputs um we can say

something like what's the weather in

SF it then streamed back our text if we

go back here we can see that the

language model decided did not have

enough um inputs from the user to select

the weather tool or any other any of the

other tools so it just sent back some

text but since we've implemented chat

history can it it said can you please

specify the state so we can just say

California and it's going to use our

chat history and our current message to

then invoke the weather tool and we see

right there there's a loading component

because it picked the weather tool sent

us back our loading component and then

went and actually hit the weather apis

and then updated this component to show

the actual weather we can also say

something like what's the deal with my

invoice and we can upload an image let's

say we upload our receipt we submit that

and this is going to use GPD 40's

multimodal capability to read our image

and then send us back this nice fully

interactable if you implemented this

component it wasn't just a demo um uh

receipt component or invoice component

um and this is all populated with the

receipt image I uploaded so it extracted

the fields and then passed in the proper

properties to this component so it could

render and then finally we can check out

our GitHub component which we

implemented so what the info on

Lang chain AI SL Lang graph and our

language model is going to be able to

recognize that this is a in a get up

repo so we submit that and then error

doing that so I wonder if I spelled

something wrong yes I spelled Lang graph

wrong try Lang chain AI

SL Lang

graph so as we saw there it gave us our

loading component but then the get of

API returned an error so it just

responded with this string when I did it

properly it gave us the proper component

and then obviously it's fully

interactable so you can click on this

and it'll bring you to the L graph

repository that's it for this video um

if you're interested in the python video

it's going to come out tomorrow or if

it's already released then we will link

in the description where we will

implement this exact same uh chatbot

demo website but with a full python

backend I will see you all in the next

video