LlamaIndex Webinar: Build an Open-Source Coding Assistant with OpenDevin

Summary

今回お届けするのは、オープンソースプロジェクト「オープンデオン」についてのウェビナーシリーズです。デオンは自律型AIソフトウェアエンジニアのオープンソース版を目指しているプロジェクトで、コミュニティが共同で開発しています。重要な貢献者であるRobertが、デモンストレーションと議論を行い、開発者が自律ソフトウェアエンジニアを安全に実行し、監視するためのツールを提供する取り組みを紹介します。また、彼らはソフトウェア開発をより創造的で魅力的なタスクに変えることを目指しており、MITライセンスで公開されています。現在2ヶ月で116人の貢献者と24,000以上のスターを獲得しており、短期間で大きな進歩を遂げています。

Takeaways

• 😀 オープンデオンは完全にオープンソースの自律型AIソフトウェアエンジニアを構築するプロジェクトです。
• 😀 自律型ソフトウェアエージェントがコードを書いたり、ソフトウェアプロジェクト全体を構築することに焦点を当てています。
• 😀 コードベース内のファイルの編集、テストの実行、デバッグ、修正などのタスクを自律的に実行できます。
• 😀 現在、116人のユニークな貢献者が参加しており、700以上のプルリクエストがマージされています。
• 😀 ChatGPTやGitHub Copilotなどのツールと比較して、エージェントは既存のコードベースと連携し、開発者のワークフローに直接組み込むことができます。
• 😀 エージェントは大規模なコードベース内で適切なファイルを見つけて編集することができます。
• 😀 自動化されたソフトウェア開発エージェントが、コードの書き換えやテストの実行、コードの修正などを行うことができます。
• 😀 プロジェクトの透明性と安全性を確保するために、Sandbox内でエージェントが実行されます。
• 😀 エージェントの能力を評価するために、SweBenchライトベンチマークを使用してパフォーマンスを測定しています。
• 😀 オープンデオンのインターフェースは、チャットウィンドウ、ターミナル、コードエディタ、ウェブブラウザを備えており、開発者がエージェントとやり取りするための統合環境を提供します。

Q & A

オープンデヴォンとは何ですか？

オープンデヴォンは、自律的なAIソフトウェアエンジニアのオープンソースプロジェクトです。MITライセンスで提供されており、コミュニティが完全に理解し、開発に参加できるようになっています。

オープンデヴォンが目指すものは何ですか？

オープンデヴォンは、ソフトウェア開発エージェントを構築し、運用するためのオープンソースプラットフォームを提供することを目指しています。具体的には、AIと連携してソフトウェアを構築するツールを提供し、開発者が自律的にコードを生成し、デバッグする手助けをします。

オープンデヴォンの主な特徴は何ですか？

オープンデヴォンの主な特徴は、エージェントが既存のコードベースと上手く連携し、開発者のワークフローに直接組み込まれ、デバッグと修正のループを実行できることです。また、Webブラウザやコマンドライン、ファイルシステムへのアクセスを持ち、幅広いタスクを自律的に実行できます。

オープンデヴォンのエージェントはどのように動作しますか？

オープンデヴォンのエージェントは、LLM（大規模言語モデル）を中心に、ユーザーからのタスクを受け取り、外部データやコードベース、ランタイム環境を利用してタスクを実行します。エージェントは繰り返しのループを通じて、プロンプトを生成し、アクションを実行し、観察結果を元に次のステップを決定します。

エージェントの性能はどのように評価されますか？

エージェントの性能は、SWEベンチという評価基準に基づいて評価されます。これは、GitHubの実際の問題を解決する能力をテストするもので、問題を修正するためのPRとその結果を評価します。

現在のオープンデヴォンのエージェントの性能はどの程度ですか？

現在のオープンデヴォンのエージェントは、SWEベンチライトの評価で約21%の問題を解決できる性能を持っています。これは、既存の最高性能のエージェントを上回る結果です。

オープンデヴォンのエージェントはどのように開発者のワークフローに統合されますか？

オープンデヴォンのエージェントは、WebインターフェースやVSコードプラグイン、コマンドラインなど、さまざまな開発環境に統合できます。また、GitHubのPRや課題のコメントにエージェントを直接組み込むことも可能です。

オープンデヴォンプロジェクトに参加する方法は？

オープンデヴォンプロジェクトに参加するには、GitHubリポジトリでプルリクエストを作成するか、イシューを作成して提案やフィードバックを提供することができます。また、既存のオープンイシューに取り組むことも歓迎されています。

今後のオープンデヴォンの開発計画は？

今後の開発計画には、エージェントの性能向上、新しいエージェントの追加、より汎用的なアーキテクチャの構築などが含まれています。また、さまざまな開発環境でエージェントを使用できるようにするためのインターフェースの改良も進められています。

オープンデヴォンを使用するための最小限のコンピュータリソースは？

オープンデヴォン自体はそれほど多くのコンピュータリソースを必要としませんが、主要な計算は外部の大規模言語モデル（LLM）に依存しています。GPT-4やClaudeなどの強力なLLMを使用することで、エージェントの性能が大幅に向上します。

Outlines

00:00

🤖オープンソースプロジェクトopen Devonの紹介

この段落では、ウェビナーで紹介されたオープンソースプロジェクトopen Devonについて説明しています。open Devonは、自律型AIソフトウェアエンジニアのオープンソースバージョンを構築することを目指しているプロジェクトです。コミュニティが共同で開発しており、MITライセンスで公開されています。プロジェクトはソフトウェア開発をより創造的で魅力的なタスクに変えることを目指しており、ボランティアによって進められています。

05:00

🔧 open Devonのビジョンとその機能

open Devonのビジョンは、ソフトウェア開発エージェントの構築と実行のためのプラットフォームを提供することです。AIとLLMsを活用してソフトウェア開発に関わるツールを提供し、研究者や開発者、そして開発エージェントを利用するソフトウェア開発者向けに使いやすいツールを目指しています。また、プロジェクトは非常に早く進んでおり、GitHub上で多くのスターとコントリビューターを獲得しています。

10:01

🛠️ エージェントの役割と機能

この段落では、エージェントが持つべき機能とその重要性について説明しています。エージェントは既存のコードベースと調和し、開発者のワークフローに組み込まれることで生産性を高めるとされています。また、デバッグやテスト実行などの反復プロセスにも活用でき、大規模なタスクを小さなステップに分割して遂行することができると強調されています。

15:03

🔬 エージェントの動作とフレームワーク

エージェントの動作について説明しており、ユーザーからのタスクを受け取り、外部データやLLM、ランタイムを活用してタスクを遂行します。また、マイクロエージェントフレームワークについても紹介されており、これは複雑なロジックを簡素化し、マークダウンベースでエージェントを作成できるようにするものです。

20:05

📊 エージェントの評価とベンチマーク

ソフトウェア開発に関するエージェントの品質を評価する基準として、sbenchというベンチマークが紹介されています。これはGitHub上で発生した現実の問題を解決する能力をテストするもので、open Devonはこのベンチマークで高い点を獲得しています。

25:06

🌐 open Devonのインターフェースとデモ

open Devonのユーザーインターフェースについて説明されており、チャットウィンドウ、ターミナル、コードエディタ、ウェブブラウザなどからなる開発環境が用意されています。さらに、デモとして、open Devonがコードベース内での問題を特定し修正する様子が紹介されています。

30:06

👥 コミュニティと貢献の呼びかけ

最後に、コミュニティへの参加と貢献を促しています。open Devonは分散型で運営されており、誰もがプロジェクトに貢献できるようになっています。ウェビナーでは、参加者がプロジェクトに積極的に参加し、新しいアイデアや機能を提供するよう呼びかけています。

Keywords

💡オープンソース

オープンソースとは、ソフトウェアのソースコードを公開し、誰でも利用・修正・配布できるようにするライセンスのことです。このビデオでは、オープンデヴォンがMITライセンスのオープンソースプロジェクトであることが強調されています。オープンソースコミュニティがどのようにプロジェクトを支えているか、またその透明性についても述べられています。

💡オートノマスAI

オートノマスAIとは、自律的に動作し、タスクを遂行する人工知能のことを指します。ビデオでは、オープンデヴォンが自律的なソフトウェアエンジニアとして機能することを目指していると説明されています。このAIは、コードの生成やテスト、修正などを自動的に行うことができるとされています。

💡エージェント

エージェントとは、特定のタスクを遂行するために設計されたソフトウェアプログラムのことです。ビデオでは、オープンデヴォンのエージェントがコードを生成し、テストし、修正する能力について述べられています。エージェントは、ユーザーの指示に基づいて行動し、ソフトウェア開発プロセスを支援します。

💡コンテキストウィンドウ管理

コンテキストウィンドウ管理とは、AIがタスクを実行するために必要な情報を適切に保持し、利用するための方法です。ビデオでは、オープンデヴォンのエージェントがコードの編集やテスト結果などの情報を管理し、次のアクションを決定するために利用すると説明されています。

💡スイベンチ

スイベンチ（SWE-Bench）は、AIエージェントの性能を測定するためのベンチマークテストです。ビデオでは、オープンデヴォンがこのベンチマークで21％のスコアを達成し、他のエージェントよりも高性能であると述べられています。

💡マイクロエージェント

マイクロエージェントとは、小規模で特定のタスクに特化したエージェントのことです。ビデオでは、オープンデヴォンがマイクロエージェントを使用して、特定のタスクを効率的に処理する方法について説明されています。例えば、コミットメッセージの生成やGitHubリポジトリの調査などに利用されています。

💡GPT-4

GPT-4は、オープンAIが開発した大規模言語モデルの一つです。ビデオでは、オープンデヴォンのエージェントがGPT-4を利用してコード生成やエラー修正を行うと述べられています。高性能な言語モデルを使用することで、エージェントの精度と効率が向上します。

💡コミュニティ駆動

コミュニティ駆動とは、プロジェクトが多くのボランティアやユーザーの貢献によって進行することを指します。ビデオでは、オープンデヴォンが完全にコミュニティ駆動であり、多くの貢献者が自発的に参加していることが強調されています。

💡開発者ワークフロー

開発者ワークフローとは、ソフトウェア開発における一連の作業手順やプロセスのことです。ビデオでは、オープンデヴォンが開発者のワークフローに直接統合され、開発プロセスを効率化する方法について述べられています。特に、コード編集やテストの自動化によって生産性が向上するとされています。

💡ベクトルデータベース

ベクトルデータベースは、データをベクトル形式で保存し、効率的な検索と類似性の評価を可能にするデータベースです。ビデオでは、オープンデヴォンがコードベースをベクトルデータベースにインデックス化して、関連するコードのスニペットを動的に取得する方法について述べられています。

Highlights

オープンデヴォンは完全にオープンソースの自律型AIソフトウェアエンジニアのプロジェクトであり、コミュニティが完全に理解し、構築できることが特徴です。

このプロジェクトは、ソフトウェア開発エージェントの構築と実行をサポートするオープンプラットフォームを提供することを目指しています。

オープンデヴォンはMITライセンスのもとで運営されており、全ての作業はコミュニティによって行われています。

このプロジェクトには116人のユニークな貢献者が参加し、700以上のPRがマージされ、GitHubで24,000以上のスターを獲得しています。

エージェントは既存のコードベースと上手く連携し、開発者のワークフローに直接組み込まれるため、効率的なデバッグやテストが可能です。

エージェントは長期的な記憶を持たず、現在のコンテキストウィンドウと事前学習された知識を基に動作します。

マイクロエージェントは特定のタスクを小さな単位で処理するために設計されており、テンプレート化されたマークダウンで定義されます。

SWEベンチライトベンチマークでは、最先端のエージェントが15〜20％の成功率を示し、オープンデヴォンの最新の評価では21％に達しました。

オープンデヴォンのユーザーインターフェースは、チャットウィンドウ、ターミナル、コードエディター、ウェブブラウザー、Jupyterノートブックを含み、様々な開発環境に対応しています。

オープンデヴォンは、エージェントが安全に動作し、何をしているのか透明性を持たせるためのツールを提供しています。

エージェントの性能向上には、ブラウザーエージェントの統合や、コードベース全体をベクトルデータベースにインデックス化することが含まれます。

オープンデヴォンは、様々なインターフェースでエージェントが動作できるように設計されており、コマンドラインやGitHubプラグインとの統合も予定されています。

エージェントの構築に関しては、全ての基盤インフラを提供し、エージェントの評価を容易にするパイプラインを提供しています。

オープンデヴォンは、エージェントの開発とユーザーのワークフローにおけるエージェントの統合を推進しており、将来的には開発者の生産性を大幅に向上させることを目指しています。

オープンデヴォンは、エージェントの強化とプラットフォームの拡張を通じて、より多くのソフトウェア開発タスクを自動化し、開発者の負担を軽減することを目指しています。

Transcripts

hey everyone uh welcome to another

episode of The Llama index webinar

series uh today we have a special guest

with us and we're excited to feature

open Devon um and so open Devon is an

open source project uh that you know is

trying to build uh an open source

version of Devon the autonomous AI

software engineer um so it's very

exciting because you know this is

actually completely open source and so

therefore the community is actually able

to understand what it takes to actually

build a fully autonomous coding

assistant obviously there's a lot of

interest in agents these days especially

around assistants that can autonomously

you know code like do tasks for you and

then actually build an entire software

project um and so excited to feature

Robert one of the core contributors to

open open Deon um who will be walking us

through it seems like a few slides as

well as a demo um and we'll have a

pretty fun discussion on you know agents

how to build them as well as the future

and what's coming next so without

further Ado I'll pass it to Robert

awesome thanks a bu Jerry I'm super

excited to be here uh and you know thank

you to the to the Llama index team as a

whole for uh for having

us um so yeah like Jerry said you know

we are we're building open Devon it's a

it's an MIT licensed open source project

uh we were really inspired when we saw

the first uh demo of De Devon um and uh

you know a community formed very quickly

uh in hopes of basically

recreating uh you know what we saw in

that demo because it was so exciting and

so so promising uh for us as Engineers

to to see just how powerful that tool

was

um so you know over the last two months

or so as we've been building that that

Vision has uh has deepened a bit um we

we consider ourselves an open source

platform for both building and running

software development agents so basically

anything that's that's interfacing with

AI with llms to build software uh we

want to be able to provide uh you know a

set of uh tools that uh academics and

agent Builders can leverage in order to

uh see their agents kind of come to life

actually work on software things like

that and then we also want to provide

agent users so you know your average

software developer who wants to run an

autonomous software engineer the tools

that they need to run those agents

safely uh to understand what they're

doing to have some transparency into

what the agent's doing how it's running

you know where it might need to be

redirected things like

that uh so big big emphasis on open you

know like I said this is an MIT licensed

project uh we're totally Community

Driven right now uh everybody who's

working on the project is is a volunteer

uh you know working working out in the

open um you know we have a bunch of

academic folks uh we have a bunch of

average developers like myself folks who

uh maybe aren't as uh as deeply uh uh

into the academic side of agent building

but are very interested in uh software

development application development as

well as you know just a bunch of end

users who you know may or may not be

software Engineers themselves but who

are interested in using autonomous

software agents to uh help build

software uh and all of us are working

together really to try and figure out uh

you know what is the best uh end user

experience that we can drive for here

and what is the best way we can get

there

technologically um and all of us really

have this goal of making software making

the development of software a more

creative engaging task you know today

there's a lot of um you know just slog

involved in uh in you know getting

something done when writing software uh

and being able to uh push away all the

parts that are kind of annoying and rote

to an autonomous engineer and really be

able to focus uh as humans on the

creative part of the task on you know

what is what does the end user want what

do I want this software to be like and

do uh if I can focus on that aspect of

things that makes the job of writing

software a lot more fun

so that's really what we're what we're

aiming for and and again we want to do

this all all in the open uh we believe

that software development is generally

uh uh lends itself to being done in the

open in an open source way and so we're

very excited about uh the open source

Community

here just some kind of quick stats on

the project uh we're very early on we're

only about two months old uh we already

have 116 unique contributors to the

project uh We've merged over 700 PRS

uh we have over 24,000 stars on GitHub

um this is a huge amount of uh of

progress for an open source project

that's this young um and uh you know

even even Beyond these numbers just the

the quality of the application that

we've been able to build and uh the the

uh quality of the agent that we have

running in there which we'll talk a

little bit about later on thanks to some

academic contributions uh really blown

me away I uh I would not have expected

to see an open source project be able to

move this quickly

and build this much um and it's been

really really impressive to see and

really exciting to just be a part

of well so you know why why are we

building this can't can't chat GPT

already write code isn't this already

possible and and that's definitely true

chat GPT can can write bits of code uh

it can even you know inside of chat GPT

it can run them to check them uh there's

there's a pretty good uh chat gbt is

actually exceptionally good as well as

the other large LS are exceptionally

good at writing code

um we also have a GitHub co-pilot uh

which hopefully you all are familiar

with this is probably you know within my

workflow the thing that has given me the

biggest productivity boost uh is having

this uh llm driven code completion right

there at my fingertips inside of my

development workflow um I I absolutely

love co-pilot it's a huge uh kind of

design inspiration for uh our long-term

Vision around open Devon uh this idea of

something that really integrates

directly into the developer workflow in

a way that's uh invisible when it's not

necessary but is always kind of there

when you when you need it when it can be

helpful and it's just you know you hit

the Tab Key and you know you're just

moving that much faster um I'm a huge

huge fan of

co-pilot uh but chat GPT and uh you know

just a raw llm and co-pilot they don't

they don't really uh you know satisfy

every need here there's some stuff that

that agents can do which make this much

much

better uh so agents play nicely with

existing code BAS

um you know the the tough thing about

say working with uh you know if you're

just like working directly inside of

chat GPT uh it's really great at writing

Green Field code if you're just like hey

I need an algorithm that does X uh it'll

write that function for you but if

you're like hey I have this 100,000 line

codebase and uh I need you to find the

right file to edit within that codebase

uh it's not really capable of doing that

you can't really just like dump your

whole codebase into the context window

and have it try and pick out the right

spot it's probably too big of a codebase

for that context window um and it's

going to cost you a lot of money to do

that um agents can also embed directly

into the developer workflow kind of

similar to what we saw with with

co-pilot uh I don't know if you all have

had this experience but very frequently

I have uh when writing code I will uh

drop into chat GPT to try and you know

write some function uh and then I copy

the contents out of chat GPT into my uh

code editor and then I run it and then I

paste an error message back into chat

GPT and there's this like really

annoying copy paste going back and forth

and uh with an agent it can really just

embed into your existing system it can

write code directly to your file system

it can run the test for you uh so it's

really it's a really neat way to kind of

embed into the developer

Loop uh it can run in this in this debug

fix Loop which is something that like

co-pilot doesn't really do well um right

it can it can edit the code uh it can

run tests it can you know run the code

itself to see what the result is uh you

know notice any problems and then fix

itself uh and really it it works like a

human engineer it can have access to a

web browser to look at

documentation um you know it's it's

instead of doing this like kind of

step-by-step process that you might do

with chat GPT where you're picking off

these small little bite-sized tasks you

can feed an agent a very large unbounded

task and it can break it down into

manageable steps and make several back

and forth trips to the large language

model to achieve those you know more

bite-sized tasks

uh so overall you how do how do agents

work um I see them as kind of a hub

between several different uh uh data

sources and um interaction points um so

at the top here you have you have the

user who's you know ultimately you know

driving driving the agent deciding you

know what do I want this agent to work

on uh so they'll send the agent a task

like you know please fix the test to my

codebase please add this new feature

whatever it is the agent can also pull

in external data U so that might be uh

you know code from your code base it

might be a data from an external uh like

knowledge base something like that uh

could be external documentation uh un

likee you know the API docs for a

particular API that's that's

out um it has access to a large language

model and this is you know really the

core uh you know it's kind of like the

CPU for the agent right it's it's what's

uh kind of driving the atomic

instructions that the that the agent you

know the atomic operations the agent is

taking all goes through the

llm uh and then it has access to some

kind of runtime right it has access to a

web browser to a command line to a file

system where it can actually uh you know

do the things that the llm is telling it

to

do here's here's kind of some pseudo

code for you know what this what this

Loop looks like um so it's it's

basically a you know a loop that's built

on top of the llm um so basically we

take the current state of things uh

which at the is probably just you know

whatever the prompt the user gave um we

send some kind of prompt to the llm and

this is kind of where uh the the core um

specifics of a particular agent live is

is within this you know how do we

generate this

prompt uh we take the response from the

llm and come up with some action to take

uh we take that action in the runtime

somehow uh and come up with an

observation so that might be the

contents of a file that might be uh the

HTML output or a screenshot of a page it

might be uh the result of running a

command on the command

line and then we update our state with

that

observation um so we basically add add

you know the output of that command or

the contents that file to the state and

then we loop back to the beginning we

generate a new prompt to send of the

LM given the output of that most recent

command or the results of that most

recent action the LM can take one step

forward to generate a new

action uh at the end of the day it's

it's really about context window

management right so the llm has uh

basically no no long-term memory it

knows exactly what you're putting into

the context window right now it has its

World Knowledge that it's that it's

learned in training uh and it has to use

those two things to figure out how to

advance uh Advance one step forward

towards our goal um and so you as the as

the agent Builder basically need to

figure out how do I intelligently based

on you know what what the LM has done so

far or what the agent has done so far uh

and based on what I know about the code

base but I know about the user's

intention what do I stick into the

context window to move one step closer

towards the goal uh that could be Co

code it could be command output it could

be a history of the actions that the

agent has taken it could be a working

plan that the agent came up with at the

beginning of the process uh could be any

number of things that you pull into that

context window this is really where the

like uh all the all the work in uh

designing an agent goes into is you know

at each turn of that Loop what am I

going to stick into the context window

for the llm in order to drive things

forward One

Step uh to kind of help with this uh

with this you know building agents and

um you know help with this Loop uh what

we've done is is um we tried to abstract

away as much of this as possible into a

framework we that we call

microagents um our our best agents are

not microagents to be clear our best

agents are written in Python they

involve some complex Logic for moving

from state to state as they progress to

towards their task um but microagents

are a very powerful way to take on nice

little bite-sized tasks um in a way that

uh kind of drives that that Loop that we

just talked about of uh you know make a

prompt take an action and then make a

new prompt and so you can see here uh

this is a micro agent that is actually

part of the open Devon system today uh

for generating a good commit message

based on what's in the git staging area

um so you can see we have this long form

this long kind of prompt uh telling the

agent you know what its job is um you

know it's a software engineer its goal

is to write a good commit message here's

how it's going to find the code changes

to describe uh and then you can see

we've templated out a bunch of kind of

the boiler plate that goes into uh

creating an agent um so at each turn of

the cycle the agent is going to get some

instructions on uh you know what um what

its history is is uh you know how to

interpret this history Json that it's

going to get and it's going to get a

history of the last 10 things that it's

done as well as the observations of

those things so that might be you know

it ran get diff and this was the result

of the get diff command it uh opened up

a particular file to read the contents

and uh this is what it saw inside that

file uh so it's going to see you know

step by step what are the last 10 things

that it's done uh and then it has three

actions that are available to it it can

run commands it can reject the task and

say you know there's nothing in the get

staging area or there's you know I'm not

in a get repo I can't do this or I can

finish and say you know here's what the

messages and then it gets some

instructions on uh basically telling it

you have to respond to Json format uh

the Json has to be in this structure it

has to represent an action um and so

basically you know this this these

templates take care of a lot of kind of

the boo plate of uh writing an agent um

and this uh this will basically this

prompt will run in a loop against the

llm uh in order to drive towards this

task of uh creating a commit message

well so once once we you've got an agent

built uh a really good question is how

good is that agent um so a lot of work

has been done in Academia to measure uh

agent quality basically how how well

does an agent uh solve uh tasks uh

related to software

engineering uh the biggest Benchmark

here is called sbench it came out of

Princeton uh and basically tests the

ability of an agent to solve real world

issues on GitHub um so these folks uh

pulled you know over 2,000 issue issue

PR pairs from popular python

repositories and basically what they did

was they looked for PRS where the pr uh

added unit tests uh and added some code

changes to address a particular issue

that was available on on the repo um and

so this makes it really easy to verify

that the agent is doing the right thing

basically you um you clone the repo at

the place where it was before the issue

was fixed you add the unit tests that

were added within that PR uh you run the

test once they fail predictably uh and

then you tell the agent hey go and uh

address the issue uh you know as it's

written you know from GitHub like that's

that's the prompt that's fed to the

agent and then the agent is allowed to

do its thing it's allowed to edit code

it's allowed to browse the web Etc it

makes a set of changes uh we apply those

changes to the repo uh we run the test

to see if they pass and if they pass you

get a thumbs up if they don't pass you

get a thumbs down uh and what we see is

that uh on on uh Alo so to to continue

forward a little bit there the sbench

folks also released a subset of those

issues uh of 300 issues that are a

little bit more self-contained a little

bit easier for agents to tackle uh and

uh you know it's a bit smaller of a set

this is actually a very expensive

evaluation to run on say GPT 4 uh cost

several thousand dollars you know

roughly speaking depending on your agent

and how it behaves uh to run a full

evaluation so swe bench light uh for for

for our eval has cost you know roughly

$500 $600 to run uh so we've been

running swe Bunch light to test our

agents uh and roughly what you see is

that uh state-of-the-art agents right

now score around uh 15 to 20% on the

sweet bench light Benchmark meaning they

can they can fix about 15 to 20% of

reasonably scoped issues that have been

found on GitHub um so there's a ton of

Headroom here right these agents are

still failing at most issues um but it's

also pretty impressive that they're able

to for you know roughly a fifth of

issues a sixth of issues they're able to

get to a solution uh and if you think

about you know if any of you manage a a

codebase uh or an open source project uh

being able to just automatically tackle

20% of your issues be a huge step

forward um so we're we're super excited

about the progress here uh and we

actually just announced uh two days ago

uh our latest score on swe bench FL is

21% um and that's a step up of three

percentage points or roughly like I

think

177% uh in absolute terms over uh the

next best agent in uh from an

Academia um so really you know really

quick movement here it's it's only in

the last few months that these SCS have

started being posted um so we're we're

super excited about our progress and

about the progress of the technology as

a whole but again there's there's a lot

of room for us to grow a lot of room for

for these agents to get better and

they're getting better every day

well so uh in terms of working with

agents uh for those of you who have seen

the Devon demo this user interface uh

will look somewhat familiar um uh this

is a snapshot of the open Devon

interface um you can see uh you know it

has the you know a main chat window

where you can interact with the agent

tell it what to do it can ask for

feedback it can stop and ask for further

instructions for for the directions um

they can check uh check in and say like

hey am I going in the right direction or

do you want me to ship

gears um and so you can kind of interact

with the agent there it has access to a

terminal where it can run commands uh it

has access to a code editor uh where you

can see all the files that are currently

in its workspace and you know it can it

can read and write to different files

and you can you know watch it as it's as

it's writing those files uh it has

access to a web browser where it can

browse the internet uh we've actually

recently added a a Jupiter notebook

where the agent can uh if you have a

task that's uh a little bit more

oriented towards like I'm just trying to

analyze some data right now you can work

with it inside of a Jupiter notebook um

so it's really uh you know kind of aare

programming environment with the

agent uh but this isn't the only way we

see folks interacting with agents uh you

know personally I'm looking for an agent

that can integrate much more directly

into my development Loop um you know I

don't really want to you know leave my

my Vim environment and like go into a

web browser and interact with an agent

there and then like drop back to Vim

once I'm done with the web browser I

want something that's a little bit more

Co like co-pilot which can come with me

you know as I edit code uh help me make

edits help me run tests in the

background um help suggest paths forward

things like that uh and so you know

we're really trying uh again to build an

open platform where we can drive not

just a a web experience like the one we

saw in the previous slide but where we

can drive vs code plugins uh Vim plugins

uh where we can interact directly inside

of GitHub where uh you know you could

just tag an issue and open Devon could

go and fix it uh where you could leave

uh comments on a PR and open Deon could

address those comments for you um where

you can interact with it on the

commandline or within a cicd environment

uh really we want agents to be able to

uh interact with your codebase anywhere

a software engineer would um they should

really be like a a remote teammate who's

collaborating with you

cool and now I have a demo to share

we're going to hope the demo goes

smoothly uh I already know for a fact

that I have ruined the file system

permissions on my current system because

I'm trying to debug an issue with users

who are running Devon as root uh so

we're going to see it fail when it tries

to edit an existing file but first we'll

start with a uh a simple task

uh kind of my favorite uh you know hello

world task right A bash script prints

hello you can see as it's going about

its work it'll kind of describe what

it's doing uh you can see it running

these commands as they go um should see

hello.sh pops up here pretty

straightforward right it just wrote a

brand new bash script um I can also you

know ask it to make edits so I can say

add a command line AR for the user's

name um and so you can do this kind of

iterative development where uh you know

it achieves some some task and you can

say okay now I have some extra

requirement that I want to add to this

um and

so you can see it edited the script to

uh add hello one uh

let's test it with the name

Jerry so we can see it you know can it

can run it test it things like

that um but you know writing writing a

hello world script not the not the most

exciting thing um what I really want to

show you

all

is uh so what I'm going to do is

attach open

Devon not to an empty staging area an

empty

workspace but I'm going to attach it to

the open Devon code base

itself um and I'll actually show you all

what I've done

here so I just changed the name of an

argument with in the open Devon code

base from DN to- X and uh that's going

to basically break this ARG par test

it's GNA say uh you know it's expecting

it to be

dashn uh so what I'm going to do

is tell open Devon

to work inside of this code base

basically you just have to uh as you

start the app point it to uh whichever

folder you want it to work in

uh so I'm going to restart the

browser share my screen

again uh well so you can see you know

now we're inside of the open Devon code

base um I should be able to

see see if I can find my way around

around here open Dev cor

config.py um this is the file that I've

edited with a problem uh I added this

this DX to be

a-n uh and I'm going to

say uh change the code so that

uh what was the name of my

test so that AR parer test passes

and now so you know by default the agent

doesn't know anything about this

codebase um so it's going to start by

saying okay you gave me a file name I'm

going to look at that file uh so it's

opened it to see the contents you can

see it you know digging through it on

the command line here um the our core

agent the coda agent which is the one

that uh

recently uh uh set a new record on the S

bench score the sweet bench light

score uh it uh does a lot of really

creative stuff to be able to navigate

through file contents bits at a time

instead of dumping whole file contents

into the context window uh that's one of

the things that makes it extremely

powerful is that it can kind of scroll

through line by line and see see what's

up

um it looks like it is now trying to uh

set up its environment to make sure that

it can actually run uh run the code here

this might take a little while I think

it's trying to install all the packages

one by

one okay here it goes just question like

the human can actually just message

stuff in the middle

right uh correct we're so we're working

on um getting that uh working a little

bit more strongly um so you can see like

right now the agent so the agent has

said okay I've identified the issue um

the command line arguments uh was

changed from DX to dashn uh and it's

saying you know

uh it's asking me to

to actually scroll for it that's

interesting uh that's the

diagnosis pleas fix it see what it does

so it will occasionally just stop and

say hey like here's here's where I'm

going uh please you know check my

thinking let me know if I'm going in the

right

direction um but there should be this

ability to like interrupt it and say

like hey maybe you should instead of

installing these packages one by one uh

just run you know poetry install and

it'll it'll figure everything

out I think one of the questions from

the audience uh was do you have some

guard rails to prevent it from doing

like rm-

RF yeah so that's a that's a really

great question and that's one of the big

pieces of kind of our core platform here

is that everything runs inside of a

Sandbox it's uh it's extremely difficult

to get things working properly if the

agent is working inside of a Sandbox

that's outside of your developer

environment you need to make sure that

as I was mentioning earlier file

permissions right now on my file system

are all screwed up because it's uh it's

very difficult to make sure that the the

edits that are being made in the sandbox

are run the same user ID that that your

user ID is and uh it takes a lot of

Plumbing to make sure that everything's

correct um so uh yeah it's it's that's a

big piece of uh the investment we've

made into making sure that the the agent

is able to run things

safely um and it's actually I think it

just I show you my console here it just

hit my file permission

issue uh yeah you can see it's trying to

edit the file but I have permission

screwed up so it can't so it's just

stuck here um but I'll I'll stop the

demo there of course the demo Gods never

never fully cooperate

but yeah this is uh this is great I um

yeah it's a super cool demo it seems

like the ux is really coming together as

well um and so that you know you really

have a lot of the core components like

the chat interface like the the brow

like the the terminal um and actually be

able to stream the responses as a as an

outputs um so that's uh that's super

exciting I um I want to go through some

of the questions from the audience uh

but maybe the the first thing I want to

ask because I think some of the

questions also touched on this is Could

you actually uh give an overview of the

current agent architecture slam like

some um examples of the micro agents so

for instance I know you mentioned like

Coda as one of the Asian Loops if you

could even just like explain at a high

level like what it's doing um maybe like

why it's it's good that type of thing

and then um uh maybe you like show some

examples of like the microagent

I think that'd be really helpful for the

audience yeah um yeah so we have at at

kind of the the lowest level we have

this agent

abstraction uh which basically says that

you know the it's a very small footprint

it basically says uh at each point in

time we're going to pass in a state that

has um a history of all the interactions

that have been done you know basically

everything that's happened that the

agent might want to know about is in

that state um and uh it's going to take

one step forward generally make one call

to another m in the step and that step

returns an action uh which is one of

about a dozen different actions that are

built into the system that are things

like run a command read a file write a

file uh browse the web Um send a message

uh that sort of

thing um so you know it's a very it's a

very tightly constrained action space uh

and then you know a very kind of broad

state that gets passed into the agent

and then kind of under the hood the

agent is you know can do whatever it

wants with that State uh and uh

basically to construct a prompt to the

llm but typically what you'll see is it

puts in uh a history of interactions it

might highlight any um the most recent

interactions and then it might like say

okay the last thing you did was run a

command uh maybe uh analyze the output

and send a message about uh you know

what you think happened there and what

your next step should

be um so that's that's kind of the rough

the rough Loop that every agent

follows um

um the uh the the idea behind micro

agents is to really um abstract away

even even more of that process so

instead of uh managing State at every

turn of the loop and instead of like

writing python code to decide you know

what goes into the prompt and like like

constructing this big prompt string by

hand uh all you have to do to write a

micro agent is write some markdown uh

with some template strings for like

where the history go U what kind of

actions the the agent should be uh

trying to run that sort of thing um and

I can I can you know share my screen and

kind of show what a few of those micro

agents are looking like uh they're

very um uh we're we're really looking

for contributions here what I what I am

hoping for is a library of like hundreds

or thousands of microagents that we can

delegate to uh for simple tasks like you

know open up a PR on GitHub uh you know

write a unit test to address the changes

that are in the G staging area uh things

like that

um so uh all of our agents are in this

uh this agent Hub directory inside of

the open Devon

repo um the uh microagents all go in

this folder um and so like for instance

we have this math agent that will just

like if there's a bit of math that needs

to be done we need to figure out you

know the area of a circle um you know it

will uh basically uh some python codee

run it and then like stick the answer

into the output

um uh we have an agent that like manages

postrest migrations uh which I haven't

actually used we just kind of like wrote

it as an example um we have an agent

that will look through a given GitHub

repository and just like investigate

what does this repository do what is it

about what is the code structure like

and it can pass that on to the next

agent that's actually supposed to write

code changes even an agent that fixes

typos in plain text docs like

readms um

we have an agent that like will verify

any changes that have been made so it'll

take the original task uh it is given

codebase where some changes have been

made and it has to like run commands to

make sure that those changes are good um

they can be kind of as big or as small

as you want um so I'm I'm kind of I'm

excited about what the community might

be able to do with micro agents and uh

contributing these kind of like you know

bite-sized tasks that agents can take on

and then uh what's neat is that these

tasks can um the these agents and any

agent can as part of its workflow it can

one of the big actions that we have that

it could take along with reading files

writing files browsing the web is

delegating to another agent um so it can

say okay here's a subtask that I want to

pass off to the math agent because I

need to know uh you know what the you

know what the result of this you know

kind of complex math problem is in order

to move forward um or I can delegate out

to uh the postgress agent to write a

postgress migration for me because it's

going to do a better job than the like

you know very vanilla like Coda agent

can um so yeah this is this is uh super

interesting so actually one related one

of the questions I was going to ask is

um how are these micro agents invoked

from like a higher level agent reasoning

Loop yeah so um it all goes through that

that uh delegate action and I can

actually show you we have this delegator

agent that uh works it's it's just

python um and basically what it does

is so you can see here's the step

function I was talking about before

where gets passed in a state and it's

going to generate an action um and

basically what the delegator agent does

is it just decides uh okay what was the

last thing I did

um uh if uh I was currently delegating

to the agent that's studying the repo

next thing I'm going to do is uh

delegate over to the agent that writes

code if the last thing I was doing was

writing code then I'm going to delegate

over to the agent that does verification

if the last thing I was doing was

verifying and the verifier said yep

you're good then I'm going to finish

otherwise I'm going to kick things back

to the coder and I'm going to tell it uh

hey uh here's here's what's wrong here's

what the verifier found um and so

basically the delegator agent is only

delegating out to other agents uh and

it's it's basically managing this in a

loop saying okay start by very start by

studying the repo and then move back and

forth between coder and verifier in

order to move towards the end

goal and then um uh so I'm assuming like

the delegator agent can also delegate

out to to micro agents or is that

handled by one of the sub agents well so

this this delegator agent is is like

kind of a demo of the delegation uh

stuff it could delegate out to any agent

hypothetically the way it's coded right

now it's just got these three agents

hardcoded um I can also show you uh

within the micro agents there's this

manager agent um which uh a little

easier to read in raw raw

format um but it's given uh the delegate

action um as a you know one of the

actions that I can take and we say hey

here's the list of all the agents that

are in the open Devon system uh that you

might want to delegate out to and we

describe each one um this isn't

something that's like working super well

today it's just like a lot of random

agents all kind of strewn together we

could definitely do a much better job of

kind of guiding it towards which agent

is good for which task ask uh I think

it's it's basically just guessing based

on the name of the agent right now um

but this is kind of how that how that

would look like is uh you know you give

it a list of agents and say Here's

here's everybody you have access to who

do you want to delegate to

next and um the the last kind of

question on this topic which is audience

question is uh so when you think about

like agent orchestration um so you have

like a delegator agent and sub agents

and coordination between them do you

think about it more as like let the

agent figure out like how to orchestrate

between the sub agents um or do you

explicitly Define flows yourself as like

a programmer um like do you kind of

hardcode like okay I must go through

this and then this or do you just like

you know agents figure that out

dynamically what's what's cool about the

system is that it allows you to kind of

work through both

workflows um so right now I actually

have a PR open to create a uh an agent

that will take whatever changes are

staged right now and open up a GitHub PR

um so it's a micro agent and it it it

has it's a very like hard-coded workflow

it's like uh add whatever file changes

are there right now that's just like a

hard-coded command um and then it has a

it has a sub agent for you know come up

with a good commit message come up with

a good Branch name uh figure out where

the Upstream repo is and where the fork

Rebo is um or if you should be pushing

to the to the you know directly to the

Upstream Rebo on a branch um so kind

like kind of figures out those pieces

dynamically uh and then sends a request

to the GitHub API uh in order to open up

the pr and so that's that's a pretty

like hard-coded

workflow um but you know as agents get

better and better they should be able to

uh uh take on you know a lot more of

that without having to hardcode

everything uh the nice thing about hard

coding is it gives you a lot of kind of

control as a developer over exactly how

the agent's going to behave what it's

going to do it gives you a lot of

transparency over like what's going to

happen um you know leaving a little bit

more open-ended requires a little bit

less upfront work and investment um but

it uh it leaves things a little bit more

open-ended for the agent it might do

something a little bit less predictable

um so we really see supporting both

workflows as

important great um the uh next question

from the audience is um what like given

that you know the performance on on

sweet bench flight is 21% what are some

of the main issues uh that you're seeing

like common sources like hallucination

um common like error modes for open de

and and this relates to the kind of like

General topic of just like challenges

with the current agent

architecture yeah great question um uh I

would say first off we're we're still

analyzing our eval results we haven't

gone through all the test cases yet to

like figure out okay where where is it

falling down and kind of like where's

the low hanging fruit for us to get the

next five 10 percentage points on the on

the

Benchmark um but the the I would say the

the broad patterns are

um edits that that require like a lot of

uh kind of like uh Meta Meta level

knowledge about how the codebase works

and functions together right like we can

you know it's very easy to fix a unit

test that where you're just editing a

single function um you know kind of like

the example I showed earlier where you

know we have a unit test failing it just

needs to make a oneline edit to an

existing function like it can find its

way around the codebase it can find the

right file to edit it can make the right

edits um

but uh something where it's like hey we

want to add this new feature uh it

involves like you know adding a database

migration and then you know changing the

API to work with the new database

structure and then changing the front

end to ingest that new API like that's a

that's a very big uh you know it's a

very there would be a very common GitHub

issue is like hey please add a new

feature uh that's going to involve

front-end work backend work database

work um and then like figuring out how

to make those those large scale edits

across an existing code base that's

really where a lot of the um the current

agents uh have room for

improvement great um the kind of uh next

question is what is sorry I'm trying to

find the question oh what's like the

minimum compute recommended to to run

this

locally so open Devon itself does not

need much compute right it basically

runs commands in a Docker file all the

compute is going on all the heavy

compute is going on in the

LM and that is

API yeah so we basically we use light

llm which is a python library that lets

us interface with any llm through the

same function basically um so uh we can

run on open AI we can run on anthropics

Claud we can run uh on any of the Azure

uh llms uh we can run against uh AMA

running locally um so BAS llm agnostic

what I will say is gp4 and Claude Far

and Away worth the best um the smaller

local llms tend to get stuck in Loops

they get tend to uh um you know not

really make the right changes the the

the the underlying llm is really uh the

power of that llm is really what drives

uh the power of the agent um and so

we're really only as powerful as the llm

that that you're using um so you know

while folks are interested in running

locally with you know llama 7 billion

and stuff like that it tends to only

work for really simple

stuff um one question for my end

actually you mentioned that you know one

of the kind of challenges of building

this is context window management of

figuring out like how to put different

things into the prompt how like what are

the specific components that you're uh

like putting into the prompt and how are

you like indexing the code base right

and are you doing like retrieval that

type of stuff to actually return

relevant

context yeah that's a that's a great

question so I think the the the hardest

part of the context window management is

managing the history of interaction so

the agent can see here's what I've done

up to now and figure out what what's my

next step um that uh works really well

up to a point and then eventually that

history gets so long especially if

you're reading like the entire contents

of files uh you end up having to one

thing we've experimented with is like

kind of progressively summarizing the

history so you know as the history gets

longer and longer and longer you know

summarize the first half of the history

so that you can cut down on the amount

of context you're shoving into that

context window assuming that those those

early actions are now like not super

relevant to what the agent is working on

now um so that's that's one bit of

context window management that we've

worked on um in terms of pulling in code

our initial agent would just dump the

entire contents of a file into the

context window if the agent was like I

need to look at f.p we just dump f.p

into the context window and sometimes

food. Pi is like 10,000 lines of code

and like you immediately hit the context

window limit uh so one thing we've done

here is

um the uh the code act agent exposes

some functions for uh searching through

code and for looking at files uh just

chunks at a time and kind of scrolling

up and down inside of that file with

these Dynamic commands and that works

really really well uh for basically

grabbing the part of the file that's

relevant to what the agent is doing

without poll including the context

window with a bunch of irrelevant

information uh so it can search it can

um uh you know it can dig through the

context window I think one uh bit of

loing fruit that we have not yet

explored is basically ingesting the

whole uh the whole code base into a

vector database uh which we we have been

experimenting with with FL index for um

in order to do some more Dynamic

retrieval of relevant Snippets of code

for the agent to work with

yeah makes sense right now you're

representing it as like almost like code

search functions that the agent can just

call and it's relatively puristic based

I'm assuming like if you search for

things it just like keyword search right

not like vector search but then if you

actually pre-index it with a bunch of

vectors then you can do some like

semantic similar up yeah I think I think

we're literally using grp under the hood

so it's like you know exact you know

yeah exact Str match yeah it makes sense

I mean grap is a pretty powerful

Baseline for what it's worth um I was

just curious how you were yeah um okay

and and so um in terms of the um like

you you you like in the uax you

mentioned some initial set of like tools

so you have like a code editor as well

as a terminal um of course like I know

the the like full Devon app has like web

search and and potentially like other

tools as well are there plans to

basically kind of expand the

capabilities to uh even solving like

more General sweet bench problems um as

opposed to like sweet bench flight or

just being able to pull in like external

information and and if so like how how

do you plan to do

that yeah so we have a a couple Folks at

CMU that are actively working on the

browsing functionality uh they've done

work on browser agents separate from

coding agents uh and they're very

interested in what they can do to push

our ability to browse the web pull an

API documentation Etc um so we've we've

uh integrated browser gym at this point

um Kodak agent is not yet using the

browser uh which is probably some of the

lowest hanging fruit for us to improve

uh the coda agent

um so we are uh that's kind of like our

next one of our next steps for pushing

codak agent forward uh to solving more

of the sbench light problem Set uh we

definitely want to get to the Fuller

sbench uh problem set and we we we also

want to do a full sbench eval of uh of

open Devon today to see how it performs

relative to Devon itself which is

published at least a partial sweep buch

score I think they evaluated like 25% of

the sweet bench set um and how it

compares to you know other academic

agents um you know like like s agent um

so that is definitely the goal is to

take on the full basically be able to

solve any any issue you find on GitHub

any any issue that a software engineer

could take on we want to be able to take

on with uh with our automated agents um

I think it's probably uh you're probably

going to see years of effort going into

that where you'll see you know some some

very quick pce development over the next

six months or so and then there's going

to be a long taale of really difficult

issues that last 20% of issues will

probably take several years to to fully

take

down and it seems like you have a few

agents in open Deon you have Kodak you

have all these micro agents you have

like browser agents that people are

working on you mentioned the overall

goal is to have a platform for uh you

know AI Engineers to just like pull in

components that build autonomous

software Engineers um how if you able to

like elaborate upon that Vision like

what is the unit of abstraction that you

expect like a developer to be able to

pull into kind of build something is it

like they're able to import uh an agent

from like a suite of different agents

that you offer is it that you just like

they can clone this entire thing and

they can just like modify the ux I'm

kind of curious how you're thinking

about

that yeah so there there's kind of two

sides to the platform right we want to

appeal to folks that are building agents

and folks that are kind of at the

bleeding edge of what agents can do and

we want to appeal to the end users who

are running these agents right um and so

on that first side what we really want

to do is take care of um all the

nitty-gritty work that goes into build

and evaluating agents um so that's uh

like a having an opinion on what what

the agent Loop looks like how actions

are structured things like that um and B

like really providing all the stuff like

the docker sandbox running commands so

you know this thing's not going to rm-

RF you know your root directory um uh

things like that so providing all that

kind of like base infrastructure for for

running the agents as you're building

them and testing them uh then we've also

actually packaged up uh sbench uh sbench

light so far but working on swe bench in

a very easy to run pipeline uh so we

want to basically make it easy for

anybody who has built an agent um to you

know wrap it in our interface uh which

should be generic enough to handle any

agent out there uh wrap it in our

interface and and very quickly be able

to run an eval and see how do you do

compared to other folks um and what we'd

love is for everybody to uh kind of

consolidate on on open Devon as a um a

standard for how you write agents even

though under the hood they could be

doing very different things they could

be structured in very different ways as

long as they kind of conform to this

very simple interface of like you know

pass in state step forward one step and

produce an action at each step um as

long as they conform to that interface

we'll be able they'll be able to plug

into all this uh amazing extra uh

architecture that we've built for

running the agents evaluating the agents

Etc

um and then on the on the user side you

know right now like I said we have a web

interface um but what we're working on

now is a re

architecture uh to enable many different

types of interfaces whether it's a CLI

or a vs code plugin to uh interact with

these agents and be able to pull in

their results and uh you know run the

actions things like

that yeah super exciting um take us

through and you know just in the last

few minutes uh yeah like what's what's

coming next like what what are things

that you're excited about uh what is

like a general landscape of um like is

it just adding more agents and adding

better agents like adding better uxes uh

like what are you excited about in the

next few months yeah I think um I think

one of the things that's that's most

exciting right now is just uh being able

to push the agent itself forward um you

know like I said we just announced the

other day uh our results on sweet bench

light which were super exciting um got a

bunch of excitement from uh from the

community which is great um and I think

we still know of a lot of loow hanging

fruit for pushing that agent forward um

and so sh who's a a PhD student at

Illinois is the brains behind the Koda

agent U has published a paper around it

uh he's he's going to be working on

really pushing that that core ability

forward further and further uh so very

excited about that and then there's

there's also the Tailwind of uh you know

llms themselves getting better you know

when GPT 5 comes out our Tech is going

to get that much better uh because we'll

just be able to flip a switch start

using the latest LM and uh the agents

will just naturally get better as a

result of that so I'm very excited about

where the agents themselves can

go uh but in addition to that you know

the thing that I'm I'm most focused on

now while Shing Yao is is focused on

pushing the agent forward is I'm I'm

focused on building this you know very

uh more more scalable more generic

architecture where uh we can uh start

interacting with agents and more than

just that web interface uh you know it

can be on GitHub issues it can be on the

command line it can be inside your code

editor um and so that's that's what I'm

really excited about personally is um

really being able to uh integrate these

agents and especially the codea agent

into my personal day-to-day

workflow makes a ton of sense um and

last question is basically how you know

we talk about agent and orchestration

how do you um you know you and the other

contributors orchestrate amongst

yourself basically I mean it's is this

like a decentralized thing do you like

meet once a week and then also you know

from the audience like if they're

interested in contributing to open de

like how can you basically becomea

contributor to open do yeah um yeah I

would say it is it is pretty

decentralized especially now that we

have an evaluation pipeline set up my My

Hope Is that it's basically a little bit

of a competition kind of like you know

the hugging face leaderboard style right

we uh we can tell who's who's the best

agent um and so if you contribute an

agent and you beat Kodak on uh on the

sweet buch light score you'll probably

become our new default agent um and uh

we really want to kind of get that that

friendly competition feeling going uh

and what's fun is like you could build

an agent that delegates 60% of its tasks

out to Koda agent and delegates the

other 40% out to a different agent uh

and as long as you're you're managing

that delegation better than anybody else

you'll do better and you'll you'll

become the default agent um and so it is

this kind of decentralized data driven

way of deciding you know what what is

the best agent out there what is the

best architecture out there Etc um and

yeah the best way to contribute is to

just you know open up a poll request um

you know maybe file an issue and let us

know what you're thinking about building

what kind of feature you want we have a

ton of open features or open issues that

are labeled like a first issue uh a ton

of you know potential feature requests a

ton of requests for uh there are a bunch

of agents that have been built in

Academia um a bunch of like interesting

agent architectures that are out there

that uh we have open issues to just like

hey let's let's build an agent that

implements this and see how it does um

so so there's there's tons of room for

for uh folks to contribute

here great well I think that's a great

way to conclude uh this webinar thanks

so much Robert for joining um for those

of you who want to check out open Deon

please check out the repo um check out

the the website as well the docs and the

demo and then uh we'll have this on

YouTube very shortly so thank you Robert

again for your time and thanks for

joining thanks so much for having me