How To Use Copilot Studio Kit for Test Automation

Summary

このビデオでは、Microsoftのpowercatグループによって開発されたco-pilot Studio kitを使用して、co-pilotを自動化テストする方法を紹介しています。Direct Line APIを通じてco-pilotとチャットすることができる機能や、Azure Application Insightsによるテレメトリの追跡、AI Builderとの連携、Dataverseのデータ管理など、キットの機能を詳細に解説しています。また、キットの入手方法、インストール手順、テストの設定方法、実行方法についても詳しく説明しています。

Takeaways

• 🛠️ 「コパイロットスタジオキット」はMicrosoftのpowercatグループによって開発された、コパイロットの自動テストを可能にするツールキットです。
• 🔍 powercatはMicrosoftのエンジニアリングチームの拡張であり、最大の顧客と連携する際に使用されます。
• 🌐 コパイロットスタジオキットには、Direct Line API、Azure Application Insights、AI Builder、Dataverseなどの機能が含まれています。
• 🔗 Direct Line APIは、Webチャットエクスペリエンスと同様にコパイロットに接続し、チャットを行うための方法です。
• 📊 Azure Application Insightsを使用すると、コパイロットの内部で何が起こっているかを追跡して確認できます。
• 🤖 AI Builderは、生成的なAIレスポンスをテストする際に必要であり、正確な一致が返されるとは限りません。
• 💾 Dataverseはコパイロットスタジオやpowercatが構築されている基盤技術であり、効果的なデータストレージや検索機能を提供します。
• 📚 aka.ms/copilot Studio kitにアクセスしてコパイロットスタジオキットのダウンロードとインストール方法を確認できます。
• 🔧 テストを実行するには、コパイロットスタジオキットをPower Apps環境にインポートし、テストスクリプトを作成する必要があります。
• 📝 テストセットとテストランを設定し、コパイロットのパフォーマンスを評価するためのテストを実行できます。

Q & A

コパイロットスタジオキットとは何ですか？

コパイロットスタジオキットはMicrosoftのpowercatグループによって開発された、コパイロットの自動テストを行うためのツールセットです。

powercatはどのようなグループですか？

powercatはMicrosoftのエンジニアリングチームの1つで、最大で影響力のある顧客と連携しています。

direct lineとは何で、どのように使われますか？

direct lineはコパイロットに接続し、チャットすることができる方法です。ウェブチャット体験ではdirect line APIが使用されます。

Azure Application Insightsとは何であり、何のために使われますか？

Azure Application Insightsは、コパイロット内部で起こっていることを追跡し、状況を確認できるツールです。

AI Builderとは何で、何のために必要なのですか？

AI Builderは生成的AIレスポンスをチェックするための機能で、直接一致するレスポンスが返されない場合に必要です。

dataverseとは何であり、どのような役割を持っていますか？

dataverseはコパイロットスタジオやpowerが構築されている基盤技術で、効果的なデータ保存や検索、インポート、エクスポート機能を提供します。

コパイロットスタジオキットをダウンロードするにはどうすればよいですか？

aka.ms/copilot-studio-kitにアクセスして、自動的にダウンロードサイトに移動し、ZIPファイルをダウンロードできます。

コパイロットスタジオキットをインポートするにはどのような手順が必要ですか？

Power Apps環境に移動し、ソリューションをインポートするオプションからダウンロードしたZIPファイルをアップロードし、インポートプロセスを開始します。

Azure Application Insightsの接続を作成するにはどうすればよいですか？

Azureポータルから取得したApplication IDとAPIキーをコパイロットスタジオキットに入力して、接続を作成します。

テストを実行するにはどうすればよいですか？

テストセットを作成し、テストケースを追加して実行します。テスト結果は自動的に記録され、詳細なレポートが生成されます。

コパイロットのテスト結果を確認するにはどうすればよいですか？

テスト実行ページからテスト結果を確認でき、各テストケースの詳細や成功/失敗の理由を閲覧することができます。

Outlines

00:00

😀 co-pilot Studio Kitの紹介と取得方法

この段落では、co-pilot Studio KitがMicrosoftのpowercatグループによって開発されたツールキットであることが説明されています。このキットは、企業のcopilotの自動化テストを可能にします。Direct Line APIを通じてcopilotとチャットすることができる機能や、Azure Application Insightsによるcopilotの動作追跡、AI Builderの応答生成能力の確認、Dataverseによるデータ管理とインポート/エクスポート機能が含まれています。また、co-pilot Studio Kitを取得するには、aka.ms copilot Studio kitにアクセスし、指示に従ってダウンロードとインストールを行うことができました。

05:04

🔧 Azure Application Insightsの接続設定

この段落では、Azure Application InsightsのインスツルメンテーションキーとAPIキーを通じてcopilot Studio Kitに接続する方法が説明されています。まず、Azure PortalでApplication Insightsを作成し、必要なキーを取得します。次に、copilot Studio Kitの設定でこれらのキーを入力し、接続を作成します。これにより、テスト結果がAzure Application Insightsに記録され、テレメトリデータとして利用可能になります。

10:06

🛠 co-pilotのテスト設定と実行

この段落では、copilot Studio Kitを使用してcopilotのテストを設定し、実行する方法が説明されています。まず、copilotを作成し、トークンエンドポイントを設定します。次に、テストセットを作成し、テストケースを追加します。各テストケースには、テスト発話と期待される応答が含まれています。テスト実行を作成し、選択したcopilotとテストセットを指定してテストを開始します。テスト結果は、成功したかどうかを示し、詳細な情報や推奨事項が提供されます。

15:08

🗣️ テストケースの詳細設定

この段落では、テストケースの詳細設定方法が説明されています。テストケースの名前を設定し、応答の種類（トピック一致、添付ファイル一致、生成的な答え）を選択することができます。また、テストのタイムアウトや開始会話イベントの有無も設定可能です。期待される応答やテスト発話の例を入力し、テストケースを保存して実行します。

20:09

📊 テスト結果の分析とレポート

最後の段落では、テスト結果の分析とレポート機能について説明されています。テスト実行が完了すると、各テストケースの成功状況を確認できます。詳細をクリックすると、応答の内容や成功の理由、推奨事項などが表示されます。また、会話のトランスクリプトや管理者機能、監査履歴などの関連情報を確認することもできます。

Keywords

💡コパイロット スタジオキット

コパイロット スタジオキットは、マイクロソフトのpowercatグループによって開発された自動化テストツールキットです。このキットを使うと、コパイロットのパフォーマンスをテストし、評価することができます。ビデオの主題はコパイロットの自動化テストであり、このキットがその中心的なツールとして紹介されています。

💡Direct Line

Direct Lineは、コパイロットに接続し、チャットすることができるインターフェースです。ビデオスクリプトでは、Direct Line APIを使ってコパイロットと対話することができる機能として説明されています。これはテスト実行において、コパイロットとのインタラクションを可能にします。

💡Azure Application Insights

Azure Application Insightsは、マイクロソフトのクラウドプラットフォームサービスで、アプリケーションのテレメトリを収集し、分析することができます。ビデオでは、このサービスを使ってコパイロットの内部動作を追跡し、テスト結果を豊富にすることが示されています。

💡AI Builder

AI Builderは、マイクロソフトが提供するサービスで、生成的AI応答を生成することができます。ビデオでは、AI Builderがテストキットに組み込まれており、生成されたAI応答のテストが可能であることが強調されています。

💡Dataverse

Dataverseは、マイクロソフトのデータ管理プラットフォームであり、効果的なデータストレージと検索機能を提供しています。ビデオでは、Dataverseがコパイロットスタジオとpowercatの基盤技術として機能し、テストキットのデータ管理に使われていることが説明されています。

💡アカウントの設定

アカウントの設定は、ビデオスクリプトで説明されているテストキットのインストールプロセスの一部です。Azure Portalから取得したApplication IDやAPIキーを設定することで、Azure Application Insightsのテレメトリを保存するための接続を作成します。

💡テスト ケース

テスト ケースは、ビデオスクリプトで作成され、実行される具体的なテストです。テストケースには、予想される応答やテストの実行時間などが含まれており、コパイロットの特定の機能を検証するために使われます。

💡トピック

トピックは、ビデオスクリプトで示されているコパイロットの機能の一部であり、特定の問い合わせに関連する応答を提供します。テストケースを作成する際に、トピックに関連する応答が期待されるかどうかを指定することができます。

💡応答マッチ

応答マッチは、テストケースで使用される機能で、コパイロットからの応答が予想される応答と一致するかどうかを検証します。ビデオでは、応答マッチを使ってテストケースの正確性を保証することが強調されています。

💡テスト実行

テスト実行は、ビデオスクリプトで説明されているプロセスで、テストケースを実際に実行し、結果を収集します。テスト実行のステータスを監視し、テスト結果を分析することで、コパイロットのパフォーマンスを評価することができます。

Highlights

介绍了如何使用Co-Pilot Studio Kit来自动化测试Co-Pilot。

Co-Pilot Studio Kit由微软的Powercat团队开发，该团队服务于微软最大的客户。

Co-Pilot Studio Kit包含Direct Line API，允许用户与Co-Pilot进行聊天。

使用Azure Application Insights跟踪Co-Pilot的性能。

AI Builder集成，用于检查生成性AI响应。

所有功能都建立在Dataverse之上，这是Co-Pilot Studio的基础技术。

通过aka.ms/copilotStudioKit访问Co-Pilot Studio Kit。

下载Co-Pilot Studio Kit后，需要在Power Apps环境中导入解决方案。

创建Azure Application Insights连接以存储遥测数据。

创建Co-Pilot测试时，可以选择关闭Entra ID认证简化测试过程。

配置测试脚本时，可以设置测试集和测试运行。

测试脚本可以设置预期的响应和测试话语。

测试运行可以自动执行并验证Co-Pilot的响应。

测试完成后，可以查看详细的测试结果和推荐。

Co-Pilot Studio Kit提供了丰富的报告和信息，帮助评估Co-Pilot的性能。

观众可以通过评论请求更多关于测试工具的视频内容。

鼓励观众订阅频道并参与评论交流。

Transcripts

[Music]

in this video I'm going to show you how

to use co-pilot Studio kit for

automating test with your co-pilot so

let's start with what is the co-pilot

Studio kit and where did this thing come

from well the co-pilot Studio kit is

built by a group at Microsoft called

powercat and powercat is an extension of

the engineering team at copilot Studio

that works with all of the biggest and

baddest customers that Microsoft has to

interface with and they went out and

decided hey it would be great to have

this automation toolkit that would allow

you to be able to test your co-pilot and

be able to find out how is it performing

so let's explore a little more on what's

inside this kit so inside of the copilot

Studio kit you actually have this

ability to test a co-pilot through

something we call direct line and direct

line is the way that you can connect to

your copilot and be able to chat with it

so if you ever think about the web chat

experience as an example it uses the

direct line API so it's going to use

that in order to be able to do your test

now we're also going to have Azure

application insights that allows you to

be able to track and see what's going on

within your co-pilot itself we it also

has capabilities with AI Builder so if

you're going to have generative AI

responses you need a way to check those

because it's not going to come back with

a direct match so that's one of the

reasons why you'll see AI Builder as

part of this and then lastly it's all

built on top of dataverse dataverse is

the foundational technology that all of

copilot studio and power is built on top

of that allows for Effective data

storage and does a lot of work to make

it so that you get this really robust

Enterprise place to store and be able to

search and find all the different

components and also allows that import

export capabilities that you've seen

within copilot studio so let's first

start with how do I go and get ah hold

of the actual copilot Studio kit and the

way that you can get this is you can go

to

aka.ms copilot Studio kit and it will

automatically take you to this site that

you see here and in this site you'll see

that it gives you information about the

co-pilot Studio kit it gives you

information about uh the different ways

that it can work you can see here it has

things about exact match attachments

match all of these different

capabilities that this thing has out of

the box and I would encourage you to

continue to look at this because uh it

enhancements of this may continue to

happen uh by the power cap team over at

Microsoft so you'll see that even the

slide that I shared earlier is here you

can see a visual representation of it

you can see other things that are in

here you can also check the

prerequisites and you can get the in

installation instructions which is the

key thing for us right now so the first

step is going to be to go ahead and go

to the

aka.ms download copilot Studio kit you

can just click this link and it will let

you download this and just from there

you're going to get a zip file that's

downloaded on your computer now once

you've downloaded that what you're going

to need to be able to do is go into a

cop Studio enabled environment such by

going to make. power

apps.com so once you get into your power

apps environment that is created as part

of this if it wasn't already created

you're going to be able to select your

environment and you can do so right up

here and you can see that I really only

have one environment in this particular

implementation so I'm going to use that

particular environment as I go forward

now it's important that the environment

is the environment in which you are uh

have your co-pilot implementation that

you're looking to test so I'm going to

go ahead and go to Solutions

and now that I'm in Solutions what I

want to do is I want to go in and import

a

solution and when I go click import a

solution I can go to browse I can

grab that file I just downloaded which

is our ZIP file for the

implementation and click

next and then I'll click next again

and notice here that you're going to

need to create an implementation for

application insights now if you haven't

already uh created one you can go create

one of these pretty easily inside of

azure I'm going to go ahead and refer

you to a video where I went through

application insights and kind of the

configuration that's already on my

channel so if you haven't actually done

this before just know that you need to

go and create one of

those so I'm going to go ahead and click

on

create you'll see here that it needs a

connection name and I'm going to go

ahead and put in a connection name and

I'll just call this uh

co-pilot um

Studio

kit then we need our application ID and

our API key now we need to get that from

from the Azure

portal so I'll grab the instrumentation

key right here and go drop it

in and we're going to drop that into the

application ID right

here now we need to get the the actual

uh API

key if you don't already have an API key

you just simply go create one and in

this I would recommend giving a

description and basically what we want

here is we want read and write and I'll

just give it all of the different

permissions and I'll take this and I'll

copy the API key that was given to me

and we'll go back and we'll drop that

in so we'll drop that into the API key

here and then we'll create and by doing

this now we have created the connection

for the Azure application insights to

store like the Telemetry that's going on

and then we have data verse which is

where our solution is being imported to

and being stored and then we can just

click import

now you'll notice that right here it

actually says that it's going through an

import process this can take a second or

two depending on your environment but

just know that it's going to go ahead

and create this solution for

us once your solution is done importing

know that it will give you a

confirmation on the screen but let's go

ahead and click over into Apps and once

we go into apps now you can see the C-

pilot Studio kit has been installed and

I can hit the play button and once I hit

the play button it's going to ask me to

authenticate if you notice this hang uh

just know that you can click off of Home

click over on co-pilots and come back

this is what you should see once once

you have your solution up and running

and at this point we should be able to

start configuring a test or a co-pilot

that we'd like to test so let's first

click over on

co-pilots and we'll click on

new and we need to create a co-pilot

that we want to test so I'm going to

test the one that I built for the

Porsche

information that I have already today

you'll need to come in and DEC decide

which is the region and in this case I'm

just going to take the default

region then over here you can see who's

the owner you can select who it is that

is actually going to be the owner of

this particular base configuration for

this new co-pilot and then we need to

get a token end point so what what I'm

going to do is I'm going to use my

Porsche product information co-pilot

that I built in an earlier video however

I'm going to make some slight changes

here just to make this simple know that

there are different configurations

depending on what you want to do but

just for the Simplicity of the video I'm

going to go ahead and I'm going to

change the security on my

co-pilot and the way I'm going to do

that is go to authentication and I'm

just going to turn off the entra ID

authentication Now by doing this it

gives me the ability to be able to uh

connect and get an endpoint without

having to worry about setting up all the

security once we've done that I'll go

back over to channels and we'll go to

the mobile app and right here you can

get your URL for the token endpoint so I

I've copied that

over and I'm going to drop that into the

token endpoint here now once we get past

this we're also going to have things

like the channel security and things of

this nature and the documentation has a

lot of great information guys for you to

go check out so if you want to look at

all the different configurations that

you can have uh and such we you can go

do that now we can

also go in and put in like enrich with

Azure application

insites by doing this it basically

enriches the test results for generative

answers test with Azure application

insights Telemetry data because of what

we're going to do in this initial test

just to show you guys how this thing

works I'm not going to be enabling that

in our uh environment but that's a place

where if you're wanting to get some

additional information on what's going

on uh behind the scenes with the

generative AI features you

can now the other option here is that we

can enrich with conversation transcripts

and again

these have to do with pulling in the

actual conversation transcripts now

again we're not going to do any advanced

configurations just calling out a couple

of key things that you're going to see

and as you can see here we also have the

generative AI

configuration as well that you can turn

on and off and notice here that you have

to say what are you going to use and the

only option is AI Builder so if you want

to test generative AI features you'll

need to do this now I may run a separate

video that shows how to be able to do

this later you guys leave something in

the comments if you want to see that now

now that we've got this all we have to

do is hit save and

close and now that we've got that done

you'll see that now you have a co-pilot

that has been configured so if we go

here you'll also see that I need to be

able able to configure test scripts and

you can see we've done no test runs

currently today all we've done is

created a co-pilot that we want to be

able to test so the first thing that we

need to do is create a test set and a

test set is going to be like a

combination of things that it's going to

go run all the tests that we want it to

run so in this case what I need to do is

I need to give it a test set and I'll

just call this um Porsche simple

and I'm just going to hit save and

close now we've got ourselves a test

set and inside of this test set you'll

see that we have no actual test that

have been

run so now what I'm going to do is I'm

going to set up some test and we'll

start by clicking on new and then we're

going to need to give it a name and

things of this nature so so let's jump

back over to copilot Studio real quick

and let's look at some of the topics

here that we have and you can see I've

got these different things that I've

been able to test with and I'm going to

start with the fine local Porsche Center

and what I'm going to need to do is I

need to grab this response right here

which is what the copilot is going to

respond with and we want to come in and

say that we'll just call this uh find a

dealer and what we're going to say here

is that we want to use the simple test

set we also um want to say how long now

if you don't know how long your your

query is going to take because you might

have like a power automate flow or

something like that that's on the back

end but the point being is what you're

wanting to look for as a an input here

is how long you think that's going to

take now if you read the instruction

manual the instruction manual will tell

you if you don't know you should put 10

now anything below 10 seconds I actually

started to see some errors and things

like that uh and so just to make sure

that it has plenty of time to respond

I'm just going to do that and

then with that we also have a setting

here and this is the setting that says

send start conversation and the send

start conversation is basically whenever

you go into a co-pilot and you first

start a conversation a lot of times

you'll see it it'll say like hello this

is my name and this is what I do but

understand that the direct line API

doesn't have to do that you have to send

in an activity or an event that is the

start conversation event so if your test

needs to have the start conversation

event triggered in order to be able to

work then you can flip this but in the

case of this I don't really need that

and I'm going to put in that the

expected response is right here

now you'll even notice I have a typo in

here and we'll play with that as we

go um but what I'll do is and I also

want to go in and come up with what is

an example of something I might ask and

so here is the utterance that I want to

grab right here and we'll jump back and

we'll put the test utterance in right

here so now we have where it's going to

ask this question and it expects this

response and we will

go ahead and hit save and

close now you can see here I've got my

find a dealer now let's go ahead and add

one more in so I'll go ahead and create

another new one and let's go in and

let's look at a different topic and in

this case I'm going to say I want to get

accessories and things of that nature

like clothes or something of that nature

so I'm going to go ahead and copy the

response that comes back and we'll give

it a name and we'll say you know

find

clothing and we're going to keep this

response match again this is where you

can go and say whether you just want to

match the topic you want to see

attachments a generative anwers so and a

topic match think of it that what you're

looking for is you want to be able to

give it a test utterance you don't

really care what it says back you just

want to know that it went to the right

topic but in the response match we want

to see the exact response so in this

case I'm going to leave it response

match I don't need the this setting here

I want it to be in uh part of this test

set and then again we'll put in

10 and then we'll put the expected

response in right here let's go get a

test utterance you can see here this is

the test utterance that I want to

have and I'll drop this in right

here and we'll go ahead and save and

close and now that we've done this you

can see now we've got our different test

sets or our test queries that we want to

run and now we can actually go and

execute this test now we'll go into test

runs and you'll see here we don't have

any that have actually run you could

also come up here and you can see that

there's been no tests that have been

recently run but let's go ahead and say

okay I want to actually run a test so

keep in mind that when we have these

tests we might want to run multiple

instances of these tests to be able to

check to see like maybe we want to run

in on a certain day or something of that

nature so in this case what I'm going to

do is I'm just going to put in that this

will be um

video demo one and what I want to test

is I want to test this test set again

that'll test all of the things in that

test set and then I also want to choose

which co-pilot and I want to do the

Porsche co-pilot that we've got now once

I hit save and close on this what's

going to happen on the back end is

you're going to see that it'll start and

it'll automatically kick this off so

it'll take a second for it to start but

we'll see it kick off and start to run

in just a

second and we're looking right here at

the Run status is what we're looking for

and you can see now it is running now

again keep in mind that we had two

different things that we needed to run

and we gave it up to 10 seconds so it

may take a second or 20 seconds for this

thing to

update and you can see we have a

completed test now let's go ahead and

click in on the test and you can see

here that we had a successful actual

match on each of these different items

and if you want to you can even click in

on them and get more details you can see

this is the actual response here's the

the reason that it was successful

all of these different things you can um

and again you can have other things that

it will give you recommendations and

things of this nature if we had pulled

in the uh information on the transcript

the transcript information would have

shown here you can also see like admin

uh functions and things of this nature

on the test set there's also related

things like an audit history um as well

that you can look at if you uh set all

of these different things up and the

also even background processes and

things that are uh that we're taking but

again this was just a very simple test

of being able to use this but you can

see the power that this will give you as

you continue to go run and have all your

different tests and you can see all the

different things that you get coming out

of this and being able to give you some

really good reporting and information

around is this thing performing and

doing what you need it to do now if

you're interested in seeing more videos

about the testing uh tools to allow us

to be able to go say run test against

generative AI things and things of that

nature again please put that in the

comments below I really want to try to

focus content on things that you guys

want to see and as always thanks for

coming and watching our my videos I

really appreciate all the support that

you guys give me know that you can like

And subscribe to my channel and as

always you can go TR copilot Studio at

aka.ms TR copilot Studio