How To Use Copilot Studio Kit for Test Automation
Summary
TLDRこのビデオでは、Microsoftのpowercatグループによって開発されたco-pilot Studio kitを使用して、co-pilotを自動化テストする方法を紹介しています。Direct Line APIを通じてco-pilotとチャットすることができる機能や、Azure Application Insightsによるテレメトリの追跡、AI Builderとの連携、Dataverseのデータ管理など、キットの機能を詳細に解説しています。また、キットの入手方法、インストール手順、テストの設定方法、実行方法についても詳しく説明しています。
Takeaways
- 🛠️ 「コパイロットスタジオキット」はMicrosoftのpowercatグループによって開発された、コパイロットの自動テストを可能にするツールキットです。
- 🔍 powercatはMicrosoftのエンジニアリングチームの拡張であり、最大の顧客と連携する際に使用されます。
- 🌐 コパイロットスタジオキットには、Direct Line API、Azure Application Insights、AI Builder、Dataverseなどの機能が含まれています。
- 🔗 Direct Line APIは、Webチャットエクスペリエンスと同様にコパイロットに接続し、チャットを行うための方法です。
- 📊 Azure Application Insightsを使用すると、コパイロットの内部で何が起こっているかを追跡して確認できます。
- 🤖 AI Builderは、生成的なAIレスポンスをテストする際に必要であり、正確な一致が返されるとは限りません。
- 💾 Dataverseはコパイロットスタジオやpowercatが構築されている基盤技術であり、効果的なデータストレージや検索機能を提供します。
- 📚 aka.ms/copilot Studio kitにアクセスしてコパイロットスタジオキットのダウンロードとインストール方法を確認できます。
- 🔧 テストを実行するには、コパイロットスタジオキットをPower Apps環境にインポートし、テストスクリプトを作成する必要があります。
- 📝 テストセットとテストランを設定し、コパイロットのパフォーマンスを評価するためのテストを実行できます。
Q & A
コパイロットスタジオキットとは何ですか?
-コパイロットスタジオキットはMicrosoftのpowercatグループによって開発された、コパイロットの自動テストを行うためのツールセットです。
powercatはどのようなグループですか?
-powercatはMicrosoftのエンジニアリングチームの1つで、最大で影響力のある顧客と連携しています。
direct lineとは何で、どのように使われますか?
-direct lineはコパイロットに接続し、チャットすることができる方法です。ウェブチャット体験ではdirect line APIが使用されます。
Azure Application Insightsとは何であり、何のために使われますか?
-Azure Application Insightsは、コパイロット内部で起こっていることを追跡し、状況を確認できるツールです。
AI Builderとは何で、何のために必要なのですか?
-AI Builderは生成的AIレスポンスをチェックするための機能で、直接一致するレスポンスが返されない場合に必要です。
dataverseとは何であり、どのような役割を持っていますか?
-dataverseはコパイロットスタジオやpowerが構築されている基盤技術で、効果的なデータ保存や検索、インポート、エクスポート機能を提供します。
コパイロットスタジオキットをダウンロードするにはどうすればよいですか?
-aka.ms/copilot-studio-kitにアクセスして、自動的にダウンロードサイトに移動し、ZIPファイルをダウンロードできます。
コパイロットスタジオキットをインポートするにはどのような手順が必要ですか?
-Power Apps環境に移動し、ソリューションをインポートするオプションからダウンロードしたZIPファイルをアップロードし、インポートプロセスを開始します。
Azure Application Insightsの接続を作成するにはどうすればよいですか?
-Azureポータルから取得したApplication IDとAPIキーをコパイロットスタジオキットに入力して、接続を作成します。
テストを実行するにはどうすればよいですか?
-テストセットを作成し、テストケースを追加して実行します。テスト結果は自動的に記録され、詳細なレポートが生成されます。
コパイロットのテスト結果を確認するにはどうすればよいですか?
-テスト実行ページからテスト結果を確認でき、各テストケースの詳細や成功/失敗の理由を閲覧することができます。
Outlines
😀 co-pilot Studio Kitの紹介と取得方法
この段落では、co-pilot Studio KitがMicrosoftのpowercatグループによって開発されたツールキットであることが説明されています。このキットは、企業のcopilotの自動化テストを可能にします。Direct Line APIを通じてcopilotとチャットすることができる機能や、Azure Application Insightsによるcopilotの動作追跡、AI Builderの応答生成能力の確認、Dataverseによるデータ管理とインポート/エクスポート機能が含まれています。また、co-pilot Studio Kitを取得するには、aka.ms copilot Studio kitにアクセスし、指示に従ってダウンロードとインストールを行うことができました。
🔧 Azure Application Insightsの接続設定
この段落では、Azure Application InsightsのインスツルメンテーションキーとAPIキーを通じてcopilot Studio Kitに接続する方法が説明されています。まず、Azure PortalでApplication Insightsを作成し、必要なキーを取得します。次に、copilot Studio Kitの設定でこれらのキーを入力し、接続を作成します。これにより、テスト結果がAzure Application Insightsに記録され、テレメトリデータとして利用可能になります。
🛠 co-pilotのテスト設定と実行
この段落では、copilot Studio Kitを使用してcopilotのテストを設定し、実行する方法が説明されています。まず、copilotを作成し、トークンエンドポイントを設定します。次に、テストセットを作成し、テストケースを追加します。各テストケースには、テスト発話と期待される応答が含まれています。テスト実行を作成し、選択したcopilotとテストセットを指定してテストを開始します。テスト結果は、成功したかどうかを示し、詳細な情報や推奨事項が提供されます。
🗣️ テストケースの詳細設定
この段落では、テストケースの詳細設定方法が説明されています。テストケースの名前を設定し、応答の種類(トピック一致、添付ファイル一致、生成的な答え)を選択することができます。また、テストのタイムアウトや開始会話イベントの有無も設定可能です。期待される応答やテスト発話の例を入力し、テストケースを保存して実行します。
📊 テスト結果の分析とレポート
最後の段落では、テスト結果の分析とレポート機能について説明されています。テスト実行が完了すると、各テストケースの成功状況を確認できます。詳細をクリックすると、応答の内容や成功の理由、推奨事項などが表示されます。また、会話のトランスクリプトや管理者機能、監査履歴などの関連情報を確認することもできます。
Mindmap
Keywords
💡コパイロット スタジオキット
💡Direct Line
💡Azure Application Insights
💡AI Builder
💡Dataverse
💡アカウントの設定
💡テスト ケース
💡トピック
💡応答マッチ
💡テスト実行
Highlights
介绍了如何使用Co-Pilot Studio Kit来自动化测试Co-Pilot。
Co-Pilot Studio Kit由微软的Powercat团队开发,该团队服务于微软最大的客户。
Co-Pilot Studio Kit包含Direct Line API,允许用户与Co-Pilot进行聊天。
使用Azure Application Insights跟踪Co-Pilot的性能。
AI Builder集成,用于检查生成性AI响应。
所有功能都建立在Dataverse之上,这是Co-Pilot Studio的基础技术。
通过aka.ms/copilotStudioKit访问Co-Pilot Studio Kit。
下载Co-Pilot Studio Kit后,需要在Power Apps环境中导入解决方案。
创建Azure Application Insights连接以存储遥测数据。
创建Co-Pilot测试时,可以选择关闭Entra ID认证简化测试过程。
配置测试脚本时,可以设置测试集和测试运行。
测试脚本可以设置预期的响应和测试话语。
测试运行可以自动执行并验证Co-Pilot的响应。
测试完成后,可以查看详细的测试结果和推荐。
Co-Pilot Studio Kit提供了丰富的报告和信息,帮助评估Co-Pilot的性能。
观众可以通过评论请求更多关于测试工具的视频内容。
鼓励观众订阅频道并参与评论交流。
Transcripts
[Music]
in this video I'm going to show you how
to use co-pilot Studio kit for
automating test with your co-pilot so
let's start with what is the co-pilot
Studio kit and where did this thing come
from well the co-pilot Studio kit is
built by a group at Microsoft called
powercat and powercat is an extension of
the engineering team at copilot Studio
that works with all of the biggest and
baddest customers that Microsoft has to
interface with and they went out and
decided hey it would be great to have
this automation toolkit that would allow
you to be able to test your co-pilot and
be able to find out how is it performing
so let's explore a little more on what's
inside this kit so inside of the copilot
Studio kit you actually have this
ability to test a co-pilot through
something we call direct line and direct
line is the way that you can connect to
your copilot and be able to chat with it
so if you ever think about the web chat
experience as an example it uses the
direct line API so it's going to use
that in order to be able to do your test
now we're also going to have Azure
application insights that allows you to
be able to track and see what's going on
within your co-pilot itself we it also
has capabilities with AI Builder so if
you're going to have generative AI
responses you need a way to check those
because it's not going to come back with
a direct match so that's one of the
reasons why you'll see AI Builder as
part of this and then lastly it's all
built on top of dataverse dataverse is
the foundational technology that all of
copilot studio and power is built on top
of that allows for Effective data
storage and does a lot of work to make
it so that you get this really robust
Enterprise place to store and be able to
search and find all the different
components and also allows that import
export capabilities that you've seen
within copilot studio so let's first
start with how do I go and get ah hold
of the actual copilot Studio kit and the
way that you can get this is you can go
to
aka.ms copilot Studio kit and it will
automatically take you to this site that
you see here and in this site you'll see
that it gives you information about the
co-pilot Studio kit it gives you
information about uh the different ways
that it can work you can see here it has
things about exact match attachments
match all of these different
capabilities that this thing has out of
the box and I would encourage you to
continue to look at this because uh it
enhancements of this may continue to
happen uh by the power cap team over at
Microsoft so you'll see that even the
slide that I shared earlier is here you
can see a visual representation of it
you can see other things that are in
here you can also check the
prerequisites and you can get the in
installation instructions which is the
key thing for us right now so the first
step is going to be to go ahead and go
to the
aka.ms download copilot Studio kit you
can just click this link and it will let
you download this and just from there
you're going to get a zip file that's
downloaded on your computer now once
you've downloaded that what you're going
to need to be able to do is go into a
cop Studio enabled environment such by
going to make. power
apps.com so once you get into your power
apps environment that is created as part
of this if it wasn't already created
you're going to be able to select your
environment and you can do so right up
here and you can see that I really only
have one environment in this particular
implementation so I'm going to use that
particular environment as I go forward
now it's important that the environment
is the environment in which you are uh
have your co-pilot implementation that
you're looking to test so I'm going to
go ahead and go to Solutions
and now that I'm in Solutions what I
want to do is I want to go in and import
a
solution and when I go click import a
solution I can go to browse I can
grab that file I just downloaded which
is our ZIP file for the
implementation and click
next and then I'll click next again
and notice here that you're going to
need to create an implementation for
application insights now if you haven't
already uh created one you can go create
one of these pretty easily inside of
azure I'm going to go ahead and refer
you to a video where I went through
application insights and kind of the
configuration that's already on my
channel so if you haven't actually done
this before just know that you need to
go and create one of
those so I'm going to go ahead and click
on
create you'll see here that it needs a
connection name and I'm going to go
ahead and put in a connection name and
I'll just call this uh
co-pilot um
Studio
kit then we need our application ID and
our API key now we need to get that from
from the Azure
portal so I'll grab the instrumentation
key right here and go drop it
in and we're going to drop that into the
application ID right
here now we need to get the the actual
uh API
key if you don't already have an API key
you just simply go create one and in
this I would recommend giving a
description and basically what we want
here is we want read and write and I'll
just give it all of the different
permissions and I'll take this and I'll
copy the API key that was given to me
and we'll go back and we'll drop that
in so we'll drop that into the API key
here and then we'll create and by doing
this now we have created the connection
for the Azure application insights to
store like the Telemetry that's going on
and then we have data verse which is
where our solution is being imported to
and being stored and then we can just
click import
now you'll notice that right here it
actually says that it's going through an
import process this can take a second or
two depending on your environment but
just know that it's going to go ahead
and create this solution for
us once your solution is done importing
know that it will give you a
confirmation on the screen but let's go
ahead and click over into Apps and once
we go into apps now you can see the C-
pilot Studio kit has been installed and
I can hit the play button and once I hit
the play button it's going to ask me to
authenticate if you notice this hang uh
just know that you can click off of Home
click over on co-pilots and come back
this is what you should see once once
you have your solution up and running
and at this point we should be able to
start configuring a test or a co-pilot
that we'd like to test so let's first
click over on
co-pilots and we'll click on
new and we need to create a co-pilot
that we want to test so I'm going to
test the one that I built for the
Porsche
information that I have already today
you'll need to come in and DEC decide
which is the region and in this case I'm
just going to take the default
region then over here you can see who's
the owner you can select who it is that
is actually going to be the owner of
this particular base configuration for
this new co-pilot and then we need to
get a token end point so what what I'm
going to do is I'm going to use my
Porsche product information co-pilot
that I built in an earlier video however
I'm going to make some slight changes
here just to make this simple know that
there are different configurations
depending on what you want to do but
just for the Simplicity of the video I'm
going to go ahead and I'm going to
change the security on my
co-pilot and the way I'm going to do
that is go to authentication and I'm
just going to turn off the entra ID
authentication Now by doing this it
gives me the ability to be able to uh
connect and get an endpoint without
having to worry about setting up all the
security once we've done that I'll go
back over to channels and we'll go to
the mobile app and right here you can
get your URL for the token endpoint so I
I've copied that
over and I'm going to drop that into the
token endpoint here now once we get past
this we're also going to have things
like the channel security and things of
this nature and the documentation has a
lot of great information guys for you to
go check out so if you want to look at
all the different configurations that
you can have uh and such we you can go
do that now we can
also go in and put in like enrich with
Azure application
insites by doing this it basically
enriches the test results for generative
answers test with Azure application
insights Telemetry data because of what
we're going to do in this initial test
just to show you guys how this thing
works I'm not going to be enabling that
in our uh environment but that's a place
where if you're wanting to get some
additional information on what's going
on uh behind the scenes with the
generative AI features you
can now the other option here is that we
can enrich with conversation transcripts
and again
these have to do with pulling in the
actual conversation transcripts now
again we're not going to do any advanced
configurations just calling out a couple
of key things that you're going to see
and as you can see here we also have the
generative AI
configuration as well that you can turn
on and off and notice here that you have
to say what are you going to use and the
only option is AI Builder so if you want
to test generative AI features you'll
need to do this now I may run a separate
video that shows how to be able to do
this later you guys leave something in
the comments if you want to see that now
now that we've got this all we have to
do is hit save and
close and now that we've got that done
you'll see that now you have a co-pilot
that has been configured so if we go
here you'll also see that I need to be
able able to configure test scripts and
you can see we've done no test runs
currently today all we've done is
created a co-pilot that we want to be
able to test so the first thing that we
need to do is create a test set and a
test set is going to be like a
combination of things that it's going to
go run all the tests that we want it to
run so in this case what I need to do is
I need to give it a test set and I'll
just call this um Porsche simple
and I'm just going to hit save and
close now we've got ourselves a test
set and inside of this test set you'll
see that we have no actual test that
have been
run so now what I'm going to do is I'm
going to set up some test and we'll
start by clicking on new and then we're
going to need to give it a name and
things of this nature so so let's jump
back over to copilot Studio real quick
and let's look at some of the topics
here that we have and you can see I've
got these different things that I've
been able to test with and I'm going to
start with the fine local Porsche Center
and what I'm going to need to do is I
need to grab this response right here
which is what the copilot is going to
respond with and we want to come in and
say that we'll just call this uh find a
dealer and what we're going to say here
is that we want to use the simple test
set we also um want to say how long now
if you don't know how long your your
query is going to take because you might
have like a power automate flow or
something like that that's on the back
end but the point being is what you're
wanting to look for as a an input here
is how long you think that's going to
take now if you read the instruction
manual the instruction manual will tell
you if you don't know you should put 10
now anything below 10 seconds I actually
started to see some errors and things
like that uh and so just to make sure
that it has plenty of time to respond
I'm just going to do that and
then with that we also have a setting
here and this is the setting that says
send start conversation and the send
start conversation is basically whenever
you go into a co-pilot and you first
start a conversation a lot of times
you'll see it it'll say like hello this
is my name and this is what I do but
understand that the direct line API
doesn't have to do that you have to send
in an activity or an event that is the
start conversation event so if your test
needs to have the start conversation
event triggered in order to be able to
work then you can flip this but in the
case of this I don't really need that
and I'm going to put in that the
expected response is right here
now you'll even notice I have a typo in
here and we'll play with that as we
go um but what I'll do is and I also
want to go in and come up with what is
an example of something I might ask and
so here is the utterance that I want to
grab right here and we'll jump back and
we'll put the test utterance in right
here so now we have where it's going to
ask this question and it expects this
response and we will
go ahead and hit save and
close now you can see here I've got my
find a dealer now let's go ahead and add
one more in so I'll go ahead and create
another new one and let's go in and
let's look at a different topic and in
this case I'm going to say I want to get
accessories and things of that nature
like clothes or something of that nature
so I'm going to go ahead and copy the
response that comes back and we'll give
it a name and we'll say you know
find
clothing and we're going to keep this
response match again this is where you
can go and say whether you just want to
match the topic you want to see
attachments a generative anwers so and a
topic match think of it that what you're
looking for is you want to be able to
give it a test utterance you don't
really care what it says back you just
want to know that it went to the right
topic but in the response match we want
to see the exact response so in this
case I'm going to leave it response
match I don't need the this setting here
I want it to be in uh part of this test
set and then again we'll put in
10 and then we'll put the expected
response in right here let's go get a
test utterance you can see here this is
the test utterance that I want to
have and I'll drop this in right
here and we'll go ahead and save and
close and now that we've done this you
can see now we've got our different test
sets or our test queries that we want to
run and now we can actually go and
execute this test now we'll go into test
runs and you'll see here we don't have
any that have actually run you could
also come up here and you can see that
there's been no tests that have been
recently run but let's go ahead and say
okay I want to actually run a test so
keep in mind that when we have these
tests we might want to run multiple
instances of these tests to be able to
check to see like maybe we want to run
in on a certain day or something of that
nature so in this case what I'm going to
do is I'm just going to put in that this
will be um
video demo one and what I want to test
is I want to test this test set again
that'll test all of the things in that
test set and then I also want to choose
which co-pilot and I want to do the
Porsche co-pilot that we've got now once
I hit save and close on this what's
going to happen on the back end is
you're going to see that it'll start and
it'll automatically kick this off so
it'll take a second for it to start but
we'll see it kick off and start to run
in just a
second and we're looking right here at
the Run status is what we're looking for
and you can see now it is running now
again keep in mind that we had two
different things that we needed to run
and we gave it up to 10 seconds so it
may take a second or 20 seconds for this
thing to
update and you can see we have a
completed test now let's go ahead and
click in on the test and you can see
here that we had a successful actual
match on each of these different items
and if you want to you can even click in
on them and get more details you can see
this is the actual response here's the
the reason that it was successful
all of these different things you can um
and again you can have other things that
it will give you recommendations and
things of this nature if we had pulled
in the uh information on the transcript
the transcript information would have
shown here you can also see like admin
uh functions and things of this nature
on the test set there's also related
things like an audit history um as well
that you can look at if you uh set all
of these different things up and the
also even background processes and
things that are uh that we're taking but
again this was just a very simple test
of being able to use this but you can
see the power that this will give you as
you continue to go run and have all your
different tests and you can see all the
different things that you get coming out
of this and being able to give you some
really good reporting and information
around is this thing performing and
doing what you need it to do now if
you're interested in seeing more videos
about the testing uh tools to allow us
to be able to go say run test against
generative AI things and things of that
nature again please put that in the
comments below I really want to try to
focus content on things that you guys
want to see and as always thanks for
coming and watching our my videos I
really appreciate all the support that
you guys give me know that you can like
And subscribe to my channel and as
always you can go TR copilot Studio at
aka.ms TR copilot Studio
Weitere ähnliche Videos ansehen
5.0 / 5 (0 votes)