How Copilot for Microsoft 365 can work with your external data

Summary

The script discusses two methods to safely integrate external data into Microsoft 365 Copilot for AI-generated responses. The first is using Microsoft Graph connectors to pre-index data with Semantic Index for Copilot, enhancing the relevance of AI responses. The second involves managed plugins for real-time API data retrieval and optional data writing back. Both methods ensure security and privacy, and can be used together. An example demonstrates resolving a DNS issue by combining a Jira plugin and an intranet Graph connector. The script also explains how Semantic Index works on SharePoint and individual user data to improve search and response accuracy. Finally, it guides on setting up connectors and plugins for external data integration.

Takeaways

• 🔌 Microsoft 365 Copilot can integrate external data sources to enhance AI-generated responses through two primary methods.
• 📚 The first method uses Microsoft Graph connectors to pre-index read-only data for retrieval with prompts, ensuring relevant AI responses.
• 🔧 The second method employs managed and trusted plugins for real-time API-based data retrieval, with optional data writing capabilities.
• 🛡️ Both options adhere to Microsoft's security, compliance, and privacy standards, respecting organizational data boundaries.
• 🔄 Demonstrated in the script, connectors and plugins can work together to respond to a single prompt, as shown with a help desk scenario.
• 🔎 Semantic Index for Copilot works by default with internal Microsoft 365 data, correlating activity signals to retrieve relevant information.
• 📈 Semantic Index operates on two levels: indexing information within SharePoint and creating a separate index for individual user documents and emails.
• 📝 The dual approach of Semantic Index increases the relevancy of AI-generated results, tailoring them to the user's approved access to information.
• 🔍 An example is given to compare SharePoint search results with and without Semantic Index, highlighting the improved search experience.
• 🔗 To utilize Semantic Index with external data, Graph connectors can be used without changes, or new connections can be set up through the Microsoft 365 Admin Center.
• 📜 Plugins offer a flexible way to retrieve data and perform actions without pre-indexing, with three types available: OpenAI plugins, Microsoft Teams message extensions, and Power Platform connectors.
• 🛠️ Plugins are developed using a shared code base with similar packaging and components, and can be managed through the Microsoft 365 Admin Center for access control.

Q & A

What are the two primary options to safely bring external information into Microsoft 365 Copilot experiences?

The two primary options are using Microsoft Graph connectors to integrate data sources with Semantic Index for Copilot, which pre-indexes defined read-only information for retrieval, and using managed and trusted plugins such as OpenAI plugins, Microsoft Teams message extensions, and Power Platform connectors, which optimize for real-time API based data retrieval and can optionally write back data.

How does Semantic Index for Copilot work with Microsoft 365 Copilot?

Semantic Index for Copilot works by correlating activity signals with the Microsoft Graph to retrieve indexed information uniquely relevant to the user. It presents a combination of the user's prompt and the retrieved information to the large language model to generate an informed response.

What is the purpose of using Microsoft Graph connectors?

Microsoft Graph connectors are used to integrate external data sources with Semantic Index for Copilot, allowing for the pre-indexing of defined read-only information that can be used to drive the most relevant AI-generated responses.

What types of plugins can be used with Microsoft 365 Copilot?

The types of plugins that can be used with Microsoft 365 Copilot include OpenAI plugins, Microsoft Teams message extensions, and Power Platform connectors.

How do plugins differ from Microsoft Graph connectors in terms of data retrieval?

Plugins optimize for real-time API-based data retrieval without pre-indexing information, and they can also perform additional actions such as writing back data, depending on user permissions.

Can both Microsoft Graph connectors and plugins work together when responding to a single prompt?

Yes, both Microsoft Graph connectors and plugins can work together when responding to a single prompt, allowing for a more comprehensive and integrated response.

What is an example of how a help desk technician might use Microsoft 365 Copilot?

A help desk technician might use Microsoft 365 Copilot to respond to users facing an issue with DNS by submitting a prompt to find who's impacted and to write an email with instructions that reference an internal knowledge base article.

How does the Semantic Index improve the search experience across Microsoft 365?

Semantic Index improves the search experience by mapping semantic signals such as people relationships and recent activities surrounding the content the user has access to, helping to determine the intent of the search and formulate a more relevant answer.

What is the process for setting up a connection to external data using Microsoft Graph connectors?

To set up a connection to external data using Microsoft Graph connectors, you need to enable the service or start a trial from the Microsoft 365 Admin Center, and then add a connection. You can define the site, who can search the data source, and whether connector results are shown in search.

How can developers make plugins available in the Microsoft 365 environment?

Developers can make plugins available in the Microsoft 365 environment by choosing from existing plugins in the store or by uploading their own plugin packages directly in the Teams Admin Center from the Microsoft 365 Admin Center.

What is the role of the YAML file in the context of plugins?

The YAML file defines the API for connecting to the external data or service, as well as any actions it can take on behalf of the user. It also includes natural language descriptions of the API operations and responses, which are used by the LLM to assess what the plugin can do and help respond to a prompt.

Outlines

00:00

🔗 Integrating External Data with Microsoft 365 Copilot

The paragraph discusses two primary methods for safely incorporating external data into Microsoft 365 Copilot to enhance AI-generated responses. The first method involves using Microsoft Graph connectors to integrate data sources with Semantic Index for Copilot, which pre-indexes information for retrieval. The second method utilizes managed and trusted plugins, such as OpenAI plugins, Microsoft Teams message extensions, and Power Platform connectors, which optimize for real-time API-based data retrieval and can optionally write back data. Both options adhere to Microsoft's security, compliance, and privacy standards, and can work together in response to a single prompt. An example is provided where a help desk technician uses both a connector and a plugin to address a DNS issue, demonstrating how Copilot generates a response by pulling information from these sources and how plugins can present adaptive cards for more information and allow for actions like editing a Jira issue directly within Teams.

05:01

🛠️ Setting Up External Data Integration with Microsoft 365 Admin Center

This paragraph provides a step-by-step guide on setting up external data integration for Microsoft 365 Copilot. It starts by explaining that existing Graph connectors will work without changes, but if not set up, they need to be enabled from the Microsoft 365 Admin Center. The process involves adding a connection to external data sources, such as enterprise websites, with options to define the site, search permissions, and visibility in search results. Advanced settings allow for more granular configurations like meta tags and semantic property labels. Once configured, integrated results can be seen using SharePoint search. The paragraph then moves on to discuss plugins, explaining how they work by retrieving information in real-time without pre-indexing. It demonstrates a plugin's use by showing how Copilot uses it to find VPN-related tickets in a Jira system. The paragraph also covers the technical aspects of plugins, including their code structure, the use of YAML files for API definitions, and how they can be made available in the Microsoft 365 environment through the Admin Center. The summary concludes with a mention of where to learn more about these integration options.

Keywords

💡Microsoft 365

Microsoft 365 is a suite of productivity and collaboration tools offered by Microsoft, which includes services like email, online meetings, file storage, and collaboration tools. In the context of the video, it is the platform on which the Copilot experiences are built, and it is where external data is being integrated to enhance AI-generated responses.

💡Microsoft 365 Copilot

Microsoft 365 Copilot is a feature that integrates AI capabilities into the Microsoft 365 suite to assist users with tasks and provide intelligent insights. The video discusses how to bring external data into the Copilot experiences to improve the relevancy and usefulness of AI-generated results.

💡Microsoft Graph connectors

Microsoft Graph connectors are tools that allow integration of external data sources with Microsoft 365. They are used to pre-index information for retrieval with prompts, which helps in driving AI-generated responses. In the video, they are one of the two primary options discussed for safely bringing external information into Copilot experiences.

💡Semantic Index

Semantic Index is a feature that works with Microsoft 365 Copilot to index and retrieve information based on user activity and relevance. It operates on two levels: within SharePoint and for individual users' email and documents. The video explains how Semantic Index increases the relevancy of AI-generated results by correlating activity signals with the user's prompt.

💡Plugins

Plugins in the context of the video refer to managed and trusted extensions that can optimize real-time API-based data retrieval and optionally write back data. They are an alternative to pre-indexing information and can work alongside connectors. The video provides examples of different types of plugins, including OpenAI plugins, Microsoft Teams message extensions, and Power Platform connectors.

💡OpenAI plugins

OpenAI plugins are a type of plugin that allows for integration with external data sources and services. They enable real-time data retrieval and can perform actions on behalf of the user, as demonstrated in the video with the Jira system for issue tracking.

💡Microsoft Teams message extensions

Microsoft Teams message extensions are a type of plugin that can be used to extend the functionality of Microsoft Teams. They allow users to interact with external services directly from within Teams, as shown in the video where a help desk technician uses them to respond to users facing DNS issues.

💡Power Platform connectors

Power Platform connectors are part of the Microsoft Power Platform and allow for integration with various data sources and services. They enable users to retrieve and manipulate data in real-time, which is highlighted in the video as one of the plugin types that can be used with Microsoft 365 Copilot.

💡Jira

Jira is a popular issue tracking and project management tool. In the video, it is used as an example of an external system that can be integrated with Microsoft 365 Copilot using plugins. This allows users to retrieve and update information related to issues directly from within the Microsoft 365 environment.

💡Adaptive card

An adaptive card is a user interface component that can be used to display information and allow users to interact with it. In the video, it is mentioned as a feature of the plugin that presents more information about a Jira ticket and allows the user to take actions like editing an issue directly within Microsoft Teams.

Highlights

Microsoft 365 Copilot can integrate external data sources for AI-generated responses.

Two primary options for integrating external data: Microsoft Graph connectors and managed plugins.

Microsoft Graph connectors pre-index information for retrieval with prompts.

Plugins optimize for real-time API-based data retrieval and can write back data.

Both options adhere to Microsoft's security compliance and privacy standards.

Demonstration of using a connector and a plugin simultaneously in Microsoft Teams.

Semantic Index for Copilot correlates activity signals to retrieve relevant information.

Semantic Index works on SharePoint and individual user data for increased relevancy.

Comparison of SharePoint search results with and without Semantic Index.

How to use Semantic Index for external data with Graph connectors in Microsoft 365.

Plugins allow for direct real-time data retrieval and user permission-based actions.

Types of plugins: OpenAI plugins, Microsoft Teams message extensions, and Power Platform connectors.

Plugins use a shared code base and open API spec for cross-platform functionality.

Plugins can be made available in the Microsoft 365 environment through the Admin Center.

Control over who can access plugins and their underlying data is maintained by the admin.

Resource for more information on Microsoft 365 Copilot extensibility.

Transcripts

(bright sound)

- A lot of the information you work

with today might sit outside of Microsoft 365 locations.

So how can you safely bring that information

into your Microsoft 365 Copilot experiences

to ground generative AI results?

There are two primary options.

First, you can use Microsoft Graph connectors

to integrate the data sources you want

with Semantic Index for Copilot.

This option pre-indexes defined read-only information used

for retrieval with your prompts

to drive the most relevant AI generated responses.

The second option uses managed and trusted plugins

which can be a few different types,

including OpenAI plugins,

Microsoft Teams message extensions,

and Power Platform connectors.

And with plugins instead of pre-indexing information,

they optimize for real-time API based data retrieval

and they can optionally even write back data.

Both options follow Microsoft's

comprehensive security compliance and privacy approach

while respecting the boundaries that you define

within your organization's data services.

And both types can even work together when responding

to a single prompt.

In fact, let me show you an example using a connector

and a plugin simultaneously.

Let's say I'm a help desk technician

in Microsoft Teams and I want to respond

to all users facing an issue with DNS.

I'll start by submitting a prompt

to help me find who's impacted

and to write an email

for me that I can send them with instructions

that reference our internal knowledge base article.

Copilot has taken my prompt

and is generating a response, in this case,

by pulling in information from both a plugin configured

for Jira which we're using for issue tracking,

as well as a Graph connector for an intranet site

which we're using as a knowledge base for our help desk.

And it responds with a matching DNS related ticket

along with the steps for flushing DNS

and even an email it authored that I can use to respond

to our user with the matching open ticket.

You'll also see at the bottom

of the response that both the plugin's Jira ticket

and the connector's intranet KB article are referenced.

Additionally, if I scroll up

and click on the Jira ticket details, you'll see

that my plugin also presents an adaptive card

to show more information

and it even allows me to take action.

I'll hit "Edit issue" and right from within Teams,

I can change the record in Jira, for example,

if I sent the email

to the user already and resolved the issue,

I can update the ticket status from "To Do," to "Done."

And by hitting "Save," it'll record that change

to our Jira system.

So we just saw how you can extend Microsoft 365 Copilot

to external data sources coming

into the Microsoft Graph using connectors

and two-way integration using plugins.

Next, let me show you what was going

on behind the scenes and how each one works.

The first option where we referenced our intranet article,

leveraged Semantic Index for Copilot

which works with Microsoft 365 Copilot

by default over internal Microsoft 365 data.

When you submit a prompt,

Copilot brings in the Microsoft Graph

with a Semantic Index to correlate activity signals,

retrieving indexed information uniquely relevant to you.

It presents the combination

of your prompt with what was retrieved

to the large language model

to generate an informed response.

Importantly, Semantic Index works on two levels.

By default, it indexes information

within SharePoint inside your organization's tenant.

And second, each user also has their own separate index

for email and their individual important documents

like files shared with them, where they've commented

or were at mentioned, and more.

This two-pronged approach is what increases relevancy,

allowing Microsoft 365 Copilot

to orchestrate AI generated results that are unique

to you as a user, again,

based on your approved access to information.

And a great way to demonstrate Semantic Index for real is

by showing two sets of search results in SharePoint,

one without leveraging the Semantic Index and one with it.

Here we're searching on Project Melbourne Shiproom.

Notice we have a typo.

On the left, without the Semantic Index,

you can see results are retrieved based

on keywords and ranking as well as last modified date

for relevance and a file summary for context

but it's not the specific information we're looking for.

Conversely, on the right with the Semantic Index,

instead of a list of links,

notice the informative answer right at the top.

You can see that the index has mapped semantic signals such

as people relationships

and recent activities surrounding the content

the user has access to,

to help determine the intent

of your search and formulate an answer.

So you can see how Semantic Index

improves the search experience across Microsoft 365

and that includes Microsoft 365 Copilot.

So how do you take advantage

of Semantic Index for your external data,

like we saw in our example?

First, if you already use Graph connectors in Microsoft 365,

they will work without any changes.

If not, before you connect to external data,

from the Microsoft 365 Admin Center,

you do need to enable the service

or start a trial and from there you can add a connection.

There are options for enterprise websites

and a few other common data sources listed here as well.

For example, if I open ServiceNow,

you'll see that there are three options

for ticketing, knowledge base, and catalog, but in my case,

I'll set up an enterprise website

and this will show me a few simple options

to define the site, who can search the data source,

and whether connector results are shown in search.

Then for even more options, I can go

into advanced settings for more granular configuration

with options to define meta tags, specific exclusions,

semantic property labels, and customizations to the schema.

And once you have a connection up and running,

you'll be able to see integrated results

using SharePoint search with the connected site.

In this case, I'll just perform a simple keyword search

for troubleshooting instructions and results are found

in SharePoint locations as well as from the connected site.

And here is the DNS article that was used as part

of the retrieval process to respond

to my prompt from before.

Beyond Microsoft Graph connectors with Semantic Index,

let's look at our second option

for data retrieval with Copilot, using plugins.

Again, starting with how they work.

So with a plugin configured, when we submitted the prompt,

Microsoft 365 Copilot leveraged that plugin

to retrieve information directly from its source,

again, in real time without pre-indexing the data.

It was also able to perform additional actions on behalf

of the user and with respect to their permissions,

like changing the status of the ticket.

This time, I'll demonstrate the plugin by itself

without connection to our knowledge base.

In Microsoft Teams, I'll write a new prompt

to look up VPN related tickets in our Jira system

and submit it.

Then Copilot responds using that information

and finds two tickets in the system related to VPN issues.

And as I mentioned, plugins work without pre-indexing

and there are three types.

These can be OpenAI plugins,

Microsoft Teams message extensions,

and Power Platform connectors.

Now I'll explain how plugins work

with a short tour of their code.

All three of these types of plugins are similar

with similar packaging and components.

This is an example of a Teams app manifest file

that uses NuGet Package Manager to find app packages

and their details.

You can see that it refers to a YAML file.

And in that file, is where the API is defined

for connecting to the external data or service

as well as any actions it can take on behalf of the user.

Also note the natural language descriptions

of the API get operation and its responses.

These are used by the LLM

to assess what the plugin can do

to help respond to a prompt.

And because these are portable components and can be used

for OpenAI, Teams, and Power Platform, as a developer,

you'll be able to maintain a single shared code base.

Each will leverage the same underlying open API spec.

Then to make plugins available

in your Microsoft 365 environment,

you'll do this from the Microsoft 365 Admin Center

in the Integrated Apps page.

Here you can choose from existing plugins

from the store, just like you would get other apps today.

You can also upload your own plugin packages directly

in the Teams Admin Center.

Either way you add the plugin, you're in control

of who can access the plugin and its underlying data.

So those are the two main options available to you

for safely bringing in data external to Microsoft 365

into your Microsoft 365 Copilot experiences

to ground generative AI results.

To learn more, check out aka.ms/CopilotExtensibility

and keep following Microsoft Mechanics

for the latest in tech updates, thanks for watching.

(bright sound)