Azure Data Factory Part 6 - Triggers and Types of Triggers

Summary

This video from the Azure Data Factory series introduces triggers, which automate pipeline execution. It explains three types: scheduled, tumbling window, and event-based triggers. The presenter demonstrates how to create and configure these triggers in Azure Data Factory, emphasizing their role in batch and real-time processing. The tutorial also covers manual triggering and monitoring triggers, providing practical insights for automating data workflows.

Takeaways

• 🔧 Triggers in Azure Data Factory are mechanisms to execute a pipeline run automatically.
• 📅 There are three main types of triggers: scheduled, tumbling window, and event-based triggers.
• ⏰ Scheduled triggers run pipelines based on a fixed time schedule, like every day at 9 a.m.
• 🔄 Tumbling window triggers operate on a periodic interval and differ from scheduled triggers in their granularity, focusing on hours and minutes.
• 📂 Event-based triggers initiate pipeline execution in response to specific events, such as the creation of a file in a storage account.
• 🛠️ Creating a trigger involves setting a name, description, start time, time zone, and recurrence pattern.
• 🔄 Recurrence settings for triggers allow for customization of execution frequency, from minutes to months.
• 🔗 Tumbling window triggers can have dependencies on other triggers, indicating a sequence of operations.
• 🚫 Advanced options for triggers include the ability to set delays, manage concurrency, and define retry policies for failed pipeline executions.
• 📝 Annotations can be added to triggers for additional information or notes.
• 🔒 Security and monitoring are implied as part of the process, with a dashboard to view trigger statistics and status.
• 🔄 The video script demonstrates the creation and management of triggers within Azure Data Factory, emphasizing automation and customization.

Q & A

What is the primary purpose of triggers in Azure Data Factory?

Triggers in Azure Data Factory are used to automate the execution of a pipeline run. They determine when a pipeline execution should be initiated, allowing for the automation of batch processing and real-time processing tasks.

How many types of triggers are discussed in the video?

The video discusses three main types of triggers: scheduled trigger, tumbling window trigger, and event-based trigger.

What is a scheduled trigger and how does it work?

A scheduled trigger is a type of trigger that invokes a pipeline on a wall clock schedule. It can be set to run a pipeline at specific times, such as every day at 9 a.m., according to a predefined schedule.

Can you explain the concept of a tumbling window trigger?

A tumbling window trigger operates on a periodic interval. It is different from a scheduled trigger in terms of its granularity, which is limited to hours and minutes, and it does not include months, days, and weeks as options.

What is an event-based trigger and how does it differ from other triggers?

An event-based trigger responds to a specific event, such as the creation or deletion of a file in a storage account. It differs from other triggers as it does not rely on a time-based schedule but rather on the occurrence of an event.

What is the difference between 'trigger now' and creating a new trigger in Azure Data Factory?

'Trigger now' is a manual trigger that initiates a pipeline run immediately, whereas creating a new trigger sets up an automated process that will run the pipeline at specified intervals or in response to an event.

How can you set up a pipeline to run every 15 minutes in Azure Data Factory?

You can set up a pipeline to run every 15 minutes by creating a scheduled trigger, setting the recurrence to every 15 minutes, and specifying the appropriate time zone.

What is the role of dependencies in tumbling window triggers?

Dependencies in tumbling window triggers allow one trigger to be dependent on the completion or start of another trigger. This ensures that the pipeline execution is coordinated based on the status of other triggers.

What options are available for configuring an event-based trigger in Azure Data Factory?

For an event-based trigger, you can configure the Azure subscription, storage account, container name, blob path with regular expressions, and specify the event type, such as blob creation or deletion, that will trigger the pipeline execution.

How can you monitor the performance and status of triggers in Azure Data Factory?

You can monitor the performance and status of triggers in Azure Data Factory using the monitoring dashboard, which provides statistics for different types of triggers, including scheduled, tumbling window, storage events, and custom events.

What are some advanced options available for configuring triggers in Azure Data Factory?

Some advanced options for configuring triggers include setting a delay, managing concurrency to control how many triggers run simultaneously, defining retry policies for failed pipeline executions, and specifying a retry interval time.

Outlines

00:00

🔄 Introduction to Triggers in Azure Data Factory

This paragraph introduces the concept of triggers in Azure Data Factory, which are mechanisms to execute pipeline runs automatically. It explains the need for automation in scenarios like batch and real-time processing and outlines the three types of triggers: scheduled, tumbling window, and event-based. The speaker illustrates how triggers can be used to define when a pipeline should run, either on a set schedule or in response to specific events, and provides an example of setting up a manual trigger versus creating a scheduled or event-based trigger within the Azure Data Factory interface.

05:01

📅 Creating and Configuring Schedule-Based Triggers

The second paragraph delves into the specifics of creating a schedule-based trigger, which is set to invoke a pipeline on a predetermined time schedule. The speaker guides through the process of naming the trigger, setting the start time, time zone, and recurrence pattern. It also touches on the advanced options for configuring the trigger, such as dependencies on other triggers, delay settings, concurrency limits, and retry policies. The paragraph emphasizes the granularity of scheduling options available, from minutes to months, and the ability to define an end date for the trigger's operation.

10:04

🔄 Understanding Tumbling Window and Event-Based Triggers

This paragraph contrasts the tumbling window trigger with the schedule-based trigger, highlighting the differences in their granularity and use cases. It explains that tumbling window triggers operate on a periodic interval, focusing on hours and minutes, and can be set up with dependencies and advanced options similar to schedule-based triggers. The speaker then transitions to event-based triggers, which are activated by specific events such as the creation or deletion of a blob in Azure storage. The paragraph outlines the configuration steps for event-based triggers, including selecting the Azure subscription, storage account, container, and defining the events that will trigger the pipeline execution.

Keywords

💡Triggers

Triggers in the context of Azure Data Factory are mechanisms that initiate the execution of a pipeline automatically or based on specific conditions. They are pivotal for automating workflows and are central to the video's theme of explaining different methods to execute pipeline runs. For example, the script mentions setting up triggers to run a pipeline 'every one hour' or 'every night around eight o'clock'.

💡Pipeline

A pipeline in Azure Data Factory is a sequence of data movements and transformations that process data. The script discusses how triggers are used to run these pipelines without manual intervention, emphasizing the importance of pipelines in data orchestration and automation.

💡Scheduled Trigger

A scheduled trigger is a type of trigger that activates a pipeline based on a fixed schedule, such as daily or hourly. The script explains this concept by stating that one can 'schedule a trigger so, a trigger that invokes a pipeline on, wall clock schedule', illustrating how it can be used to run a pipeline at '9 a.m. every day'.

💡Tumbling Window Trigger

This trigger operates on a periodic interval and is different from a scheduled trigger in terms of its operational window. The script clarifies the distinction by pointing out that a tumbling window trigger deals with 'hours and minutes' as opposed to the broader 'months, days, and weeks' scope of a scheduled trigger.

💡Event-based Trigger

An event-based trigger initiates a pipeline in response to a specific event, such as the creation of a file in Azure storage. The script provides an example of an event-based trigger where 'whenever the blob is created', it triggers the pipeline, highlighting its use in reactive data processing scenarios.

💡ADF (Azure Data Factory)

Azure Data Factory is a cloud-based data integration service that allows for the creation of data-driven workflows for data movement and transformation. The script refers to ADF as the platform where triggers and pipelines are configured and managed, with the video focusing on how to set up and use triggers within this service.

💡Data Sets

Data sets in the script refer to the collections of data that are used as inputs and outputs for the copy activities within an Azure Data Factory pipeline. The script mentions creating 'source and sync data sets' which are essential for defining what data the pipeline will process.

💡Blob Storage

Blob storage is a service provided by Azure for storing unstructured data, such as text or binary data. In the script, blob storage is used as an example when setting up an event-based trigger, where the creation of a blob in a storage container acts as the triggering event.

💡Automation

Automation is the process of making a system or process run without manual intervention. The script emphasizes the role of triggers in automating the execution of pipelines in Azure Data Factory, eliminating the need for manual initiation of processes.

💡Recurrence

Recurrence in the context of triggers refers to the frequency at which a trigger will activate a pipeline. The script explains that one can configure triggers to recur 'every 15 minutes' or at other specified intervals, demonstrating how recurrence is configured for both scheduled and tumbling window triggers.

💡Dependencies

Dependencies in the script refer to the conditional activation of a trigger based on the completion or initiation of another trigger. This concept is introduced when the script discusses advanced options for configuring triggers, allowing for complex orchestration of pipeline executions.

Highlights

Introduction to triggers in Azure Data Factory and their role in executing pipeline runs.

Explanation of how triggers can automate the running of pipelines without manual intervention.

Types of triggers in Azure Data Factory: Scheduled, Tumbling Window, and Event-based triggers.

Scheduled triggers for running pipelines at specific times based on a wall clock schedule.

Tumbling window triggers for operating on a periodic interval, differing from scheduled triggers.

Event-based triggers that respond to specific events, such as file copy completion.

Demonstration of creating a demo pipeline in Azure Data Factory with copy activity.

Creating and managing triggers directly from the Azure Data Factory interface.

Options to trigger a pipeline manually or to create a new scheduled, event-based, or tumbling window trigger.

The process of creating a new trigger with a specific start time and recurrence.

Configuring the granularity of trigger recurrence, such as minutes, hours, days, or weeks.

Setting an end date for when a trigger should no longer execute the pipeline.

Adding annotations and configuring dependencies between different triggers.

Advanced options for triggers, including delay, concurrency, and retry settings.

Monitoring dashboard in Azure Data Factory for viewing trigger statistics and types.

Practical example of creating an event-based trigger for blob storage events.

Custom events configuration for more specific or complex event-based trigger scenarios.

Encouragement to subscribe to the channel for more informative sessions on Azure Data Factory.

Transcripts

hi everyone welcome back to azure data

factory video series part six so in this

section we'll be discussing about uh

triggers and different types of triggers

so without any further ado let's get

into the concept triggers so triggers

are basically just a way that you can

execute a pipeline run

now let's say you have created your

pipeline and that is ready to be

deployed in the protection now you

wanted to run the pipeline so it's it's

it's

it's not a job that you go there and you

run it manually every day for example

you have batch processings and you have

real time processing for example and you

are instructing

to the adf that you run this pipeline

every one hour or maybe every night

around eight o'clock end of the day

reports for example for end of the day

batch right so for that you need to have

an automatic process in place so

basically triggers are used to automate

the pipeline run so you you can

advise or you can define that to run a

specific pipeline on a specific time

and you can also schedule that so

triggers represent

a unit of processing that determines

when a pipeline execution needs to be

kicked off so mainly this is when a

pipeline execution needs to be kicked

off so basically you are defining or you

are telling to adf using triggers when

you want to execute a pipeline in an

automatic fashion okay and what are the

different types of triggers you have

basically you have three types of

triggers scheduled trigger and then you

have tumbling window trigger and event

based trigger so let's try to see one by

one so what is schedule trigger so i

think the name itself says it's

basically self explanatory so basically

you are trying to schedule a trigger so

a trigger that invokes a pipeline on

wall clock

schedule basically i say that every day

morning 9 a.m you trigger my pipeline so

that comes under schedule trigger then

you have another type which is called

tumbling window trigger so a trigger

that operates on a periodic interval

basically that is called a tumbling

window trigger so there is a slight

difference between schedule trigger and

tumbling window trigger we will look

into that and then you have a third type

of trigger which is called event based

trigger

so basically

this is nothing but a trigger that

responds to an event so let's say i do

not want my pipeline to run every day or

every one hour i want my pipeline to

respond or i want my pipeline to run

based on a specific event

event could be

copying a file from one location to

another location once that file is being

copied to specific location then you

trigger my pipeline basically my

pipeline picks that file and sends it to

another location that could be a use

case okay so basically these are the

three types of triggers we have

then if i

look into

the azure data factory for example

so if you remember that we've used uh

we've created this demo pipeline one in

section three creating uh your first

azure data factory pipeline there we've

used this copy activity to copy from

source

right test data input and also we have

created a sync data sets basically we've

created two data sets source and sync

and we used copy data

pipeline uh

maybe copy data activity and create a

demo pipeline basically to copy the file

from source to sync

right now we have created the pipeline

so let me close this so that we have

more space so now i want to run this

pipeline so what are the options i have

so if you see

you have a option called add trigger

here

right so if i click on this guy so

basically if i click on this guy i have

two options

either i can trigger it now

so basically trigger now which is like

manual trigger or i can create a trigger

any one of the type like schedule event

based or tumbling window i can create

any of that type of trigger

and i can tell or i can define saying

that you run this automatically every

day for example right so you can also

have a manual trigger if i click on

trigger now basically so it says that

okay

it takes the last published

configuration that's fine if i click on

ok

so basically now it's running uh the

pipeline so basically this pipeline does

copy a file from source sensing so this

is a manual trigger but i do not want to

manually trigger this every day in the

morning that's not my job i want to

automate this process so for that you

could

create a new trigger here then choose

the trigger basically i do not have any

triggers created okay that is succeeded

okay so i do not have any triggers

created either i can click new which

will allow me to create a new triggers

or

i could go to manage and then i could

also create a new triggers from this

section so you have multiple places

where you can create the triggers and if

i go to monitor

just to show you something here you have

a monitoring

dashboard with different types of

trigger so if you see here if you click

on odd basically if you have any

triggers you get the statistics of those

triggers here but you have schedule

based triggers

tumbling window storage events and

custom emails basically these two come

under event-based triggers okay so

currently we do not have any triggers

let's try to create one trigger and see

how to do that

so if i come here if you are under

triggers then you click on new

and then it asks you to create the name

of the trigger okay

so i put a

schedule hyphen

okay it doesn't like hyphen okay so you

have a specific naming convention then i

say schedule underscore trigger

right and then add some description here

so basically i wanted to show you what

are the different options you have so

when it comes here on the type basically

you have three different four different

types schedule tumbling and event and

event has two different types so now i

want schedule okay you select that then

it asks you what is the start time for

this so when you want to choose when you

want to start this trigger so basically

you can fill in the start time then you

can also give the time zone as well so

which time zone you want

the trigger to follow and then you can

say that what is the recurrence time so

whether you want to run it like every

day or every two days every 15 minutes

okay so if you click on this guy here

you have couple of other options so

basically you can configure whether you

want to run it at minutes or hours days

and weeks right this is the granularity

and you also have months as

so let's put it for minutes so what i'm

trying to say is you run this every 15

minutes you run the pipeline every 15

minutes you can also specify the end

date when you want to end this complete

pipeline you do not want to run it after

the specific end date you can also add

some annotations and here you have a

check box basically it says that

if you enable this check box and then

you click on ok it's you are saying that

you start the trigger immediately after

creating this trigger so basically you

don't have to enable this so

this is start date and every 15 minutes

you run this trigger so this is schedule

based trigger okay so then i have

tumbling trigger

so

when it comes to tumbling trigger

if i select tumbling trigger let's try

to see what are the different options so

i think you need to understand the

slight difference between tumbling

trigger and also schedule based trigger

so basically the start date is same for

both and then the recurrence is also

same but if i click on these units

basically the granularity here for

tumbling is

for hours and minutes whereas for

schedule it was still months days and

weeks right this is a major difference

between these two different types of

triggers and then if you click on this

advanced option you also have an option

to add dependencies for example if i

click on this so you could also choose

if there is any other trigger

let's say i am saying that

you depend this trigger on another

trigger once the other trigger is

finished or started then you

run this trigger so basically i'm adding

a dependencies here with different

triggers so you also have an option to

do that and then if i click on advanced

and you can also configure few other

options you can add delay you can have

concurrency how many triggers you want

to run simultaneously

if the pipeline fails you

can also say that retry for us three or

four times again right because for

network related sometimes if you're

running this under vnet if if the

network is down it could also fail it

doesn't mean that the pipeline is wrong

you could retry as well and what is a

retry interval time as well in seconds

you can also define that you can add

annotations and this is the same star

trigger on creation basically this is a

tumbling trigger let's also look into

the storage event so basically event

based trigger

so event

trigger

yeah so here with for event based

triggers so basically the

the options you have here to configure

is so you select azure subscription okay

you you fill in all the information what

is the storage account and things like

that maybe let's try to fill in this

information and then i have different

storage accounts okay let me fill in

this information you also need to

provide the container name

okay for example container is not

written

let me look into the container name what

i have

within my azure storage account

so if i go to

portal.azure.com i just want to check my

storage accounts and container names for

example if i go to the storage accounts

and then within this storage account i

just wanted to check what is the

container i have okay i have source and

sync

maybe i can give um

sou or ce source

and this okay this is in the same in the

puff the correct format it's expecting

now

you need to give

block path let's say blob path begins

with blob path ends with you can also

give those a regular expressions here

okay so what is the event here right

choose which events are associated with

this trigger so

when you want to run this trigger what

is the event you are defining so

whenever the blob is created you trigger

this pipeline or whenever the blob is

deleted you create this pipeline

basically this is the event i'm giving

here right so this could be one use case

if there is a new blob created then you

just trigger this

this trigger is

based on the event

which is block creation or population

and then you have another type which is

a custom events so more or less you have

the same options here and then you it is

basically customized so when it comes to

storage accounts it's specific to

storage but this could be more

customized right and you have couple of

options here which you can configure as

well and you can also add some advanced

filters

and even types you can also add you can

also give some event types but uh for

custom events we will look into it with

an example when we do more practical

sessions

with the mapping data flows and with

different control

flow activities but for now uh you have

to understand that there are types of

triggers right and how to configure them

basically

and if i go back to

my session yeah so basically these are

the three different types of triggers we

have in azure data factory so i hope uh

you like this session or

just this video if you think you've

gained any knowledge out of this by this

i kindly request you to subscribe our

channel and also share it with your