How to Start an AWS Step Function Workflow From Lambda | STEP BY STEP GUIDE

Summary

This AWS tutorial demonstrates how to initiate a Step Function from a Lambda function. The video guides viewers through creating a Step Function instance and a Lambda function in Python, setting the necessary permissions, and writing code to interact with Step Functions. It showcases the process of defining a unique transaction ID using the UUID library, crafting a JSON input, and invoking the Step Function's start execution API. The tutorial concludes with testing the Lambda function and verifying successful Step Function instances.

Takeaways

• 😀 The video demonstrates how to start a Step Function instance from a Lambda function in AWS.
• 🔗 The presenter has already created a Step Function instance and predefined some AWS Step Language (ASL) code for a simple task involving an SQS queue.
• 📝 The ASL code takes an input and communicates with the SQS 'SendMessage' API, posting a JSON object with a transaction ID and type derived from the Step Function input.
• 🆔 The Step Function input includes a unique transaction ID and a type, both of which are strings.
• 👉 The presenter copies the Amazon Resource Name (ARN) of the Step Function for use in the Lambda function creation process.
• 🤖 The Lambda function named 'step function invoker' is created in Python 3.6 or 3.7, with permissions set to access the Step Function.
• 🔑 It's important to use the least privilege model for permissions, but the presenter uses full access for simplicity in this tutorial.
• 💾 The Lambda function's code uses the 'boto3' library to interact with AWS services, specifically the Step Functions client.
• 🔄 A unique transaction ID is generated using the 'UUID' library to ensure the Step Function instance has a unique key.
• 📑 The input for the Step Function is defined as a JSON object, with the transaction ID and type hardcoded as 'purchase' in the example.
• 🚀 The 'start_execution' API of the Step Functions client is called with the state machine ARN, a unique name (transaction ID), and the JSON payload as parameters.

Q & A

What is the main topic of the video?

The main topic of the video is demonstrating how to start a Step Function instance from a Lambda function in AWS.

What is ASL in the context of the video?

ASL in this context refers to Amazon States Language, which is used to define the state machine for Step Functions in AWS.

What is the purpose of the SQS SendMessage API in the Step Function?

The SQS SendMessage API is used in the Step Function to post a JSON object containing a transaction ID and type to an SQS queue.

What are the inputs required for the Step Function in the script?

The Step Function requires an input containing a transaction ID and a type, both of which are strings.

What is the ARN used for in the Lambda function?

The ARN (Amazon Resource Name) is used to reference the Step Function that the Lambda function will interact with.

Why is it important to use the least privilege model when creating an IAM role for the Lambda function?

Using the least privilege model ensures that the Lambda function has only the permissions it needs to perform its tasks, enhancing security by minimizing potential access to sensitive resources.

What is the role of the UUID library in the Lambda function code?

The UUID library is used to generate a unique transaction ID for each Step Function instance, ensuring that each instance has a unique identifier.

How does the Lambda function interact with the Step Functions client?

The Lambda function uses the boto3 library to create an instance of the Step Functions client and then calls the start_execution API to start a Step Function instance.

What is the significance of the 'name' field in the start_execution API call?

The 'name' field serves as the unique identifier for the Step Function instance, allowing AWS to distinguish between different instances.

How does the Lambda function handle the input for the Step Function?

The Lambda function constructs a JSON object with the required transaction ID and type, converts it to a string using the JSON library, and passes it as the input to the start_execution API call.

How can you verify that the Lambda function has successfully started the Step Function instances?

You can verify the success by checking the logs in the Lambda function and by looking at the Step Functions console to see the instances and their progress.

Outlines

00:00

🚀 Starting a Step Function from Lambda

This paragraph introduces the video's objective to demonstrate initiating a Step Function instance from within a Lambda function. The presenter has already set up a Step Function instance and predefined ASL code for a task that interacts with the SQS 'send message' API. The input for the Step Function consists of a transaction ID and a type, both of which are strings. The presenter emphasizes the importance of using unique identifiers for Step Function instances and proceeds to show how to create a Lambda function with the necessary permissions to start the Step Function.

05:01

🛠 Creating and Testing the Lambda Function

The second paragraph details the process of creating a Lambda function named 'step function invoker' in Python 3.6 or 3.7. The function requires specific permissions for accessing the Step Function service, and the presenter advises using the least privilege model. The Lambda function's code is written in an external editor, with the presenter explaining the need for libraries such as 'json', 'moto3', and 'uuid'. The code generates a unique transaction ID using the 'uuid' library and constructs a JSON input object for the Step Function. The Lambda function then uses the 'boto3' Step Function client to start the execution of the Step Function, passing the state machine ARN, a unique name, and the JSON payload. The paragraph concludes with instructions on testing the Lambda function by creating a test event and verifying successful execution in the Step Functions console.

Keywords

💡AWS

AWS stands for Amazon Web Services, which is a comprehensive cloud computing platform provided by Amazon. It offers a wide range of services including computing power, storage, databases, and more. In the context of the video, AWS is the platform where the user is demonstrating the integration of various services like Lambda and Step Functions.

💡Lambda

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. It automatically scales your application by running code in response to events. In the video, the user is showing how to integrate Lambda functions with Step Functions for event orchestration.

💡Step Functions

AWS Step Functions is a service that enables you to coordinate multiple AWS services into serverless workflows so you can build and update apps quickly. The video is focused on demonstrating the process of starting a Step Function instance from a Lambda function.

💡Event Orchestration

Event orchestration refers to the process of managing the flow of events in a system to ensure that they are processed in a specific order or manner. In the script, the user explains how to use Lambda to orchestrate events within Step Functions for a more automated workflow.

💡ASL

ASL stands for Amazon States Language, which is a JSON-based language specific to AWS Step Functions that lets you define your serverless workflows. The script mentions predefined ASL as part of the setup for the Step Function.

💡SQS

SQS stands for Amazon Simple Queue Service, a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. The video discusses using SQS to send messages as part of the Step Function's task.

💡JSON

JSON is a lightweight data interchange format that is easy for humans to read and write and for machines to parse and generate. In the video, JSON is used to structure the input for the Step Function, containing a transaction ID and a type.

💡IAM Role

An IAM Role in AWS is an AWS identity that you can use to permit the AWS Services to access resources securely. The script mentions using an existing IAM role with Step Function access permissions for the Lambda function.

💡UUID

UUID stands for Universally Unique Identifier, which is a 128-bit number used to uniquely identify some object or entity. In the script, the UUID library is used to generate a unique transaction ID for the Step Function instance.

💡Start Execution API

The Start Execution API is a method provided by AWS Step Functions to initiate the execution of a state machine. The video demonstrates using this API within a Lambda function to start a Step Function instance with specific input.

💡Moto3

Moto3 is a library that allows you to use boto3, the Amazon Web Services (AWS) SDK for Python, to create mock AWS services for testing without actually making API calls to AWS. The script mentions using the Moto3 library, presumably for testing the Lambda function's interaction with AWS services.

Highlights

Introduction to integrating AWS Lambda with Step Functions for event orchestration.

Demonstration of starting a Step Function instance from a Lambda function.

Creation of a Step Function instance named 'step function from lambda' with predefined ASL code.

Explanation of the Step Function's task to communicate with the SQS Send Message API.

Details on the JSON object structure for the transaction ID and type from the Step Function input.

Copying the ARN for use in creating the Lambda function to start the Step Function instance.

Creating a new Lambda function named 'step function invoker' in Python 3.6.

Importance of using the correct permissions for the Lambda function to access Step Functions.

Using the least privilege model to assign only necessary permissions for the Lambda function.

Writing code in the Lambda function to interact with Step Functions using the boto3 client.

Generating a unique transaction ID using the UUID library for the Step Function instance.

Defining the input object for the Step Function with a hardcoded 'purchase' type.

Calling the Step Function's start_execution API with the necessary parameters.

Conversion of the Python dictionary object to a JSON string for the API call.

Testing the Lambda function with a test event to ensure successful Step Function execution.

Verification of multiple successful Step Function instances in the AWS console.

Exploration of individual Step Function instances to track progress and success.

Encouragement to check out additional videos on Step Functions and related AWS services.

Transcripts

hello everyone and welcome back to AWS

simplified in my previous videos I've

shown you how to integrate lambdas into

step functions as part of event

orchestration but in this video I want

to show you how to start a step function

instance from a lambda so let's jump

right into it so here we are in the step

function console so I've actually

already created a step function instance

and it's called step function from

lambda and I predefined some ASL also

some step function code that does a

fairly trivial task so let's just take a

look at this really quick so it takes an

input and it's going to be communicating

with the sqs send message API and here

is the URL to the Q that I've created in

advance of this video and the body that

it's going to be posting to this sq bus

it's going to be a JSON object that has

a transaction ID and the value is going

to be coming from the input of the step

function and the type same deal so

there's a type key and the value is

going to be coming from the input so

that's to say our step function input is

going to contain a transaction ID which

is a string and a type which is also a

string okay so it's a very

straightforward application here so this

all looks well and good I'm going to

copy the AR n because we're gonna need

it when we create our lambda function

and actually try to start an instance of

the step function so this is all fine

here now let's go over to lambda and

actually create our lambda function

that's going to do this okay going to

click on create function now we're going

to be doing this from scratch get a name

my function names step function invoker

and we're going to be doing this in

Python 3.6 3.7 rather and then this is

actually quite important the permission

section so let's scroll down here so I'm

going to be using on an existing role

that actually has the step function

access associated with it however if you

are doing this from scratch you need to

go out and create a role that has the

permission that you need I'm using full

access but I encourage you to use lease

privilege models so only use the

permissions that you need for this

exercise so you actually need the

functions start execution policy and you

need to associate that with a role so

I've already done that here except that

I'm using the full access policy so I

get all those permissions for free so

after that's done you can go ahead and

click on create a function in the bottom

right here you can sometimes take a bit

that was pretty quick that's great okay

so that was created successfully so

let's scroll down here so now we can

actually write our code to interact with

step functions so you can do this in the

editor if you choose I'm gonna do it in

sublime just because I'm a little bit

more comfortable there so let's switch

over to sublime the first step I want to

do here is just paste in the AR n that

corresponds with the step function that

we created in the previous step and

we're gonna need that in a moment or so

when we interact with the client library

that needs an AR n that's an input and

then let's just take whatever we have

here as a templates I get rid of the

stuff we don't need

alright perfect okay so let's get to it

so we're gonna need the JSON library so

that's so let's leave that as it is I'm

gonna need the moto3 library and we're

also going to need the UUID library so

we're gonna be using a function from

this library that generates a random

hash of values and this will make sense

in a moment it's actually the first step

here so I'll probably have to get into

it right away and then the other thing

that we need is an instance of the boat

of three step function client so we need

to say client is equal to boat o 3 dot

clients then you say step functions so

now we have a reference to that client

ok so the first thing that we need is

the transaction ID but before we do that

let's just recall what the input to this

step function is going to look like so

the input is going to be a JSON that

looks like this so we're gonna have

transaction ID and then something excuse

me there we go foo and then we have type

and that can be either purchase or

refund we're just gonna hard code this

is purchase here just in this example so

just a reminder that's what our input is

going to look like ok so the first thing

we need to do is actually generate a

value that corresponds with the

transaction ID and this can be anything

you want to be a random string of values

it could be something meaningful like a

customer ID plus a date

it could be absolutely anything the only

condition though is that this value

whatever you're using as the key for the

step function instance needs to be

unique you cannot have two step function

instances that correspond with the same

key value now since we're using

transaction ID is the kind of identifier

in this example it makes sense to have a

unique value here so let's actually do

that right now through the UUID library

so let's say our transaction ID and

we're going to store this in a variable

is equal to uu ID dot u u ID 1 and what

this thing will generate is just a

random string of characters separated by

dashes so it'll look like like something

like this and many many more of these

variables I think it's limited to 4

delimiter so probably 30 or so

characters so that's what it's gonna

look like so we need to actually convert

this to string since UUID 1 actually

returns an object so now we have a

string instance of that so this is going

to be the value that we populate into

the body of the step function it's also

going to be the name field or the

identifier of the step function instance

as well okay so let's proceed to the

next step and actually define our input

object so let's say our input is equal

to a json and it's gonna look similar to

what i have above here on line 10 and

let's say the transaction ID is equal to

what we just defined so that's going to

be that UUID and we're gonna say the

type surrounded in quotes of course and

we're gonna say purchase okay so now we

have a JSON sorry dictionary object in

Python that corresponds to this object

here that we're looking at okay so next

we need to actually call the step

function API it's called the start

execution API and we can store the

result in a response object if we want

but we can say response is equal to

client dot start

underscore execution and we have a

couple things that we need to pass in

here okay so the first one that's

absolutely mandatory state machine AR N

and you can see now why I copied this

from the previous step because we need

to paste this in here so let's do that

all right now based on

make sure you put a comma at the end

here let's just Center this guy all

right cool

and the second field that you need is

the name so this is the unique

identifier of this step function

instance so it's going to be transaction

ID which is the same value that we're

populating in the JSON payload and the

third value is the actual JSON payload

that you want to pass in and this can be

used throughout your step function for

orchestration or to identify certain key

values to determine what you want to do

in this step function so for our

purposes is this JSON object for yours

it can be anything you like so let's say

input is equal to a now there's a key

step here recall that this is a Python

dictionary object this API requires a

string so we need to string Phi and

convert this dictionary object into a

JSON and so the way we do that is just

use the JSON library JSON dump s and

pass in that object the input object was

which corresponds to the dictionary

object we created previously okay so

everything looks good here let's copy

this out you can go back to our staff

function he's not in click on save in

the top right in order to test it out we

need to create a test event you can put

whatever you want in the event name

leave this as default we're not reading

off the input so it doesn't matter if

you were reading off the input in your

lab well you may want to change this but

we don't care so we don't need it

clicking on create in the bottom right

okay and let's click on test now to see

if this worked okay you can see that

it's exceeded a whole bunch of jargony

we don't care about let's click this a

few times test plus whatever how many

times you want okay so it looks like

it's succeeding from the logs here let's

head over to step functions and actually

verify that there's multiple instances

so we can see eight instances have been

successful if we actually click into

here now now we can look at the separate

instances that correspond to each click

of the button on that lambda function so

these values all kind of our first

glance may look at the same but you can

see here ad 580 0 so these are all

unique values that have all their own

unique key so if we click on one of

these we can see what the step function

was doing and its progress so you can

see through the green box here that it

was successful at this step and you can

see here

you got a 200 okay when we were

communicating with the SQS sendmessage

api sofa lee found this video useful I

have quite a few videos on step

functions and set function orchestration

including dynamo and SQS and us and us

how they suggest to check those out I'll

put a playlist in the pop up that's

coming up in a second thanks so much

guys and I'll see you next time