InternDiaries
Start an EC2 Instance using AWS SQS & Lambda
It’s been around one month since I got my first internship opportunity at LiveRoom as an intern software engineer. Throughout this month I was assigned to many tasks in various projects. In my first project, our team needed a mechanism to process batches of objects which are coming in a sequence manner on a pre-build deep learning model. So we needed a cost-effective and efficient way to do so and as the first task, I started working on this issue.
Our initial idea was to use the AWS Auto Scaling group for the scale-up and scale in of the EC2 instances using AWS cloud watch metrics and SQS. As we want a cost-effective solution our plan was to use the t2.micro instance as it is a free tier and it was enough for the processing task of an incoming object. Although the scaling up procedure didn’t work well as we expect (It was because the SQS metric didn’t respond for new incoming messages. SQS metric worked according to a threshold and for our task, it wasn’t fit)
Then we thought of another way to achieve our target. AWS Lambda was the solution.
Getting started with AWS SQS and Lambda Trigger to start an ec2 instance
The summary of what we wanted to achieve here was :
- Send objects to the FIFO queue using the S3 bucket in a sequence manner
- When the FIFO queue received a message lambda function will trigger
- The function will start a new EC2 instance (Here we used t2.micro instance and we need one instance per processing of an object)
- After the processing part is done automatically terminate the instance
NOTE: This procedure was done actually for testing purposes, not for production purposes.
AWS SQS, Lambda function, and EC2 are some main services provide by AWS, and for this particular task, I and my mentor decided to use these services to achieve our target.
Here I am not going to tell you about how to set up the AWS account. Instead, I'll dive into the concepts of SQS and lambda function. Below I have given a small definition for both SQS and Lambda by referring to Amazon Documentation
Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. It provides a generic web services API that you can access using any programming language that the AWS SDK supports. Amazon SQS supports both standard and FIFO queues.
AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. With Lambda, you can run code for virtually any type of application or backend service — all with zero administration. Just upload your code as a ZIP file or container image, and Lambda automatically and precisely allocates compute execution power and runs your code based on the incoming request or event, for any scale of traffic. You can set up your code to automatically trigger from over 200 AWS services and SaaS applications or call it directly from any web or mobile app. You can write Lambda functions in your favorite language (Node.js, Python, Go, Java, and more) and use both serverless and container tools, such as AWS SAM or Docker CLI, to build, test, and deploy your functions.
In this task, we use the S3 bucket to push objects to the queue and we used the FIFO queue for the task as the FIFO manner was crucial when processing the objects.
To use all the AWS services make sure you have the right access and permission to the services.
Make sure to create ‘roles’ using IAM especially for lambda events that use triggers.
After getting relevant access you can simply create SQS using Amazon SQS.
Before creating the lambda function I created an IAM role for the lambda function that has a lambda function execution permission and full access SQS permissions.
Then using the AWS lambda console I created a lambda function. To invoke the function I created a lambda SQS trigger with a batch size of 1 (Because we want to trigger the function when a new message is pushed into the queue)
So what is happening inside the function? 😕
Boto3 is the name of the Python SDK for AWS. It allows you to directly create, update, and delete AWS resources from your Python scripts.
Make sure to pass the proper AMI details
#AMI details
AMI = ‘ami-xxxxxxx’
INSTANCE_TYPE = ‘t2.micro’
KEY_NAME = ‘xxxx’
SUBNET_ID = ‘xxxxxx’
lambda_handler(event, context) is the function that executes when the event is triggered.
In the lambda function console you can configure these:
- The Handler is set to lambda_function.lambda_handler
- lambda_function.py is the name of our .py file
- lambda_handler is the name of the function in our code
event argument is the data that's passed to the function upon execution. Here A message arriving on an SQS Queue is the event and we can access the specific message using:
event[“Records”][0]
context arg is the data about the execution environment of the function
UserData
When you launch an instance in Amazon EC2, you have the option of passing user data to the instance that can be used to perform common automated configuration tasks and even run scripts after the instance starts.
Here I add UserData to the instance using the bash command so I can save the message ID and message body in a text file.
userData1 = f”#!/bin/bash \n echo ‘message id : {messageId} , body : {body}’ > /home/ec2-user/message.txt \n”
Terminating the Instance
As I mentioned earlier in summary we wanted to terminate the instance after the processing is done. So what I did was write a bash script to terminate the instance and pass the script to the instance via userData. So after processing the object we can execute that bash script.
userData2 = “””touch /home/ec2-user/file
cat << EOF > /home/ec2-user/file
aws configure set aws_access_key_id xxxxxx
aws configure set aws_secret_access_key xxxxxx
aws configure set default.region xxxxx
ec2InstanceId=$(ec2-metadata — instance-id | cut -d “ “ -f 2)
aws ec2 terminate-instances — instance-ids $(ec2-metadata — instance-id | cut -d “ “ -f 2)
EOF
chmod +x /home/ec2-user/file”””
Here I created a bash file named “file”. But to terminate an instance it should configure the AWS details. So make sure to fill in the proper aws_access_key and secret_key values along with the region.
After processing the object just execute the bash file by typing ./file
So the particular instance will terminate automatically.
Testing
You can create a custom test plan using the lambda function console or test via simply sending an SQS message. I'll explain the second approach.
To test this you can simply send a message to the FIFO queue using SQS console. { When sending messages make sure to put the same group ID (Messages that belong to the same message group are processed in a FIFO manner. However, messages in different message groups might be processed out of order) and different duplicate ID for every message }
By monitoring the EC2 console you could see that new instances are being created and the FIFO queue becomes empty because when the function is triggered the particular message also will be automatically deleted.
Conclusion
I hope you guys got an idea about how I achieve the target and apology for not including images of steps that I followed. I am currently having a permission issue with my AWS account so I can’t capture screenshots.
This is my first medium article and could have tiny mistakes and unclear points. However, feel free to give your comments, and hope to fix those little bugs 😉 and see you in the next article soon.
Thank you for reading the article and if you want more to learn on AWS please refer to the AWS doc.
