OOPS ! I left my EC2 Instance Running

This is the biggest fear of any AWS user; leaving an EC2 instance on. The first stop is always to use AWS Budgets. It sends alerts when AWS Costs surpass a specified amount. However, is it possible that AWS has such a facility for CPU uptime ? Send alerts when an EC2 instance has exceed my budgeted uptime. And also, don’t alert once, keep alerting recursively when the budgeted uptime for my EC2 instance is exceeded.

This feature would help be more aware of the usage of EC2 instances never leave them running unwittingly.

Amazon EventBridge Rules

Amazon Eventbridge Rules was clearly the place to go. Create a rule on EventBridge to send an SNS Notification depending on a specified EC2 condition.

Amazon Eventbridge Rule - A rule watches for specific types of events. When a matching event occurs, the event is routed to the targets associated with the rule. A rule can be associated with one or more targets.

However, upon examining the events being tracked, none of them have anything to do with CPU Uptime. The only event that is close enough is EC2 Instance State Change Notification. It sends a notification when the state of an EC2 instance changes, for example, from running to stopped or terminated. The EventBridge cannot monitor and track intricate details of an EC2 instance such as CPU Uptime. Another way has to be considered

AWS Lambda

One way to observe the CPU uptime is through AWS Lambda.

AWS Lambda - A serverless compute service for running code without having to provision or manage servers

Through the library boto3, you can capture the CPU Uptime of an EC2 instance. For example:

ec2_client = boto3.client("ec2")

cpu_uptime = current_time - ec2_client.describe_instances().Reservations.Instances.InstanceId.LaunchTime

However, it could only capture the CPU uptime once, when it was being run. How can you make this a repeatable process ?

Amazon EventBridge Scheduler

Using an Amazon EventBridge Scheduler, create a cron-job that runs the AWS Lambda Function repeatedly after a specified period of time.

Amazon EventBridge Scheduler - Amazon EventBridge Scheduler is a serverless scheduler that allows you to create, run, and manage tasks from one central, managed service.

That means you can create a cron-job using Amazon EventBridge Scheduler to run the AWS Lambda Function above that tracks the cpu uptime after every few minutes.

However, this means that for as long as you have an AWS Account, this EventBridge Scheduler would always be on the look out. The cost implications were unclear, but it felt inefficient still. How can you run an EventBridge Scheduler only when you have EC2 instances Running ?

1. Create an SNS Topic called ec2-uptime-alert

This SNS topic will send an email to the specified address when the ec2 uptime has exceeded the defined limits.

2. Create a Lambda Function called send-uptime-notification

This Lambda Function will check all the instances that are running in your account.

It will then calculate the ec2 uptime by subtracting the time the instance was launched against the current time.

If the ec2 uptime is longer than the specified MAX_UPTIME_MINS, it will send a notification through the SNS Topic above.

import datetime
import boto3
import json

SCHEDULE_NAME = "ec2-uptime-cron"
scheduler_client = boto3.client("scheduler")
sns_client = boto3.client("sns")
TOPIC_ARN="<ec2-uptime-alert Topic ARN>"
SUBJECT="[WARNING]: Excessive usage of EC2 Instance" 
ec2_client = boto3.client("ec2")
MAX_UPTIME_MINS=15

def lambda_handler(event, context):
    response = ec2_client.describe_instances()
    running_instances=[]
    for reservation in response["Reservations"]:
        for instance in reservation["Instances"]:
            instance_id = instance["InstanceId"]
            launch_time = instance["LaunchTime"]
            instance_state = instance['State']['Name']
            print(instance_id)
            print(launch_time)
            print(instance_state)
            if instance_state != "running":
                continue

            # all instances beyond this point are running
            running_instances.append(instance_id)
            uptime = datetime.datetime.now(tz=launch_time.tzinfo) - launch_time
            uptime_minutes = uptime.total_seconds()/60
            if float(uptime_minutes) > MAX_UPTIME_MINS:
                message=f'You cannot afford to have instance {instance_id} running any longer. It\'s been up for {uptime_minutes} minutes'
                sns_client.publish(
                    TopicArn=TOPIC_ARN,
                    Message= message,
                    Subject=SUBJECT
                )
    # if 0 running instances, disable the cronjob of the eventbridge scheduler
    if len(running_instances) < 1:
            current = scheduler_client.get_schedule(
                 Name=SCHEDULE_NAME
            )

            scheduler_client.update_schedule(
                Name=SCHEDULE_NAME,
                FlexibleTimeWindow=current["FlexibleTimeWindow"],
                ScheduleExpression=current["ScheduleExpression"],
                Target=current["Target"],
                State="DISABLED"
            )

    return {
        "statusCode": 200,
        "body": json.dumps(response)
    }

3. Create an Amazon EventBridge Scheduler called ec2-uptime-cron

The Lambda Function above should be run according to a recurring schedule every 10 minutes:

Select the AWS Lambda Function with the name send-uptime-notification:

4. Create an AWS Lambda Function trigger_eventscheduler

Create a lambda function called trigger_eventscheduler. It enables the ec2-uptime-cron rule above.

import json
import boto3

scheduler_client = boto3.client("scheduler")
SCHEDULE_NAME = "ec2-uptime-cron"

def lambda_handler(event, context):
    current = scheduler_client.get_schedule(
        Name=SCHEDULE_NAME
    )
    print(event)
    response = scheduler_client.update_schedule(
        Name=SCHEDULE_NAME,
        FlexibleTimeWindow=current["FlexibleTimeWindow"],
        ScheduleExpression=current["ScheduleExpression"],
        Target=current["Target"],
        State="ENABLED"
    )
    return {
        "statusCode": 200,
        "body": json.dumps(response)
    }

5. Create an Amazon EventBridge Rule called start-uptime-cron

The start-uptime-cron will be triggered when an EC2 instance is started and is in running state.

Once it is triggered, it will go ahead to trigger the Lambda Function trigger_eventscheduler. Set the target as below:

Conclusion

Now, you can expect emails that look as below when the EC2 instances have exceeded your budgeted uptime: