Trying Out AWS Fargate for Daily Job on Docker

Having a use case to run a short-lived docker image in daily manner, somehow I stumbled upon newly released managed-container service from AWS: AWS Fargate.

AWS Fargate’s quite neat, I only need to specify where my image reside and how much capacity I need. It is also possible to schedule the job using CloudWatch. Currently it is only available at N. Virginia region.

The use case is to schedule a daily job to check repo for secret and keys using this cool tool called gitleaks, and alert when it find something.

Why don’t just use Lambda?

Yes using lambda is simpler and might be more suitable for this use case, it is for exploration purpose. However, I think Fargate has several benefits over Lambda:

  1. Fill the gap on use cases where we need short-lived job, but the job could take more than 5 minutes (which is currently hard limit of Lambda)

Again, it goes back to what your use case is. Imho Fargate fill the gap on spectrum between simple function that suit on Lambda and full fledge dockerized app on ECS or Kube.

AWS Fargate Concepts

There are several concepts we need to understand:

Image for post
Image for post

Container is defined by specifying image repo URL, port mapping, etc. Container then wrapped by a Task. On Task, we define the size (CPU & RAM). On Service, we could define application load balancer and autoscaling configs. On Cluster, we define physical machine type, networking configuration (VPC, secgroup, etc).

The Plan

Goal of the use case is to send alarm when gitleaks which run daily has findings. The plan is to

  1. Schedule a Fargate Task daily, output to CloudWatch Log

Schedule

To schedule a task, go to Cluster and click Schedule a Task tab. It’s convenient to define since it supports cron syntax.

Image for post
Image for post

To test whether your task definition works well, you could run task manually on Tasks tab and here is the sample result. (Note: I’m using gitleaks own github which is predefined with leaks.)

Image for post
Image for post

Please note that we need to set task number to 0 for daily job. If you set task number to 1, then everytime the job done it will start another job and so on and so forth.

Set Metric Filter and Alert

Go to CloudWatch Log and click Create Metric Filter. I’m putting “reason” on filter pattern based on log written by gitleaks. Aftewards, go to Alarm, click Create Alarm and choose Custom Metrics with predefined filter as metric.

Image for post
Image for post

And voila, by triggering a task manually, I got my first alarm! (Again, note: I’m using gitleaks own github which is predefined with leaks.) (Pardon my typo on alarm name)

Image for post
Image for post

Crazy dad. Data technology enthusiast. Youtube: Insinyur Data

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store