Trying Out AWS Fargate for Daily Job on Docker

Having a use case to run a short-lived docker image in daily manner, somehow I stumbled upon newly released managed-container service from AWS: AWS Fargate.

AWS Fargate’s quite neat, I only need to specify where my image reside and how much capacity I need. It is also possible to schedule the job using CloudWatch. Currently it is only available at N. Virginia region.

The use case is to schedule a daily job to check repo for secret and keys using this cool tool called gitleaks, and alert when it find something.

Why don’t just use Lambda?

  1. Fill the gap on use cases where we need short-lived job, but the job could take more than 5 minutes (which is currently hard limit of Lambda)
  2. Docker! Better devops support! CI/CD for docker is ubiquituous compare to development-testing-release flow of Lambda, it felt more natural
  3. A lot less hard limit compare to Lambda

Again, it goes back to what your use case is. Imho Fargate fill the gap on spectrum between simple function that suit on Lambda and full fledge dockerized app on ECS or Kube.

AWS Fargate Concepts

Container is defined by specifying image repo URL, port mapping, etc. Container then wrapped by a Task. On Task, we define the size (CPU & RAM). On Service, we could define application load balancer and autoscaling configs. On Cluster, we define physical machine type, networking configuration (VPC, secgroup, etc).

The Plan

  1. Schedule a Fargate Task daily, output to CloudWatch Log
  2. Put a Metric Filter for the log on the findings
  3. Set CloudWatch Alert to notify via SNS to email when finding > 0

Schedule

To test whether your task definition works well, you could run task manually on Tasks tab and here is the sample result. (Note: I’m using gitleaks own github which is predefined with leaks.)

Please note that we need to set task number to 0 for daily job. If you set task number to 1, then everytime the job done it will start another job and so on and so forth.

Set Metric Filter and Alert

And voila, by triggering a task manually, I got my first alarm! (Again, note: I’m using gitleaks own github which is predefined with leaks.) (Pardon my typo on alarm name)

Crazy dad. Data technology enthusiast. Youtube: Insinyur Data

Crazy dad. Data technology enthusiast. Youtube: Insinyur Data