Trying Out AWS Fargate for Daily Job on Docker
Having a use case to run a short-lived docker image in daily manner, somehow I stumbled upon newly released managed-container service from AWS: AWS Fargate.
AWS Fargate’s quite neat, I only need to specify where my image reside and how much capacity I need. It is also possible to schedule the job using CloudWatch. Currently it is only available at N. Virginia region.
The use case is to schedule a daily job to check repo for secret and keys using this cool tool called gitleaks, and alert when it find something.
Why don’t just use Lambda?
Yes using lambda is simpler and might be more suitable for this use case, it is for exploration purpose. However, I think Fargate has several benefits over Lambda:
- Fill the gap on use cases where we need short-lived job, but the job could take more than 5 minutes (which is currently hard limit of Lambda)
- Docker! Better devops support! CI/CD for docker is ubiquituous compare to development-testing-release flow of Lambda, it felt more natural
- A lot less hard limit compare to Lambda
Again, it goes back to what your use case is. Imho Fargate fill the gap on spectrum between simple function that suit on Lambda and full fledge dockerized app on ECS or Kube.
AWS Fargate Concepts
There are several concepts we need to understand:
Container is defined by specifying image repo URL, port mapping, etc. Container then wrapped by a Task. On Task, we define the size (CPU & RAM). On Service, we could define application load balancer and autoscaling configs. On Cluster, we define physical machine type, networking configuration (VPC, secgroup, etc).
The Plan
Goal of the use case is to send alarm when gitleaks which run daily has findings. The plan is to
- Schedule a Fargate Task daily, output to CloudWatch Log
- Put a Metric Filter for the log on the findings
- Set CloudWatch Alert to notify via SNS to email when finding > 0
Schedule
To schedule a task, go to Cluster and click Schedule a Task tab. It’s convenient to define since it supports cron syntax.
To test whether your task definition works well, you could run task manually on Tasks tab and here is the sample result. (Note: I’m using gitleaks own github which is predefined with leaks.)
Please note that we need to set task number to 0 for daily job. If you set task number to 1, then everytime the job done it will start another job and so on and so forth.
Set Metric Filter and Alert
Go to CloudWatch Log and click Create Metric Filter. I’m putting “reason” on filter pattern based on log written by gitleaks. Aftewards, go to Alarm, click Create Alarm and choose Custom Metrics with predefined filter as metric.
And voila, by triggering a task manually, I got my first alarm! (Again, note: I’m using gitleaks own github which is predefined with leaks.) (Pardon my typo on alarm name)
