AWS Batch

25 Jun 2024 - Shyam Mohan

What is AWS Batch?

AWS Batch allows developers, scientists, and engineers to run thousands of computing jobs in the AWS platform. It is a managed service that dynamically maintains the optimal compute resources like CPU, Memory based on the volume of submitted jobs.The User just has to focus on the applications (like shell scripts,

Linux codes or java programs). It executes workloads on EC2 (including Spot instances) and AWS Fargate.

Components:

● Jobs - are the fundamental applications running on Amazon EC2 machines in containerised form.

● Job Definitions – define how the job is meant to be run. Like the associated IAM role, vCPU requirement, and container properties.

● Job Queues – Jobs reside in the Job queue where they wait till they are scheduled.

● Compute Environments – Each job queue is linked to a computing environment which in itself contains the EC2 instance to run containerized applications.

● There are two types of environments: Managed where the user gives min and max vCPU, EC2 instance type and AWS runs it on your behalf and

Unmanaged where you have your own ECS agent.

● Scheduler – maintains the execution of jobs submitted to the queue as time and dependencies.

Best Practices:

● Use Fargate if you want to run the application without getting into EC2 infrastructure details. Let the AWS batch manage it.

● Use EC2 if your work scale is very large and you want to get into machine specifications like memory, CPU, GPU.

● Jobs running on Fargate are faster on startup as there is no time lag in scale-out operation, unlike EC2 where launching new instances may take time.

Use Cases:

● Stock markets and Trading – The trading business involves daily processing of large scale data and loading them into a Data warehouse for analytics. So that your predictions and decisions are quick enough to make a business grow on a regular basis.

● Media houses and the Entertainment industry – Here a large amount of data in the forms of audio, video and photos are being processed daily to cater to their customers. These application workloads can be moved to containers on AWS Batch.

Pricing:

● There is no charge for AWS Batch rather you pay for the resources like EC2 and Fargate you use.