AWS Infrastructure Investigation

Investigate AWS infrastructure issues affecting Buildkite build agents. Covers EC2 instances, AutoScaling Groups, the autoscaling Lambda, and related resources.

Prerequisites

• AWS CLI installed (brew install awscli)
• AWS SSO profile mockserver-build configured (SSO region: eu-west-2)
• Active SSO session: aws sso login --profile mockserver-build
• Corporate TLS proxy (if applicable): export AWS_CA_BUNDLE=$NODE_EXTRA_CA_CERTS (only if NODE_EXTRA_CA_CERTS is set)
• macOS + Python 3.14 + Homebrew: if you get pyexpat symbol errors, export DYLD_LIBRARY_PATH=/opt/homebrew/opt/expat/lib

Infrastructure Overview

There are two build agent stacks. Investigate the current stack first; fall back to the legacy stack only if the current one has not been provisioned yet.

Current: Terraform-managed (eu-west-2)

Managed by terraform/buildkite-agents/ using the official Buildkite Elastic CI Stack module.

Resource names are generated by Terraform with a random suffix. To find them:

# Get ASG name from Terraform state
cd terraform/buildkite-agents
terraform output auto_scaling_group_name

# Or find ASGs with the Buildkite tag
aws autoscaling describe-auto-scaling-groups \
  --region eu-west-2 --profile mockserver-build \
  --query 'AutoScalingGroups[?contains(Tags[?Key==`Stack`].Value | [0], `buildkite-mockserver`)].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Min:MinSize,Max:MaxSize,Instances:Instances[*].{ID:InstanceId,State:LifecycleState}}'

Legacy: CloudFormation-managed (us-east-1)

Being replaced by the Terraform stack above. May still be active during migration.

AWS CLI Prefix

All commands require --region and --profile flags:

# Current stack (eu-west-2)
aws ... --region eu-west-2 --profile mockserver-build

# Legacy stack (us-east-1)
aws ... --region us-east-1 --profile mockserver-build

Investigation Workflow

Step 1: Determine Active Stack

Check which stack is currently running agents:

# Check current stack (eu-west-2) — look for ASGs tagged with buildkite-mockserver
aws autoscaling describe-auto-scaling-groups \
  --region eu-west-2 --profile mockserver-build \
  --query 'AutoScalingGroups[?contains(Tags[?Key==`Stack`].Value | [0], `buildkite-mockserver`)].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Instances:Instances[*].{ID:InstanceId,State:LifecycleState}}'

# Check legacy stack (us-east-1)
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names "buildkite-AgentAutoScaleGroup-VGG28FR0DE6Q" \
  --region us-east-1 --profile mockserver-build \
  --query 'AutoScalingGroups[0].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Instances:Instances[*].{ID:InstanceId,State:LifecycleState}}'

Use whichever stack has instances (or non-zero desired capacity) for the remaining steps. Substitute the correct --region and ASG name accordingly.

Step 2: Quick Health Check

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names "<ASG_NAME>" \
  --region <REGION> --profile mockserver-build \
  --query 'AutoScalingGroups[0].{Desired:DesiredCapacity,Min:MinSize,Max:MaxSize,Instances:Instances[*].{ID:InstanceId,State:LifecycleState,Health:HealthStatus}}'

Expected healthy state:

• If queue is empty: Desired = 0 can be healthy (scale-to-zero)
• If queue has pending jobs: desired capacity should increase above 0 within 1-2 scaler intervals
• Active instances should be InService and Healthy

Problem indicators:

• Desired: 0 — no agents requested (scaler not seeing jobs, or Lambda not running)
• Desired > 0 but no instances — launch failures
• Instances in Pending for >5 min — launch issues
• Instances Unhealthy — failing health checks

Step 3: Check EC2 Instance Status

aws ec2 describe-instances \
  --filters "Name=tag:aws:autoscaling:groupName,Values=<ASG_NAME>" \
  --region <REGION> --profile mockserver-build \
  --query 'Reservations[].Instances[].{ID:InstanceId,State:State.Name,Type:InstanceType,Launch:LaunchTime,AZ:Placement.AvailabilityZone}'

For running instances, check system/instance status:

aws ec2 describe-instance-status \
  --instance-ids <instance-id-1> <instance-id-2> \
  --region <REGION> --profile mockserver-build

Step 4: Check Scaling Activities

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name "<ASG_NAME>" \
  --region <REGION> --profile mockserver-build \
  --max-items 10

Look for:

• "user request explicitly set group desired capacity" — the Lambda scaler adjusted capacity
• "an instance was taken out of service" — scale-in event
• Failed status codes — launch failures (AMI issues, capacity, subnet exhaustion)

Step 5: Check the Autoscaling Lambda

Find the scaler Lambda by listing functions with a Buildkite-related name:

aws lambda list-functions \
  --region <REGION> --profile mockserver-build \
  --query 'Functions[?contains(FunctionName, `buildkite`) && (contains(FunctionName, `scaler`) || contains(FunctionName, `caling`))].{Name:FunctionName,Runtime:Runtime,State:State,LastModified:LastModified}'

Then check its logs:

# Recent invocations (last hour)
aws logs filter-log-events \
  --log-group-name "/aws/lambda/<LAMBDA_FUNCTION_NAME>" \
  --region <REGION> --profile mockserver-build \
  --start-time $(python3 -c "import time; print(int((time.time() - 3600) * 1000))") \
  --limit 20

# Error logs (last hour)
aws logs filter-log-events \
  --log-group-name "/aws/lambda/<LAMBDA_FUNCTION_NAME>" \
  --region <REGION> --profile mockserver-build \
  --start-time $(python3 -c "import time; print(int((time.time() - 3600) * 1000))") \
  --filter-pattern "ERROR" \
  --limit 10

Step 6: Check EC2 Console Output

For instances that are running but not registering as Buildkite agents:

aws ec2 get-console-output \
  --instance-id <instance-id> \
  --region <REGION> --profile mockserver-build \
  --query 'Output' --output text

Step 7: Check Suspended ASG Processes

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names "<ASG_NAME>" \
  --region <REGION> --profile mockserver-build \
  --query 'AutoScalingGroups[0].SuspendedProcesses'

Note: AZRebalance is intentionally suspended to prevent killing running builds. Other suspended processes may indicate problems.

Failure Patterns

Emergency: Manually Scale Up Agents

If the Lambda is broken and you need agents immediately:

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name "<ASG_NAME>" \
  --desired-capacity <TEMP_CAPACITY_LEQ_MAX_SIZE> \
  --region <REGION> --profile mockserver-build

Choose a temporary capacity that does not exceed the ASG MaxSize from Step 2.

Warning: The Lambda scaler may override this on its next invocation if it sees no pending jobs.

Output — Structured Data Return

Return this structure in your final message:

{
  "schema": "aws-investigation/v1",
  "timestamp": "<ISO8601>",
  "active_stack": "terraform-eu-west-2 | legacy-us-east-1",
  "asg": {
    "name": "<ASG name>",
    "region": "<region>",
    "desired_capacity": 0,
    "min_size": 0,
    "max_size": "<max_size>",
    "instances": [
      {
        "instance_id": "<id>",
        "state": "InService|Pending|Terminating",
        "health": "Healthy|Unhealthy",
        "availability_zone": "<az>"
      }
    ],
    "suspended_processes": ["<process names>"]
  },
  "lambda": {
    "function_name": "<name>",
    "state": "Active|Inactive",
    "runtime": "<runtime>",
    "recent_errors": ["<error messages>"],
    "last_invocation": "<ISO8601 or null>"
  },
  "root_cause": {
    "summary": "<one-line description>",
    "detail": "<technical explanation>",
    "category": "<category from failure patterns>",
    "evidence": "<relevant log lines or CLI output>"
  },
  "recommended_fix": "<actionable steps>",
  "warnings": ["<deprecation notices, capacity concerns, etc.>"]
}

After returning the JSON, provide a brief summary (2-3 lines).

Notes

• Always run Step 1 first to determine which stack is active
• The Lambda scaler logs are the most valuable data source for understanding scaling decisions
• Always check if the Buildkite agent token is still valid if agents start but don't register