Implementing Metrics, Alarms, Monitoring, and Logging

Monitoring on AWS starts with Amazon CloudWatch, the service that ingests metrics from every native AWS service, stores them with a defined resolution and retention, and lets you attach alarms that react when a measurement crosses a threshold. CloudWatch is not a single product but a family: Metrics, Alarms, Logs, Dashboards, Events (now Amazon EventBridge), Container Insights, Lambda Insights, Application Signals, Synthetics, RUM, ServiceLens, and Internet Monitor all live under the CloudWatch umbrella. A CloudOps engineer is expected to know which lens to reach for, which data is automatic versus opt-in, and which knobs trade resolution for cost.

Every CloudWatch metric is identified by three things: a namespace (for example AWS/EC2 or a custom namespace like Acme/Payments), a metric name (CPUUtilization, OrderLatencyP95), and a set of dimensions (key-value pairs such as InstanceId=i-0abc...). Two data points written with the same namespace, name, and dimension set go to the same metric stream; change any dimension value and you get a new metric. Statistics (Sum, Average, Minimum, Maximum, SampleCount, percentiles like p99) are computed at query time over a period. The default resolution is one minute (standard resolution); high-resolution custom metrics support a one-second granularity and short data points are retained at full fidelity for three hours, then aggregated upward. The retention ladder is: high resolution and one-minute data for fifteen days, five-minute data for sixty-three days, one-hour data for fifteen months. After fifteen months CloudWatch drops the data; export to S3 or to a long-term store if you need it longer.

Alarms watch one metric (a metric alarm) or a boolean expression over other alarms (a composite alarm). Each metric alarm holds five settings that determine its behavior: the metric and statistic, the period (the time window over which the statistic is computed), the evaluation period (how many consecutive periods are inspected), the data-points-to-alarm count (how many of those periods must breach), and the threshold itself. The state machine has three values: OK, ALARM, and INSUFFICIENT_DATA. The fifth state, INSUFFICIENT_DATA, is the one that trips operators most often, and it is covered in depth later in the chapter. Alarms can also be configured to treat missing data as breaching, not breaching, ignore, or missing, and that single setting is the difference between an alarm that pages on a real outage and one that flaps every time a workload goes quiet at night.

Logging in AWS is split between two services. CloudWatch Logs holds application logs, OS logs, and managed-service logs (Lambda function logs, RDS audit logs, VPC flow logs, Route 53 query logs, and so on) inside log groups and log streams; a log group has a retention policy (one day to ten years, or never expire), an optional KMS key for encryption at rest, and an optional metric filter that scans incoming events and emits a CloudWatch metric. CloudTrail records API activity (the who-did-what audit trail), is automatically enabled for management events for the last ninety days, and writes longer-term trails to S3, optionally also forwarding to CloudWatch Logs and EventBridge.

Alarm Invokable Actions: Direct Invoke versus EventBridge

A CloudWatch alarm has two ways to do work when it transitions. The first is a direct alarm action, registered with aws cloudwatch put-metric-alarm --alarm-actions <arn>. The supported direct actions are an Amazon SNS topic, an EC2 action ARN (stop, reboot, recover, or terminate the specific instance the alarm is watching), an Auto Scaling policy ARN (scale-out or scale-in by changing capacity), and a Systems Manager OpsItem or incident in Incident Manager. The second is to wire the alarm state-change as an event source for Amazon EventBridge; EventBridge then fans out to Lambda functions, Step Functions state machines, ECS tasks, SQS queues, SSM Automation runbooks, API destinations, and dozens of other targets. The choice is not stylistic. Direct actions are the simplest path for single-instance self-healing; EventBridge is the right path the moment you need filtering on top of state, enrichment, multiple downstream consumers, or cross-account routing.

# Direct-action alarm: reboot a stuck EC2 instance after 3 minutes of failed status checks.
aws cloudwatch put-metric-alarm \
 --alarm-name acme-ec2-reboot-on-status-fail \
 --alarm-description "Auto-reboot when both EC2 status checks fail" \
 --namespace AWS/EC2 \
 --metric-name StatusCheckFailed \
 --dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f67890 \
 --statistic Maximum \
 --period 60 \
 --evaluation-periods 3 \
 --datapoints-to-alarm 3 \
 --threshold 1 \
 --comparison-operator GreaterThanOrEqualToThreshold \
 --treat-missing-data breaching \
 --alarm-actions arn:aws:automate:us-east-1:ec2:reboot \
 --ok-actions arn:aws:sns:us-east-1:111122223333:acme-ops-noisy

The EC2 action ARNs are well-known constants of the form arn:aws:automate:<region>:ec2:<stop|reboot|recover|terminate>. They only act on the instance referenced by the InstanceId dimension of the alarm, so an alarm with a per-instance dimension and an EC2 reboot action is the canonical pattern for instance-level self-heal. Recover is the action for hardware-degradation scenarios; it migrates the EBS-backed instance to new underlying hardware and preserves the same private IP, EBS volumes, and elastic IP attachments. Terminate is for stateless members of an Auto Scaling group where replacement is cheaper than recovery.

# EventBridge route: forward all ALARM transitions of a critical alarm into a Systems Manager runbook.
aws events put-rule \
 --name acme-alarm-to-runbook \
 --event-pattern '{
 "source": ["aws.cloudwatch"],
 "detail-type": ["CloudWatch Alarm State Change"],
 "detail": {
 "alarmName": ["acme-payments-latency-p99"],
 "state": { "value": ["ALARM"] }
 }
 }'

aws events put-targets --rule acme-alarm-to-runbook --targets '[
 {
 "Id": "RemediationRunbook",
 "Arn": "arn:aws:ssm:us-east-1::automation-definition/AWS-RestartEC2Instance:$DEFAULT",
 "RoleArn": "arn:aws:iam::111122223333:role/EventBridgeInvokeSSM",
 "InputTransformer": {
 "InputPathsMap": { "instance": "$.detail.configuration.metrics[0].metricStat.metric.dimensions.InstanceId" },
 "InputTemplate": "{\"InstanceId\":[<instance>]}"
 }
 }
]'

💡 Exam Trap: A direct EC2 alarm action only acts on the instance carried in the alarm's InstanceId dimension. If you build an aggregate alarm across an Auto Scaling group (a metric math expression with no InstanceId dimension) and attach a reboot action, the call fails with no clear feedback at runtime. Aggregate alarms must use an Auto Scaling policy ARN, an SNS topic, or EventBridge as the action.

💡 Exam Trap: An EC2 recover action requires the instance to use only EBS volumes and live on a supported instance type; bare-metal and instance-store instances are not eligible. Questions that propose recover for an i3.metal are testing whether you know that boundary.

EBS Performance Metrics and Volume-Type Optimization

The other half of this section is the storage layer that those EC2 alarms watch. EBS publishes a fixed metric set under AWS/EBS (per volume) and AWS/EC2 (per attached instance). The volume-level metrics include VolumeReadOps, VolumeWriteOps, VolumeTotalReadTime, VolumeTotalWriteTime, VolumeIdleTime, VolumeQueueLength, and BurstBalance (for gp2/st1/sc1 only). Average queue length and per-operation latency are the two derived signals that decide whether a volume is the bottleneck: queue length consistently above one or two on a single-attachment volume means the workload is waiting for I/O.

Picking the right volume type is a per-workload decision. gp3 (general-purpose SSD) is the baseline that ships three thousand IOPS and one hundred twenty-five mebibytes per second of throughput at any size, with provisioned IOPS up to sixteen thousand and provisioned throughput up to one thousand mebibytes per second decoupled from capacity. io2 and io2 Block Express deliver up to two hundred fifty-six thousand and two hundred fifty-six thousand IOPS respectively for the highest-IOPS workloads (the io2 Block Express ceiling reaches two hundred fifty-six thousand per volume on supported Nitro instances), with 99.999% durability versus the 99.8 to 99.9% of gp2/gp3. st1 is throughput-optimized HDD for sequential big-block workloads (log processing, data warehouses): cheap throughput, lousy random-access latency. sc1 is cold HDD for archive volumes that are rarely touched.

# Modify a gp2 volume to gp3 with 5,000 IOPS and 500 MB/s provisioned throughput.
aws ec2 modify-volume \
 --volume-id vol-0a1b2c3d4e5f67890 \
 --volume-type gp3 \
 --iops 5000 \
 --throughput 500

# Watch the modification progress.
aws ec2 describe-volumes-modifications --volume-ids vol-0a1b2c3d4e5f67890

The modify operation is online: the volume continues to serve I/O while EBS rebalances behind the scenes; only the optimization-progress percentage moves until it completes. A common cost play is moving a fleet of gp2 volumes to gp3, since gp3 ships at roughly twenty percent lower cost per gigabyte and the included IOPS and throughput baseline is usually higher than what gp2 delivers at the same size.

📊 Limit: A single io2 Block Express volume on a Nitro-based instance supports up to two hundred fifty-six thousand IOPS, four thousand mebibytes per second throughput, and sixty-four tebibytes of capacity, the highest single-volume ceiling on EBS.

📊 Limit: Standard-resolution CloudWatch metric data is kept at one-minute resolution for fifteen days, five-minute for sixty-three days, and one-hour for fifteen months; high-resolution data is kept at one-second resolution for three hours before downsampling.

⚠️ Anti-Pattern: Treating BurstBalance as a generic gp3 signal. BurstBalance is published only for gp2, st1, and sc1 (volume types that use a burst credit model). gp3 has no burst credits because its baseline performance is provisioned; an alarm on BurstBalance for a gp3 volume sits in INSUFFICIENT_DATA forever.

Decision Anchor: Choose a direct alarm action when the remediation is a single, well-known operation on the same resource the alarm watches (reboot, recover, terminate, scale this group, notify an SNS topic). Choose an EventBridge target when the remediation needs filtering, enrichment, retry policies, multiple downstream consumers, or cross-account fan-out.

Operational Flow: From Telemetry to Action

🎯 Scenario: A payments service runs on a single c6i.4xlarge EC2 instance behind an Application Load Balancer. Every few weeks the instance hangs and CPU drops to zero. An operator wants the instance to self-heal without a human in the loop. The right pattern is two alarms: one on StatusCheckFailed_System with the EC2 recover action (hardware fault), and one on StatusCheckFailed_Instance with the EC2 reboot action (guest hang). Both alarms set treat-missing-data to breaching so they do not silently sit in INSUFFICIENT_DATA when the agent itself stops reporting.