Data Operations and Support Study Guide — DEA-C01

Data Operations and Support

This domain accounts for 22% of the AWS DEA-C01 exam — the second-largest domain. It tests your ability to keep data pipelines healthy in production: monitoring pipeline health with CloudWatch and CloudTrail, troubleshooting failures in Glue and Kinesis, controlling costs, enforcing data quality, and optimizing query and cluster performance. Expect scenario-based questions where you must pick the right monitoring metric, the right cost lever, or the right fix for a broken pipeline.

Amazon CloudWatch: Metrics, Logs, Alarms, and Dashboards

CloudWatch is the primary observability layer for all AWS data services. Understand its three pillars for data pipelines:

Metrics

• Namespaces are service-specific: AWS/Glue, AWS/Kinesis, AWS/Redshift, AWS/EMR, AWS/States.
• Standard resolution: 1-minute granularity. High resolution: 1-second granularity (custom metrics only).
• Metric Math lets you combine metrics (e.g., compute error rate = Errors / Invocations).
• Default retention: 3 hours for 1-second metrics, 15 days for 1-minute, 63 days for 5-minute, 15 months for 1-hour.

Logs

• Log Groups → Log Streams. Glue jobs write to /aws-glue/jobs/output and /aws-glue/jobs/error by default.
• Metric Filters extract numeric values from log events and publish them as custom metrics.
• CloudWatch Logs Insights runs SQL-like queries against log groups — use for ad-hoc pipeline debugging.
• Subscription Filters stream logs in near-real-time to Kinesis, Lambda, or Firehose.

Alarms

• States: OK, ALARM, INSUFFICIENT_DATA.
• Composite Alarms combine multiple alarms with AND/OR logic — use to reduce alert noise across pipeline stages.
• Anomaly Detection uses ML to set dynamic thresholds — useful for variable pipeline throughput.

Dashboards

• Cross-account and cross-region dashboards require CloudWatch cross-account observability (sharing via Organizations).
• Dashboards are global resources; they do not live in a specific region.

Common traps:

• CloudWatch does not monitor data inside S3 objects — it monitors S3 request metrics, not content.
• INSUFFICIENT_DATA is the initial state, not an error state — do not treat it as a failure alarm.
• Metric Filters only apply to new log events after the filter is created; they are not retroactive.
• Dashboards are global but metrics are regional — a dashboard can pull metrics across regions.

AWS CloudTrail: Auditing API Calls in Data Workflows

CloudTrail records API calls (who called what, when, from where). It answers "who deleted the Glue table?" or "which IAM role called StartJobRun?"

Key Concepts

• Management events: control-plane operations (create, delete, modify resources) — enabled by default.
• Data events: object-level operations (S3:GetObject, DynamoDB:PutItem, Glue BatchGetTable) — must be explicitly enabled, billed additionally.
• Insights events: detect unusual write API activity — useful for detecting runaway Glue job launches.
• Trail: configuration that delivers events to an S3 bucket (and optionally CloudWatch Logs and EventBridge).
• A multi-region trail is best practice — single-region trails miss global service events.
• Event history: 90-day free lookup in the console, no trail required.

Data Pipeline Use Cases

• Audit CreateJob, StartJobRun, StopJobRun in Glue.
• Detect unauthorized GetObject calls on S3 data lake buckets.
• Track CreateCluster / TerminateCluster for EMR cost auditing.
• EventBridge integration: route CloudTrail events to Lambda or Step Functions for automated remediation.

Common traps:

• CloudTrail is not real-time — typical delivery to S3 is within 15 minutes.
• CloudTrail logs API calls; CloudWatch logs application/service behavior. Do not confuse them.
• Data events for S3 are logged at the object level, not the bucket level, but you configure them at the bucket level.
• Disabling a trail does not delete the S3 logs already delivered.