Data Ingestion and Transformation Study Guide — DEA-C01

Data Ingestion and Transformation

This domain accounts for 34% of the AWS DEA-C01 exam — the single largest domain. It tests your ability to design, build, and optimize data ingestion pipelines (both batch and streaming), apply transformations at scale, and orchestrate complex workflows. Expect questions on selecting the right AWS service for a given requirement, understanding service limits and defaults, and troubleshooting pipeline failures. Scenario-based questions dominate: you will be given a business constraint (latency, cost, throughput, ordering) and asked to choose and configure the correct combination of AWS services.

AWS Glue

AWS Glue is a fully managed serverless ETL service. It has several distinct components the exam tests separately.

Glue Crawlers

A crawler scans data stores (S3, JDBC, DynamoDB) and automatically infers schema, creating or updating tables in the Glue Data Catalog. Crawlers use classifiers to determine format. Default behavior: a crawler creates one table per S3 folder (prefix). Crawlers can be scheduled or triggered on demand.

• Partition detection: Crawlers recognize Hive-style partitions (year=2024/month=01/) and add them as partition keys in the catalog.
• Incremental crawls: Only crawl new/changed files using the "crawl new folders only" setting.

Glue Data Catalog

The Data Catalog is a central metadata repository — a managed Apache Hive Metastore compatible with Athena, EMR, and Redshift Spectrum. Each AWS account has one Data Catalog per region. It stores databases, tables, partitions, and connections.

Glue Jobs

Glue Jobs execute ETL scripts in Apache Spark (Python/Scala) or Python shell. Three job types:

DPU (Data Processing Unit): 4 vCPUs + 16 GB RAM. Default Spark job = 10 DPUs. You pay per second (minimum 1 minute).

Job bookmarks: Track processed data to avoid reprocessing. Works for S3 and JDBC sources. Enable per-job.

Glue Studio and DataBrew

Common traps

• Crawlers do not automatically update partition metadata after new data lands — use MSCK REPAIR TABLE in Athena or re-run the crawler, or use Glue partition indexing.
• Glue Job bookmarks track jobs, not crawlers.
• Python Shell jobs cannot use Glue's DynamicFrames — only DataFrames.
• Glue DataBrew does not run Spark — it's a separate profiling/recipe service; don't confuse it with a Glue job.
• Glue catalog is regional — cross-region access requires catalog federation or resource-based policies.

Worked scenario: A data team needs non-technical analysts to clean and normalize a CSV dataset in S3 without writing code, then output to Parquet. → Glue DataBrew (visual recipes, no-code, supports Parquet output). Not Glue Studio (requires understanding ETL nodes).

Amazon Kinesis Data Streams

Amazon Kinesis Data Streams (KDS) provides real-time, durable streaming with configurable retention and shard-based throughput.

Shards and Throughput

• 1 shard = 1 MB/s write, 2 MB/s read (shared across all standard consumers on that shard — not per consumer)
• Enhanced Fan-Out (EFO): Each registered consumer gets 2 MB/s per shard via HTTP/2 push — independent of other consumers. Reduces latency to ~70 ms vs ~200 ms for standard polling.
• Partition key: Determines which shard a record goes to (MD5 hash). Use high-cardinality keys to avoid hot shards.

Retention and Limits

On-demand mode: Auto-scales shards; no capacity planning. Scales up to 200 MB/s write, 400 MB/s read. Billed per GB ingested/retrieved — more expensive at steady high throughput. Provisioned mode: Manual shard management, cheaper at predictable load.

Common traps

• ProvisionedThroughputExceededException on reads in standard mode often means too many consumers — fix with EFO, not adding shards.
• Increasing shards does not automatically increase consumer read throughput if they are polling the same shard (standard mode limit is shared across all standard consumers on that shard).
• Sequence numbers guarantee ordering per shard, not across shards.
• Resharding (splitting/merging) is not instantaneous — parent shards remain readable until their data expires.