Data Engineering Study Guide — MLS-C01

Data Engineering for AWS MLS-C01

This domain accounts for 20% of the exam and tests your ability to select the right AWS services and configurations for ingesting, storing, transforming, and validating data that feeds ML workloads. Expect scenario-based questions where you must choose between services, justify storage formats, or identify what breaks a pipeline. The emphasis is on decision-making, not just service definitions.

Amazon S3: Data Lake Patterns and Storage

S3 is the default storage layer for ML datasets on AWS. Key exam decisions:

• Partitioning strategy: Partition by features used in query filters (e.g., year/month/day for time-series, label/ for training splits). Hive-style partitioning (key=value/) is recognized natively by Glue crawlers and Athena.
• Storage classes: Use S3 Intelligent-Tiering for datasets with unpredictable access. S3 Glacier is appropriate for archiving raw data you must retain but rarely re-train on. Standard-IA suits infrequently accessed feature stores.
• Trap: S3 itself does not enforce schema or data quality—that's Glue or Lake Formation's job. Don't confuse S3 access policies with Lake Formation column-level permissions.

AWS Glue: ETL, Crawlers, and Data Catalog

Glue is the primary managed ETL service for ML pipelines.

• Crawlers infer schema from S3, RDS, Redshift, and populate the Data Catalog. The catalog acts as a central metastore usable by Athena, EMR, and Redshift Spectrum.
• Glue ETL jobs run PySpark or Python shell scripts. Use Glue DynamicFrames (not Spark DataFrames) when schema is inconsistent across records—DynamicFrames handle schema evolution more gracefully.
• Glue DataBrew is a no-code/low-code tool for data profiling and transformation. It's the right answer when the question mentions non-technical users, visual data prep, or pre-built transformation recipes.
• Glue Workflows orchestrate crawlers + ETL jobs in sequence.
• Trap: Glue is not real-time. For streaming transforms, use Kinesis Data Streams + Kinesis Data Analytics or a Glue Streaming job (micro-batch, not true real-time).

Amazon Kinesis: Streaming Ingestion for Real-Time ML

• Kinesis Data Streams (KDS): Low-latency ingestion, 1–7 day retention, multiple consumers. Use when downstream ML inference needs near-real-time features.
• Kinesis Data Firehose: Managed delivery to S3, Redshift, OpenSearch. Supports inline Lambda transforms before delivery. Use for landing streaming data into a data lake without managing consumers.
• Kinesis Data Analytics: SQL or Apache Flink for real-time stream processing—feature computation on live data before feeding a model endpoint.
• Decision rule: KDS → when you need replay or fan-out to multiple consumers. Firehose → when you need zero-management delivery to a sink. Analytics → when you need stateful stream transformations.
• Trap: Firehose is not a real-time query engine. It buffers (minimum 60 seconds or 1 MB) before delivery.

AWS Lake Formation: Governed Data Lakes

Lake Formation sits on top of S3 and Glue to provide fine-grained access control.

• Column- and row-level security for data accessed via Athena, Glue, or Redshift Spectrum. This is the answer when a question asks about restricting PII columns from certain IAM roles or teams.
• Cross-account data sharing: Lake Formation allows sharing Data Catalog databases/tables across AWS accounts without copying data.
• Governed Tables support ACID transactions and automatic compaction—useful for ML feature tables that receive concurrent writes.
• Trap: Lake Formation permissions layer on top of S3 bucket policies. Both must allow access; Lake Formation alone is not sufficient if the S3 bucket policy denies the principal.