AWS DEA-C01: Data Ingestion and Transformation Practice Questions

A broadcast join (also called a map-side join) sends a copy of the smaller dataset to every executor, completely eliminating the shuffle for that join. At 2 GB, the product dimension table may be large for default Spark broadcast thresholds (10 MB), but in Glue/EMR you can explicitly set spark.sql.autoBroadcastJoinThreshold or use a broadcast hint (broadcast()) to force it. This is the correct optimization for repeated shuffles of a known-small dimension table. Adding more DPUs increases parallelism and memory but does not eliminate the shuffle — it just makes it faster, which doesn't address the root cause. Repartitioning all datasets to aligned partition counts can reduce some skew but does not eliminate shuffles during joins; it may even add an extra shuffle step. Glue job bookmarks track processed data to avoid reprocessing but have no impact on join execution plans or shuffle behavior within a single job run.

Kinesis Data Firehose supports dynamic partitioning, which allows you to extract values from record fields (using inline parsing or Lambda) and use them to construct custom S3 prefix paths. By enabling dynamic partitioning with a JQ expression or inline parsing on the 'sensorType' field, Firehose automatically routes each record to a prefix like s3://bucket/sensorType=temperature/. This requires no additional services. A transformation Lambda can preprocess records but cannot redirect them to different S3 prefixes — the output destination is controlled by the delivery stream's configuration, not the Lambda function's return value. Creating multiple delivery streams would work functionally but requires routing logic at the producer level and managing multiple streams, which adds operational overhead and constitutes introducing additional resources/complexity. Record format conversion (e.g., to Parquet) changes the file format but does not split records to different prefixes based on field values.

The most likely cause is that the crawler's 'Schema updates in existing tables' and 'Object deletion handling' configuration is set in a way that prevents adding new partitions — specifically, the crawler behavior for 'Add new columns only' or similar restrictive update behavior can prevent new partition discovery if the crawler is configured to not create new partitions in existing tables. In Glue crawler settings, you can configure what happens when the crawler finds new partitions: if set to update existing only, new partition paths are ignored. Option D is incorrect because Glue crawlers fully support Parquet. Option C (IAM permissions) would cause the crawler to fail entirely or log access-denied errors on all paths, not selectively skip new ones. Option A (S3 versioning) has no bearing on Glue crawler partition discovery behavior.

Firehose natively supports JSON-to-Parquet (and ORC) conversion using a schema defined in the AWS Glue Data Catalog — no Lambda required for format conversion. Dynamic partitioning allows using JQ expressions to extract fields from the payload (e.g., .event_date) to build custom S3 prefix paths. Together these features achieve serverless, schema-aware Parquet conversion with payload-based partitioning. Option A works but requires managing Lambda code and is not necessary when native conversion is available. Option C uses the Firehose ingestion timestamp (not the event's own event_date field), which does not meet the requirement of partitioning by the event's date. Option E is entirely incorrect — KMS encryption has no relationship to S3 partitioning.

Option B (resharding): With only 5 shards, each shard handles up to 1 MB/s ingest and 2 MB/s read. At 15,000 events/sec, more shards both increase ingest capacity and add more independent processing lanes for Lambda.

Option E (Enhanced Fan-Out): Standard consumers share the 2 MB/s shard read limit across all readers. Enhanced Fan-Out gives each registered consumer its own dedicated 2 MB/s per shard via a push subscription, directly reducing iterator age caused by read throttling.

Option C (batch size + parallelization factor): Increasing the batch size means each Lambda invocation processes more records, reducing invocation overhead. The parallelization factor (up to 10) allows multiple Lambda invocations per shard concurrently, multiplying throughput.

Option A (SSE) is a security feature; encryption/decryption is handled transparently by KMS and adds negligible latency. It does not affect consumer throughput or iterator age.

Option F (SQS FIFO) would require rearchitecting the pipeline. SQS FIFO has a 300 TPS limit per message group (per player), which would be even more constraining. This is not an effective solution for the throughput problem.

Option D (Firehose as intermediary) is architecturally incorrect. Firehose is a delivery service to destinations like S3/Redshift; inserting it between Kinesis and Lambda does not reduce Lambda consumer lag and adds unnecessary complexity.