AWS MLS-C01: Data Engineering Practice Questions

Kinesis Data Streams with enhanced fan-out provides dedicated 2 MB/s per-shard throughput to each consumer with ~70ms propagation latency, enabling low-latency, high-throughput ingestion that can handle spikes via shard scaling. AWS Lambda triggered from Kinesis Data Streams processes records in near-real-time (sub-second), meeting the 500ms latency requirement without managing servers. Kinesis Data Firehose has a minimum buffer interval of 60 seconds, making it unsuitable for sub-500ms latency requirements. SQS Standard Queue does not guarantee ordering and its polling model introduces variable latency that may exceed 500ms at high throughput. Amazon MSK is a valid high-throughput streaming option but adds operational complexity; combined with Lambda it is less native than Kinesis+Lambda, and MSK alone does not address the downstream processing latency requirement.

AWS Glue Data Quality, which is powered by the open-source AWS Deequ library, provides a declarative rules engine for defining and enforcing data quality constraints such as completeness checks (missing values), statistical profile comparisons (drift detection), and range/constraint validations (e.g., ColumnValues between 0 and 120). It integrates natively with the Glue Data Catalog and can be embedded in ETL pipelines. Amazon CloudWatch Metrics can monitor S3 storage metrics but has no ability to inspect record-level data content or statistical properties. SageMaker Clarify is designed for bias and fairness detection and explainability, not general-purpose data quality rule enforcement. AWS Lambda with custom scripts is feasible but is not purpose-built, requires significant custom development and maintenance, and lacks the built-in statistical profile comparison and rule management that Deequ provides.

Increasing the batch window and batch size reduces the number of Lambda invocations by processing more records per call, reducing per-invocation overhead and helping Lambda keep up with the stream. Switching to SQS allows Lambda to scale more elastically, as SQS event sources support higher concurrency scaling and are not bound by Kinesis shard limits. Increasing Lambda's reserved concurrency allows more concurrent executions to process records in parallel, directly addressing throttling. Enhanced Fan-Out provides dedicated 2 MB/s throughput per consumer per shard and reduces read latency, but it does not resolve Lambda throttling caused by insufficient concurrency or processing rate — the bottleneck here is Lambda, not read throughput from Kinesis. Enabling server-side encryption adds negligible overhead to Lambda and does not affect throttling. Kinesis Data Firehose is a delivery service for destinations like S3, Redshift, or OpenSearch — it cannot write directly to DynamoDB and would not resolve this issue.

Amazon Kinesis Data Streams is purpose-built for real-time, high-throughput streaming ingestion and can handle hundreds of thousands of records per second with sub-second latency, making it ideal for real-time fraud detection. AWS Glue streaming ETL can process data from Kinesis but is a transformation layer on top of a streaming source, not an ingestion mechanism on its own. Amazon SQS FIFO queues provide reliable message queuing but impose a throughput limit of 3,000 messages per second per API action and are not designed for real-time analytics pipelines at this scale. AWS DMS is designed for database migration and change data capture, not for high-throughput clickstream ingestion.

CloudTrail data events combined with Lake Formation audit logging provide a complete record of who accessed which resources, satisfying requirement 1. Lake Formation row-level security (data filters) can restrict which rows each principal sees at query time—data scientists get a filter excluding underage patients, compliance analysts get the full dataset—satisfying requirement 2. IAM permission boundaries on the compliance analyst IAM roles denying s3:PutObject except on approved buckets enforce the export restriction at the AWS API level, satisfying requirement 3. S3 Object Lock in governance mode prevents object deletion/overwrite by non-privileged users within the same bucket; it does not prevent principals from copying objects to different personal buckets. Amazon Macie detects sensitive data through classification and generates findings, but it does not enforce real-time blocking of S3 PUT operations—it is a detection, not a prevention control. Lake Formation transaction logging tracks table-level DML transactions in governed tables but does not provide the requester-identity audit trail equivalent to CloudTrail for all access types.