Data Engineer Associate · 34% of the exam

Data Ingestion and Transformation: free practice questions

5 sample questions from our 85-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real AWS exam.

1. A company needs to replicate data from an on-premises Oracle database to Amazon S3 using AWS DMS. The source tables contain BLOB columns storing binary document files up to 500 MB each. The DMS task consistently fails with replication errors when processing rows with large BLOBs. Which configuration change should the engineer make FIRST?

  • A. Increase the DMS replication instance class to a larger instance type (e.g., dms.r5.4xlarge)
  • B. Enable the 'Limit LOB mode' setting and set the maximum LOB size to 600 MB to fully accommodate the largest BLOBs
  • C. Use 'Full LOB mode' in the DMS task settings to ensure complete LOB data is replicated without truncation✓ Correct
  • D. Exclude the BLOB columns from replication using column-level transformation rules, and replicate the BLOBs separately using a custom script
Explanation

AWS DMS offers three LOB (Large Object) handling modes: Limited LOB mode (truncates LOBs above a set size), Full LOB mode (replicates complete LOBs regardless of size, at lower performance), and Inline LOB mode. When BLOBs up to 500 MB are failing, switching to Full LOB mode ensures the entire LOB is fetched and replicated without truncation. Option A (larger instance) may help with memory pressure but does not address the root cause — LOB replication mode configuration. Option B (Limit LOB mode at 600 MB) sounds correct but Limited LOB mode still reads LOB data inline up to the limit and can cause memory issues with very large BLOBs — Full LOB mode uses a different, more robust fetch strategy for large objects. Option D (excluding BLOBs) avoids the problem but results in incomplete data replication.

2. A company's security policy requires that all data passing through Amazon Kinesis Data Streams must be encrypted at rest using customer-managed keys (CMK) stored in AWS KMS. Additionally, only a specific IAM role used by the producer application should be able to publish records to the stream. Which TWO configurations enforce BOTH requirements? (Select TWO)

  • A. Enable server-side encryption on the Kinesis Data Stream and specify the ARN of the customer-managed KMS key.✓ Correct
  • B. Attach an IAM policy to the producer IAM role that allows kinesis:PutRecord and kinesis:PutRecords on the specific stream ARN, and deny these actions for all other principals using a Kinesis resource-based policy.✓ Correct
  • C. Enable Kinesis Data Streams Enhanced Fan-Out on the stream and associate it with the producer's IAM role.
  • D. Configure a VPC endpoint for Kinesis and restrict access to the stream using VPC endpoint policies.
  • E. Use AWS Secrets Manager to store the Kinesis stream name and rotate it daily to prevent unauthorized access.
Explanation

Option A directly addresses the encryption requirement: Kinesis Data Streams supports SSE with CMKs via KMS. Specifying the CMK ARN ensures all data is encrypted at rest under the customer-managed key. Option B addresses the access restriction requirement: an IAM policy grants only the producer role PutRecord/PutRecords permissions on the specific stream, while a resource-based policy (Kinesis stream policy) can explicitly deny all other principals, enforcing least privilege. Option C (Enhanced Fan-Out) is a consumer-side feature for high-throughput consumption using dedicated 2 MB/s per-shard read throughput. It has nothing to do with producer access control or encryption. Option D (VPC endpoint + endpoint policy) adds network-level restriction but does not enforce IAM-identity-based producer restrictions or CMK encryption on its own — it is a complementary control, not a complete solution for either requirement. Option E (Secrets Manager for stream name) is not a security control. Rotating the stream name does not prevent unauthorized access; access is controlled by IAM and resource policies, not by obscuring the stream name.

3. A company runs an AWS Step Functions Standard Workflow that orchestrates a nightly data pipeline. The workflow invokes an AWS Glue job and then waits for it to complete before proceeding. A junior engineer implements the Glue job invocation using a Task state with the Glue:startJobRun API and polls every 30 seconds using a retry loop built with a Wait state and a Lambda function that calls GetJobRun. A senior engineer suggests replacing this with a single Task state using the '.sync:2' resource integration. What is the PRIMARY advantage of the '.sync:2' (synchronous SDK integration) approach?

  • A. The .sync:2 integration runs the Glue job inside the Step Functions execution context, reducing job startup latency
  • B. The .sync:2 integration uses Step Functions' built-in polling mechanism, eliminating the need for a custom Lambda polling function and reducing state transitions and cost✓ Correct
  • C. The .sync:2 integration automatically retries the Glue job up to three times on failure without requiring an explicit Retry configuration in the state definition
  • D. The .sync:2 integration enables the Glue job to write its output directly to the Step Functions execution history for auditing
Explanation

The optimistic integration pattern (.sync:2, also known as the 'Run a Job' pattern) causes Step Functions to start the Glue job and then wait for it to complete by polling the job status internally using AWS SDK calls — without requiring any custom Lambda function or explicit Wait/Choice state loop. This simplifies the workflow definition, reduces the number of state transitions (which directly reduces cost for Standard Workflows, billed per state transition), and eliminates the operational overhead of maintaining a polling Lambda. The .sync:2 integration does NOT run the Glue job inside Step Functions — Glue jobs still run on Glue's infrastructure. Automatic retries must still be explicitly configured via the Retry field in the Task state; .sync:2 does not add implicit retry behavior. Glue job output is written to S3 or other targets as configured in the Glue job itself — it is not written to Step Functions execution history regardless of integration type.

4. A company uses AWS Glue Studio to run a PySpark ETL job that reads 50 GB of Apache Parquet files from Amazon S3, applies several column-level transformations, and writes results back to S3. A data engineer wants to reduce the job's runtime. After profiling, they find that 80% of the job time is spent in a wide transformation (groupBy + aggregation) that causes a full shuffle. Which TWO actions would MOST directly reduce the shuffle overhead? (Select TWO)

  • A. Increase the number of Glue DPUs allocated to the job.✓ Correct
  • B. Repartition the input data by the groupBy key before the aggregation using .repartition().✓ Correct
  • C. Enable Glue job bookmarks on the job.
  • D. Use .coalesce(1) immediately before the groupBy aggregation.
  • E. Cache the input DataFrame in memory before the groupBy step using .cache().
  • F. Switch the output format from Parquet to CSV to reduce serialization cost.
Explanation

Adding more DPUs gives Spark more executor cores and memory, so each shuffle partition is processed faster and less data is spilled to disk. Pre-partitioning the DataFrame by the groupBy key using .repartition() co-locates records with the same key on the same executor, reducing the amount of data that must cross the network during the shuffle. Option C (job bookmarks) only tracks already-processed data to avoid reprocessing; it has no effect on shuffle performance. Option D (.coalesce(1)) reduces parallelism to a single partition, which serializes all work onto one executor and would dramatically increase runtime. Option E (caching) helps when the DataFrame is reused multiple times in the DAG, but does not reduce shuffle data volume for a single aggregation pass. Option F (CSV output) has no effect on shuffle overhead during the transformation phase.

5. A data engineering team needs to ingest customer order records from an on-premises relational database into Amazon S3 in Parquet format for analytics. The records arrive in micro-batches every 15 minutes, and the team wants to minimize operational overhead. They also need automatic schema detection and cataloging so downstream Amazon Athena queries work immediately after each load. Which combination of AWS services best satisfies these requirements?

  • A. AWS DMS to replicate records to an Amazon S3 staging prefix in CSV, then an AWS Glue crawler to update the Data Catalog, then an AWS Glue job to convert CSV to Parquet in a separate prefix.
  • B. Amazon Kinesis Data Firehose with a built-in format conversion to Parquet (backed by an AWS Glue Data Catalog table schema), delivering directly to the final S3 prefix.✓ Correct
  • C. AWS Lambda triggered every 15 minutes by Amazon EventBridge to query the source database and write Parquet files directly to S3, with manual Athena table DDL maintained by the team.
  • D. Amazon MSK Connect with a JDBC source connector writing to an S3 sink connector in Parquet format, with an AWS Glue crawler scheduled every 15 minutes.
  • E. AWS Glue DataBrew scheduled job reading from the source database via a JDBC connection and outputting Parquet files to S3 every 15 minutes.
Explanation

Amazon Kinesis Data Firehose natively supports dynamic Parquet/ORC format conversion using a schema defined in the AWS Glue Data Catalog — no separate conversion job is required. It writes directly to S3 and the Glue Data Catalog table is already available for Athena, minimizing operational overhead. Option A works but is more complex: it requires three separate components (DMS, crawler, Glue job) and adds latency between landing and Parquet availability. Option C involves custom Lambda code to handle JDBC queries and Parquet serialization, which increases operational burden and is error-prone. Option D (MSK Connect) is a valid Kafka-based approach but involves deploying and managing MSK Connect workers, which is higher operational overhead than Firehose for a simple 15-minute micro-batch case. Option E: AWS Glue DataBrew is primarily a data preparation and profiling tool; it supports scheduled runs but is not designed as a recurring ingestion pipeline and lacks a native JDBC source for streaming/micro-batch ingest.

80 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free