Databricks Data Engineer Professional: Debugging and Deploying Practice Questions

Chispa is a PySpark-specific testing library that provides assert_df_equality, which natively handles row-order independence, schema comparison, and null semantics — exactly what is needed for DataFrame assertions. pytest-spark does not provide an assert_frame_equal helper. Converting to pandas with .toPandas() works in small tests but does not scale and is not order-agnostic by default. Using unittest.assertEqual on .collect() results is fragile because list equality is order-sensitive, and manually sorting adds complexity without a guarantee of correctness across all data types.

Chispa's assert_dataframes_equal supports an ignore_row_order=True parameter that sorts both DataFrames before comparing, making it safe when the output order is non-deterministic—which is common after window-based deduplication. Option C is wrong because default Chispa comparison is order-sensitive and would produce false failures when rows are shuffled. Option D is wrong because assert_approx_dataframes_equal is for numeric tolerance, not row ordering. Option B is wrong because assert_schema_equality only validates structure, not row-level data correctness.

Using databricks runs submit --wait with --new-cluster provisions an ephemeral cluster tied to that single run; the cluster is automatically terminated when the run finishes, eliminating the need for manual teardown or relying solely on auto-termination. Pinning the Databricks Runtime version to match production ensures that library versions, Spark behaviors, and Delta protocol features are consistent between tests and production, which is critical for integration test fidelity. Option A (single-node) may be insufficient for integration tests that test distributed behavior. Option B (auto-termination only) is a fallback, not a primary strategy — the cluster could still run for up to 10 minutes after completion needlessly. Option C (weekend scheduling) reduces test frequency and slows feedback loops, which defeats CI/CD goals. Option F (shared cluster pool) introduces test isolation issues since other workloads may be running on the pool.

Databricks Asset Bundles define jobs, pipelines, and configuration as code (YAML) for versioned, repeatable deployment across environments. Manual UI recreation (B) and emailing exports (C) are error-prone and unrepeatable, and a cluster (D) isn't a deployment mechanism.

Option A (generating synthetic fixtures) is the gold standard for unit/integration test data management: the test controls its own data, the data is minimal and deterministic, and cleanup is straightforward. Option F (DEEP CLONE into a test container) is the recommended approach for integration tests that need realistic data shapes/schemas: a known-good snapshot is cloned before tests run, providing a stable, isolated baseline that is not affected by production data changes. Option D (RESTORE TO VERSION 0) is only reliable if the table was originally created with version 0 as the seed; transaction log retention and vacuum operations can make this fragile. Option E (reading from production) violates test isolation — production data changes will cause test flakiness. Option C (disabling log compaction) has no bearing on test repeatability. Option B (Photon clusters) is irrelevant to test isolation and repeatability.