Machine Learning Associate · 22% of the exam

Exploratory Data Analysis: free practice questions

5 sample questions from our 55-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Databricks exam.

1. A data science team is performing EDA on a PySpark DataFrame `clickstream_df` with 50 million rows. A team member proposes: 'Let's use `pyspark.pandas` (pandas-on-Spark) so we can write familiar Pandas code while keeping data distributed.' A second team member responds: 'We should just call `.toPandas()` and work in native Pandas — it's simpler.' Which statement BEST captures the key trade-off between these two approaches?

  • A. Pandas-on-Spark always produces identical results to native Pandas, so the only difference is syntax verbosity.
  • B. Calling `.toPandas()` on 50 million rows collects all data to the driver node, risking memory exhaustion, whereas pandas-on-Spark executes operations in a distributed fashion across the cluster.✓ Correct
  • C. Pandas-on-Spark requires converting the DataFrame to a Pandas DataFrame internally before each operation, making it slower than native Pandas for large datasets.
  • D. Calling `.toPandas()` is preferred for large datasets because native Pandas uses vectorized C operations that are faster than Spark's JVM-based execution.
Explanation

B is correct: `.toPandas()` on 50 million rows forces all data onto the driver node's memory, which is likely to cause OutOfMemoryErrors on a typical driver. Pandas-on-Spark (pyspark.pandas) translates Pandas-style operations into Spark execution plans, keeping data distributed across worker nodes. A is incorrect because pandas-on-Spark does not guarantee identical results in all cases — some behaviors differ (e.g., default index handling, certain API gaps). C is incorrect because pandas-on-Spark does NOT convert data to a Pandas DataFrame before each operation; it submits Spark jobs instead. D is incorrect because while native Pandas is faster for small data due to C vectorization, it is completely impractical for 50 million rows due to driver memory constraints.

2. During EDA, a data scientist converts a large PySpark DataFrame `clickstream_df` to a Pandas DataFrame using `.toPandas()` before plotting with Matplotlib. The cluster has 16 worker nodes with 32 GB RAM each, but the driver node has only 8 GB RAM. The DataFrame has approximately 200 million rows. What is the MOST LIKELY consequence of this approach?

  • A. The operation will succeed because Spark distributes the Pandas conversion across worker nodes
  • B. The operation will trigger a driver out-of-memory (OOM) error because all data is collected to the driver✓ Correct
  • C. The Matplotlib plot will fail to render because Matplotlib cannot accept data converted from Spark
  • D. The operation will silently sample the DataFrame to fit available driver memory
Explanation

Calling .toPandas() on a PySpark DataFrame collects all data from the distributed worker nodes and materializes it as a single Pandas DataFrame on the driver node. With 200 million rows, the resulting in-memory object will almost certainly exceed the driver's 8 GB RAM, causing an out-of-memory error. The worker nodes' RAM is irrelevant because .toPandas() does not distribute processing — it centralizes all data on the driver. Matplotlib has no issue accepting Pandas DataFrames; the problem is in the data transfer step, not the plotting step. PySpark does not silently sample data during .toPandas(); it attempts to collect everything, which is why this pattern is dangerous on large datasets.

3. A data scientist wants to explore a Delta Lake table `clickstream` to understand how the distribution of the `page_views` column has changed over the past 7 daily snapshots (the table is updated once per day and history is available). Which approach correctly leverages Delta Lake's time travel feature for this EDA task?

  • A. Run `DESCRIBE HISTORY clickstream` to retrieve the `page_views` statistics for all historical versions in a single result set.
  • B. Use `spark.read.format('delta').option('timestampAsOf', '<date>').load('/mnt/data/clickstream')` or the SQL equivalent `SELECT * FROM clickstream TIMESTAMP AS OF '<date>'` for each of the 7 daily snapshots, and then compare summary statistics.✓ Correct
  • C. Enable Delta Lake's `autoOptimize` setting and run `OPTIMIZE clickstream` — this rebuilds historical snapshots and surfaces distribution changes automatically.
  • D. Use `spark.read.format('delta').option('versionAsOf', 'all').load('/mnt/data/clickstream')` to load all versions simultaneously into a single DataFrame for comparison.
Explanation

Option B is correct. Delta Lake time travel allows reading a table at a specific point in time using either `timestampAsOf` (for timestamp-based travel) or `versionAsOf` (for version-based travel). By reading each of the 7 daily snapshots separately and computing summary statistics (e.g., `describe()` or `dbutils.data.summarize()`), the data scientist can compare how the `page_views` distribution evolved over time. Option A is wrong: `DESCRIBE HISTORY` returns metadata about table operations (version number, operation type, timestamp, user) but does NOT return column-level statistics or data distributions. Option C is wrong: `OPTIMIZE` compacts small files for performance and has nothing to do with surfacing historical distributions. `autoOptimize` is a write-time setting, not an EDA feature. Option D is wrong: There is no `versionAsOf='all'` option. Delta Lake time travel requires specifying a single version or timestamp at a time; you cannot load all versions simultaneously with one read call.

4. A data science team is exploring a PySpark DataFrame `events_df` with 10 million rows for EDA. A team member suggests plotting a Seaborn histogram of the `session_duration` column. What is the MOST important prerequisite step before using Seaborn, and why?

  • A. Register `events_df` as a temporary SQL view using `events_df.createOrReplaceTempView('events')`, because Seaborn can directly query Spark SQL views.
  • B. Convert the relevant column(s) to a Pandas DataFrame or Series (e.g., via `.toPandas()` or after sampling), because Seaborn operates only on Pandas/NumPy objects and cannot consume PySpark DataFrames.✓ Correct
  • C. Cache `events_df` in memory using `events_df.cache()` before passing it to Seaborn to avoid repeated recomputation during plotting.
  • D. Install the `seaborn` library using `%pip install seaborn` in every notebook cell before calling any Seaborn functions.
Explanation

Option B is correct. Seaborn is a Python library built on Matplotlib and operates exclusively on Pandas DataFrames, NumPy arrays, or Python lists. It cannot accept PySpark DataFrames as input. Therefore, the critical prerequisite is to either sample the data first (e.g., `df.sample(fraction=0.01).toPandas()`) and then convert to Pandas, or collect a manageable subset before plotting. Option A is wrong: Seaborn has no ability to query Spark SQL views; creating a temp view does not make the data accessible to Seaborn. Option C is wrong: Caching helps with repeated Spark computations but does not enable Seaborn to read PySpark DataFrames. Caching is a Spark optimization, not a Seaborn integration step. Option D is wrong: Seaborn comes pre-installed on Databricks clusters with ML runtime; `%pip install` is not required in most cases, and even if it were, it is not the 'most important prerequisite' compared to the data conversion requirement.

5. A data engineer is preparing a large PySpark DataFrame for EDA and wants to use Pandas syntax (e.g., `.groupby()`, `.plot()`) without converting the entire dataset to a Pandas DataFrame. Which approach should they use?

  • A. Install the `pandas` library on the cluster and call df.toPandas() within a try-except block to handle memory errors gracefully.
  • B. Use `import pyspark.pandas as ps` and create a pandas-on-Spark DataFrame, which supports Pandas-like syntax on distributed data.✓ Correct
  • C. Use `spark.sql()` to write SQL queries that mimic Pandas groupby behavior, then collect results.
  • D. Enable Arrow-based optimization with `spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'true')` and call .toPandas().
Explanation

The Pandas API on Spark (pyspark.pandas, formerly Koalas) allows users to write Pandas-compatible code that executes distributedly on a Spark cluster. It is the correct solution when the dataset is too large for the driver but Pandas syntax is desired. Option A (.toPandas() in a try-except) does not solve the memory problem — it still attempts to collect all data to the driver; catching the error doesn't prevent the OOM condition. Option C (spark.sql() mimicking groupby) requires writing SQL rather than Pandas syntax, which defeats the purpose; it also requires manual collection of results. Option D (Arrow optimization + .toPandas()) speeds up the serialization of data to the driver but still collects all data to the driver — it does not enable distributed Pandas execution.

50 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free
Exploratory Data Analysis — Free Machine Learning Associate Practice Questions | DataCertPrep — Certification Prep