Pass the Databricks Certification Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions and answers with ExamsMirror

When queries frequently filter on certain columns, partitioning by those columns ensures partition pruning, allowing Spark to scan only relevant directories instead of the entire dataset.

Correct code:

final.write.partitionBy("event_year", "event_month").parquet("events.liveLatest")

This improves read performance dramatically for filters like:

SELECT * FROM events.liveLatest WHERE event_year = 2024 AND event_month = 5;

bucketBy() helps in clustering and joins, not in partition pruning for file-based tables.

Why the other options are incorrect:

B: Bucket count changes parallelism, not query pruning.

C: sortBy organizes data within files, not across partitions.

D: Partitioning by only one column limits pruning benefits.

The .coalesce(numPartitions) function is used to reduce the number of partitions in a DataFrame. It does not increase the number of partitions. If the specified number of partitions is greater than the current number, it will not have any effect.

From the official Spark documentation:

“coalesce() results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim one or more of the current partitions.”

However, if you try to increase partitions using coalesce (e.g., from 10 to 20), the number of partitions remains unchanged.

Hence, df.coalesce(20) will still return a DataFrame with 10 partitions.

Apache Spark is a distributed data processing engine designed for large-scale, cluster-based computation.

Advantages:

Horizontal Scalability: Spark can distribute tasks across many machines, handling datasets larger than the memory of a single node.

Fault Tolerance: Spark automatically recovers from node or task failures using the lineage graph (RDD recovery mechanism) and retry logic.

These two features allow Spark to process huge datasets efficiently and reliably, unlike standard Python scripts that are limited to one machine and fail on single-node errors.

Why the other options are incorrect:

B: Spark runs on commodity hardware; no specialized machines required.

C: Spark emphasizes in-memory processing, not disk-only operations.

D: Spark still requires user code in Python, Scala, SQL, or Java.

Spark automatically uses a broadcast hash join when one side of the join is small enough to fit within the broadcast threshold.

Default threshold:

spark.sql.autoBroadcastJoinThreshold = 10MB (as of Spark 3.5)

Since df2 is 8 MB, Spark automatically broadcasts it to all executors. This avoids a shuffle on the large dataset (df1) and speeds up the join.

Why the other options are incorrect:

A: 8 MB < 10 MB threshold → broadcast join is efficient.

B: AQE is not required for broadcast joins; it’s a static optimization.

C: Broadcast hints are optional — Spark infers automatically.

df.cache() is shorthand for df.persist(StorageLevel.MEMORY_AND_DISK)

df.persist() allows specifying any storage level such as MEMORY_ONLY, DISK_ONLY, MEMORY_AND_DISK_SER, etc.

By default, persist() uses MEMORY_AND_DISK, unless specified otherwise.

To enable fault tolerance and ensure that Spark can resume from the last committed offset after failure, you must configure a checkpoint location using the correct option key: "checkpointLocation".

From the official Spark Structured Streaming guide:

“To make a streaming query fault-tolerant and recoverable, a checkpoint directory must be specified using .option("checkpointLocation", "/path/to/dir").”

Explanation of options:

Option A uses an invalid option name: "checkpoint" (should be "checkpointLocation")

Option B is correct: it sets checkpointLocation properly

Option C lacks checkpointing and won’t resume after failure

Option D also lacks checkpointing configuration

To exclude sensitive (PII) columns from a DataFrame, the easiest method is to use the .drop() function with the list of column names to remove.

Correct syntax:

df_user_non_pii = df_user.drop("name", "email", "birthdate")

This creates a new DataFrame containing all remaining columns.

Why the other options are incorrect:

B: .dropFields() is not valid for standard DataFrames — it’s used for struct fields only.

C: .select() would keep only PII columns, not remove them.

D: .remove() does not exist in Spark DataFrame API.

To persist a table and specify the save path, use:

users.write.option("path", "/some/path").saveAsTable("default_table")

The .option("path", ...) must be applied before calling saveAsTable.

Option A uses invalid syntax (write(path=...)).

Option B applies .option() after .saveAsTable()—which is too late.

Option D uses incorrect syntax (no path parameter in saveAsTable).

Adaptive Query Execution (AQE) is a Spark SQL feature introduced to dynamically optimize queries at runtime based on actual data statistics collected during execution.

Key benefits include:

Runtime plan adaptation: Spark adjusts the physical plan after some stages complete.

Skew handling: Automatically splits skewed partitions to balance work distribution.

Join strategy optimization: Dynamically switches between shuffle join and broadcast join depending on partition sizes.

Coalescing shuffle partitions: Reduces the number of small tasks for better performance.

Example configuration:

spark.conf.set("spark.sql.adaptive.enabled", True)

This enables AQE globally in Spark 3.5.

Why the other options are incorrect:

A: AQE adapts during runtime, not only before execution.

B: Task distribution is a base Spark feature, not specific to AQE.

C: AQE specifically addresses runtime skew and join adjustments.

Spark's best practice is to estimate partition count based on data volume and a reasonable partition size — typically 128 MB to 256 MB per partition.

With 1 TB of data: 1 TB / 128 MB ≈ ~8000 partitions

This ensures that tasks are distributed across available CPUs for parallelism and that each task processes an optimal volume of data.

Option A (equal to cores) may result in partitions that are too large.

Option B (fixed 200) is arbitrary and may underutilize the cluster.

Option C (nodes) gives too few partitions (10), limiting parallelism.