Pass the Databricks Certification Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions and answers with ExamsMirror

Comprehensive and Detailed Explanation From Exact Extract:

In Apache Spark, the execution hierarchy is structured as follows:

Application: The highest-level unit, representing the user program built on Spark.

Job: Triggered by an action (e.g.,collect(),count()). Each action corresponds to a job.

Stage: A job is divided into stages based on shuffle boundaries. Each stage contains tasks that can be executed in parallel.

Task: The smallest unit of work, representing a single operation applied to a partition of the data.

When thecollect()action is invoked, Spark initiates a job. This job is then divided into stages at points where data shuffling is required (i.e., wide transformations). Each stage comprises tasks that are distributed across the cluster's executors, operating on individual data partitions.

This hierarchical execution model allows Spark to efficiently process large-scale data by parallelizing tasks and optimizing resource utilization.

Comprehensive and Detailed Explanation From Exact Extract:

ThegroupBy()operation causes a shuffle because it requires all values for a specific key to be brought together, which may involve moving data across partitions.

In contrast:

filter()andselect()are narrow transformations and do not cause shuffles.

coalesce()tries to reduce the number of partitions and avoids shuffling by moving data to fewer partitions without a full shuffle (unlikerepartition()).

broadcast(df1) tells Spark to send the small DataFrame (df1) to all worker nodes.

This eliminates the need for shuffling df1 during the join.

Broadcast joins are optimized for scenarios with one large and one small table.

Comprehensive and Detailed Explanation From Exact Extract:

Shuffling occurs in operations likegroupBy,reduceByKey, orjoin—which cause data to be moved across partitions. Therepartition()operation can also cause a shuffle, but in this context, it follows an aggregation.

InOption D, thegroupByfollowed byaggresults in a shuffle due to grouping across nodes.

Therepartition(10)is a partitioning transformation but does not involve a new shuffle since the data is already grouped.

This sequence — shuffle (groupBy) followed by non-shuffling (repartition) — is correct.

Option A does the opposite: thefilterdoes not cause a shuffle, butgroupBydoes — this makes it the wrong order.

Comprehensive and Detailed Explanation From Exact Extract:

Apache Spark employs a lazy evaluation model for transformations. This means that when transformations (e.g.,map(),filter()) are applied to a DataFrame, Spark does not execute them immediately. Instead, it builds a logical plan (lineage) of transformations to be applied.

Execution is deferred until an action (e.g.,collect(),count(),save()) is called. At that point, Spark's Catalyst optimizer analyzes the logical plan, optimizes it, and then executes the physical plan to produce the result.

This lazy evaluation strategy allows Spark to optimize the execution plan, minimize data shuffling, and improve overall performance by reducing unnecessary computations.

To persist a table and specify the save path, use:

users.write.option("path","/some/path").saveAsTable("default_table")

The .option("path", ...) must be applied before calling saveAsTable.

Option A uses invalid syntax (write(path=...)).

Option B applies.option()after.saveAsTable()—which is too late.

Option D uses incorrect syntax (no path parameter in saveAsTable).

Comprehensive and Detailed Explanation:

The approx_percentile function in Spark is a performance-optimized alternative to percentile. It takes an optional accuracy parameter:

approx_percentile(column, percentage, accuracy)

Higher accuracy values → more precise results, but increased memory/computation.

Lower values → faster but less accurate.

From the documentation:

“Increasing the accuracy improves precision but increases memory usage.”

Final Answer: D

Comprehensive and Detailed Explanation From Exact Extract:

To flatten an array of structs into individual rows and access fields within each struct, you must:

Useexplode()to expand the array so each struct becomes its own row.

Access the struct fields via dot notation (e.g.,record_exploded.sensor_id).

Option C does exactly that:

First, explode therecordarray column into a new columnrecord_exploded.

Then, access fields of the struct using the dot syntax inselect.

This is standard practice in PySpark for nested data transformation.

Final Answer: C

spark.executor.cores determines how many concurrent tasks an executor can run.

For example, if set to 4, each executor can run up to 4 tasks in parallel.

Other settings:

spark.task.maxFailures controls task retry logic.

spark.driver.cores is for the driver, not executors.

spark.executor.memory sets memory limits, not task concurrency.

Comprehensive and Detailed Explanation From Exact Extract:

To write a structured streaming DataFrame to Parquet files, the correct way to specify the format and output directory is:

writeStream

format("parquet")

option("path", "path/to/destination/dir")

According to Spark documentation:

“When writing to file-based sinks (like Parquet), you must specify the path using the .option("path", ...) method. Unlike batch writes, .save() is not supported.”

Option A incorrectly uses.option("location", ...)(invalid for Parquet sink).

Option B incorrectly sets the format via.option("format", ...), which is not the correct method.

Option C repeats the same issue.

Option D is correct:.format("parquet")+.option("path", ...)is the required syntax.

Final Answer: D