Pass the Databricks Certification Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions and answers with ExamsMirror

Comprehensive and Detailed Explanation From Exact Extract:

To extract unique values from a column and sort them alphabetically:

distinct()is required to remove duplicate values.

orderBy()is needed to sort the results alphabetically (ascending by default).

Correct code:

df.select("Name").distinct().orderBy(df["Name"])

This is directly aligned with standard DataFrame API usage in PySpark, as documented in the official Databricks Spark APIs. Option A is incorrect because it may not remove duplicates. Option C omits sorting. Option D sorts in descending order, which doesn't meet the requirement for alphabetical (ascending) order.

dropDuplicates() with no column list removes duplicates based on all columns.

It's the most efficient and semantically correct way to deduplicate records that are completely identical across all fields.

From the PySpark documentation:

dropDuplicates(): Return a new DataFrame with duplicate rows removed, considering all columns if none are specified.

— Source:PySpark DataFrame.dropDuplicates() API

Comprehensive and Detailed Explanation From Exact Extract:

The.coalesce(numPartitions)function is used to reduce the number of partitions in a DataFrame. It does not increase the number of partitions. If the specified number of partitions is greater than the current number, it will not have any effect.

From the official Spark documentation:

“coalesce() results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim one or more of the current partitions.”

However, if you try to increase partitions using coalesce (e.g., from 10 to 20), the number of partitions remains unchanged.

Hence,df.coalesce(20)will still return a DataFrame with 10 partitions.

Comprehensive and Detailed Explanation From Exact Extract:

According to the PySpark API documentation:

"getOrCreate(): Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder."

This means Spark maintains a global singleton session within a JVM process. Repeated calls togetOrCreate()return the same session, unless explicitly stopped.

Option A is incorrect: the method does not destroy any session.

Option B incorrectly ties uniqueness toappName, which does not influence session reusability.

Option D is incorrect: it contradicts the fundamental behavior ofgetOrCreate().

(Source:PySpark SparkSession API Docs)

Comprehensive and Detailed Explanation:

Spark Connect introduces a decoupled client-server architecture. Its key feature is enabling Spark job submission and execution from remote clients — in Python, Java, etc.

From Databricks documentation:

“Spark Connect allows remote clients to connect to a Spark cluster and execute Spark jobs without being co-located with the Spark driver.”

A is close, but "any language" is overstated (currently supports Python, Java, etc., not literally all).

B refers to REST, which is not Spark Connect's mechanism.

D is incorrect; Spark Connect isn’t focused on ingestion.

Final Answer: C

A broadcast join is a type of join where the smaller DataFrame is replicated (broadcast) to all worker nodes in the cluster. This avoids shuffling the large DataFrame across the network.

Benefits:

Eliminates shuffle for the smaller dataset.

Greatly improves performance when one side of the join is small enough to fit in memory.

Correct usage example:

df_result = df2.join(broadcast(df1), "id")

This is a map-side join, where each executor joins its local partition of the large dataset with the broadcasted copy of the small one.

Why the other options are incorrect:

A: Broadcasting does not change partition sizes.

B: Joins always match on key equality; this is not specific to broadcast joins.

D: Broadcasting does not filter; it distributes data for faster joins.

In Spark’s client mode, the driver runs on the local machine that submitted the job. If that machine is resource-constrained (e.g., low memory), performance degrades.

From the Spark documentation:

"In cluster mode, the driver runs inside the cluster, benefiting from cluster resources and scalability."

Option A is incorrect — executors do not help the driver directly.

Option B might help short-term but does not scale.

Option C is correct — switching to cluster mode moves the driver to the cluster.

Option D (local mode) is for development/testing, not production.

Final Answer: C

dropDuplicates() with no column list removes duplicates based on all columns.

It's the most efficient and semantically correct way to deduplicate records that are completely identical across all fields.

From the PySpark documentation:

dropDuplicates(): Return a new DataFrame with duplicate rows removed, considering all columns if none are specified.

— Source: PySpark DataFrame.dropDuplicates() API

Adaptive Query Execution (AQE) is a powerful optimization framework introduced in Apache Spark 3.0 and enabled by default since Spark 3.2. It dynamically adjusts query execution plans based on runtime statistics, leading to significant performance improvements. The key benefits of AQE include:

Dynamic Join Strategy Selection: AQE can switch join strategies at runtime. For instance, it can convert a sort-merge join to a broadcast hash join if it detects that one side of the join is small enough to be broadcasted, thus optimizing the join operation .

Handling Skewed Data: AQE detects skewed partitions during join operations and splits them into smaller partitions. This approach balances the workload across tasks, preventing scenarios where certain tasks take significantly longer due to data skew .

Coalescing Post-Shuffle Partitions: AQE dynamically coalesces small shuffle partitions into larger ones based on the actual data size, reducing the overhead of managing numerous small tasks and improving overall query performance .

These runtime optimizations allow Spark to adapt to the actual data characteristics during query execution, leading to more efficient resource utilization and faster query processing times.

The original code has several issues:

It references a variable answer that is undefined.

The function is annotated to return a str, but the logic attempts numeric multiplication.

The UDF return type is declared as T.IntegerType() but the function performs a floating-point operation, which is incompatible.

Option B correctly:

Uses DoubleType to reflect the fact that the multiplication involves a float (3.14159).

Declares the input as float, which aligns with the multiplication.

Returns a float, which matches both the logic and the schema type annotation.

This structure aligns with how PySpark expects User Defined Functions (UDFs) to be declared:

"To define a UDF you must specify a Python function and provide the return type using the relevant Spark SQL type (e.g., DoubleType for float results)."

Example from official documentation:

from pyspark.sql.functions import udf

from pyspark.sql.types import DoubleType

@udf(returnType=DoubleType())

def multiply_by_pi(x: float) -> float:

return x * 3.14159

This makes Option B the syntactically and semantically correct choice.