Pass the Databricks Certification Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions and answers with ExamsMirror

The Spark UI is the standard and most effective way to inspect executor logs, task time, input size, and shuffles.

From the Databricks documentation:

“You can monitor job execution via the Spark Web UI. It includes detailed logs and metrics, including task-level execution time, shuffle reads/writes, and executor memory usage.”

(Source: Databricks Spark Monitoring Guide)

Option A is incorrect: logs are not guaranteed to be in /tmp, especially in cloud environments.

B. —verbose helps during job submission but doesn't give detailed executor logs.

D. spark-sql is a CLI tool for running queries, not for inspecting logs.

Hence, the correct method is using the Spark UI → Stages tab → Executor logs.

The split() function in PySpark splits strings into an array based on a given delimiter.

Then, .getItem(index) extracts a specific element from the array.

Correct usage:

from pyspark.sql.functions import split, col

customerDF = customerDF \

withColumn("username", split(col("email"), "@").getItem(0)) \

withColumn("domain", split(col("email"), "@").getItem(1))

This creates two new columns derived from the email field:

"username" → text before @

"domain" → text after @

Why the other options are incorrect:

B: regexp_replace only replaces text; does not split into multiple columns.

C: .select() cannot alias multiple derived columns like this.

D: Column objects are not native Python strings; cannot use standard .split().

In Spark, a shuffle occurs when data needs to be redistributed across partitions or nodes, typically due to operations that require grouping, joining, or sorting.

groupBy triggers a shuffle because it aggregates data based on key values, requiring data from multiple partitions to be moved across the cluster.

Operations like filter and select are narrow transformations (no shuffle).

coalesce can reduce the number of partitions without a full shuffle (unlike repartition).

Why the other options are incorrect:

A (filter) – Narrow transformation, no shuffle.

B (select) – Simple projection, no data movement.

D (coalesce) – Reduces partitions locally without shuffle.

Comprehensive Explanation:

To cover structured data processing, SQL querying, and machine learning in Apache Spark, the correct combination of components is:

Spark DataFrames: for structured data processing

Spark SQL: to execute SQL queries over structured data

MLlib: Spark's scalable machine learning library

This trio is designed for exactly this type of use case.

Why other options are incorrect:

A: GraphX is for graph processing — not needed here.

B: Pandas API on Spark is useful, but MLlib is essential for ML, which this option omits.

C: Spark Streaming is legacy; GraphX is irrelevant here.

Spark Connect introduces a decoupled client-server architecture. Its key feature is enabling Spark job submission and execution from remote clients — in Python, Java, etc.

From Databricks documentation:

“Spark Connect allows remote clients to connect to a Spark cluster and execute Spark jobs without being co-located with the Spark driver.”

A is close, but "any language" is overstated (currently supports Python, Java, etc., not literally all).

B refers to REST, which is not Spark Connect's mechanism.

D is incorrect; Spark Connect isn’t focused on ingestion.

Final Answer: C

Spark Connect enables a decoupled client-server architecture, allowing remote clients to run Spark code via gRPC.

However, as of Spark 3.5, Spark Connect supports DataFrame and SQL APIs, but not RDD APIs.

Limitation:

Applications that rely heavily on RDD-based transformations or actions cannot be migrated directly to Spark Connect.

These APIs require tight driver integration, which Spark Connect intentionally decouples.

Thus, complete Spark API compatibility is not yet achieved — this is the key adoption blocker.

Why the other options are incorrect:

A: Debugging is possible through IDE integration and logs on the client side.

B: Spark Connect actually supports upgradable clients independent of the driver — this is an advantage, not a limitation.

D: Spark Connect provides strong isolation between the client and driver processes.

When writing a DataFrame using .partitionBy() in Spark, the data is physically organized into directory structures corresponding to unique combinations of the partition columns.

Example:

/path/to/output/color=Red/taste=Sweet/part-0001.parquet

/path/to/output/color=Green/taste=Sour/part-0002.parquet

This structure improves query performance by pruning partitions when filtering on these columns.

Why the other options are incorrect:

A: Appending requires .mode("append"), which isn’t used here.

B: Null values in partition columns are handled; they don’t raise errors.

D: Partitioning prevents storing all data in a single file.

To create a Python dictionary from a Spark DataFrame, you can first collect the data to the driver node and then convert it into a Python dictionary using dict().

Steps:

Select only relevant columns.

Order by region_id to get the smallest ones.

Limit to 3 rows.

Map each row into key–value pairs.

Collect results to the driver and convert to a dictionary.

Correct code:

regions_dict = dict(

regions.orderBy("region_id")

limit(3)

rdd.map(lambda x: (x.region_id, x.region_name))

collect()

)

This produces a dictionary like:

{10: 'North', 12: 'East', 14: 'West'}

Why the other options are incorrect:

A/B: take(3) returns a list of Row objects, not key–value pairs.

C: Doesn’t order or limit by smallest IDs, so the result may not be correct.

To read all files recursively within a nested directory structure, Spark requires the recursiveFileLookup option to be explicitly enabled. According to Databricks official documentation, when dealing with deeply nested Parquet files in a directory tree (as shown in this example), you should set:

df = spark.read.option("recursiveFileLookup", "true").parquet("/path/events/data/")

This ensures that Spark searches through all subdirectories under /path/events/data/ and reads any Parquet files it finds, regardless of the folder depth.

Option A is incorrect because while it includes an option, inferSchema is irrelevant here and does not enable recursive file reading.

Option C is incorrect because wildcards may not reliably match deep nested structures beyond one directory level.

Option D is incorrect because it will only read files directly within /path/events/data/ and not subdirectories like /2023/01/01.

Databricks documentation reference:

"To read files recursively from nested folders, set the recursiveFileLookup option to true. This is useful when data is organized in hierarchical folder structures" — Databricks documentation on Parquet files ingestion and options.

Spark supports multiple broadcast joins in a single query plan, as long as each broadcasted DataFrame is small enough to fit under the configured threshold.

Execution Plan:

Spark broadcasts df2 to all executors.

Joins df1 (big) with broadcasted df2.

Then broadcasts df3 and performs another join with the intermediate result.

The result is efficient and avoids shuffling large data.

Why the other options are incorrect:

B: Multiple broadcast joins are supported in Spark 3.x.

C: The join condition is correct since all use id as the key.

D: broadcast() can be used inline; it’s valid syntax.