Pass the Databricks Certification Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Questions and answers with ExamsMirror

Explanation

Correct code block:

transactionsDf.repartition(1).write.option("sep", "\t").option("nullValue", "n/a").csv(csvPath)

It is important here to understand that the QUESTION NO: specifically asks for writing the DataFrame as a single CSV file. This should trigger you to think about partitions. By default, every partition

is written as a separate file, so you need to include repatition(1) into your call. coalesce(1) works here, too!

Secondly, the QUESTION NO: is very much an invitation to search through the parameters in the Spark documentation that work with DataFrameWriter.csv (link below). You will also need to know

that you need an option() statement to apply these parameters.

The final concern is about the general call structure. Once you have called accessed write of your DataFrame, options follow and then you write the DataFrame with csv. Instead of csv(csvPath),

you could also use save(csvPath, format='csv') here.

More info: pyspark.sql.DataFrameWriter.csv — PySpark 3.1.1 documentation

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 52 (Databricks import instructions)

Explanation

This is QUESTION NO: is hard. Let's assess the different answers one-by-one.

Spark will only broadcast DataFrames that are much smaller than the default value.

This is correct. The default value is 10 MB (10485760 bytes). Since the configuration for spark.sql.autoBroadcastJoinThreshold expects a number in bytes (and not megabytes), the code block sets

the limits to merely 20 bytes, instead of the requested 20 * 1024 * 1024 (= 20971520) bytes.

The command is evaluated lazily and needs to be followed by an action.

No, this command is evaluated right away!

Spark will only apply the limit to threshold joins and not to other joins.

There are no "threshold joins", so this option does not make any sense.

The correct option to write configurations is through spark.config and not spark.conf.

No, it is indeed spark.conf!

The passed limit has the wrong variable type.

The configuration expects the number of bytes, a number, as an input. So, the 20 provided in the code block is fine.

Explanation

Output of correct code block:

+-------------------+

| supplier|

+-------------------+

|Sports Company Inc.|

+-------------------+

Key to managing this QUESTION NO: is understand which operator to use to do the opposite of an operation – the ~ (not) operator. In addition, you should know that there is no unique() method.

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 55 (Databricks import instructions)

The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.

Correct. In cluster mode, the cluster manager is located on a node other than the client machine. From there it starts and ends executor processes on the cluster nodes as required by the Spark

application running on the Spark driver.

The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.

Wrong, there is no "remote" execution mode in Spark. Available execution modes are local, client, and cluster.

The cluster manager allocates resources to the DataFrame manager

Wrong, there is no "DataFrame manager" in Spark.

The cluster manager schedules tasks on the cluster in client mode.

No, in client mode, the Spark driver schedules tasks on the cluster – not the cluster manager.

The cluster manager schedules tasks on the cluster in local mode.

Wrong: In local mode, there is no "cluster". The Spark application is running on a single machine, not on a cluster of machines.

More info: Running Spark: an overview of Spark’s runtime architecture - Manning (https://bit.ly/2RPmJn9)

Explanation

Correct code block:

transactionsDf.select(["transactionId", "predError", "value", "f"])

The DataFrame.select returns specific columns from the DataFrame and accepts a list as its only argument. Thus, this is the correct choice here. The option using col(["transactionId", "predError",

"value", "f"]) is invalid, since inside col(), one can only pass a single column name, not a list. Likewise, all columns being specified in a single string like "transactionId, predError, value, f" is not valid

syntax.

filter and where filter rows based on conditions, they do not control which columns to return.

Static notebook | Dynamic notebook: See test 2, QUESTION NO: 49 (Databricks import instructions)

transactionsDf.coalesce(24)

No, the coalesce() method can only reduce, but not increase the number of partitions.

transactionsDf.repartition()

No, repartition() requires a numPartitions argument.

transactionsDf.repartition("itemId", 24)

No, here the cols and numPartitions argument have been mixed up. If the code block would be transactionsDf.repartition(24, "itemId"), this would be a valid solution.

transactionsDf.repartition(24, boost=True)

No, there is no boost argument in the repartition() method.

Explanation

spark.createDataFrame(throughputRates, FloatType())

Correct! spark.createDataFrame is the correct operator to use here and the type FloatType() which is passed in for the command's schema argument is correctly instantiated using the parentheses.

Remember that it is essential in PySpark to instantiate types when passing them to SparkSession.createDataFrame. And, in Databricks, spark returns a SparkSession object.

spark.createDataFrame((throughputRates), FloatType)

No. While packing throughputRates in parentheses does not do anything to the execution of this command, not instantiating the FloatType with parentheses as in the previous answer will make this

command fail.

spark.createDataFrame(throughputRates, FloatType)

Incorrect. Given that it does not matter whether you pass throughputRates in parentheses or not, see the explanation of the previous answer for further insights.

spark.DataFrame(throughputRates, FloatType)

Wrong. There is no SparkSession.DataFrame() method in Spark.

spark.createDataFrame(throughputRates)

False. Avoiding the schema argument will have PySpark try to infer the schema. However, as you can see in the documentation (linked below), the inference will only work if you pass in an "RDD of

either Row, namedtuple, or dict" for data (the first argument to createDataFrame). But since you are passing a Python list, Spark's schema inference will fail.

More info: pyspark.sql.SparkSession.createDataFrame — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 55 (Databricks import instructions)

Explanation

Correct code block:

transactionsDf.filter(col('predError').isin([3, 6])).count()

The isin method is the correct one to use here – the in method does not exist for the Column object.

More info: pyspark.sql.Column.isin — PySpark 3.1.2 documentation

Explanation

Output of correct code block:

+---------+-----+

|predError|value|

+---------+-----+

| 6| 7|

| null| null|

| 3| 2|

+---------+-----+

This is not an easy QUESTION NO: to solve. You need to know that % stands for the module operator in Python. % 2 will return true for every second row. The statement using spark.sql gets it

almost right (the modulo operator exists in SQL as well), but % 2 = 2 will never yield true, since modulo 2 is either 0 or 1.

Other answers are wrong since they are missing quotes around the column names and/or use filter or select incorrectly.

If you have any doubts about SparkSQL and answer options 3 and 4 in this question, check out the notebook I created as a response to a related student question.

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 53 (Databricks import instructions)