Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Databricks Certified Associate Developer for Apache Spark 3.5 – Python Exam Practice Test

The repartition(n) transformation reshuffles data into exactly n partitions.

Unlike coalesce(), repartition() always causes a shuffle to evenly redistribute the data.

Correct behavior:

df2 = df.repartition(20)

df2.rdd.getNumPartitions() # returns 20

Thus, the resulting DataFrame will have 20 partitions.

Why the other options are incorrect:

A/D: Doesn’t retain old partition count — it’s explicitly set to 20.

C: Number of partitions is not automatically tied to executors.

dropDuplicates() with no column list removes duplicates based on all columns.

It's the most efficient and semantically correct way to deduplicate records that are completely identical across all fields.

From the PySpark documentation:

dropDuplicates(): Return a new DataFrame with duplicate rows removed, considering all columns if none are specified.

— Source: PySpark DataFrame.dropDuplicates() API

The Spark History Server provides a web UI for viewing past completed applications, including event logs, stages, and performance metrics.

If disabled:

Spark jobs still run normally,

But users lose the ability to review historical job metrics, DAGs, or logs after completion.

Thus, debugging, performance analysis, and audit capabilities are lost.

Why the other options are incorrect:

A: Disabling History Server doesn’t manage logs.

B/D: Minimal overhead; disabling doesn’t improve runtime speed or executor performance.

Pandas API on Spark provides a distributed implementation of the Pandas DataFrame API on top of Apache Spark.

Advantages:

Executes transformations in parallel across all nodes and cores in the cluster.

Maintains Pandas-like syntax, making it easy for Python users to transition.

Enables scaling of existing Pandas code to handle large datasets without memory limits.

Therefore, it combines Pandas usability with Spark’s distributed power, offering both speed and scalability.

Why the other options are incorrect:

B: While it uses Python, that’s not its main advantage.

C: It runs distributed across the cluster, not on a single node.

D: Pandas API on Spark uses lazy evaluation, not eager computation.

In Spark’s DataFrame API, the dropna() (or equivalently, DataFrameNaFunctions.drop()) method removes rows containing null values.

Behavior:

how="any" → drops rows where any column has a null value.

how="all" → drops rows where all columns are null.

Since the data scientist wants to drop records with any null field, the correct parameter is how="any".

Correct syntax:

filtered_users = raw_users.dropna(how="any")

This will remove all records that have at least one null value in any column.

Why the other options are incorrect:

A: Uses na.drop("any") but missing parentheses context (works only as raw_users.na.drop("any"), which is equivalent to option C).

B/D: how="all" only removes rows where all values are null — too strict for this use case.

Apache Arrow is used under the hood to optimize conversion between Pandas and PySpark DataFrames. The correct configuration setting is:

spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

From the official documentation:

“This configuration must be enabled to allow for vectorized execution and efficient conversion between Pandas and PySpark using Arrow.”

Option B is correct.

Options A, C, and D are invalid config keys and not recognized by Spark.

Final Answer: B

In client mode, the driver runs on the same machine that submitted the job (often a developer’s workstation). If the driver has insufficient memory or CPU, it becomes a bottleneck.

Solution: Run the job in cluster mode.

In cluster mode, the driver runs inside the cluster on a worker node, benefiting from distributed cluster resources and improved performance for large workloads.

Why the other options are incorrect:

B: Executors handle tasks, not driver overhead.

C: May help temporarily but doesn’t scale; cluster mode is best practice.

D: Local mode runs everything on one JVM — worse for large workloads.

To create a Python dictionary from a Spark DataFrame, you can first collect the data to the driver node and then convert it into a Python dictionary using dict().

Steps:

Select only relevant columns.

Order by region_id to get the smallest ones.

Limit to 3 rows.

Map each row into key–value pairs.

Collect results to the driver and convert to a dictionary.

Correct code:

regions_dict = dict(

regions.orderBy("region_id")

limit(3)

rdd.map(lambda x: (x.region_id, x.region_name))

collect()

)

This produces a dictionary like:

{10: 'North', 12: 'East', 14: 'West'}

Why the other options are incorrect:

A/B: take(3) returns a list of Row objects, not key–value pairs.

C: Doesn’t order or limit by smallest IDs, so the result may not be correct.

The method saveAsTable() creates a new table and optionally fails if the table exists.

From Spark documentation:

"The mode 'ErrorIfExists' (default) will throw an error if the table already exists."

Thus:

Option A is correct.

Option B (Overwrite) would overwrite existing data — not acceptable here.

Option C and D use save(), which doesn't create a managed table with metadata in the metastore.

Final Answer: A

The function date_sub(start, days) subtracts the number of days from the start date. If a negative number is passed, the behavior becomes a date addition.

Example:

SELECT date_sub('2024-05-01', -5)

-- Returns: 2024-05-06

So, a negative value effectively adds the absolute number of days to the date.

To exclude sensitive (PII) columns from a DataFrame, the easiest method is to use the .drop() function with the list of column names to remove.

Correct syntax:

df_user_non_pii = df_user.drop("name", "email", "birthdate")

This creates a new DataFrame containing all remaining columns.

Why the other options are incorrect:

B: .dropFields() is not valid for standard DataFrames — it’s used for struct fields only.

C: .select() would keep only PII columns, not remove them.

D: .remove() does not exist in Spark DataFrame API.

spark.executor.cores determines how many concurrent tasks an executor can run.

For example, if set to 4, each executor can run up to 4 tasks in parallel.

Other settings:

spark.task.maxFailures controls task retry logic.

spark.driver.cores is for the driver, not executors.

spark.executor.memory sets memory limits, not task concurrency.

Spark Connect enables decoupling of the client and Spark driver processes, allowing remote access. Spark supports configuring the remote Spark Connect server in multiple ways:

From Databricks and Spark documentation:

Option B (--remote "sc://localhost") is a valid command-line argument for the pyspark shell to connect using Spark Connect.

Option C (setting SPARK_REMOTE environment variable) is also a supported method to configure the remote endpoint.

Option A is incorrect because Spark Connect uses the sc:// protocol, not https:// .

Option D requires modifying the code, which the question explicitly avoids.

Option E configures the port on the server side but doesn’t start a client connection.

Final Answers: B and C

The partitionBy() method in Spark organizes output into subdirectories based on unique combinations of the specified columns:

e.g.

/path/to/output/color=red/fruit=apple/part-0000.parquet

/path/to/output/color=green/fruit=banana/part-0001.parquet

This improves query performance via partition pruning.

It does not consolidate into a single file.

Null values are allowed in partitions.

It does not "append" unless .mode("append") is used.

Spark Connect enables a decoupled client-server architecture, allowing remote clients to run Spark code via gRPC.

However, as of Spark 3.5, Spark Connect supports DataFrame and SQL APIs, but not RDD APIs.

Limitation:

Applications that rely heavily on RDD-based transformations or actions cannot be migrated directly to Spark Connect.

These APIs require tight driver integration, which Spark Connect intentionally decouples.

Thus, complete Spark API compatibility is not yet achieved — this is the key adoption blocker.

Why the other options are incorrect:

A: Debugging is possible through IDE integration and logs on the client side.

B: Spark Connect actually supports upgradable clients independent of the driver — this is an advantage, not a limitation.

D: Spark Connect provides strong isolation between the client and driver processes.

The setLogLevel() method of SparkContext sets the logging level on the driver, which controls the verbosity of logs emitted during job execution. Supported levels are inherited from log4j and include the following:

ALL

DEBUG

ERROR

FATAL

INFO

OFF

TRACE

WARN

According to official Spark and Databricks documentation:

"Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, and WARN."

Among the choices provided, only option B (ERROR, WARN, TRACE, OFF) includes four valid log levels and excludes invalid ones like "FAIL" or "NONE".

To process data in fixed micro-batch intervals, use the .trigger(processingTime="interval") option in Structured Streaming.

Correct usage:

query = df.writeStream \

outputMode("append") \

trigger(processingTime="5 seconds") \

start()

This instructs Spark to process available data every 5 seconds.

Why the other options are incorrect:

B: continuous triggers are for continuous processing mode (different execution model).

C: once=True runs the stream a single time (batch mode).

D: Default trigger runs as fast as possible, not fixed intervals.

To filter rows based on a condition and display them in Spark, use filter(...).show():

employees_df.filter(employees_df.tenure >= 5).show()

Option A is correct and shows the results.

Option B filters but doesn’t display them.

Option C uses Python’s built-in filter, not Spark.

Option D collects the results to the driver, which is unnecessary if .show() is sufficient.

Final Answer: A

To enable a Structured Streaming query to recover from failures or intentional shutdowns, it is essential to specify the checkpointLocation option during the writeStream operation. This checkpoint location stores the progress information of the streaming query, allowing it to resume from where it left off.

According to the Databricks documentation:

"You must specify the checkpointLocation option before you run a streaming query, as in the following example:

option("checkpointLocation", "/path/to/checkpoint/dir")

toTable("catalog.schema.table")

— Databricks Documentation: Structured Streaming checkpoints

By setting the checkpointLocation during writeStream, Spark can maintain state information and ensure exactly-once processing semantics, which are crucial for reliable streaming applications.

In PySpark, when performing a join between two DataFrames, the columns do not have to share the same name. You can explicitly provide a join condition by comparing specific columns from each DataFrame.

This syntax is correct and fully supported:

df1.join(df2, df1.employee_id == df2.emp_id, how='inner')

This will perform an inner join between df1 and df2 using the employee_id from df1 and emp_id from df2.

To create an external table and enable schema merging, the correct syntax is:

CREATE EXTERNAL TABLE users

USING json

OPTIONS (

path '/data/input.json',

mergeSchema 'true'

)

mergeSchema is the correct option key (not schemaMerge)

EXTERNAL allows Spark to query files without managing their lifecycle

RDD actions like reduce() aggregate values across all partitions and return the result to the driver.

To compute the maximum processing time, reduce() is ideal because it combines results from all tasks efficiently.

Example:

max_time = rdd_times.reduce(lambda x, y: max(x, y))

This aggregates maximum values from all executors into a single result on the driver.

Why the other options are incorrect:

A: Broadcast variables distribute read-only data; they cannot aggregate results.

B: Spark UI provides visualization, not programmatic collection.

D: Accumulators support additive operations only (e.g., counters, sums), not non-associative ones like max.

Shuffling occurs in operations like groupBy, reduceByKey, or join—which cause data to be moved across partitions. The repartition() operation can also cause a shuffle, but in this context, it follows an aggregation.

In Option D, the groupBy followed by agg results in a shuffle due to grouping across nodes.

The repartition(10) is a partitioning transformation but does not involve a new shuffle since the data is already grouped.

This sequence — shuffle (groupBy) followed by non-shuffling (repartition) — is correct.

Option A does the opposite: the filter does not cause a shuffle, but groupBy does — this makes it the wrong order.

Partitioning a dataset divides data into separate directories based on partition column values. When queries filter on partitioned columns, Spark can prune irrelevant partitions — meaning it only reads files that match the filter criteria.

Advantage:

Reduces I/O and improves performance by scanning only relevant subsets of data.

Example:

/data/sales/year=2023/month=10/...

/data/sales/year=2024/month=01/...

A query filtering WHERE year = 2024 reads only the relevant partition.

Why the other options are incorrect:

A: Compression is independent of partitioning.

B: Spark does not automatically clean partitions unless managed manually.

C: Partitioning does not cause Spark to load entire data into memory.

In the Spark architecture, the driver node is responsible for orchestrating the execution of a Spark application. It converts user-defined transformations and actions into a logical plan, optimizes it into a physical plan, and then splits the plan into tasks that are distributed to the executor nodes.

As per Databricks and Spark documentation:

“The driver node is responsible for maintaining information about the Spark application, responding to a user's program or input, and analyzing, distributing, and scheduling work across the executors.”

This means:

Option A is correct because the driver schedules and coordinates the job execution.

Option B is incorrect because the driver does more than just UI monitoring.

Option C is incorrect since data and computations are distributed across executor nodes.

Option D is incorrect; results are returned to the driver but not stored long-term by it.

The Spark configuration spark.executor.cores defines how many concurrent tasks can be executed within a single executor process.

Each executor is assigned a number of CPU cores.

Each core executes one task at a time.

Therefore, increasing spark.executor.cores allows an executor to run more tasks concurrently.

Example:

--conf spark.executor.cores=4

→ Each executor can run 4 parallel tasks.

Why the other options are incorrect:

B (spark.task.maxFailures): Sets retry attempts for failed tasks.

C (spark.executor.memory): Sets executor memory, not concurrency.

D (spark.sql.shuffle.partitions): Defines number of shuffle partitions, not executor concurrency.

To persist a table and specify the save path, use:

users.write.option("path", "/some/path").saveAsTable("default_table")

The .option("path", ...) must be applied before calling saveAsTable.

Option A uses invalid syntax (write(path=...)).

Option B applies .option() after .saveAsTable()—which is too late.

Option D uses incorrect syntax (no path parameter in saveAsTable).

A Spark job is triggered by an action (e.g., count, show).

The job is broken into stages, typically one per shuffle boundary.

Each stage is divided into multiple tasks, which are distributed across worker nodes.

Adaptive Query Execution (AQE) is a powerful optimization framework introduced in Apache Spark 3.0 and enabled by default since Spark 3.2. It dynamically adjusts query execution plans based on runtime statistics, leading to significant performance improvements. The key benefits of AQE include:

Dynamic Join Strategy Selection: AQE can switch join strategies at runtime. For instance, it can convert a sort-merge join to a broadcast hash join if it detects that one side of the join is small enough to be broadcasted, thus optimizing the join operation .

Handling Skewed Data: AQE detects skewed partitions during join operations and splits them into smaller partitions. This approach balances the workload across tasks, preventing scenarios where certain tasks take significantly longer due to data skew .

Coalescing Post-Shuffle Partitions: AQE dynamically coalesces small shuffle partitions into larger ones based on the actual data size, reducing the overhead of managing numerous small tasks and improving overall query performance .

These runtime optimizations allow Spark to adapt to the actual data characteristics during query execution, leading to more efficient resource utilization and faster query processing times.

Spark Connect enables remote execution of Spark jobs by decoupling the client from the driver using the Spark Connect protocol (gRPC).

It allows users to run Spark code from different environments (like notebooks, IDEs, or remote clients) while executing jobs on the cluster.

Key Features:

Enables remote interaction between client and Spark driver.

Supports interactive development and lightweight client sessions.

Improves developer productivity without needing driver resources locally.

Why the other options are incorrect:

A: Spark Connect is not limited to ingestion tasks.

B: It allows multi-language clients (Python, Scala, etc.) but runs via Spark Connect API, not arbitrary remote code.

C: Uses gRPC protocol, not REST.

The correct way to aggregate information (e.g., max value) from distributed workers back to the driver is using RDD actions such as reduce() or aggregate().

From the documentation:

“To perform global aggregations on distributed data, actions like reduce() are commonly used to collect summaries such as min/max/avg.”

Accumulators (Option B) do not support max operations directly and are not intended for such analytics.

Broadcast (Option C) is used to send data to workers, not collect from them.

Spark UI (Option D) is a monitoring tool — not an analytics collection interface.

Final Answer: A

Apache Spark follows a lazy evaluation model, meaning transformations (like filter(), select(), map()) are not executed immediately. Instead, they build a logical plan (lineage graph) that represents the sequence of operations to be applied.

Execution only begins when an action (e.g., count(), collect(), save(), show()) is called. At that point, Spark’s engine:

Optimizes the logical plan into a physical plan.

Divides it into stages and tasks.

Executes them across the cluster.

This design helps Spark optimize execution paths and avoid unnecessary computations.

Why the other options are incorrect:

A: Transformations do not execute immediately; they are deferred.

B: Optimization happens during job execution (after an action), not during transformations.

D: Execution starts automatically once an action is triggered, no manual intervention needed.

To sort a PySpark DataFrame by multiple columns with mixed sort directions, the correct usage is:

python

CopyEdit

df.orderBy("age", "salary", ascending=[True, False])

age will be sorted in ascending order

salary will be sorted in descending order

The orderBy() and sort() methods in PySpark accept a list of booleans to specify the sort direction for each column.

Documentation Reference: PySpark API - DataFrame.orderBy

The SparkSession is the entry point to Spark functionality in modern versions (2.x and later). It unifies the SparkContext, SQLContext, and HiveContext into a single object.

Best Practice:

Use one SparkSession for the entire application.

Create it once at the start using SparkSession.builder.getOrCreate().

Reuse it across all transformations and actions.

Stop it only after all operations are completed.

Example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MyApp").getOrCreate()

# Perform transformations and actions

spark.stop()

Why the other options are incorrect:

B: SparkSession is the recommended interface; SparkContext alone is deprecated for SQL/DataFrame APIs.

C: Creating multiple sessions increases overhead and wastes resources.

D: Restarting SparkSession breaks lineage and adds unnecessary startup costs.

In Spark, a shuffle occurs when data needs to be redistributed across partitions or nodes, typically due to operations that require grouping, joining, or sorting.

groupBy triggers a shuffle because it aggregates data based on key values, requiring data from multiple partitions to be moved across the cluster.

Operations like filter and select are narrow transformations (no shuffle).

coalesce can reduce the number of partitions without a full shuffle (unlike repartition).

Why the other options are incorrect:

A (filter) – Narrow transformation, no shuffle.

B (select) – Simple projection, no data movement.

D (coalesce) – Reduces partitions locally without shuffle.

Structured Streaming supports three output modes:

Append: Writes only new rows since the last trigger.

Update: Writes only updated rows.

Complete: Writes the entire result table after every trigger execution.

For aggregations like groupBy().count(), only complete mode outputs the entire table each time.

Example:

aggDF.writeStream \

outputMode("complete") \

format("console") \

start()

Why the other options are incorrect:

A: “AGGREGATE” is not a valid output mode.

C: “REPLACE” does not exist.

D: “APPEND” writes only new rows, not the full table.

The groupBy() operation causes a shuffle because it requires all values for a specific key to be brought together, which may involve moving data across partitions.

In contrast:

filter() and select() are narrow transformations and do not cause shuffles.

coalesce() tries to reduce the number of partitions and avoids shuffling by moving data to fewer partitions without a full shuffle (unlike repartition()).

The default broadcast join threshold in Spark is:

spark.sql.autoBroadcastJoinThreshold = 10MB

Since df2 is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.

From the Spark documentation:

“If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors.”

A is incorrect because Spark does support auto broadcast even with static plans.

B is correct: Spark will automatically broadcast df2.

C and D are incorrect because Spark's default logic handles this optimization.

Final Answer: B

According to Spark’s watermarking rules:

“Records that are older than the watermark (event time < current watermark) are considered too late and are dropped.”

So, if a record’s event_time is earlier than (max event_time seen so far - 10 minutes), it is discarded.