When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, one of the first topics you should understand is Azure Databricks compute, often informally called a cluster.

Many exam questions are not really testing whether you remember a button name. They are testing whether you can choose the right compute type for a workload:

- interactive development - production ingestion - Lakeflow Spark Declarative Pipelines - batch ETL jobs - cost optimization - machine learning workloads - troubleshooting out-of-memory failures

In the uploaded DP-750 question bank, several questions focus directly on clusters, autoscaling, job compute, serverless compute, Photon, auto termination, and Spark UI troubleshooting.

---

1. What is a cluster in Azure Databricks?

In Azure Databricks, a cluster is a compute resource used to run notebooks, jobs, Spark workloads, machine learning code, and data engineering pipelines.

More generally, Azure Databricks documentation now often uses the word compute instead of only “cluster”. Microsoft’s documentation describes configuration settings for both all-purpose compute and job compute, and explains that many users create compute through policies that control which settings are available.

A Databricks compute resource usually includes:

- a driver node, which coordinates the Spark application - one or more worker nodes, which execute distributed tasks - a Databricks Runtime, which provides Spark, libraries, connectors, and Databricks optimizations - optional features such as autoscaling, auto termination, Photon, and compute policies

For DP-750, the most important point is not the internal architecture. The most important point is choosing the right compute for the right workload.

---

2. Main compute types you must know for DP-750

All-purpose cluster

An all-purpose cluster is designed for interactive work.

Typical use cases:

- notebook development - data exploration - debugging - ad hoc analysis - collaborative development

In exam questions, if you see words such as interactive development, users working in notebooks, or development cluster, the answer often points to an all-purpose cluster.

However, all-purpose clusters are usually not the best choice for production pipelines because they can be shared by many users, stay idle, and allow development activity to affect production workloads.

---

Job cluster

A job cluster is created for running a job and is usually terminated after the job finishes.

Typical use cases:

- production ETL - scheduled ingestion - automated workflows - isolated pipeline execution - repeatable job runs

In DP-750 questions, when the requirement says:

> production workloads must run as scheduled, non-interactive pipelines

or:

> prevent development activity from affecting production pipelines

you should think about job compute, serverless compute, or Lakeflow pipeline compute, not a shared all-purpose development cluster.

---

Serverless compute

Serverless compute means Databricks manages the infrastructure for you. You do not manually provision the underlying compute resources. Microsoft describes serverless compute as an Azure Databricks-managed service for notebooks, workflows, and Lakeflow Spark Declarative Pipelines; Databricks automatically allocates and manages the required compute resources, which reduces idle time and management effort.

For Lakeflow Jobs, serverless compute lets you run jobs without configuring and deploying infrastructure. Databricks manages, optimizes, and scales the compute resources, and autoscaling and Photon are automatically enabled for the compute resources that run the job.

For Lakeflow Spark Declarative Pipelines, Databricks recommends serverless compute for new pipelines. Serverless pipelines use enhanced autoscaling and can scale both horizontally and vertically based on workload demand.

This is very important for DP-750:

> If the question asks for the lowest operational effort for a new Lakeflow Spark Declarative Pipeline and serverless compute is available as an option, serverless compute is usually the best answer.

---

3. Autoscaling vs auto termination

These two features are often tested together, but they solve different problems.

Autoscaling

Autoscaling automatically adds or removes worker nodes based on workload demand.

Use autoscaling when:

- workload size changes - ingestion volume spikes - users run variable workloads - you need to scale up and scale down automatically

In DP-750 language:

> “Automatically add and remove worker nodes” = Autoscaling

---

Auto termination

Auto termination shuts down compute after it has been idle for a configured period.

Use auto termination when:

- users forget to stop clusters - all-purpose clusters remain idle - you want to reduce unnecessary compute cost - active workloads must not be affected

In DP-750 language:

> “Automatically shut down when idle” = Auto termination

Microsoft’s compute best-practice documentation also recommends enabling auto termination to ensure compute is terminated after inactivity, and considering autoscaling based on the analyst’s workload.

---

4. Photon acceleration

Photon is Databricks’ native vectorized query engine. It accelerates SQL workloads, DataFrame API calls, ETL pipelines, and stateless streaming workloads. It is compatible with Apache Spark APIs, so existing Spark code can often run without code changes.

In DP-750, Photon is usually associated with:

- performance improvement - ETL acceleration - SQL workloads - DataFrame workloads - cost reduction per workload when faster execution reduces total compute usage

However, Photon is not a magic answer for every cost problem.

If a cluster is overprovisioned and CPU utilization is very low, enabling Photon is not necessarily the best answer. In that case, the better solution may be to reduce the number of workers or right-size the cluster.

---

5. Cost optimization logic for cluster questions

DP-750 cluster questions often test cost optimization. Here are the most useful decision rules.

Case 1: The workload is variable or bursty

Use autoscaling or serverless compute.

Example signals:

- telemetry spikes - unpredictable volume - workload demand changes - new records arrive frequently - pipeline must scale automatically

---

Case 2: The cluster is idle for long periods

Use auto termination.

Example signals:

- users finish work but clusters keep running - all-purpose clusters remain idle - reduce cost without affecting active workloads

---

Case 3: The workload is predictable and overprovisioned

Reduce the number of workers.

Example signals:

- CPU utilization remains below 20% - workload does not spike - current node type already meets requirements - need to reduce cost without increasing duration

In this situation, disabling autoscaling and reducing workers can be better than simply enabling Photon or changing the auto-termination timeout.

---

Case 4: Production and development are sharing compute

Separate them.

Example signals:

- production and development workloads run on the same all-purpose clusters - development activity affects production pipelines - production ingestion should be scheduled and non-interactive

The answer will usually involve job compute, serverless compute, or a production-specific pipeline configuration.

---

6. Cluster libraries and Unity Catalog

Sometimes the exam asks how to install libraries on a cluster while using Unity Catalog for access control.

A cluster-scoped library can be used by notebooks and jobs running on that cluster, and Microsoft’s documentation explains that libraries can be installed on a specific cluster through the Azure Databricks workspace UI, REST API, CLI, Terraform, or policies.

For DP-750, the key point is:

> If the solution must use Unity Catalog for access control, avoid unmanaged or ad hoc installation patterns. Prefer workspace-managed or governed library installation approaches.

---

7. Machine learning cluster traps

A common exam trap is confusing Databricks Runtime for Machine Learning with GPU capability.

Databricks Runtime ML includes common machine learning and deep learning libraries, but GPU acceleration still requires GPU-enabled compute. Microsoft’s GPU documentation says that to create GPU compute, the worker type must be a GPU instance type.

So if a cluster is:

- single-node - general-purpose VM - Databricks Runtime ML - Python supported

then it can run Python ML code using preinstalled libraries, but it cannot automatically train GPU-based deep learning models unless the node type is GPU-enabled. A single-node cluster also cannot distribute workloads across multiple worker nodes.

---

8. Troubleshooting cluster failures with Spark UI

When a job fails because of out-of-memory errors, you should not only look at the notebook output. You need to understand what happened during execution.

The Spark UI is the key tool for analyzing runtime behavior. Microsoft’s Spark memory troubleshooting documentation explains that memory errors can be generic and may come from several causes, such as shuffle partitions, large broadcasts, UDFs, skew, and streaming state.

For DP-750:

> OOM root-cause analysis = Spark UI > Memory/runtime behavior = executors, tasks, stages, shuffle, spill, skew

If the question asks specifically for execution behavior and root cause, Spark UI is usually stronger than cluster event logs or notebook output.

---

Real Exam Questions

Question 1

Useful case information

Contoso has a single Azure Databricks workspace named Workspace1 in the West US Azure region. Workspace1 is enabled for Unity Catalog.

Workspace1 contains all-purpose clusters for both development and production workloads.

The company’s existing analytics environment has several compute issues:

- Production and development workloads run on the same all-purpose clusters. - Production and development workloads do NOT support autoscaling or workload isolation.

Contoso identifies the following environment and compute requirements:

- Ensure that production ingestion workloads run on compute clusters that can scale automatically during telemetry spikes. - Prevent development activity from affecting production pipelines. - Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared interactive development clusters.

You need to configure compute for the ingestion of telemetry data. The solution must meet the data ingestion and processing requirements.

What should you do?

A. Move the ingestion pipelines to shared compute.

B. Enable Photon acceleration for a job compute cluster. ✅ Correct Answer

C. Increase an all-purpose cluster to a larger fixed node type.

D. Disable autoscaling for a job compute cluster.

---

Question 2

You have an Azure Databricks workspace.

You are creating a Lakeflow Spark Declarative Pipelines (SDP) pipeline that scales automatically.

You need to configure compute for the pipeline. The solution must minimize operational costs and effort.

What should you use?

A. the existing SQL warehouse

B. an all-purpose cluster that uses autoscaling

C. a job cluster that uses autoscaling ✅ Correct Answer

D. a single-node, all-purpose cluster

---

Question 3

You have an Azure Databricks workspace that contains an all-purpose compute cluster named Cluster1. Cluster1 is used for interactive development.

You need to configure Cluster1 to meet the following requirements:

- Automatically add and remove worker nodes based on workload demand. - Automatically shut down when the cluster has been idle for a specific period.

What should you configure for each requirement?

|Requirement|Answer| |---|---| |Automatically add and remove worker nodes|Autoscaling ✅ Correct Answer| |Automatically shut down|Auto termination ✅ Correct Answer|

---

Question 5

You have an Azure Databricks workspace named Workspace1.

You create a compute cluster named Cluster1 that will be used to ingest data.

You need to install the required libraries on Cluster1. The solution must use Unity Catalog for access control.

What should you do?

A. Install the libraries by using pip3.

B. Create a custom dependency management script and run the script from a Databricks notebook.

C. Upload the libraries to Workspace1 and install the libraries on Cluster1. ✅ Correct Answer

D. Install the libraries on Cluster1 and manually restart the cluster.

---

Question 7

You have an Azure Databricks workspace that contains an all-purpose cluster named Cluster1.

You need to configure Cluster1 to meet the following requirements:

- Scale up automatically when workloads increase. - Scale down automatically when workloads decrease. - Minimize costs.

Which two actions should you perform? Each correct answer presents part of the solution.

A. Disable Photon acceleration.

B. Enable autoscaling for Cluster1. ✅ Correct Answer

C. Apply a compute policy that enables users to manage the cluster settings.

D. Specify a fixed number of workers.

E. Configure Cluster1 to terminate after 30 minutes of inactivity. ✅ Correct Answer

---

Question 10

You have an Azure Databricks workspace that contains a cluster named Cluster1.

Performance monitoring shows that Cluster1 is consistently overprovisioned for its batch workload:

- CPU utilization remains below 20 percent, including peak processing periods. - The workload is highly predictable and does not spike. - The current node type already meets the workload requirements.

You need to reduce compute costs without increasing job duration.

What should you do?

A. Enable Photon acceleration.

B. Configure Cluster1 to use a larger node type.

C. Decrease the autotermination timeout of Cluster1.

D. Disable autoscaling and reduce the number of worker nodes. ✅ Correct Answer

---

Question 11

You have an Azure Databricks workspace.

You are creating a Lakeflow Spark Declarative Pipelines (SDP) pipeline that scales automatically.

You need to configure compute for the pipeline. The solution must minimize operational costs and administrative effort.

What should you use?

A. serverless compute ✅ Correct Answer

B. a single-node, all-purpose cluster

C. an all-purpose cluster that uses autoscaling

D. an existing SQL warehouse

E. a job cluster that uses autoscaling

---

Question 13

You have an Azure Databricks workspace that contains a cluster named Cluster1.

Cluster1 has the following characteristics:

- Configured as a single node cluster - Uses a general purpose virtual machine node type

The cluster runtime environment has the following configurations:

- Uses Databricks Runtime for Machine Learning - Includes common machine learning libraries - Supports Python workloads

For each of the following statements, select Yes if the statement is true. Otherwise, select No.

|Statement|Answer| |---|---| |Cluster1 can be used to train deep learning models that require GPU acceleration.|No ✅ Correct Answer| |Cluster1 can distribute machine learning workloads across multiple nodes.|No ✅ Correct Answer| |Cluster1 can run Python workloads that rely on preinstalled machine learning libraries.|Yes ✅ Correct Answer|

---

Question 61

You have an Azure Databricks workspace that contains an all-purpose cluster named Cluster1.

You discover that out-of-memory, OOM, errors intermittently cause jobs running on Cluster1 to fail.

You need to identify the root cause of the failures by analyzing the runtime execution behavior.

What should you do?

|Area|Answer| |---|---| |Diagnostic tool to use|The Apache Spark UI ✅ Correct Answer| |Execution level to analyze|Executors ✅ Correct Answer|

---

Question 70

You have an Azure Databricks workspace that contains multiple all-purpose clusters.

You discover that some clusters remain idle for long periods after users finish their work.

You need to reduce compute costs without affecting active workloads.

What should you do?

A. Enable autoscaling.

B. Convert the clusters into job clusters.

C. Use spot instances.

D. Configure automatic termination. ✅ Correct Answer