DP-750: Structured Streaming, Checkpointing, Schema Evolution, and Change Data Feed Explained and with Real Exam Questions

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to understand how Azure Databricks handles streaming and incremental data processing.

This topic includes several important concepts:

• Apache Spark Structured Streaming
• checkpointing
• exactly-once processing
• availableNow trigger
• schema drift and schema evolution
• Auto Loader schema evolution modes
• Delta table streaming reads and writes
• Change Data Feed, CDF

The related questions in your DP-750 question bank are Q8, Q43, Q47, Q48, Q54, Q59, and Q71.

---

1. What is Structured Streaming?

Apache Spark Structured Streaming is Spark’s stream processing engine. In Azure Databricks, it is commonly used to process data continuously or incrementally from sources such as:

• cloud storage files
• Delta tables
• Event Hubs
• Kafka
• Auto Loader
• streaming tables

In DP-750, Structured Streaming often appears in questions about:

• avoiding duplicate processing after failure
• writing to Delta tables
• checkpoint locations
• schema changes
• Event Hubs ingestion
• streaming recovery
• change data capture

For exam purposes, the most important idea is:

> Structured Streaming needs checkpointing to remember progress and recover after failures.

---

2. What is checkpointing?

A checkpoint stores streaming query progress and metadata.

For example, if a streaming job has already processed files or events, the checkpoint helps the job remember what has already been processed.

A checkpoint location is usually configured like this:

.option("checkpointLocation", checkpoint_path)

Azure Databricks documentation states that checkpointLocation is required for fault tolerance and exactly-once processing guarantees, and each streaming query must use a unique checkpoint location.

For DP-750, the key pattern is:

> Streaming job reprocesses old data after restart = missing or wrong checkpoint location.

So if the question says:

After the cluster restarts, the streaming job reprocesses previously ingested data.

the answer is:

Configure a checkpoint location for the streaming query.

---

3. Exactly-once processing

Exactly-once processing means each record is processed into the target exactly once, even if there is a failure and restart.

In Databricks, exactly-once behavior is usually achieved through:

• Structured Streaming
• checkpointing
• Delta Lake transaction log
• idempotent writes
• unique checkpoint location per query

Databricks documentation says Delta Lake is deeply integrated with Structured Streaming and that the Delta transaction log guarantees exactly-once processing when writing data into a Delta table using Structured Streaming.

For DP-750, remember:

Streaming + Delta target + checkpointLocation = fault tolerance / no reprocessing

---

4. Checkpoint location vs schema location

When using Auto Loader, you often see both:

.option("cloudFiles.schemaLocation", "/cloud/schema")
.option("checkpointLocation", "/data/checkpoint")

They are not the same thing.

|Option|Purpose| |---|---| |checkpointLocation|Stores streaming progress and state| |cloudFiles.schemaLocation|Stores inferred schema information for Auto Loader|

Auto Loader documentation says specifying cloudFiles.schemaLocation enables schema inference and evolution. The same directory can be used for checkpointLocation, but the two options serve different purposes.

For DP-750:

• If the issue is reprocessing after restart, choose checkpointLocation.

• If the issue is schema inference or schema evolution, choose cloudFiles.schemaLocation or schema evolution settings.

---

5. Schema drift and schema evolution

Schema drift means the source data structure changes over time.

Example:

{
  "sensor_id": "S001",
  "temperature": 21.5
}

Later, the source sends a new column:

{
  "sensor_id": "S001",
  "temperature": 21.5,
  "humidity": 62
}

If the pipeline cannot handle the new column, it may fail.

Schema evolution allows the target schema to evolve when new columns appear.

Databricks documentation says Auto Loader supports column changes and type widening, and schema evolution can be configured with cloudFiles.schemaEvolutionMode.

For DP-750:

> New columns are added to the source and pipeline writes fail = enable schema evolution.

Do not choose row filters. Row filters control which rows users can see; they do not solve schema drift.

---

6. mergeSchema

When writing to Delta, mergeSchema allows additive schema changes, such as adding new columns.

Example:

.option("mergeSchema", "true")

This is commonly used when incoming data may contain new columns that should be added to the target Delta table.

In the DP-750 question bank, Q43 uses:

.option("mergeSchema", "true")

This means new columns from incoming JSON payloads can be added automatically to the target Delta table.

---

7. Auto Loader schema evolution mode: failOnNewColumns

Auto Loader has several schema evolution modes.

One of them is:

.option("cloudFiles.schemaEvolutionMode", "failOnNewColumns")

This mode can be confusing.

With failOnNewColumns, when Auto Loader detects new columns, the stream fails so that the schema can evolve. After restart, the evolved schema is used. Databricks documentation describes this pattern: new columns fail the stream and evolve the schema, and the stream can then be restarted with the updated inferred schema.

For DP-750, if a question asks:

> When a new element is found in the source data, will the element be written successfully?

and the code uses:

.option("cloudFiles.schemaEvolutionMode", "failOnNewColumns")

then the answer is:

No

Because the stream fails when it detects the new column.

---

8. availableNow trigger

The availableNow trigger processes all available data as an incremental batch and then stops.

It is useful when you want streaming-style incremental processing without keeping the stream running forever.

Example:

.trigger(availableNow=True)

For DP-750, availableNow is commonly used with:

• Auto Loader
• file ingestion
• incremental batch processing
• structured streaming jobs that should process currently available data and then stop

In the exam, if the code uses:

.trigger(availableNow=True)
.outputMode("append")

then it does not mean each batch overwrites the target table. With append mode, new records are appended.

---

9. Delta table as streaming target

A common Structured Streaming write pattern is:

df.writeStream \
    .format("delta") \
    .option("checkpointLocation", checkpoint_path) \
    .option("mergeSchema", "true") \
    .trigger(availableNow=True) \
    .table(raw_events)

For DP-750, remember:

|Requirement|Code option| |---|---| |Write to Delta table|.format("delta")| |Resume after failure without reprocessing|.option("checkpointLocation", checkpoint_path)| |Add new columns automatically|.option("mergeSchema", "true")| |Process available data and stop|.trigger(availableNow=True)|

---

10. Change Data Feed, CDF

Change Data Feed, or CDF, lets you track row-level changes from a Delta table.

It is useful when downstream pipelines need to consume:

• inserts
• updates
• deletes

Databricks documentation describes Change Data Feed as a feature used to track row-level changes for Delta Lake and Apache Iceberg v3 tables.

A streaming read from a Delta table’s change feed can look like this:

spark.readStream \
    .format("delta") \
    .option("readChangeFeed", "true") \
    .table("db1.sales_orders")

For DP-750, if the requirement says:

Ingest all changes from a Delta table, including inserts, updates, and deletes.

then the key option is:

readChangeFeed = true

---

11. skipChangeCommits

When streaming from Delta tables, some options control how to handle source table updates and deletes.

One important option is:

.option("skipChangeCommits", "false")

For the DP-750 question in your bank, the correct dropdown combination is:

readChangeFeed = true
skipChangeCommits = false

This means the stream reads the change feed rather than skipping commits that contain changes.

For exam purposes:

|Requirement|Option| |---|---| |Read CDF changes|readChangeFeed| |Do not skip change commits|skipChangeCommits = false|

---

12. Checkpointing vs watermarking vs CDF

These options solve different problems.

|Feature|What it solves| |---|---| |Checkpointing|Recovery and progress tracking| |Watermarking|Late data handling in stateful aggregations| |Change Data Feed|Read inserts, updates, and deletes from Delta table| |Trigger interval|How often micro-batches run| |Schema evolution|Handles new columns or schema changes|

For DP-750:

• Reprocessing after restart → checkpointing

• New columns causing failure → schema evolution

• Need inserts, updates, deletes → CDF

• Late arriving event-time data → watermarking

---

13. DP-750 decision table for streaming

|Scenario in the question|Best answer pattern| |---|---| |Streaming job reprocesses data after restart|Configure checkpoint location| |Job must resume from failure point|Implement checkpointing| |Need exactly-once write to Delta|Structured Streaming + Delta + checkpoint| |New source columns cause write failure|Enable schema evolution| |Auto Loader code uses failOnNewColumns|New column fails the stream initially| |Code uses mergeSchema=true|New columns can be added to target| |Code uses outputMode("append")|New data is appended, not overwritten| |Need to ingest inserts, updates, deletes from Delta|Read Change Data Feed| |CDF streaming option|readChangeFeed = true| |Write stream to Delta|.format("delta")| |Failure recovery option|checkpointLocation| |Process available data then stop|availableNow=True|

---

Real Exam Questions

Question 8

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You have an Apache Spark Structured Streaming job that writes data to a Delta table.

After the cluster restarts, the streaming job reprocesses previously ingested data.

You need to prevent the streaming job from reprocessing the data after the cluster restarts.

What should you do?

A. Configure a checkpoint location for the streaming query. ✅ Correct Answer

B. Increase the trigger interval of the streaming query.

C. Enable change data feed, CDF, for the target table.

D. Configure a watermark for the streaming query.

---

Question 43

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You plan to run the following PySpark code.

(
    df.writeStream
        .format("delta")
        .option("checkpointLocation", CHECKPOINT_LOCATION)
        .outputMode("append")
        .option("mergeSchema", "true")
        .trigger(availableNow=True)
        .toTable(BRONZE_TABLE)
)

For each of the following statements, select Yes if the statement is true. Otherwise, select No.

|Statement|Answer| |---|---| |New columns from incoming JSON payload will be added automatically to the target Delta table.|Yes ✅ Correct Answer| |Previously processed data will not be reprocessed if the pipeline fails.|Yes ✅ Correct Answer| |Each new batch of telemetry data will overwrite the existing data in the target Delta table.|No ✅ Correct Answer|

---

Question 47

You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named db1.sales_orders.

db1.sales_orders is updated nightly and has change data feed, CDF, enabled.

You need to ingest all the changes from the db1.sales_orders table, including inserts, updates, and deletes, into a downstream pipeline.

How should you complete the PySpark code segment?

spark.readStream \
    .format("delta") \
    .option(______, "true") \
    .option(______, "false") \
    .table("db1.sales_orders")

|Area|Answer| |---|---| |First dropdown|"readChangeFeed" ✅ Correct Answer| |Second dropdown|"skipChangeCommits" ✅ Correct Answer|

---

Question 48

You have an Azure Databricks workspace that uses Unity Catalog.

You have a Lakeflow Spark Declarative Pipelines, SDP, pipeline that ingests data into a managed Delta table named Table1. Table1 is used for analytics.

New columns are added to the source data, causing pipeline failures during writes to Table1.

You need to prevent the pipeline failures. The solution must ensure that schema changes are detected and handled.

What should you do?

A. Disable schema enforcement for Table1.

B. Use row filters to exclude records that have new columns.

C. Enable schema evolution. ✅ Correct Answer

D. Create a separate table for each schema version.

---

Question 54

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You plan to run the following PySpark code.

(
    spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .option("cloudFiles.schemaLocation", "/cloud/schema")
        .option("cloudFiles.schemaEvolutionMode", "failOnNewColumns")
        .load("/data/source")
        .writeStream
        .option("checkpointLocation", "/data/checkpoint")
        .toTable("table1")
)

For each of the following statements, select Yes if the statement is true. Otherwise, select No.

|Statement|Answer| |---|---| |When a new element is found in the source data, the element will be written to table1 successfully.|No ✅ Correct Answer| |The streaming process will resume from a failure without data loss.|Yes ✅ Correct Answer| |Data will be written to a JSON file.|No ✅ Correct Answer|

---

Question 59

Useful case information

Contoso identifies the following data ingestion and processing requirements:

• Handle schema drift for the maintenance and telemetry data.

• Store all the ingested data in a format that supports incremental processing.

• Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics.

• Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events.

• The pipelines must support schema drift and resume processing after failures without reprocessing the data.

You need to complete the PySpark code for the Spark Structured Streaming pipelines. The solution must meet the data ingestion and processing requirements.

How should you complete the code segment?

telemetry_path = "abfss://telemetry@contosodata.dfs.core.windows.net/"
checkpoint_path = "abfss://telemetry@contosodata.dfs.core.windows.net/checkpoints/raw_events"
raw_events = "analytics.telemetry.raw_events"

df = (
    spark.readStream
        .format("json")
        .load(telemetry_path)
)

df.writeStream
    .format( ______ )
    .option( ______ )
    .option("mergeSchema", "true")
    .trigger(availableNow=True)
    .table(raw_events)

|Area|Answer| |---|---| |First dropdown|"delta" ✅ Correct Answer| |Second dropdown|("checkpointLocation", checkpoint_path) ✅ Correct Answer|

---

Question 71

You have an Azure Databricks workspace.

You have an Apache Spark Structured Streaming job named Job1 that processes data continuously and fails periodically due to transient errors.

You need to ensure that Job1 meets the following requirements:

• Resumes processing from the point that Job1 failed

• Minimizes how long it takes to restart Job1

• Minimizes the costs to restart Job1

What should you do?

A. Decrease the retry interval.

B. Increase the minimum number of nodes in the cluster.

C. Add an alert and manually restart Job1.

D. Implement checkpointing. ✅ Correct Answer

---

Key takeaways

For DP-750, Structured Streaming questions are usually about recovery, incremental processing, and schema changes.

Remember these patterns:

• Checkpointing prevents reprocessing after restart.

• checkpointLocation is required for fault tolerance and exactly-once guarantees.

• Each streaming query should use a unique checkpoint location.

• Delta tables are the normal target for reliable streaming writes.

• mergeSchema=true allows new incoming columns to be added.

• Schema evolution handles new source columns.

• failOnNewColumns causes the stream to fail when new columns appear, so the schema can evolve.

• availableNow=True processes available data and then stops.

• outputMode("append") appends data; it does not overwrite existing records.

• Change Data Feed is used to ingest inserts, updates, and deletes from a Delta table.

• For CDF streaming reads, use readChangeFeed=true.

If you can identify whether the issue is failure recovery, schema drift, Delta change capture, or output behavior, you can usually choose the correct DP-750 streaming answer quickly.