DP-750: Data Ingestion Patterns Explained and with Real Exam Questions

When preparing for DP-750: Microsoft Certified: Azure Databricks Data Engineer Associate, you need to know how to choose the right ingestion pattern for different data sources.

This is a common exam topic because Azure Databricks supports several ingestion options:

• Auto Loader
• COPY INTO
• Lakeflow Connect
• Apache Spark Structured Streaming
• Databricks notebooks
• Lakeflow Jobs triggers

The related questions in your DP-750 question bank are Q37, Q44, Q46, and Q56.

---

1. Why data ingestion patterns matter in DP-750

Data ingestion means loading data from source systems into the lakehouse.

In DP-750, the source system is often one of these:

|Source type|Typical ingestion pattern| |---|---| |Files in cloud storage|Auto Loader or COPY INTO| |Frequently arriving files|Auto Loader| |Simple SQL-based file loading|COPY INTO| |SaaS applications or databases|Lakeflow Connect| |Azure Event Hubs|Spark Structured Streaming| |REST API|Databricks notebook or custom code|

The exam usually gives you requirements such as:

• minimize custom code
• handle schema drift
• process new files incrementally
• support near-real-time ingestion
• ingest from operational databases
• read streaming events
• preserve existing table data
• use Unity Catalog governance

Your job is to match the requirement to the right ingestion tool.

---

2. Auto Loader

Auto Loader is one of the most important ingestion features for DP-750.

Auto Loader incrementally discovers and processes new files as they arrive in cloud object storage. Databricks documentation describes Auto Loader as a way to incrementally and efficiently process new data files as they arrive in cloud storage. It can load files into Delta tables and supports both Python and SQL.

Auto Loader is commonly used with:

spark.readStream.format("cloudFiles")

Example:

df = (
    spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/path/to/source")
)

Use Auto Loader when the question says:

• new files arrive frequently
• data must be available in near-real time
• minimize custom code
• handle large-scale file ingestion
• support schema drift or schema evolution
• ingest JSON, CSV, Parquet, or other file-based data

---

3. Auto Loader and schema drift

Schema drift means the source data structure changes over time.

For example, today’s JSON files contain:

{
  "sensor_id": "S001",
  "temperature": 21.5
}

Tomorrow’s JSON files contain a new field:

{
  "sensor_id": "S001",
  "temperature": 21.5,
  "humidity": 62
}

This is common in IoT telemetry and operational logs.

Auto Loader supports schema inference and schema evolution. Databricks documentation says Auto Loader can automatically detect the schema of loaded data and supports schema evolution options such as cloudFiles.schemaEvolutionMode.

For DP-750:

> File-based ingestion + schema drift + minimal operational effort = Auto Loader

---

4. COPY INTO

COPY INTO is a SQL command for loading files from cloud storage into a Delta table.

Example:

COPY INTO main.sales.orders
FROM 'abfss://raw@storage1.dfs.core.windows.net/orders/'
FILEFORMAT = CSV;

Databricks documentation says COPY INTO can load data incrementally and idempotently from cloud storage into a Delta table. It is often used for incremental or bulk loading from cloud object storage.

Use COPY INTO when the question says:

• load files into an existing table
• use SQL
• load incrementally
• minimize processing effort
• source is cloud object storage
• source files are CSV, JSON, Parquet, Avro, ORC, text, or similar

For DP-750:

> SQL-based incremental file loading = COPY INTO

---

5. COPY INTO vs Auto Loader

Both Auto Loader and COPY INTO can load files from cloud storage, but they are used in slightly different situations.

|Requirement|Better choice| |---|---| |Near-real-time file ingestion|Auto Loader| |Frequently arriving files|Auto Loader| |Large-scale file discovery|Auto Loader| |Schema drift handling|Auto Loader| |SQL-based incremental loading|COPY INTO| |Simple one-command file ingestion|COPY INTO| |Thousands of files|COPY INTO can be suitable| |Millions of files or advanced ingestion|Auto Loader is usually better|

Databricks documentation says COPY INTO works well for data sources that contain thousands of files, while Auto Loader is recommended for loading millions of files and advanced use cases.

For DP-750, if the question says:

> New rows are appended frequently and must be available near-real-time.

choose Auto Loader.

If the question says:

> Complete the SQL statement to incrementally load CSV files.

choose COPY INTO.

---

6. Lakeflow Connect

Lakeflow Connect is used to ingest data from systems such as SaaS applications, databases, cloud storage, and message buses.

Databricks documentation describes Lakeflow Connect as a set of connectors for ingesting data from local files, enterprise applications, databases, cloud storage, message buses, and more. Managed connectors are governed by Unity Catalog, powered by serverless compute and Lakeflow Spark Declarative Pipelines, and support efficient incremental reads and writes.

Use Lakeflow Connect when the question says:

• ingest from an operational database
• ingest from SQL Server
• ingest from Salesforce, ServiceNow, Google Analytics, or supported applications
• minimize custom ingestion code
• use a managed connector
• load data into the lakehouse

For DP-750:

> Structured operational database ingestion + managed connector = Lakeflow Connect

---

7. Lakeflow Connect vs foreign catalog

This is an important distinction.

|Requirement|Correct choice| |---|---| |Ingest data from external system into Databricks|Lakeflow Connect| |Query external database tables without copying data|Foreign catalog| |Data should appear in Unity Catalog but remain in source system|Foreign catalog| |Data should be loaded into lakehouse tables|Lakeflow Connect|

In the previous article, we used foreign catalog for read-only federation.

In this article, Lakeflow Connect is about ingestion.

For DP-750:

> If data is copied or ingested into Databricks, think Lakeflow Connect. > If data is not copied and only queried externally, think foreign catalog.

---

8. Azure Event Hubs ingestion

Azure Event Hubs is used for streaming event data.

In Azure Databricks, Event Hubs data can be read with Spark Structured Streaming. A common pattern is:

events = (
    spark.readStream
        .format("eventhubs")
        .option("eventhubs.connectionString", "<auth_value>")
        .load()
)

Event Hubs event payloads are stored in the body field, so you commonly convert the body to a string:

payload = events.selectExpr("CAST(body AS STRING) AS payload")

Microsoft documentation for Azure Event Hubs with Databricks describes using Structured Streaming to ingest Event Hubs data into raw landing tables. Databricks also documents Event Hubs as a pipeline data source for Lakeflow pipelines.

For DP-750:

> Event Hubs + streaming ingestion = Spark Structured Streaming

And for the PySpark code:

Connection option = "eventhubs.connectionString"
Payload expression = "CAST(body AS STRING) AS payload"

---

9. REST API ingestion

Some data sources do not have a managed connector.

For example:

• external weather API
• custom business API
• third-party REST endpoint
• internal HTTP service

In these cases, a Databricks notebook is often the simplest answer.

The notebook can:

1. call the REST API
2. parse the JSON response
3. convert it into a DataFrame
4. write it to a Delta table
5. run on a schedule through Lakeflow Jobs

For DP-750, if the source is an external REST API and no managed connector option is available, the answer is often:

> A Databricks notebook

This is especially true when the question says the external data is retrieved from a REST API and written to cloud storage or Delta tables.

---

10. Incremental processing and Delta tables

DP-750 often includes the requirement:

> Store all ingested data in a format that supports incremental processing.

The answer is usually Delta Lake.

Delta tables support:

• ACID transactions
• scalable metadata
• time travel
• streaming reads and writes
• change data feed
• incremental processing patterns

For ingestion questions, this means the target should usually be a Delta table, especially a managed Delta table when Unity Catalog is enabled.

---

11. DP-750 decision table for ingestion patterns

|Scenario in the question|Best answer pattern| |---|---| |Ingest new files from cloud storage with minimal effort|Auto Loader| |Files arrive frequently and near-real-time is required|Auto Loader| |File-based telemetry with schema drift|Auto Loader| |Maintenance logs with unstructured files|Auto Loader| |Simple SQL command to load CSV files incrementally|COPY INTO| |Complete SQL statement: [?] table [?] path FILEFORMAT = CSV|COPY INTO table FROM path| |Operational database ingestion|Lakeflow Connect managed connector| |SQL Server ingestion into lakehouse|Lakeflow Connect| |External database query without copying data|Foreign catalog| |External REST API|Databricks notebook| |Azure Event Hubs streaming|Spark Structured Streaming| |Event Hubs authentication option|"eventhubs.connectionString"| |Event Hubs payload column|"CAST(body AS STRING) AS payload"|

---

Real Exam Questions

Question 37

Useful case information

Contoso ingests the following operational and business data:

• Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift.

• Maintenance logs: Maintenance systems generate historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts.

• Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server.

• External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts.

Contoso identifies the following data ingestion and processing requirements:

• Auto-scale ingestion pipelines to handle bursty workloads.
• Handle schema drift for the maintenance and telemetry data.
• Ingest file-based telemetry data by using minimal operational effort.
• Store all the ingested data in a format that supports incremental processing.
• Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server.

Which ingestion option should you recommend for each data source?

|Data source|Answer| |---|---| |Telemetry Data|Auto Loader ✅ Correct Answer| |Operational maintenance data|Lakeflow Connect managed connector ✅ Correct Answer| |Maintenance logs|Auto Loader ✅ Correct Answer| |External weather data|A Databricks notebook ✅ Correct Answer|

---

Question 44

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You plan to ingest data from CSV files stored in Azure Data Lake Storage Gen2. New rows are appended frequently.

You need to implement a data ingestion solution that meets the following requirements:

• New data must be available in near-real-time, NRT.
• The data must be stored in managed Delta tables.
• The solution must minimize custom code and maintenance effort.

What should you include in the solution?

A. scheduled Apache Spark batch jobs B. an Azure Data Factory pipeline C. Auto Loader ✅ Correct Answer D. an external table that references the CSV files

---

Question 46

You have an Azure Databricks workspace.

You need to ingest streaming data from Azure Event Hubs by using Apache Spark Structured Streaming. The solution must authenticate to Event Hubs and read the event payload.

How should you complete the PySpark code segment?

events = (
    spark.readStream
        .format("eventhubs")
        .option(______, "<auth_value>")
        .load()
)

payload = events.selectExpr(______ )

|Area|Answer| |---|---| |First dropdown|"eventhubs.connectionString" ✅ Correct Answer| |Second dropdown|"CAST(body AS STRING) AS payload" ✅ Correct Answer|

---

Question 56

You have an Azure Databricks workspace that is enabled for Unity Catalog.

You need to ingest data from the CSV files stored in an external Azure Data Lake Storage Gen2 container into a managed Delta table. The solution must meet the following requirements:

• Load data incrementally as new files arrive.

• Minimize custom code and operational effort.

How should you complete the SQL statement?

[Dropdown 1] main.sales.orders

[Dropdown 2] 'abfss://raw@storage1.dfs.core.windows.net/orders/'

FILEFORMAT = CSV

|Area|Answer| |---|---| |Dropdown 1|COPY INTO ✅ Correct Answer| |Dropdown 2|FROM ✅ Correct Answer|

Completed statement:

COPY INTO main.sales.orders
FROM 'abfss://raw@storage1.dfs.core.windows.net/orders/'
FILEFORMAT = CSV

---

Key takeaways

For DP-750, ingestion questions are mostly about matching the source and requirement to the right tool.

Remember these patterns:

• Auto Loader is best for incremental cloud file ingestion, especially near-real-time ingestion and schema drift.

• COPY INTO is a SQL-based way to load files incrementally into Delta tables.

• Lakeflow Connect is used for managed ingestion from databases and enterprise applications.

• Foreign catalog is for querying external databases without copying data, not ingestion.

• Databricks notebooks are useful for custom ingestion, especially REST APIs.

• Azure Event Hubs streaming ingestion uses Spark Structured Streaming.

• Event Hubs payload is in the body column.

• Use "eventhubs.connectionString" for the Event Hubs connection option.

• Use "CAST(body AS STRING) AS payload" to extract the event payload.

If you can identify whether the source is cloud files, operational database, Event Hubs, or REST API, you can usually choose the correct DP-750 ingestion pattern quickly.