Databricks-Certified-Data-Engineer-Associate Databricks Certified Data Engineer Associate Exam Questions and Answers

In a Databricks customer ' s cloud account, the primary elements stored include:

Data: This is the central type of content stored in the customer ' s cloud account. Data might include various datasets, tables, and files that are used and managed through Databricks platforms.

Notebooks: These are also stored within a customer ' s cloud account. Notebooks include scripts, notes, and other information necessary for data analysis and processing tasks.

Cluster management metadata is indeed managed through the cloud, but it ' s primarily handled by Databricks rather than stored directly in the customer ' s account. The Databricks web application itself is not stored within the customer ' s cloud account; rather, it ' s a service provided by Databricks.

Unity Catalog attribute-based access control policies provide centralized, tag-driven governance. Administrators define policies at a catalog or schema scope and use governed tags to identify the tables or columns to which the policies apply. When matching tags are assigned to current or future objects, the applicable policy is evaluated automatically, avoiding repetitive manual attachment of filters and masks to individual tables. This approach improves scalability and consistency when many objects require the same protection rules. It is important that the policy applies to objects matching its tag conditions, rather than indiscriminately to every object. The other choices incorrectly describe supported object types, multiple-filter behavior, or implementation languages. Databricks specifically positions attribute-based access control as the centralized approach for consistently applying row filtering and column masking across many governed tables. Therefore, option A is correct.

================

he approach that can be used to identify the owner of new_table is to review the Owner field in the table’s page in Data Explorer. Data Explorer is a web-based interface that allows users to browse, create, and manage data objects such as tables, views, and functions in Databricks1. The table’s page in Data Explorer provides various information about the table, such as its schema, partitions, statistics, history, and permissions2. The Owner field shows the name and email address of the user who created or owns the table3. The data engineer can use this information to contact the table owner and request for permission to access the table.

The other options are not correct or reliable for identifying the owner of new_table. Reviewing the Permissions tab in the table’s page in Data Explorer can show the users and groups who have access to the table, but not necessarily the owner4. Reviewing the Owner field in the table’s page in the cloud storage solution can be misleading, as the owner of the data files may not be the same as the owner of the table5. There is a way to identify the owner of the table, as explained above, so option E is false.

1: Data Explorer | Databricks on AWS

2: Table details | Databricks on AWS

3: Set owner when creating a view in databricks sql - Databricks - 9978

4: Table access control | Databricks on AWS

5: External tables | Databricks on AWS

The expected behavior when a batch of data containing data that violates the expectation is processed is that the job will fail. This is because the expectation clause has the ON VIOLATION FAIL UPDATE option, which means that if any record in the batch does not meet the expectation, the entire batch will be rejected and the job will fail. This option is useful for enforcing strict data quality rules and preventing invalid data from entering the target dataset.

Option A is not correct, as the ON VIOLATION FAIL UPDATE option does not drop the records that violate the expectation, but fails the entire batch. To drop the records that violate the expectation and record them as invalid in the event log, the ON VIOLATION DROP RECORD option should be used.

Option C is not correct, as the ON VIOLATION FAIL UPDATE option does not drop the records that violate the expectation, but fails the entire batch. To drop the records that violate the expectation and load them into a quarantine table, the ON VIOLATION QUARANTINE RECORD option should be used.

Option D is not correct, as the ON VIOLATION FAIL UPDATE option does not add the records that violate the expectation, but fails the entire batch. To add the records that violate the expectation and record them as invalid in the event log, the ON VIOLATION LOG RECORD option should be used.

Option E is not correct, as the ON VIOLATION FAIL UPDATE option does not add the records that violate the expectation, but fails the entire batch. To add the records that violate the expectation and flag them as invalid in a field added to the target dataset, the ON VIOLATION FLAG RECORD option should be used.

Delta Live Tables Expectations

[Databricks Data Engineer Professional Exam Guide]

A SQL Serverless Warehouse is the most suitable option for running SQL-based exploratory data analysis at scale while minimizing operational and cost overhead. Serverless SQL warehouses are fully managed by Databricks , meaning Databricks automatically provisions and scales compute resources, applies Photon acceleration , and optimizes query execution without requiring the user to manage infrastructure. This makes them ideal for workloads involving large numbers of ad hoc or interactive SQL queries , which is common during EDA. Because the data resides in a custom-defined cloud network , serverless SQL warehouses can still securely access governed data when Unity Catalog and networking requirements are satisfied. All-purpose clusters (option A) are designed for collaborative development and mixed workloads, not cost-efficient high-concurrency SQL querying. Classic SQL warehouses (option D) are legacy and lack the performance and cost optimizations of newer serverless offerings. Pro SQL warehouses (option B) require user-managed sizing decisions, whereas serverless warehouses optimize both performance and cost automatically. Databricks documentation recommends SQL Serverless Warehouses for fast, scalable, and cost-effective SQL analytics.

=========

For small-scale, ad hoc Python workloads that run frequently and should start quickly and stop without leaving idle infrastructure running, serverless compute is the best fit. Databricks documentation explains that serverless compute allows users to run workloads without provisioning compute resources manually , and Databricks automatically allocates and manages the required resources. This improves startup and scaling speed and minimizes idle time, which directly matches the requirement that compute should “wind down quickly” after the work completes. Databricks also recommends serverless compute broadly when the workload is supported because it is the simplest and most reliable option. An all-purpose cluster is better suited to interactive collaborative development and can remain running longer than needed. A job cluster is ephemeral and often a good batch choice, but it still requires more explicit configuration than serverless and is not as optimized for minimal operational overhead. A SQL warehouse is intended for SQL workloads, not general ad hoc Python script execution. Therefore, serverless compute is the most appropriate choice.

 The spark.sql method can be used to execute SQL queries programmatically and return the result as a DataFrame. The spark.sql method accepts a string argument that contains a valid SQL statement. The data engineer can use a formatted string literal (f-string) to insert the Python variable table_name into the SQL query. The other methods are either invalid or not suitable for running SQL queries. References: Running SQL Queries Programmatically, Formatted string literals, spark.sql

 To grant full permissions on a table to a user or a group, you can use the GRANT ALL PRIVILEGES ON TABLE statement. This statement will grant all the possible privileges on the table, such as SELECT, CREATE, MODIFY, DROP, ALTER, etc. Option A is the only code block that follows this syntax correctly. Option B is incorrect, as it does not grant all the possible privileges on the table, but only a subset of them. Option C is incorrect, as it only grants the SELECT privilege on the table, which is not enough to fully manage the project. Option D is incorrect, as it grants the USAGE privilege on the table, which is not a valid privilege for tables. Option E is incorrect, as it grants all the privileges on the table team to the user or group sales, which is the opposite of what the question asks. References: Grant privileges on a table using SQL | Databricks on AWS, Grant privileges on a table using SQL - Azure Databricks, SQL Privileges - Databricks

Deploying a bundle creates or updates the resources defined in its configuration, but deployment does not automatically execute those resources. The databricks bundle run command starts a deployed job or pipeline by using its resource key from the bundle configuration. Because the job is keyed as etl_job, the correct command is databricks bundle run etl_job. The command bundle deploy run is not valid CLI syntax, and bundle execute is not the command used to run a bundle resource. databricks jobs submit creates and runs a one-time job from a supplied JSON definition rather than executing the job already managed by the bundle. Therefore, option D correctly triggers the deployed job while preserving the bundle-managed resource configuration.

================

According to the medallion lakehouse architecture, gold tables are the final layer of data that powers analytics, machine learning, and production applications. They are often highly refined and aggregated, containing data that has been transformed into knowledge, rather than just information. Silver tables, on the other hand, are the intermediate layer of data that represents a validated, enriched version of the raw data from the bronze layer. They provide an enterprise view of all its key business entities, concepts and transactions, but they may not have all the aggregations and calculations that are required for specific use cases. Therefore, gold tables are more likely to contain aggregations than silver tables. References:

What is the medallion lakehouse architecture?

What is a Medallion Architecture?

Option A uses the supported ALTER TABLE ... SET MANAGED conversion for a Unity Catalog external Delta table. Databricks documents that this conversion retains the table name, permissions, dependent objects, settings, and table history while minimizing downtime for readers and writers. After conversion, Databricks manages the table’s storage lifecycle, satisfying the requirement for platform-controlled cleanup when the managed table is dropped. Dropping the original table with PURGE destroys the object instead of migrating it. SET LOCATION changes the storage path referenced by an external table but does not convert the table into a managed table. Creating a replacement table with CREATE TABLE AS SELECT introduces a new object and requires manual migration of permissions, dependencies, and workloads. Therefore, ALTER TABLE catalog.schema.sales SET MANAGED is the correct action.

================

The best action that the data engineer can perform to improve the start up time for the clusters used for the Job is to use clusters that are from a cluster pool. A cluster pool is a set of idle clusters that can be used by jobs or interactive sessions. By using a cluster pool, the data engineer can avoid the cluster creation time and reduce the latency of the tasks. Cluster pools also offer cost savings and resource efficiency, as they can be shared by multiple users and jobs.

Option A is not relevant, as endpoints available in Databricks SQL are used for creating and managing SQL analytics workloads, not for improving cluster start up time.

Option B is not correct, as jobs clusters and all-purpose clusters have similar start up times. Jobs clusters are clusters that are dedicated to run a single job and are terminated when the job is completed. All-purpose clusters are clusters that can be used for multiple purposes, such as interactive sessions, notebooks, or multiple jobs. Both types of clusters can benefit from using a cluster pool.

Option C is not advisable, as configuring the clusters to be single-node will reduce the parallelism and performance of the tasks. Single-node clusters are clusters that have only one worker node and are typically used for testing or development purposes. They are not suitable for running production jobs that require high scalability and fault tolerance.

Option E is not helpful, as configuring the clusters to autoscale for larger data sizes will not affect the start up time of the clusters. Autoscaling is a feature that allows clusters to dynamically adjust the number of worker nodes based on the workload. It can help optimize the resource utilization and cost efficiency of the clusters, but it does not speed up the cluster creation process.

Cluster Pools

Jobs

Clusters

[Databricks Data Engineer Professional Exam Guide]

The File Arrival trigger and Auto Loader perform separate, complementary functions. The trigger monitors the configured storage location and starts the Lakeflow Job after detecting new files. After the notebook starts, Auto Loader examines its own checkpoint and file-tracking state to determine which files have not yet been committed successfully. This separation provides incremental processing and exactly-once ingestion behavior. With availableNow=True, Auto Loader processes all data available when the query starts and then stops, allowing the job compute to terminate and control costs. File Notification mode is not required solely because a File Arrival trigger is used. Disabling checkpoints would remove progress tracking and could cause duplicate processing. The trigger also does not eliminate the need to configure the Auto Loader source path. Therefore, option A is correct.

================

In Databricks Asset Bundles , the databricks.yml file located at the repository root serves as the single authoritative configuration file for the entire project. This file defines key elements such as the bundle name, deployment targets (for example, dev, test, prod), workspace configurations, permissions, and resource definitions including jobs and pipelines. Databricks documentation specifies that there must be one primary databricks.yml file at the root , ensuring consistency and simplifying deployment processes. However, to support modularity and maintainability, this root configuration file can reference additional YAML files using the include mapping. This allows teams to organize configurations across directories while still maintaining a single source of truth. Option A is incorrect because multiple root-level configuration files are not supported as independent authorities. Option C is invalid since configuration files are not restricted to hidden directories. Option D introduces ambiguity and contradicts the requirement for a single authoritative configuration. Therefore, using one root-level configuration file with optional includes is the correct and recommended approach.

 The keyword that can be used to compact the small files associated with a Delta table is OPTIMIZE. The OPTIMIZE command performs file compaction on a Delta table by rewriting a set of small files into a set of larger files1. This can improve the performance of queries that scan the table by reducing the number of files that need to be read and the amount of metadata that needs to be processed1. The OPTIMIZE command can also optionally sort the data within each file by a given set of columns, which can further improve the query performance by enabling data skipping and predicate pushdown1. The OPTIMIZE command can be applied to the whole table or to a specific partition of the table1.

The other keywords are not suitable for compacting the small files associated with a Delta table. REDUCE is a keyword used in the SQL syntax for aggregating data using a user-defined function2. COMPACTION is not a valid keyword in SQL or Python. REPARTITION is a keyword used in the Python syntax for changing the number of partitions of a DataFrame or an RDD3. VACUUM is a keyword used to remove files that are no longer referenced by a Delta table and are older than a retention threshold4.

1: OPTIMIZE | Databricks on AWS

2: REDUCE | Databricks on AWS

3: repartition | Databricks on AWS

4: VACUUM | Databricks on AWS

https://community.databricks.com/t5/data-engineering/sequential-vs-concurrency-optimization-questions-from-query/td-p/36696

The MERGE INTO command is used to perform upserts, which are a combination of insertions and updates, based on a source table into a target Delta table1. The MERGE INTO command can handle scenarios where the target table cannot contain duplicate records, such as when there is a primary key or a unique constraint on the target table. The MERGE INTO command can match the source and target rows based on a merge condition and perform different actions depending on whether the rows are matched or not. For example, the MERGE INTO command can update the existing target rows with the new source values, insert the new source rows that do not exist in the target table, or delete the target rows that do not exist in the source table1.

The INSERT INTO command is used to append new rows to an existing table or create a new table from a query result2. The INSERT INTO command does not perform any updates or deletions on the existing target table rows. The INSERT INTO command can handle scenarios where the location of the data needs to be changed, such as when the data needs to be moved from one table to another, or when the data needs to be partitioned by a certain column2. The INSERT INTO command can also handle scenarios where the target table is an external table, such as when the data is stored in an external storage system like Amazon S3 or Azure Blob Storage3. The INSERT INTO command can also handle scenarios where the source table can be deleted, such as when the source table is a temporary table or a view4. The INSERT INTO command can also handle scenarios where the source is not a Delta table, such as when the source is a Parquet, CSV, JSON, or Avro file5.

1: MERGE INTO | Databricks on AWS

2: [INSERT INTO | Databricks on AWS]

3: [External tables | Databricks on AWS]

4: [Temporary views | Databricks on AWS]

5: [Data sources | Databricks on AWS]

 A serverless SQL endpoint is a compute resource that is automatically managed by Databricks and scales up or down based on the workload. A serverless SQL endpoint can be used to run queries and dashboards without requiring manual configuration or management. A serverless SQL endpoint is only active when it is needed and shuts down automatically when idle, minimizing the total running time and cost. A serverless SQL endpoint can be created and assigned to a dashboard using the Databricks SQL UI or the SQL Analytics API. References:

Create a serverless SQL endpoint

Assign a SQL endpoint to a dashboard

SQL Analytics API

A temporary view is a relational object that is defined in the metastore and points to an existing DataFrame. It does not copy or store any physical data, but only saves the query that defines the view. The lifetime of a temporary view is tied to the SparkSession that was used to create it, so it does not persist across different sessions or applications. A temporary view is useful for accessing the same data multiple times within the same notebook or session, without incurring additional storage costs. The other options are either materialized (A, E), persistent (B, C), or not relational objects ©. References: Databricks Documentation - Temporary View, Databricks Community - How do temp views actually work?, Databricks Community - What’s the difference between a Global view and a Temp view?, Big Data Programmers - Temporary View in Databricks.

A Gold table is a table that contains highly refined and aggregated data that powers analytics, machine learning, and production applications. It represents data that has been transformed into knowledge, rather than just information. A Gold table is typically the final output of a medallion lakehouse architecture, where data flows from Bronze to Silver to Gold tables, with each layer improving the structure and quality of data. A job that queries aggregated data designed to feed into a dashboard is an example of a data workload that will utilize a Gold table as its source, as it requires data that is ready for consumption and analysis. The other options are either data workloads that will use a Bronze or Silver table as their source, or data workloads that will produce a Gold table as their output. References: Databricks Documentation - What is the medallion lakehouse architecture?, Databricks Documentation - What is a Medallion Architecture?, K21Academy - Delta Lake Architecture & Azure Databricks Workspace.

In Python, the == operator is used to compare the values of two variables, while the = operator is used to assign a value to a variable. Therefore, option A and E are incorrect, as they use the = operator for comparison. Option B and C are also incorrect, as they compare the review_period variable to a string value  " True " , which is different from the boolean value True. Option D is the correct answer, as it uses the == operator to compare the day_of_week variable to the integer value 1, and the and operator to check if both conditions are true. If both conditions are true, then the final block of the Python program will be executed. References: [Python Operators], [Python If … Else]

JSON data is a text-based format that represents data as a collection of name-value pairs. By default, when Auto Loader infers the schema of JSON data, it treats all columns as strings. This is because JSON data can have varying data types for the same column across different files or records, and Auto Loader does not attempt to reconcile these differences. For example, a column named “age” may have integer values in some files, but string values in others. To avoid data loss or errors, Auto Loader infers the column as a string type. However, Auto Loader also provides an option to infer more precise column types based on the sample data. This option is called cloudFiles.inferColumnTypes and it can be set to true or false. When set to true, Auto Loader tries to infer the exact data types of the columns, such as integers, floats, booleans, or nested structures. When set to false, Auto Loader infers all columns as strings. The default value of this option is false. References: Configure schema inference and evolution in Auto Loader, Schema inference with auto loader (non-DLT and DLT), Using and Abusing Auto Loader’s Inferred Schema, Explicit path to data or a defined schema required for Auto loader.

In Unity Catalog, the Metastore Admin role has the highest level of authority within a metastore and is responsible for governing all data assets registered in that metastore. Metastore admins can transfer ownership of catalogs , manage privileges across all securable objects (catalogs, schemas, tables, views), and administer storage credentials, external locations, and Delta Sharing objects such as shares and recipients. This makes the role essential when a platform owner departs and continuity must be maintained at an organizational level. Workspace admins (option B) manage workspace-level settings but do not automatically have authority over Unity Catalog ownership unless explicitly granted. Catalog owners (option D) are limited to a single catalog and cannot perform actions across the entire metastore. Account admins (option A) operate at the account level and are primarily responsible for creating metastores and assigning admins, not for day-to-day governance within an existing metastore. Databricks documentation clearly defines the Metastore Admin as the role required to centrally manage ownership, privileges, and data sharing across the metastore in Databricks.

Structured Streaming provides native Kafka integration and processes incoming records in repeated micro-batches, making it suitable for the required sub-minute clickstream latency. For the customer data, an incremental Lakeflow Connect ingestion pattern tracks source progress and processes only new or changed records during subsequent executions. This avoids repeatedly transferring and processing unchanged Snowflake rows. Option B performs a complete extraction every four hours, so the later MERGE does not eliminate unnecessary source reading and network transfer. Option C relies on a time filter that may miss late changes, updates without a suitable timestamp, or records affected by clock boundaries. Option D explicitly performs full refreshes and therefore violates the requirement to prevent reprocessing unchanged customer data. Consequently, option A is the only approach that satisfies both the streaming-latency and incremental-ingestion requirements.

================

 The VACUUM command is used to remove files that are no longer referenced by a Delta table and are older than the retention threshold1. The default retention period is 7 days2, but it can be changed by setting the delta.logRetentionDuration and delta.deletedFileRetentionDuration configurations3. If the VACUUM command was run on the table with a retention period shorter than 3 days, then the data files that were needed to restore the table to a 3-day-old version would have been deleted. The other commands do not delete data files from the table. The TIME TRAVEL command is used to query a historical version of the table4. The DELETE HISTORY command is not a valid command in Delta Lake. The OPTIMIZE command is used to improve the performance of the table by compacting small files into larger ones5. The HISTORY command is used to retrieve information about the operations performed on the table. References: 1: VACUUM | Databricks on AWS 2: Work with Delta Lake table history | Databricks on AWS 3: [Delta Lake configuration | Databricks on AWS]  4: Work with Delta Lake table history - Azure Databricks 5: [OPTIMIZE | Databricks on AWS] : [HISTORY | Databricks on AWS]

The Continuous Pipeline Mode for Delta Live Tables allows the pipeline to run continuously and process data as it arrives. This mode is suitable for streaming ingest and CDC workloads that require low-latency updates. The Development mode for Delta Live Tables allows the pipeline to run on a dedicated cluster that is not shared with other pipelines. This mode is useful for testing and debugging the pipeline logic before deploying it to production. Therefore, the correct answer is B, because the pipeline will run continuously on a dedicated cluster until it is manually stopped, and the compute resources will be released only after the pipeline is shut down. References: Databricks Documentation - Configure pipeline settings for Delta Live Tables, Databricks Documentation - Continuous vs. triggered pipeline execution, Databricks Documentation - Development vs. production mode.

Auto Loader in Databricks utilizes Spark Structured Streaming for processing data incrementally. This allows Auto Loader to efficiently ingest streaming or batch data at scale and to recognize new data as it arrives in cloud storage. Spark Structured Streaming provides the underlying engine that supports various incremental data loading capabilities like schema inference and file notification mode, which are crucial for the dynamic nature of data lakes.

With schema evolution enabled, Auto Loader detects source fields that are not present in the current schema and evolves the destination schema to include them. The new userAgent field is therefore added to the Delta table as a column, and values from newly processed records are written to it. Existing rows cannot be retroactively populated because the earlier source records did not contain that field, so they receive NULL for the newly added column. The column is consequently nullable unless a later transformation explicitly enforces another constraint. Option A describes behavior without suitable schema evolution. Option B incorrectly claims that historical rows receive a default value and that the column becomes non-nullable. Option D contradicts schema-evolution behavior. Therefore, option C correctly describes how the new JSON field is handled.

Option B changes duplicate detection from all-column comparison to comparison on the two required business keys. With no subset argument, dropDuplicates() considers every column, so records that differ in event_timestamp are not duplicates. Databricks documents that passing a list such as [ " user_id " , " event_type " ] makes those columns the duplicate-comparison subset while returning a DataFrame with the original schema. groupBy(...).count() aggregates the data and returns counts instead of one complete event row. Dropping event_timestamp and then calling dropDuplicates() still compares every remaining column, not only the two specified keys. Calling distinct() before selecting the keys also performs distinctness on complete rows and then discards the non-key columns. Therefore, df.dropDuplicates([ " user_id " , " event_type " ]) is the precise change, making option B correct.

In Unity Catalog, managed tables and external tables have different lifecycle behavior. When you drop a managed table , Databricks removes the table metadata and also schedules deletion of the underlying managed data files according to the managed-table retention behavior. By contrast, when you drop an external table , Unity Catalog removes only the table metadata and does not delete the underlying files in cloud storage. This distinction exists specifically so that external data can remain available to systems outside Databricks. In the scenario described, ephemeral staging outputs that should disappear completely are best represented by managed tables, while tables backed by shared cloud storage should be external tables so that dropping them does not remove files used by downstream jobs. Therefore, option C is the correct answer. Options A, B, and D contradict documented Unity Catalog behavior because external table files are not automatically deleted, while managed table data is lifecycle-managed by Databricks.

=========

Lakeflow Connect managed connectors are designed to simplify ingestion from supported enterprise applications and databases. They provide managed authentication, incremental processing, schema handling, and integration with Unity Catalog. Salesforce is supported as a managed application source, while SQL Server ingestion can accommodate database connectivity patterns, including privately connected or on-premises environments when the required networking is configured. Auto Loader is the appropriate service for incrementally discovering and processing new JSON files arriving in ADLS. It supports schema inference and evolution and can be used within Lakeflow Spark Declarative Pipelines. Option B would require substantial custom API and CDC code. Option D incorrectly treats cloud files as the same connector use case as Salesforce and SQL Server. Therefore, the combination in option C minimizes maintenance while matching each source with its appropriate ingestion mechanism.

================

Option A is the supported migration path for bringing existing workspace resources under Declarative Automation Bundle management. The bundle generate command creates bundle configuration for supported existing resources, including jobs, pipelines, and dashboards, and can download referenced artifacts where applicable. The bundle deployment bind command then associates the bundle resource key with the existing workspace resource so that future deployments update that resource rather than creating a duplicate. The completed bundle project can be committed to Git and promoted through CI/CD. Exporting REST API JSON does not create a complete bundle workflow, manually recreating every resource is unnecessary, and bundle init creates a project from a template but does not automatically discover existing resources. Therefore, option A provides the intended generation and binding process.

================

In the given command, a data engineer is trying to create a table in Databricks using data from an SQLite database. The correct option to fill in the blank is “sqlite” because it specifies the type of database being connected to in a JDBC connection string. The USING clause should be followed by the format of the data, and since we are connecting to an SQLite database, “sqlite” would be appropriate here. References:

Create a table using JDBC

JDBC connection string

SQLite JDBC driver

A webhook alert destination is a notification destination that allows Databricks to send HTTP POST requests to a third-party endpoint when an alert is triggered. This enables the data engineer to integrate Databricks alerts with their preferred messaging or collaboration platform, such as Slack, Microsoft Teams, or PagerDuty. To set up a webhook alert destination, the data engineer needs to create and configure a webhook connector in their messaging platform, and then add the webhook URL to the Databricks notification destination. After that, the data engineer can create an alert for their Databricks SQL query, and select the webhook alert destination as the notification destination. The alert can be configured with a custom condition, such as when the number of stores with $0 in sales is greater than zero, and a custom message template, such as “Alert: {number_of_stores} stores have $0 in sales”. The alert can also be configured with a recurrence interval, such as every hour, to check the query result periodically. When the alert condition is met, the data engineer and their team will receive a notification via the messaging webhook, with the custom message and a link to the Databricks SQL query. The other options are either not suitable for sending notifications via a messaging webhook (A, B, E), or not suitable for sending recurring notifications ©. References: Databricks Documentation - Manage notification destinations, Databricks Documentation - Create alerts for Databricks SQL queries, Databricks Documentation - Configure alert conditions and messages.

https://www.w3schools.com/python/python_functions.asp

https://www.geeksforgeeks.org/python-functions/

When migrating to Delta Live Tables (DLT) with a data pipeline that involves different programming languages across various data layers, the migration does not require unifying the pipeline into a single language. Delta Live Tables support multi-language pipelines, allowing data engineers and data analysts to work in their preferred languages, such as Python for data engineering tasks (raw, bronze, and silver layers) and SQL for data analytics tasks (gold layer). This capability is particularly beneficial in collaborative settings and leverages the strengths of each language for different stages of data processing.

Lakehouse Federation is the Databricks feature built for querying external systems without moving all data into Databricks . Databricks documentation describes it as the platform for query federation , enabling users to run queries against multiple external data sources while keeping governance centralized through Unity Catalog . Databricks also documents support for external systems such as Amazon Redshift and other databases through connections and foreign catalogs, allowing read-only access to external data while managing permissions in Unity Catalog. This aligns directly with the requirement to minimize duplication and still maintain centralized governance. Databricks Connect is for local development against Databricks compute, not federated querying. MLflow is for machine learning lifecycle management. Delta Lake is a storage format and table layer, not a federation framework. Therefore, when the goal is unified governance across Databricks and external systems like MySQL, Redshift, and BigQuery without first ingesting the data, Lakehouse Federation is the documented answer. Databricks does note that for high-volume production ingestion, managed connectors may sometimes be preferred, but for direct querying without data movement, Lakehouse Federation is the correct feature.

A data engineer can create a multi-task job in Databricks that consists of multiple tasks that run in a specific order. Each task can have one or more dependencies, which are other tasks that must run before the current task. The Depends On field of a new Databricks Job Task allows the data engineer to specify the dependencies of the task. The data engineer should select a task in the Depends On field when they want the new task to run only after the selected task has successfully completed. This can help the data engineer to create a logical sequence of tasks that depend on each other’s outputs or results. For example, a data engineer can create a multi-task job that consists of the following tasks:

Task A: Ingest data from a source using Auto Loader

Task B: Transform the data using Spark SQL

Task C: Write the data to a Delta Lake table

Task D: Analyze the data using Spark ML

Task E: Visualize the data using Databricks SQL

In this case, the data engineer can set the dependencies of each task as follows:

Task A: No dependencies

Task B: Depends on Task A

Task C: Depends on Task B

Task D: Depends on Task C

Task E: Depends on Task D

This way, the data engineer can ensure that each task runs only after the previous task has successfully completed, and the data flows smoothly from ingestion to visualization.

The other options are incorrect because they do not describe valid scenarios for selecting a task in the Depends On field. The Depends On field does not affect the following aspects of a task:

Whether the task needs to be replaced by another task

Whether the task needs to fail before another task begins

Whether the task has the same dependency libraries as another task

Whether the task needs to use as little compute resources as possible References: Create a multi-task job, Run tasks conditionally in a Databricks job, Databricks Jobs.

Option C extracts each JSON field into a separate Silver-table column and casts age to the appropriate integer type. get_json_object accepts a JSON string and a JSONPath expression rooted at $; paths such as $.user_id and $.age therefore identify the required top-level fields correctly. This produces analytics-ready columns and prevents downstream queries from repeatedly parsing the original JSON string. Option A returns one nested struct and incorrectly defines age as STRING, so further projection or casting would still be required. Option B attempts field notation directly on a STRING column, which is not valid until the JSON has been parsed into a structured type. Option D omits the required $ JSONPath root and leaves age uncast. Therefore, option C most completely standardizes the source data for the Silver layer.

================

Delta Sharing is an open protocol developed by Databricks that enables secure, real-time sharing of data across organizations without requiring recipients to have a Databricks account. It is specifically designed for scenarios where external partners need read-only access to data stored in Delta format. With Delta Sharing, data providers create a share and grant access to recipients via a secure mechanism (such as a profile file or URL-based access), ensuring that the data cannot be modified by the consumer. This approach eliminates the need for data duplication or manual file transfers, maintaining data freshness and governance. Option B is incorrect because Unity Catalog governs internal access within an organization, not external sharing without accounts. Option C is inefficient and breaks data lineage and freshness. Option D violates security best practices by granting excessive permissions. Delta Sharing ensures fine- grained access control, auditability, and secure external collaboration , making it the correct solution for this requirement.

=========

Option D is directly associated with the failure because toPandas() collects the complete Spark DataFrame into a single pandas DataFrame in the driver process. With 50 million wide rows, the collected result can exceed the memory assigned to the Spark driver and produce a driver-side OutOfMemoryError. Databricks and PySpark documentation warn that toPandas() should be used only when the resulting dataset is expected to be small because all records are loaded into driver memory. spark.executor.memory controls the executor processes that perform distributed transformations, not the final local pandas object. spark.sql.shuffle.partitions controls the number of shuffle partitions, while spark.memory.fraction controls Spark execution and storage memory allocation. Therefore, the relevant configuration among the available choices is spark.driver.memory, making option D correct.

================

In Production mode, the pipeline runs continuously and updates the output tables whenever new data is available in the input sources. The compute resources are allocated on demand and released when the pipeline is stopped. This mode is suitable for production workloads that require high availability and reliability. References: Configure pipeline settings for Delta Live Tables, Tutorial: Run your first Delta Live Tables pipeline, Building Reliable Data Pipelines Using DataBricks’ Delta Live Tables

File notification mode is the recommended option for scalable file discovery. On the stream’s first execution, Auto Loader performs a complete directory listing to discover existing files and establish a valid read position in its checkpoint and file-events cache. It can therefore ingest the 50,000 historical files without requiring a separate directory-listing stream. Subsequent executions discover newly arriving files through cloud-storage events instead of repeatedly scanning the entire directory, improving discovery performance and reducing cloud API and compute costs. Auto Loader maintains its checkpoint state and exactly-once processing guarantees when using either detection mode. Although switching modes across restarts is supported, it is unnecessary for this requirement. Options B, C, and D incorrectly imply that notification mode cannot efficiently initialize from existing files. Therefore, option A is correct.

================

The < = > operator performs null-safe equality in Databricks SQL. It returns TRUE when both operands contain the same non-NULL value and also when both operands are NULL. It returns FALSE when only one operand is NULL. By contrast, the standard equality operator returns UNKNOWN when either side is NULL, so it cannot match two NULL values in a WHERE condition. Option C implements inequality logic and does not provide null-safe equality. Option D can incorrectly treat a genuine empty string as equivalent to NULL, changing the meaning of the data. Therefore, option A is the only predicate that preserves SQL values while implementing the required NULL comparison behavior. This question was also checked against the existing Word question bank.

================

The query performing a streaming hop from raw data to a Bronze table is identified by using the Spark streaming read capability and then writing to a Bronze table. Let ' s analyze the options:

Option A: Utilizes .writeStream but performs a complete aggregation which is more characteristic of a roll-up into a summarized table rather than a hop into a Bronze table.

Option B: Also uses .writeStream but calculates an average, which again does not typically represent the raw to Bronze transformation, which usually involves minimal transformations.

Option C: This uses a basic .write with .mode( " append " ) which is not a streaming operation, and hence not suitable for real-time streaming data transformation to a Bronze table.

Option D: It employs spark.readStream.load() to ingest raw data as a stream and then writes it out with .writeStream, which is a typical pattern for streaming data into a Bronze table where raw data is captured in real-time and minimal transformation is applied. This approach aligns with the concept of a Bronze table in a modern data architecture, where raw data is ingested continuously and stored in a more accessible format.