Databricks-Machine-Learning-Professional Databricks Certified Machine Learning Professional Questions and Answers

Concept drift is a change in the relationship between input and output data over time, which can affect the performance and accuracy of the model. To monitor concept drift, one needs to compare the predicted values from the model with the observed (actual) label values from the data source. A common way to do this is to compute an evaluation metric, such as accuracy, precision, recall, F1-score, etc., using the observed and predicted values. This metric can indicate how well the model is performing on the current data, and how it differs from the baseline performance on the training data. If the metric shows a significant decline over time, it may suggest that concept drift has occurred and the model needs to be updated123 References:

• Monitoring Concept Drift - Databricks
• Data drift, concept drift, and how to monitor for them - Mona Labs
• 8 Concept Drift Detection Methods To Use With ML Models - Aporia

The correct way to log a scikit-learn model using MLflow is to use the mlflow.sklearn.log_model function, which logs the model as an MLflow artifact for the current run1. This function takes the scikit-learn model object and an artifact path as required arguments, and optionally accepts other arguments such as conda environment, code paths, serialization format, etc2. The other options are incorrect because they either use the wrong flavor (spark instead of sklearn), the wrong function (load_model or track_model instead of log_model), or do not exist (track_model is not a valid MLflow function). References: MLflow Models, mlflow.sklearn

Real-time deployment is a type of model deployment where predictions are computed on-demand at query time. This can be useful when the feature values are not available until the time of delivery, and results are needed exceedingly fast for one record at a time. Real-time deployment can leverage Databricks Model Serving, which allows users to host MLflow models as REST endpoints on Databricks1. The other options are incorrect because they either do not meet the requirements or are not supported by Databricks Machine Learning. Edge/on-device deployment is a type of model deployment where predictions are computed on the edge device, such as a smartphone or a sensor. This can be useful when there is limited or no network connectivity, or when privacy and security are important. However, edge deployment is not supported by Databricks Machine Learning2. Streaming deployment is a type of model deployment where predictions are computed on streaming data, such as sensor readings or web logs. This can be useful when the data volume is high and the results need to be updated continuously. However, streaming deployment is not supported by Databricks Model Serving1. Batch deployment is a type of model deployment where predictions are computed in advance and stored for later use. This can be useful when the feature values are known ahead of time and the predictions do not need to be delivered quickly. However, batch deployment does not meet the requirements of this scenario, as the feature values are not available until the time of delivery, and results are needed exceedingly fast for one record at a time. References: MLflow Model Serving, Model deployment patterns

 In a continuous integration, continuous deployment (CI/CD) process for machine learning pipelines, the arrival of a new model version in the MLflow Model Registry is a common event that triggers the execution of automated testing. This is because the MLflow Model Registry is a centralized repository for managing the full lifecycle of MLflow models. It provides model lineage, model versioning, stage transitions, and annotations. When a new model version is registered, it can be automatically tested for quality, performance, and compliance using various tools and frameworks, such as MLflow Projects, MLflow Models, MLflow Tracking, and MLflow Model Serving. The automated testing can help ensure that the new model version meets the desired criteria and standards before it is deployed to production or other stages. The automated testing can also help detect and prevent any errors, bugs, or drifts that may affect the model functionality or accuracy. References:

• MLflow Model Registry Documentation, p. 1-2
• MLOps: Continuous delivery and automation pipelines in machine learning, Continuous integration and delivery (CI/CD) for ML, p. 5-6
• Machine Learning Engineering with MLflow, Testing and Validating ML Models, p. 1-2
• CI/CD for Machine Learning: What it is & Benefits in 2023 - AIMultiple, CI/CD for Machine Learning, p. 1-2

The DESCRIBE HISTORY command can be used to view the commit history of a Delta table, including the schema changes, operations, and timestamps. This command can help identify when a column was dropped from the feature set and by which operation. The other commands are either invalid or do not provide the required information. References:

• Delta Lake - View Commit History
• Databricks Certified Machine Learning Professional Exam Guide - Section 1: Experimentation - Data Management

Drift is the change over time in the statistical properties of the data that was used to train a machine learning model. This can cause the model to become less accurate or perform differently than it was designed to1. Drift can be detected by monitoring the statistics of the input and output data over time and comparing them with the baseline statistics from the training data2. Depending on the type and severity of the drift, different responses may be appropriate. Some possible responses are:

• Retraining and deploying a model on more recent data: This can help the model adapt to the changes in the data and improve its performance. However, this may require frequent retraining and deployment cycles, which can be costly and time-consuming. Also, retraining may not be sufficient if the drift is caused by a change in the underlying concept or relationship between the input and output variables3.
• Rebuilding the machine learning application with a new label variable: This can help the model capture the new concept or relationship that has emerged in the data. However, this may require a significant redesign of the application and the data pipeline, as well as collecting and labeling new data. Also, rebuilding may not be feasible if the concept or relationship is constantly changing or unknown3.
• Sunsetting the machine learning application: This can help avoid the risks and costs of maintaining a model that is no longer reliable or useful. However, this may mean losing the benefits and value of the application and the data. Also, sunsetting may not be an option if the application is critical or mandatory for the business or the users3.

Therefore, all of these responses are probable, depending on the situation and the trade-offs involved. References:

• Databricks Machine Learning Professional Exam Guide, Section 4: Solution and Data Monitoring, p. 5
• Databricks Machine Learning Documentation, Monitoring ML Models, Data Drift Detection, p. 2-3
• A Gentle Introduction to Concept Drift in Machine Learning, Types of Concept Drift, p. 3-4
• Understanding Data Drift and Model Drift: Drift Detection in Python, Types of Drift, p. 2-3

The Model Registry is a centralized model store that allows you to manage the full lifecycle of MLflow Models. It provides model lineage, versioning, stage transitions, and annotations. You can use the Model Registry to discover and share models, collaborate on moving them from experimentation to production, integrate with approval and governance workflows, and monitor model deployments and performance1. The other options are incorrect because:

• Option A: Models is a component of MLflow that allows you to manage and deploy models from a variety of ML libraries to a variety of model serving and inference platforms. Models is not a centralized model store, but rather a convention for packaging machine learning models that offers self-contained code, environments, and models2.
• Option C: Model Serving is a feature of Databricks that allows you to host MLflow Models as REST endpoints. Model Serving is not a centralized model store, but rather a model deployment option that enables low-latency inference for real-time applications3.
• Option D: Feature Store is a feature of Databricks that allows you to store, discover, and share features for machine learning. Feature Store is not a centralized model store, but rather a centralized feature store that enables feature reuse, consistency, and governance across ML projects4.
• Option E: Experiments is a component of MLflow that allows you to track experiments to record and compare parameters and results. Experiments is not a centralized model store, but rather a logging service that stores runs, metrics,parameters, tags, and artifacts for each experiment. References: MLflow Model Registry, MLflow Models, MLflow Model Serving, Databricks Feature Store, [MLflow Tracking]

The mlflow.register_model operation can be used to add a new version of an MLflow Model to an existing MLflow Model Registry model. This operation takes the URI of the model source (such as an MLflow run or a local path) and the name of the registered model as arguments, and returns a ModelVersion object that contains metadata about the registered model version. The mlflow.register_model operation can also create a new registered model if the name does not exist in the Model Registry.

The other options are incorrect for the following reasons:

• B. MlflowClient.update_registered_model is used to update the metadata of an existing registered model, such as the name or description, not to add a new version of the model.
• C. mlflow.add_model_version does not exist as a valid MLflow operation.
• D. MlflowClient.get_model_version is used to fetch the metadata of a specific version of a registered model, not to add a new version of the model.
• E. The machine learning engineer does not need to create an entirely new MLflow Model Registry model, as they can use the mlflow.register_model operation to add a new version of the model to the existing “project” model.

References: Adding an MLflow Model to the Model Registry, mlflow.register_model

The correct code block is the one that matches the event type and the model name specified in the webhook. The event type is MODEL_VERSION_TRANSITIONED_TO_STAGING, which means the webhook will be triggered when a model version is transitioned to the staging stage. The model name is model, which means the webhook will only apply to the registered model with that name. Therefore, the code block that satisfies both conditions is:

client.transition_model_version_stage (name=model, version=model_version, stage=“Staging”)

This code block will transition the model version to the staging stage and invoke the webhook to run the associated job.

References: MLflow Model Registry Webhooks on Databricks, MLflow Python API

The MLflow Model Registry allows users to manage the full lifecycle of MLflow Models. It provides model lineage (which MLflow Experiment and Run produced the model), model versioning, stage transitions (for example from staging to production), and annotations. Users can use the MLflow Client API to interact with the Model Registry programmatically.

To move a model version from one stage to another, users can use the transition_model_version_stage method of the MLflow Client. This method takes the following parameters:

• name: The name of the registered model.
• version: The version number of the model.
• stage: The name of the stage to transition the model version to.
• archive_existing_versions: (Optional) If True, archive all other model versions in the same stage.

Therefore, to move the model version model_version for the model model from the Staging stage to the Production stage, and archive any existing model versions in the Production stage, the correct code block is:

PythonAI-generated code. Review and use carefully. More info on FAQ.

client.transition_model_version_stage(

name=model,

version=model_version,

stage="Production",

archive_existing_versions=True

)

References:

• [MLflow Model Registry - Databricks]
• [MLflow Client API - Databricks]
• [MLflow Model Registry API - Databricks] Message has links.

 To read data from a stream source, such as Kafka, socket, or rate, the spark.readStream method should be used instead of spark.read. The spark.readStream method returns a streaming DataFrame that represents the unbounded input data stream. The spark.readStream method supports the same options and formats as the spark.read method, such as schema, delta, csv, json, etc. The spark.readStream method can also read from a Delta table as a stream source, by specifying the format("delta") and the path or table name of the Delta table123

The other options are incorrect because:

• A. Replacing “inference” with the path to the location of the Delta table does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the path or table name of the Delta table can be specified as an option or argument.
• B. Replacing schema(schema) with option("maxFilesPerTrigger", 1) does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the schema can be specified as an option or argument. The option("maxFilesPerTrigger", 1) is an optional configuration that limits the number of files processed in each trigger for file-based stream sources, such as delta, csv, json, etc. It does not affect the reading of data from a stream source4
• D. Replacing format("delta") with format("stream") does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the format can be specified as an option or argument. The format("stream") is not a valid format for reading data from a stream source. The supported formats are delta, kafka, socket, rate, etc1
• E. Replacing predict with a stream-friendly prediction function does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the prediction function can be applied to the streaming DataFrame as usual. The predict function does not need to be changed, as long as it can accept a streaming DataFrame as input and return a column of predictions as output5

References:

• Input Sources - Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Structured Streaming + Delta Lake - Databricks
• Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Configuration - Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Machine Learning with Structured Streaming - Databricks

The Databricks REST APIs allow users to programmatically access Databricks features and services, such as clusters, jobs, workspaces, libraries, secrets, and more. The Jobs API enables users to create, edit, delete, run, and list jobs, as well as manage job runs. Users can use the Jobs API to programmatically create a new Databricks job whose schedule depends on the result of some automated tests in a machine learning pipeline. For example, users can use the create endpoint to create a new job with a cron_schedule parameter that specifies the cron expression for the job schedule. Users can also use the run_now endpoint to run an existing job with a run_name parameter that specifies the name of the run. Users can use the runs/get endpoint to get the run result state of a job run, and use conditional logic to trigger another job run based on the result. Users can also use the runs/list endpoint to list all the runs of a job, and use the runs/cancel endpoint to cancel a job run if needed. References:

• Databricks REST API — Databricks Documentation, p. 1
• Jobs API — Databricks Documentation, p. 1-2
• Create and run Databricks Jobs | Databricks on AWS, Create and run jobs using the CLI, API, or notebooks, p. 1

MLflow provides a module called mlflow.pyfunc that can be used to deploy any Python model as a PySpark UDF. The mlflow.pyfunc.spark_udf function takes a Spark session and a model URI, and returns a Spark UDF that can be applied to a Spark DataFrame. The model URI can be a local path, a remote path, or a run ID with a relative path within the MLflow tracking server. The mlflow.pyfunc.spark_udf function can be used with any model that has a predict function, such as scikit-learn, xgboost, etc12

The other code blocks are incorrect because:

• A. The mlflow.pyfunc.spark_udf function requires a Spark session as the first argument, not a Spark DataFrame.
• B. It is possible to deploy a scikit-learn model on a Spark DataFrame using mlflow.pyfunc.spark_udf or other methods, such as broadcasting the model or using Fugue.
• C. There is no module called sklearn.spark_udf. To use a scikit-learn model as a Spark UDF, mlflow.pyfunc.spark_udf or other methods should be used.
• D. There is no method called spark.spark_udf. To create a Spark UDF, the spark.udf.register or pyspark.sql.functions.udf methods should be used.

References:

• mlflow.pyfunc — MLflow 2.9.1 documentation
• Deploy models for batch inference and prediction - Databricks

Containers are a type of software packaging that can assist in real-time deployments by packaging software with its own application, tools, and libraries. Containers are isolated and portable environments that run on a shared operating system, but have their ownfile system, dependencies, configuration, and resources. Containers can help developers and operators to build, test, and deploy software faster, easier, and more reliably, as they ensure that the software runs the same way in any environment, whether it is a developer’s laptop, a test server, or a production cluster. Containers can also help to achieve scalability, high availability, and load balancing, as they can be easily created, destroyed, replicated, and orchestrated using tools such as Kubernetes, Docker Swarm, or Amazon ECS. Containers can also integrate with other tools and services, such as MLflow, Databricks, AWS CodeDeploy, and Google Cloud Deployment Manager, to enable end-to-end machine learning pipelines with tracking, reproducibility, and governance1234 References:

• How to Package Your Application’s Infrastructure with Docker
• 21 open source DevOps tools (and what they do)
• Microservices [What is it & Top Benefits]
• 9 Best Software Deployment Tools for 2023 (Paid & Free)

A model signature is a specification of the input and output schema of a model, which defines the names, data types, and shapes of the model inputs and outputs. Logging a model signature with an MLflow model has several benefits, such as:

• The signature can be used to document what type of data is expected as input and output by the model, which can help users and developers understand how to use the model correctly.
• The signature can be used to enforce schema validation when serving models via REST API, which can help prevent errors and ensure data quality. For example, Azure Machine Learning enforces compliance with the model signature for MLflow models that contain a signature1.
• The signature can be used to enable automatic conversion of input data to the appropriate format for the model, such as pandas DataFrame, numpy array, or PyTorch tensor. This can simplify the data preprocessing and inference steps and reduce the code complexity.

References:

• MLflow Models Documentation, Model Signature and Input Example, p. 6-7
• mlflow.models.signature Documentation, p. 1-2
• Guidelines for deploying MLflow models - Azure Machine Learning, p. 1