ISTQB CT-AI Dumps Questions Answers

 Challenges for Acquiring Test Data in ML Systems:Compliance needs, the changing nature of data over time, and sourcing data from public sources are significant challenges. Data being generated quickly is generally not a challenge; it can actually be beneficial as it provides more data for training and testing.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Data Preparation and Data Quality Issues​​​​.

The problem described in the question is a classic case ofconcept drift. Concept drift occurs when the relationship between input variables and the output variable changes over time, leading to a decline in model accuracy​.

In this scenario, theaverage passenger and baggage weightsused in the model changed due to seasonal variations, but the model was not updated accordingly. This resulted in inaccurate predictions for fuel requirements in the winter season. This is an example ofseasonal drift, where model behavior changes periodically due to recurring trends (e.g., higher luggage weights in winter compared to summer).

To prevent such problems:

Themodel should be regularly testedfor concept drift against agreed ML functional performance criteria​.

Exploratory Data Analysis (EDA)should be performed periodically to detect gradual changes in input distributions.

Retraining of the modelwith updated training data should be done to maintain accuracy.

If drift is detected, mitigation techniques such asincremental learning, retraining with new data, or adjusting model parametersshould be employed.

Option B (Easing the performance standard instead of addressing drift): Lowering the performance standard is not a solution; it only masks the problem without fixing it. Instead, regular testing and retraining should be used to handle drift properly​.

Option C (Corruption and reloading the model): Model corruption is unrelated to this issue. Corruption refers to accidental or malicious damage to the model or data, whereas this case is due to a changing data environment​.

Option D (Lack of transparency): Transparency refers to how understandable the model’s decisions are, but the problem here is a change in data distributions, making drift the primary concern​.

ISTQB CT-AI Syllabus (Section 7.6: Testing for Concept Drift)

"The operational environment can change over time without the trained model changing correspondingly. This phenomenon is known as concept drift and typically causes the outputs of the model to become increasingly less accurate and less useful."​

"Systems that may be prone to concept drift should be regularly tested against their agreed ML functional performance criteria to ensure that any occurrences of concept drift are detected soon enough for the problem to be mitigated."​

ISTQB CT-AI Syllabus (Section 7.7: Selecting a Test Approach for an ML System)

"If concept drift is detected, it may be mitigated by retraining the system with up-to-date training data followed by confirmation testing, regression testing, and possibly A/B testing where the updated system must outperform the original system."​

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Since the question describes a situation whereseasonal variations affected input data distributions, the correct answer isA: The model suffers from drift and therefore should be regularly tested to ensure that any occurrences of drift are detected soon enough for the problem to be mitigated.

Detecting biases in the ML pipeline involves various tests to ensure fairness and accuracy throughout the ML process.

Testing the distribution shift in the training data for inappropriate bias (A): This involves checking if there is any shift in the data distribution that could lead to bias in the model. It is an important test but not the most direct method for detecting biases.

Test the model during model evaluation for data bias (B): This is a critical stage where the model is evaluated to detect any biases in the data it was trained on. It directly addresses potential data biases in the model.

Testing the data pipeline for any sources for algorithmic bias (C): This test is crucial as it helps identify biases that may originate from the data processing and transformation stages within the pipeline. Detecting sources of algorithmic bias ensures that the model does not inherit biases from these processes.

Check the input test data for potential sample bias (D): While this is an important step, it focuses more on the input data and less on the overall data pipeline.

Hence, the most likely useful test to help detect different kinds of biases in the ML pipeline isB. Test the model during model evaluation for data bias.

References:

ISTQB CT-AI Syllabus Section 8.3 on Testing for Algorithmic, Sample, and Inappropriate Bias discusses various tests that can be performed to detect biases at different stages of the ML pipeline.

Sample Exam Questions document, Question #32 highlights the importance of evaluating the model for biases.