CT-AI Sample Questions Answers

 Type of Testing for Stock-Price Prediction Models:Concept drift refers to the change in the statistical properties of the target variable over time. Severe weather storms causing massive losses in coffee production and affecting stock prices would require testing for concept drift to ensure that the model adapts to new patterns in data over time.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 7.6 Testing for Concept Drift, which explains the need to test for concept drift in models that might be affected by changing external factors​​.

Pairwise testing is used to handle the large number of combinations of parameters that can arise in complex systems like self-driving cars. The question asks which of the given options isleast likelyto be a reason for the explosion in the number of parameters.

Different Road Types (A): Self-driving cars must operate on various road types, such as highways, city streets, rural roads, etc. Each road type can have different characteristics, requiring the car's system to adapt and handle different scenarios. Thus, this is a significant factor contributing to the growth of parameters.

Different Weather Conditions (B): Weather conditions such as rain, snow, fog, and bright sunlight significantly affect the performance of self-driving cars. The car's sensors and algorithms must adapt to these varying conditions, which adds to the number of parameters that need to be considered.

ML Model Metrics to Evaluate Functional Performance (C): While evaluating machine learning (ML) model performance is crucial, it does not directly contribute to the explosion of parameter combinations in the same way that road types, weather conditions, and car features do. Metrics are used to measure and assess performance but are not themselves variable conditions that the system must handle.

Different Features like ADAS, Lane Change Assistance, etc. (D): Advanced Driver Assistance Systems (ADAS) and other features add complexity to self-driving cars. Each feature can have multiple settings and operational modes, contributing to the overall number of parameters.

Hence, theleast likelyreason for the incredible growth in the number of parameters isC. ML model metrics to evaluate the functional performance.

References:

ISTQB CT-AI Syllabus Section 9.2 on Pairwise Testing discusses the application of this technique to manage the combinations of different variables in AI-based systems, including those used in self-driving cars.

Sample Exam Questions document, Question #29 provides context for the explosion in parameter combinations in self-driving cars and highlights the use of pairwise testing as a method to manage this complexity.

AI-generated test cases are often created using machine learning (ML) models or heuristic algorithms. While these can be effective in generating large numbers of test cases quickly, they oftensuffer from the "test oracle problem."

Test Oracle Problem:A test oracle is the mechanism used to determine the expected output of a test case. AI-generated test cases oftenlack expected resultsbecause AI-based tools do not inherently understand what the correct output should be​.

Difficulty in Verification:Without expected results, verifying test cases becomes challenging. Testers mustrely on heuristics, anomaly detection, or significant failures, rather than traditional pass/fail conditions​.

A (Slow Execution Time):AI-generated tests are typically automated and designed for efficiency. They are not inherently slow and often executefasterthan manually written tests​.

B (Defect-Prone Due to Nuance Issues):While AI-generated tests may struggle with some complexities in requirements, they primarilylack expected results, rather than failing due to an inability to detect nuances​.

C (Complicated Debugging Due to Many Steps):AI-generated testsreducedebugging complexity by limiting the number of steps required to reproduce failures​.

ISTQB CT-AI Syllabus (Section 11.3: Using AI for Test Case Generation)

"AI-generated test cases often lack expected results, making it difficult to verify correctness without a test oracle."​.

"Verification often relies on detecting significant failures rather than having predefined expected results."​.

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Since AI-generated test cases frequentlylack expected results, verification becomes difficult, requiring testers tofocus on major failuresrather than precise pass/fail conditions. Thus, thecorrect answer is D.

 Challenges for Acquiring Test Data in ML Systems:Compliance needs, the changing nature of data over time, and sourcing data from public sources are significant challenges. Data being generated quickly is generally not a challenge; it can actually be beneficial as it provides more data for training and testing.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Data Preparation and Data Quality Issues​​​​.

When written requirements are given in text documents, the best way to generate test cases is by using Natural Language Processing (NLP). Here's why:

Natural Language Processing (NLP): NLP can analyze and understand human language. It can be used to process textual requirements to extract relevant information and generate test cases. This method is efficient in handling large volumes of textual data and identifying key elements necessary for testing.

Why Not Other Options:

Analyzing source code for generating test cases: This is more suitable for white-box testing where the code is available, but it doesn't apply to text-based requirements.

Machine learning on logs of execution: This approach is used for dynamic analysis based on system behavior during execution rather than static textual requirements.

GUI analysis by computer vision: This is used for testing graphical user interfaces and is not applicable to text-based requirements.

References:This aligns with the methodology discussed in the syllabus under the section on using AI for generating test cases from textual requirements​​.

Precision is a performance metric used to evaluate the accuracy of positive predictions in a classification model. It is defined by the formula:

Precision=TPTP+FP×100%\text{Precision} = \frac{TP}{TP + FP} \times 100\%Precision=TP+FPTP​×100%

Where:

TP (True Positives)= Number of correctly predicted positive cases

FP (False Positives)= Number of incorrectly predicted positive cases

The confusion matrix provided in the question would typically list these values. Based on ISTQB's guidelines for calculating precision, selecting the correct number of true positives and false positives from the given data should yield94.2%as the precision​.

Section 5.1 - Confusion Matrix and ML Functional Performance Metricsexplains the calculation ofprecisionusing the confusion matrix​.

Reference from ISTQB Certified Tester AI Testing Study Guide:

Probabilistic and non-deterministic AI-based systemsdo not always produce the same output for identical inputs. This makes traditional testing approaches ineffective. Instead, the best approach is torun tests multiple times and analyze results statistically.

Statistical Validity:Running tests multiple times ensures that observed results are statistically significant. Instead of relying on a single test run,analyzing multiple iterations helps determine trends, probabilities, and outliers​.

Expected Result Tolerance:AI-based systems may produce different results within an acceptable range. Defining acceptable tolerances (e.g., "result must be within 2% of the optimal value") improves test effectiveness​.

A (Run Several Times for the Same Correct Result):AI systems are ofteninherently non-deterministicand may not return the exact same result every time. Expecting identical outputs contradicts the nature of these systems​.

B & C (Decomposing Tests into Data Ingestion Tests):While data ingestion quality is important, it does notdirectlysolve the issue of probabilistic test results. Statistical analysis is the key approach​.

ISTQB CT-AI Syllabus (Section 8.4: Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems)

"For probabilistic systems, running a test multiple times may be necessary to obtain a statistically valid test result."​.

"Where a single definitive output is not possible, results should be analyzed statistically rather than relying on individual test cases."​.

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Sinceprobabilistic AI systems do not always return the same result, the best approach is torun multiple test iterations and validate results statistically. Hence, thecorrect answer is D.

Bias in an AI system occurs when the training data contains inherent prejudices that cause the model to make unfair predictions. Experience-based testing, particularlyExploratory Data Analysis (EDA), helps uncover these biases by analyzing patterns, distributions, and potential discriminatory factors in the training data​.

Option A:“Experience-based testing should be used to confirm that the training data set is operationally relevant. This can include applying exploratory data analysis (EDA) to check for bias within the training data set.”

This is the correct answer. EDA involves examining the dataset for bias, inconsistencies, or missing values, ensuring fairness in ML model predictions​.

Option B:“Back-to-back testing should be used to compare the model created using the training data set to another model created using the test data set. If the two models significantly differ, it will indicate there is bias in the original model.”

Back-to-back testing is used for regression testing and to compare versions of an AI system but is not primarily used to detect bias​.

Option C:“Acceptance testing should be used to make sure the algorithm is suitable for the customer. The team can re-work the acceptance criteria such that the algorithm is sure to correctly predict the remaining applicants that have been set aside for the validation data set ensuring no bias is present.”

Acceptance testing focuses on meeting predefined business requirements rather than detecting and mitigating bias​.

Option D:“A/B testing should be used to verify that the test data set does not detect any bias that might have been introduced by the original training data. If the two models significantly differ, it will indicate there is bias in the original model.”

A/B testing is used for evaluating variations of a model rather than for explicitly identifying bias​.

Bias Testing Methods:"AI-based systems should be tested for algorithmic bias, sample bias, and inappropriate bias. Experience-based testing and EDA are useful for detecting bias"​.

Exploratory Data Analysis (EDA):"EDA helps uncover potential bias in training data through visualization and statistical analysis"​.

Analysis of the Answer Options:ISTQB CT-AI Syllabus References:Thus,Option A is the best choice for detecting bias in the loan applicant model.

 A. This statement is incorrect. Current AI is true AI and there is no reason to believe that this fact will change over time.

AI is an evolving field, and the definition of what constitutes AI can change as technology advances.

 B. This statement is correct. In general, what is considered AI today may change over time.

The term AI is dynamic and has evolved over the years. What is considered AI today might be viewed as standard computing in the future. Historically, as technologies become mainstream, they often cease to be considered "AI".

 C. This statement is incorrect. What is considered AI today will continue to be AI even as technology evolves and changes.

This perspective does not account for the historical evolution of the definition of AI. As new technologies emerge, the boundaries of AI shift.

 D. This statement is correct. In general, today the term AI is utilized incorrectly.

While some may argue this, it is not a universal truth. The term AI encompasses a broad range of technologies and applications, and its usage is generally consistent with current technological capabilities.

To calculate the accuracy, recall, and specificity from the confusion matrix provided, we use the following formulas:

Confusion Matrix:

Actually Rotten: 45 (True Positive), 8 (False Positive)

Actually Fresh: 5 (False Negative), 42 (True Negative)

Accuracy:

Accuracy is the proportion of true results (both true positives and true negatives) in the total population.

Formula: Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN​

Calculation: Accuracy=45+4245+42+8+5=87100=0.87\text{Accuracy} = \frac{45 + 42}{45 + 42 + 8 + 5} = \frac{87}{100} = 0.87Accuracy=45+42+8+545+42​=10087​=0.87

Recall (Sensitivity):

Recall is the proportion of true positive results in the total actual positives.

Formula: Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}Recall=TP+FNTP​

Calculation: Recall=4545+5=4550=0.9\text{Recall} = \frac{45}{45 + 5} = \frac{45}{50} = 0.9Recall=45+545​=5045​=0.9

Specificity:

Specificity is the proportion of true negative results in the total actual negatives.

Formula: Specificity=TNTN+FP\text{Specificity} = \frac{TN}{TN + FP}Specificity=TN+FPTN​

Calculation: Specificity=4242+8=4250=0.84\text{Specificity} = \frac{42}{42 + 8} = \frac{42}{50} = 0.84Specificity=42+842​=5042​=0.84

Therefore, the correct combinations of accuracy, recall, and specificity are 0.87, 0.9, and 0.84 respectively.

References:

ISTQB CT-AI Syllabus, Section 5.1, Confusion Matrix, provides detailed formulas and explanations for calculating various metrics including accuracy, recall, and specificity.

"ML Functional Performance Metrics" (ISTQB CT-AI Syllabus, Section 5).

 Factors Affecting ML Functional Performance:The data pipeline, quality of the labeling, and biased data are all factors that significantly affect the performance of machine learning models. The number of classes, while relevant for the model structure, is not a direct factor affecting the performance metrics such as accuracy or bias.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Data Quality and its Effect on the ML Model and ML Functional Performance Metrics​​​​.

Object recognition issues in test automation often arise whendevelopers frequently change object identifiersin the GUI. AI can enhance the stability of GUI automation by:

Using multiple criteria for object identification

AI cantrack UI elements using multiple attributessuch asXPath, label, ID, class, and screen coordinatesrather than relying on a single identifier that may change over time.

This approach makes the automationless brittle and more adaptive to changes in the UI​.

Why other options are incorrect?

B (Ignore unrecognizable objects to avoid failures): Ignoring objects instead of identifying them properly wouldlead to incomplete or incorrect test execution.

C (Dynamically name objects and alter source code): AI-based testing tools donot modify application source code; they work byadjusting the recognition strategy.

D (Anticipate developer changes and pre-alter automation code): While AI can adapt,it does not predict future changes to the GUI, making this option unrealistic​.

Thus,Option A is the best answer, as AI tools enhance object recognitionby dynamically selecting the most stable and persistent identification methods, improving test automation reliability.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 11.6.1 (Using AI to Test Through the Graphical User Interface (GUI))

ISTQB CT-AI Syllabus v1.0, Section 11.6.2 (Using AI to Test the GUI)​.

AI systems, particularly machine learning models, need to exhibit adaptability and flexibility to handle slight variations in input data without requiring retraining. The ISTQB CT-AI syllabus outlines adaptability as a crucial feature of AI systems, especially when the system is exposed to variations in its operational environment​.

Option A:“It has been trained to recognize cats and is given an image of a dog.”

This scenario introduces an entirely new class (dogs), which is outside the AI system’s expected scope. If the AI was only trained to recognize cats, it would not be expected to recognize dogs correctly without retraining. This does not demonstrate adaptability as expected from an AI system.

Option B:“It has been trained to recognize human faces at a particular resolution and it is given a human face image captured with a higher resolution.”

This is an example of an AI system encountering a variation of its training data rather than entirely new data. Most AI-based image processing models can adapt to different resolutions by applying downsampling or other pre-processing techniques. Since the data remains within the domain of human faces, the model should be able to process the higher-resolution image without significant issues.

Option C:“It has been trained to analyze mathematical models and is given a set of landscape pictures to classify.”

This represents a complete shift in the data type from structured numerical data to unstructured image data. The AI system is unlikely to adapt effectively, as it has not been trained on image classification tasks.

Option D:“It has been trained to analyze customer buying trend data and is given information on supplier cost data.”

This introduces a significant domain shift. Customer buying trends focus on consumer behavior, while supplier cost data relates to pricing structures and logistics. The AI system would likely require retraining to process the new data meaningfully.

Adaptability Requirements:The syllabus discusses that AI-based systems must be able to adapt to changes in their operational environment and constraints, including minor variations in input quality (such as resolution changes)​.

Autonomous Learning & Evolution:AI systems are expected to improve and handle evolving inputs based on prior experience​.

Challenges in Testing Self-Learning Systems:AI systems should be tested to ensure they function correctly when encountering new but related data, such as different resolutions of the same object​.

Analysis of the Answer Options:ISTQB CT-AI Syllabus References:Thus,option Bis the best choice as it aligns with the adaptability characteristics expected from AI-based systems.

Detecting biases in the ML pipeline involves various tests to ensure fairness and accuracy throughout the ML process.

Testing the distribution shift in the training data for inappropriate bias (A): This involves checking if there is any shift in the data distribution that could lead to bias in the model. It is an important test but not the most direct method for detecting biases.

Test the model during model evaluation for data bias (B): This is a critical stage where the model is evaluated to detect any biases in the data it was trained on. It directly addresses potential data biases in the model.

Testing the data pipeline for any sources for algorithmic bias (C): This test is crucial as it helps identify biases that may originate from the data processing and transformation stages within the pipeline. Detecting sources of algorithmic bias ensures that the model does not inherit biases from these processes.

Check the input test data for potential sample bias (D): While this is an important step, it focuses more on the input data and less on the overall data pipeline.

Hence, the most likely useful test to help detect different kinds of biases in the ML pipeline isB. Test the model during model evaluation for data bias.

References:

ISTQB CT-AI Syllabus Section 8.3 on Testing for Algorithmic, Sample, and Inappropriate Bias discusses various tests that can be performed to detect biases at different stages of the ML pipeline.

Sample Exam Questions document, Question #32 highlights the importance of evaluating the model for biases.

Clustering is a type of machine learning technique used to group similar data points into clusters. It is a key concept in unsupervised learning, where the algorithm tries to find patterns or groupings in data without prior knowledge of output classes. Let's analyze each option:

A. Clustering is classification of a continuous quantity.

This is incorrect. Classification typically involves discrete categories, whereas clustering involves grouping similar data points. Classification of continuous quantities is generally referred to as regression.

B. Clustering is supervised learning.

This is incorrect. Clustering is an unsupervised learning technique because it does not rely on labeled data.

C. Clustering is done without prior knowledge of output classes.

This is correct. In clustering, the algorithm groups data points into clusters without any prior knowledge of the classes. It discovers the inherent structure in the data.

D. Clustering requires you to know the classes.

This is incorrect. Clustering does not require prior knowledge of classes. Instead, it aims to identify and form the classes or groups based on the data itself.

Therefore, the correct answer isCbecause clustering is an unsupervised learning technique done without prior knowledge of output classes​​.

Whentesting an AI-based chat assistantfor e-commerce customers, thelack of sufficient specificationsmakes it difficult to use structured test techniques. TheISTQB CT-AI Syllabusrecommendsexploratory testingin such cases:

Why Exploratory Testing?

Exploratory testing is usefulwhen specifications are incomplete or unclear.

AI-based systems, particularly those usingnatural language processing (NLP),may not behave deterministically, making scripted test cases ineffective.

Thetester interacts dynamicallywith the system, identifying unexpected behaviorsnot documented in the specification​.

Analysis of Answer Choices:

A (Exploratory testing)→Correct, as it is the best approach when specifications are incomplete.

B (Static analysis)→ Incorrect, as static analysis checks code without execution, which isnot helpfulfor AI chatbots.

C (Equivalence partitioning)→ Incorrect, asthis technique requires well-defined inputs and outputs, which are missing due toinsufficient specifications.

D (State transition testing)→ Incorrect, as state-based testingrequires knowledge of valid and invalid transitions, which is difficult with a chatbot lacking a clear specification​.

Thus,Option A is the correct answer, asexploratory testing is the best approach when dealing with insufficient specifications in AI-based systems.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 7.7 (Selecting a Test Approach for an ML System)

ISTQB CT-AI Syllabus v1.0, Section 9.6 (Experience-Based Testing of AI-Based Systems)​.

 Coverage for Neuron Activation Values:Sign change coverage is used to ensure that test cases cause each neuron to achieve both positive and negative activation values. This type of coverage ensures that the neurons are thoroughly tested under different activation states.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 6.2 Coverage Measures for Neural Networks, which details different types of coverage measures, including sign change coverage​​.

 Technology Most Typically Used to Implement AI:Genetic algorithms are a well-known technique used in AI. They are inspired by the process of natural selection and are used to find approximate solutions to optimization and search problems. Unlike search engines, procedural programming, or case control structures, genetic algorithms are specifically designed for evolving solutions and are commonly employed in AI implementations.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 1.4 AI Technologies, which identifies different technologies used to implement AI​​.

AI-based systems can sometimes behave likeblack boxes, where the internal decision-making process is unclear.Transparencyrefers to theability to inspect and understand the training data, algorithms, and decision-making processof the AI system.

Transparency ensures that testers and stakeholders can review how an AI system was trained.

Access totraining datais a key factor in transparency because it allows testers toanalyze biases, completeness, and representativenessof the dataset.

Transparency is an essential characteristic of explainable AI (XAI).

Having access to training data means that testers can investigate how data influences AI behavior.

Regulatory and ethical AI guidelines emphasize transparency.

Many AI ethics frameworks, such asGDPR and Trustworthy AI guidelines, recommend transparency to ensurefair and explainable AI decision-making.

(A) Autonomy❌

Autonomy refers to an AI system’s ability to make decisions independentlywithout human intervention. However,having access to training data does not relate to autonomy, which is more about self-learning and decision-making without human control.

(B) Explorability❌

Explorability refers to the ability to test AI systems interactivelyto understand their behavior, but it does not directly relate to accessing training data.

(D) Accessibility❌

Accessibility refers to the ease with which people can use the system, not the ability to inspect the training data.

Transparency is the ease with which the training data and algorithm used to generate a model can be understood."Transparency: This is considered to be the ease with which the algorithm and training data used to generate the model can be determined."​

Why is Option C Correct?Why Other Options are Incorrect?References from ISTQB Certified Tester AI Testing Study GuideThus,option C is the correct answer, astransparency involves access to training data, allowing testers to understand AI decision-making processes.

In AI-based systems, particularly those where theinternal structure has been generated by another software system, the complexity often makes it difficult for human testers to analyze the inner workings. As per the ISTQB Certified Tester AI Testing (CT-AI) Syllabus:

Black-box testingis particularly useful when dealing with AI systems that have been generated by another system because:

It allows testingwithout requiring knowledge of the internal logic.

The AI model may be too complex for human testers to comprehend, making white-box testing ineffective.

Black-box testing evaluates theinputs and outputs, ensuring functional correctnesswithout needing insight into how the system reaches a decision​.

Why other options are incorrect?

A (Test automation and black-box testing): While automation is possible,black-box testing is not primarily about automationbut aboutabstracting the internal complexity.

B (Understanding the logic of the software): This contradicts the premise of black-box testing, which is designed totest functionality without needing to understandthe inner workings.

C (Checking transparency of the algorithm):Black-box testing does not check algorithm transparency—that would requirewhite-box testing or explainability techniques​.

Thus, the best choice isOption D, as black-box testingremoves the need to analyze the internal structure of AI systems, making it the most appropriate testing method in this case.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 8.5 (Challenges Testing Complex AI-Based Systems)

ISTQB CT-AI Syllabus v1.0, Section 8.6 (Testing the Transparency, Interpretability, and Explainability of AI-Based Systems)​

The question asks which test is least likely to be performed during the ML model testing phase. Let's consider each option:

Testing the accuracy of the classification model (A): Accuracy testing is a fundamental part of the ML model testing phase. It ensures that the model correctly classifies the data as intended and meets the required performance metrics.

Testing the API of the service powered by the ML model (B): Testing the API is crucial, especially if the ML model is deployed as part of a service. This ensures that the service integrates well with other systems and that the API performs as expected.

Testing the speed of the training of the model (C): This is least likely to be part of the ML model testing phase. The speed of training is more relevant during the development phase when optimizing and tuning the model. During testing, the focus is more on the model's performance and behavior rather than how quickly it was trained.

Testing the speed of the prediction by the model (D): Testing the speed of prediction is important to ensure that the model meets performance requirements in a production environment, especially for real-time applications.

References:

ISTQB CT-AI Syllabus Section 3.2 on ML Workflow and Section 5 on ML Functional Performance Metrics discuss the focus of testing during the model testing phase, which includes accuracy and prediction speed but not the training speed.