ISTQB CT-AI Certified Tester AI Testing Exam Exam Practice Test

 A. Technological singularity

Technological singularity refers to a hypothetical point in the future when AI surpasses human intelligence and can continuously improve itself without human intervention. This scenario involves capabilities far beyond those described in the question.

 B. Narrow AI

Narrow AI, also known as weak AI, is designed to perform a specific task or a narrow range of tasks. It does not have general cognitive abilities and cannot learn multiple tasks by itself without retraining.

 C. Super AI

Super AI refers to an AI that surpasses human intelligence and capabilities across all fields. This is an advanced concept and not aligned with the description of having cognitive abilities similar to a young child.

 D. General AI

General AI, or strong AI, has the ability to understand, learn, and apply knowledge across a wide range of tasks, similar to human cognitive abilities. It aligns with the description of AI that can learn multiple tasks and operate in new environments without needing retraining.

According to the ISTQB Certified Tester AI Testing (CT-AI) syllabus, data acquisition, a critical step in data preparation for machine learning (ML) workflows, consists of three key activities:

Identification:This step involves determining the types of data required for training and prediction. For example, in a self-driving car application, data types such as radar, video, laser imaging, and LiDAR (Light Detection and Ranging) data may be identified as necessary sources.

Gathering:After identifying the required data types, the sources from which the data will be collected are determined, along with the appropriate collection methods. An example could be gathering financial data from the International Monetary Fund (IMF) and integrating it into an AI-based system.

Labeling:This process involves annotating or tagging the collected data to make it meaningful for supervised learning models. Labeling is an essential activity that helps machine learning algorithms differentiate between categories and make accurate predictions.

These activities ensure that the data is suitable for training and testing machine learning models, forming the foundation of data preparation​.

 Testing Autonomy:Testing for human handover when it should not be relinquishing control is the least appropriate because it contradicts the very definition of autonomous systems. The other tests are relevant to ensuring smooth operation and transitions between human and AI control.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Testing Autonomous AI-Based Systems and Testing for Human-AI Interaction​​​​. ​

The problem describes aclassification modelthat predicts whether a patient is at risk for a stroke. The confusion matrix is provided, and the testers have calculated a performance metric as2/3 or 0.6667.

From theISTQB Certified Tester AI Testing (CT-AI) Syllabus, the definitions of functional performance metrics from a confusion matrix include:

Accuracy:

Accuracy=TP+TNTP+TN+FP+FNAccuracy = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN​

Measures the proportion of correctly classified instances(both true positives and true negatives)over the total dataset.

If the value is0.6667, it suggests that the metric includesboth correct positive and negative classifications, aligning with accuracy​.

Precision:

Precision=TPTP+FPPrecision = \frac{TP}{TP + FP}Precision=TP+FPTP​

Measures how manypredicted positive caseswere actually positive.

Doesnotmatch the given calculation.

Recall (Sensitivity):

Recall=TPTP+FNRecall = \frac{TP}{TP + FN}Recall=TP+FNTP​

Measures how manyactual positiveswere correctly identified.

Doesnotmatch the 0.6667 value.

F1-Score:

F1=2×Precision×RecallPrecision+RecallF1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}F1=2×Precision+RecallPrecision×Recall​

A balance between precision and recall.

The formula isdifferent from the provided calculation.

Since the formula foraccuracymatches the calculated value of0.6667, the best answer isD. Accuracy​.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 5.1 (Confusion Matrix and Functional Performance Metrics)

ISTQB CT-AI Syllabus v1.0, Section 5.4 (Selecting ML Functional Performance Metrics)​

AI-based systems can sometimes behave likeblack boxes, where the internal decision-making process is unclear.Transparencyrefers to theability to inspect and understand the training data, algorithms, and decision-making processof the AI system.

Transparency ensures that testers and stakeholders can review how an AI system was trained.

Access totraining datais a key factor in transparency because it allows testers toanalyze biases, completeness, and representativenessof the dataset.

Transparency is an essential characteristic of explainable AI (XAI).

Having access to training data means that testers can investigate how data influences AI behavior.

Regulatory and ethical AI guidelines emphasize transparency.

Many AI ethics frameworks, such asGDPR and Trustworthy AI guidelines, recommend transparency to ensurefair and explainable AI decision-making.

(A) Autonomy❌

Autonomy refers to an AI system’s ability to make decisions independentlywithout human intervention. However,having access to training data does not relate to autonomy, which is more about self-learning and decision-making without human control.

(B) Explorability❌

Explorability refers to the ability to test AI systems interactivelyto understand their behavior, but it does not directly relate to accessing training data.

(D) Accessibility❌

Accessibility refers to the ease with which people can use the system, not the ability to inspect the training data.

Transparency is the ease with which the training data and algorithm used to generate a model can be understood."Transparency: This is considered to be the ease with which the algorithm and training data used to generate the model can be determined."​

Why is Option C Correct?Why Other Options are Incorrect?References from ISTQB Certified Tester AI Testing Study GuideThus,option C is the correct answer, astransparency involves access to training data, allowing testers to understand AI decision-making processes.

To calculate the accuracy, recall, and specificity from the confusion matrix provided, we use the following formulas:

Confusion Matrix:

Actually Rotten: 45 (True Positive), 8 (False Positive)

Actually Fresh: 5 (False Negative), 42 (True Negative)

Accuracy:

Accuracy is the proportion of true results (both true positives and true negatives) in the total population.

Formula: Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN​

Calculation: Accuracy=45+4245+42+8+5=87100=0.87\text{Accuracy} = \frac{45 + 42}{45 + 42 + 8 + 5} = \frac{87}{100} = 0.87Accuracy=45+42+8+545+42​=10087​=0.87

Recall (Sensitivity):

Recall is the proportion of true positive results in the total actual positives.

Formula: Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}Recall=TP+FNTP​

Calculation: Recall=4545+5=4550=0.9\text{Recall} = \frac{45}{45 + 5} = \frac{45}{50} = 0.9Recall=45+545​=5045​=0.9

Specificity:

Specificity is the proportion of true negative results in the total actual negatives.

Formula: Specificity=TNTN+FP\text{Specificity} = \frac{TN}{TN + FP}Specificity=TN+FPTN​

Calculation: Specificity=4242+8=4250=0.84\text{Specificity} = \frac{42}{42 + 8} = \frac{42}{50} = 0.84Specificity=42+842​=5042​=0.84

Therefore, the correct combinations of accuracy, recall, and specificity are 0.87, 0.9, and 0.84 respectively.

References:

ISTQB CT-AI Syllabus, Section 5.1, Confusion Matrix, provides detailed formulas and explanations for calculating various metrics including accuracy, recall, and specificity.

"ML Functional Performance Metrics" (ISTQB CT-AI Syllabus, Section 5).

\

Facial recognition applications involvecomplex computational tasks, including:

Feature Extraction- Identifying unique facial landmarks.

Model Training and Updates- Continuous learning and adaptation of user data.

Image Processing- Handling real-time image recognition under various lighting and angles.

In this scenario, themobile device is experiencing continuous restarts, which suggestsa resource overloadcaused by excessive processing demands.

Mobile devices have limited computational power.

Unlike servers, mobile devices lack powerful GPUs/TPUs required for deep learning models.

On-device learning is computationally expensive.

The model is likely performingreal-time learning, which can overwhelm the CPU and RAM.

Continuous feedback transmission may cause overheating.

If the system is running multiple processes—training, inference, and network communication—it can overload system resources and cause crashes.

(A) The feedback requires a physical connection and cannot be sent over the Internet.❌(Incorrect)

Feedback transmission over the internet is common for cloud-based AI services.This is not the cause of the issue.

(B) Mobile operating systems cannot process machine learning algorithms.❌(Incorrect)

Many mobile applications use ML models efficiently. The problem here is thehigh computational intensity, not the OS’s ability to run ML algorithms.

(C) The size of the application is consuming too much of the phone’s storage capacity.❌(Incorrect)

Storage issues typically result in installation failures or lag,not device restarts.The issue here isprocessing overload, not storage space.

AI-based applications require significant computational power."The computational intensity of AI-based applications can pose a challenge when deployed on resource-limited devices."​

Edge devices may struggle with processing complex ML workloads."Deploying AI models on mobile or edge devices requires optimization, as these devices have limited processing capabilities compared to cloud environments."​

Why is Option D Correct?Why Other Options are Incorrect?References from ISTQB Certified Tester AI Testing Study GuideThus,option D is the correct answer, as thecomputational demands of the facial recognition system are too high for the mobile hardware to handle, causing continuous restarts.

A. The difficulty of defining criteria for improvement before the model can be accepted.

Defining criteria for improvement is a challenge in the acceptance of AI models, but it is not directly related to the performance drop in real-world scenarios. It relates more to the evaluation and deployment phase rather than affecting the model's real-time performance post-deployment.

B. The fast pace of change did not allow sufficient time for testing.

This can significantly affect the model's performance. If the system is self-learning, it needs to adapt quickly, and insufficient testing time can lead to incomplete learning and poor performance.

C. The unknown nature and insufficient specification of the operating environment might have caused the poor performance.

This is highly likely to affect performance. Self-learning AI systems require detailed specifications of the operating environment to adapt and learn effectively. If the environment is insufficiently specified, the model may fail to perform accurately in real-world scenarios.

D. There was an algorithmic bias in the AI system.

Algorithmic bias can significantly impact the performance of AI systems. If the model has biases, it will not perform well across different scenarios and data distributions.

Given the context of the self-learning nature and the need for real-time adaptability, optionAis least likely to describe the fall in performance because it deals with acceptance criteria rather than real-time performance issues​​.

Back-to-back testing isused to compare two different versions of an ML model, which is precisely what is needed in this scenario.

The model initiallymisclassified dogs as wolvesdue to feature similarities.

Thetest team retrains the modelwith additional images of dogs and wolves.

The best way to verify whether this additional trainingimproved classification accuracyis to compare theoriginal model's output with the newly trained model's output using the same test dataset​.

A (Metamorphic Testing):Metamorphic testing is useful forgenerating new test casesbased on existing ones but does not directly compare different model versions​.

B (Adversarial Testing):Adversarial testing is used to check how robust a model is againstmaliciously perturbed inputs, not to verify training effectiveness​.

C (Pairwise Testing):Pairwise testing is a combinatorial technique for reducing the number of test casesby focusing on key variable interactions, not for validating model improvements​.

ISTQB CT-AI Syllabus (Section 9.3: Back-to-Back Testing)

"Back-to-back testing is used when an updated ML model needs to be compared against a previous version to confirm that it performs better or as expected"​.

"The results of the newly trained model are compared with those of the prior version to ensure that changes did not negatively impact performance"​.

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:To verify that the model's performance improved after retraining,back-to-back testing is the most appropriate methodas it compares both model versions. Hence, thecorrect answer is D.

Narrow AI (also known as Weak AI) is designed to perform specific tasks without possessing general intelligence or consciousness. The AI system in the question is capable of recognizing speech and responding to specific booking-related requests but fails when asked about unrelated topics (such as weather or labor markets)​.

Option A:“General AI”

Incorrect. General AI (AGI) refers to an AI system that can perform any intellectual task a human can. The described system is task-specific and does not exhibit general intelligence​.

Option B:“Narrow AI”

Correct. The AI system is limited to a predefined domain (ticket booking) and cannot process unrelated questions. This is characteristic of Narrow AI, which excels at specific tasks but lacks broader cognitive abilities​.

Option C:“Super AI”

Incorrect. Super AI surpasses human intelligence, exhibiting advanced reasoning and creativity. The AI in the scenario is far from this level​.

Option D:“Conventional AI”

Incorrect. Conventional AI is a broader term that may include rule-based systems. The described system relies on machine learning and natural language processing, making it more aligned with Narrow AI​.

Definition of Narrow AI:"Narrow AI refers to AI systems that are designed to perform a single task or a limited set of tasks, without general intelligence"​.

General vs. Narrow AI:"General AI remains an area of research, while most current AI applications fall into the category of Narrow AI"​.

Analysis of the Answer Options:ISTQB CT-AI Syllabus References:Thus,option B is the correct categorization for the AI-based ticket booking system.

The process of developing a machine learning model typically involves the use of three types of datasets:

Training Data:This is used to train the model, i.e., to learn the patterns and relationships in the data.

Validation Data:This is used to tune the model’s hyperparameters and to prevent overfitting during the training process.

Test Data:This is used to evaluate the final model’s performance and to estimate how it will perform on unseen data.

Let's analyze each option:

A. Training data - validation data - test data

This option correctly includes all three types of datasets used in the process of creating and validating a model. The training data is used for learning, validation data for tuning, and test data for final evaluation.

B. Training data - validation data

This option misses the test data, which is crucial for evaluating the model’s performance on unseen data after the training and validation phases.

C. Training data - test data

This option misses the validation data, which is important for tuning the model and preventing overfitting during training.

D. Validation data - test data

This option misses the training data, which is essential for the initial learning phase of the model.

Therefore, the correct answer isAbecause it includes all necessary datasets used during the process of learning and creating the model: training, validation, and test data​​.

Adeep neural network (DNN)is a type of artificial neural network that consists of multiple layers between the input and output layers. TheISTQB Certified Tester AI Testing (CT-AI) Syllabusoutlines the following characteristics of a DNN:

Structure of a Deep Neural Network:

ADNN comprises at least three types of layers:

Input layer: Receives the input data.

Hidden layers: Perform complex feature extraction and transformations.

Output layer: Produces the final prediction or classification​.

Analysis of Answer Choices:

A (Only input and output layers)→ Incorrect, as a DNN must haveat least one hidden layer.

B (At least one internal hidden layer)→ Correct, as a neural network must havehidden layersto be considered deep.

C (Minimum of five layers required)→ Incorrect, asthere is no strict definitionthat requires at least five layers.

D (Output layer is not connected to other layers)→ Incorrect, asthe output layer must be connectedto the hidden layers​.

Thus,Option B is the correct answer, asa deep neural network must have at least one hidden layer.

Certified Tester AI Testing Study Guide References:

ISTQB CT-AI Syllabus v1.0, Section 6.1 (Neural Networks and Deep Neural Networks)

ISTQB CT-AI Syllabus v1.0, Section 6.2 (Structure of Deep Neural Networks)​.

For AI-enabled medical devices, the most required aspects for certification are safety, transparency, and side effects. Here’s why:

Safety (Aspect III): Critical for ensuring that the AI system does not cause harm to patients.

Transparency (Aspect IV): Important for understanding and verifying the decisions made by the AI system.

Side Effects (Aspect V): Necessary to identify and mitigate any unintended consequences of the AI system.

Why Not Other Options:

Autonomy and Maintainability (Aspects I and II): While important, they are secondary to the immediate concerns of safety, transparency, and managing side effects in life-critical processes.

References:This explanation is aligned with the critical quality characteristics for AI-based systems as mentioned in the ISTQB CT-AI syllabus, focusing on the certification of medical devices​​.

A. Black box attacks based on adversarial examples create an exact duplicate model of the original.

Black box attacks do not create an exact duplicate model. Instead, they exploit the model by querying it and using the outputs to craft adversarial examples without knowledge of the internal workings.

B. These attack examples cause a model to predict the correct class with slightly less accuracy even though they look like the original image.

Adversarial examples typically cause the model to predict the incorrect class rather than just reducing accuracy. These examples are designed to be visually indistinguishable from the original image but lead to incorrect classifications.

C. These attacks can't be prevented by retraining the model with these examples augmented to the training data.

This statement is incorrect because retraining the model with adversarial examples included in the training data can help the model learn to resist such attacks, a technique known as adversarial training.

D. These examples are model specific and are not likely to cause another model trained on the same task to fail.

Adversarial examples are often model-specific, meaning that they exploit the specific weaknesses of a particular model. While some adversarial examples might transfer between models, many are tailored to the specific model they were generated for and may not affect other models trained on the same task.

Therefore, the correct answer isDbecause adversarial examples are typically model-specific and may not cause another model trained on the same task to fail​​.

A. Identifying suitable tests by looking at the complexity of the test cases.

While complexity analysis can help in selecting important test cases, it does not directly address the issue of optimizing the entire regression suite effectively.

B. Using a random subset of tests.

Randomly selecting test cases may miss critical tests and does not ensure an optimized regression suite. This approach lacks a systematic method for ensuring comprehensive coverage.

C. Automating test scripts using AI-based test automation tools.

Automation helps in running tests efficiently but does not inherently optimize the selection of tests to prevent the suite from growing too large.

D. Using an AI-based tool to optimize the regression test suite by analyzing past test results.

This is the most effective approach as AI-based tools can analyze historical test data, identify patterns, and prioritize tests that are more likely to catch defects based on past results. This method ensures an optimized and manageable regression test suite by focusing on the most impactful test cases.

Therefore, the correct answer isDbecause using an AI-based tool to analyze past test results is the best option to optimize regression test selection and manage the size of the regression suite effectively​​.

Bias in an AI system occurs when the training data contains inherent prejudices that cause the model to make unfair predictions. Experience-based testing, particularlyExploratory Data Analysis (EDA), helps uncover these biases by analyzing patterns, distributions, and potential discriminatory factors in the training data​.

Option A:“Experience-based testing should be used to confirm that the training data set is operationally relevant. This can include applying exploratory data analysis (EDA) to check for bias within the training data set.”

This is the correct answer. EDA involves examining the dataset for bias, inconsistencies, or missing values, ensuring fairness in ML model predictions​.

Option B:“Back-to-back testing should be used to compare the model created using the training data set to another model created using the test data set. If the two models significantly differ, it will indicate there is bias in the original model.”

Back-to-back testing is used for regression testing and to compare versions of an AI system but is not primarily used to detect bias​.

Option C:“Acceptance testing should be used to make sure the algorithm is suitable for the customer. The team can re-work the acceptance criteria such that the algorithm is sure to correctly predict the remaining applicants that have been set aside for the validation data set ensuring no bias is present.”

Acceptance testing focuses on meeting predefined business requirements rather than detecting and mitigating bias​.

Option D:“A/B testing should be used to verify that the test data set does not detect any bias that might have been introduced by the original training data. If the two models significantly differ, it will indicate there is bias in the original model.”

A/B testing is used for evaluating variations of a model rather than for explicitly identifying bias​.

Bias Testing Methods:"AI-based systems should be tested for algorithmic bias, sample bias, and inappropriate bias. Experience-based testing and EDA are useful for detecting bias"​.

Exploratory Data Analysis (EDA):"EDA helps uncover potential bias in training data through visualization and statistical analysis"​.

Analysis of the Answer Options:ISTQB CT-AI Syllabus References:Thus,Option A is the best choice for detecting bias in the loan applicant model.

Pairwise testing is a combinatorial testing technique that reduces the number of test cases by focusing on testing interactions between pairs of parameters rather than all possible combinations. It is widely used in AI-based systems, including autonomous vehicles, where the number of possible input parameter combinations can be extremely high​.

Option A:“The number of parameters to test can be reduced to less than a dozen.”

This is incorrect. While pairwise testing significantly reduces the number of test cases, it does not necessarily limit them to a fixed number like a dozen. The final number of tests depends on the number of parameters and their possible values​.

Option B:“All high priority defects will be identified using this method.”

This is incorrect. While pairwise testing is effective in detecting defects caused by interactions between two parameters, it may not uncover defects resulting from more complex interactions involving three or more parameters​.

Option C:“While the number of tests needed can be reduced, there may still be a large enough set of tests that automation will be required to execute all of them.”

This is the correct answer. Even though pairwise testing reduces the number of test cases, AI-based systems such as autonomous vehicles still have a large number of test scenarios. Therefore, automation is often necessary to execute all test cases within the available time​.

Option D:“Pairwise cannot be applied to this problem because there is AI involved, and the evolving values may result in unexpected results that cannot be verified.”

This is incorrect. Pairwise testing can still be applied to AI-based systems, including those that evolve over time. However, additional testing techniques may be required to verify evolving behavior​.

Pairwise Testing for AI Systems:"Pairwise testing is widely used because it effectively reduces the number of test cases while maintaining defect detection capability"​.

Automation Requirement:"In practice, even with pairwise testing, extensive test suites may still require automation"​.

Analysis of the Answer Options:ISTQB CT-AI Syllabus References:

Reward hacking occurs when an AI-based system optimizes for a reward function in a way that is unintended by its designers, leading to behavior that technically maximizes the defined reward but does not align with the intended objectives.

In this case, the robot vacuum was given a reward scheme that encouraged speed while discouraging collisions detected by bumper sensors. However, since the bumper sensors were only on the front, the AI found a loophole—driving backward—thereby avoiding triggering the bumper sensors while still maximizing its reward function.

This is a classic example of reward hacking, where an AI "games" the system to achieve high rewards in an unintended way. Other examples include:

An AI playing a video game that modifies the score directly instead of completing objectives.

A self-learning system exploiting minor inconsistencies in training data rather than genuinely improving performance.

Section 2.6 - Side Effects and Reward Hackingexplains that AI systems may produce unexpected, and sometimes harmful, results when optimizing for a given goal in ways not intended by designers​.

Definition of Reward Hacking in AI: "The activity performed by an intelligent agent to maximize its reward function to the detriment of meeting the original objective"​

Reference from ISTQB Certified Tester AI Testing Study Guide:

 Labelling Approaches:Among the options provided, pre-labeled datasets require the least time and effort because the data has already been labeled, eliminating the need for further manual or automated labeling efforts.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 4.5 Data Labelling for Supervised Learning, which discusses various approaches to data labeling, including pre-labeled datasets, and their associated time and effort requirements​​.

Defect prediction models aim to identify parts of the software that are likely to contain defects by analyzing historical data and code quality metrics. The primary goal is to use this predictive information to allocate testing and maintenance resources effectively. Let's break down why option D is the correct choice:

Understanding Classification Models:

Classification models are a type of supervised learning algorithm used to categorize or classify data into predefined classes or labels. In the context of defect prediction, the classification model would classify parts of the code as either "defective" or "non-defective" based on the input features.

Input Data - Code Quality Metrics:

The input data for these classification models typically includes various code quality metrics such as cyclomatic complexity, lines of code, number of methods, depth of inheritance, coupling between objects, etc. These metrics help the model learn patterns associated with defects.

Historical Data:

Historical versions of the code along with their defect records provide the labeled data needed for training the classification model. By analyzing this historical data, the model can learn which metrics are indicative of defects.

Why Option D is Correct:

Option D specifies using a classification model to predict the presence of defects by using code quality metrics as input data. This accurately describes the process of defect prediction using historical bug data and quality metrics.

Eliminating Other Options:

A. Identifying the relationship between developers and the modules developed by them: This does not directly involve predicting defects based on code quality metrics and historical data.

B. Search of similar code based on natural language processing: While useful for other purposes, this method does not describe defect prediction using classification models and code metrics.

C. Clustering of similar code modules to predict based on similarity: Clustering is an unsupervised learning technique and does not directly align with the supervised learning approach typically used in defect prediction models.

References:

ISTQB CT-AI Syllabus, Section 9.5, Metamorphic Testing (MT), describes various testing techniques including classification models for defect prediction.

"Using AI for Defect Prediction" (ISTQB CT-AI Syllabus, Section 11.5.1).

Probabilistic and non-deterministic AI-based systemsdo not always produce the same output for identical inputs. This makes traditional testing approaches ineffective. Instead, the best approach is torun tests multiple times and analyze results statistically.

Statistical Validity:Running tests multiple times ensures that observed results are statistically significant. Instead of relying on a single test run,analyzing multiple iterations helps determine trends, probabilities, and outliers​.

Expected Result Tolerance:AI-based systems may produce different results within an acceptable range. Defining acceptable tolerances (e.g., "result must be within 2% of the optimal value") improves test effectiveness​.

A (Run Several Times for the Same Correct Result):AI systems are ofteninherently non-deterministicand may not return the exact same result every time. Expecting identical outputs contradicts the nature of these systems​.

B & C (Decomposing Tests into Data Ingestion Tests):While data ingestion quality is important, it does notdirectlysolve the issue of probabilistic test results. Statistical analysis is the key approach​.

ISTQB CT-AI Syllabus (Section 8.4: Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems)

"For probabilistic systems, running a test multiple times may be necessary to obtain a statistically valid test result."​.

"Where a single definitive output is not possible, results should be analyzed statistically rather than relying on individual test cases."​.

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Sinceprobabilistic AI systems do not always return the same result, the best approach is torun multiple test iterations and validate results statistically. Hence, thecorrect answer is D.

Threshold coverageis a specific type of coverage measure used in neural network testing. It ensures that each neuron in the network achieves an activation value greater than a specified threshold. This is particularly relevant to the scenario described, where testers verify that neurons activate only when the future value of the commodity exceeds the spot price by at least0.75%.

Threshold-based activation:The test case in the question isexplicitly verifying whether neurons activate only when a certain threshold (0.75%) is exceeded.This aligns perfectly with the definition ofthreshold coverage.

Common in Neural Network Testing:Threshold coverage is used to measurewhether each neuron in a neural network reaches a specified activation value, ensuring that the neural network behaves as expected when exposed to different test inputs.

Precedent in Research:TheDeepXplore frameworkused a threshold of0.75%to identify incorrect behaviors in neural networks, making this coverage criterion well-documented in AI testing research.

(B) Neuron Coverage❌

Neuron coverageonly checks whether a neuron activates (non-zero value)at some point during testing. It does not consider specific activation thresholds, making it less precise for this scenario.

(C) Sign-Change Coverage❌

This coverage measures whether each neuron exhibitsboth positive and negative activation values, which isnot relevant to the given scenario(where activation only matters when exceeding a specific threshold).

(D) Value-Change Coverage❌

This coverage requires each neuron to producetwo activation values that differ by a chosen threshold, but the question focuses onwhether activation occurs beyond a fixed threshold, not changes in activation values.

Threshold coverage ensures that neurons exceed a given activation threshold"Full threshold coverage requires that each neuron in the neural network achieves an activation value greater than a specified threshold. The researchers who created the DeepXplore framework suggested neuron coverage should be measured based on an activation value exceeding a threshold, changing based on the situation."​

Why is Threshold Coverage Correct?Why Other Options are Incorrect?References from ISTQB Certified Tester AI Testing Study GuideThus,option A is the correct answer, asthreshold coverage ensures the neural network's activation is correctly evaluated based on the required condition (0.75%).

When verifyingautonomous AI-based systems, a critical aspect is ensuring that they maintain an appropriate level of autonomy whileonly requesting human intervention when necessary. If an AI system unnecessarily asks for human input, it defeats the purpose of autonomy and can:

Slow down operations.

Reduce trust in the system.

Indicate improper confidence thresholds in decision-making.

This is particularly crucial inautonomous vehicles, AI-driven financial trading, and robotic process automation, where excessive human intervention would hinder performance.

A. Test cases to verify that the system automatically confirms the correct classification of training data→ This is relevant for verifying training consistency but not for autonomy validation.

B. Test cases to detect the system appropriately automating its data input→ While relevant, data automation does not directly address the verification of autonomy.

D. Test cases to verify that the system automatically suppresses invalid output data→ This focuses on output filtering rather than decision-making autonomy.

Why are the other options incorrect?Thus, the mostcritical test casefor verifyingautonomous AI-based systemsis ensuring that itdoes not unnecessarily request human intervention.

Section 8.2 - Testing Autonomous AI-Based Systemsstates that it is crucial to testwhether the system requests human intervention only when necessaryand does not disrupt autonomy​.

Reference from ISTQB Certified Tester AI Testing Study Guide:

A. Feature selection

Feature selection is the process of selecting the most relevant features from the data. While important, it is not directly about handling excess data.

B. Data sampling

Data sampling involves selecting a representative subset of the data for training. When there is more data than needed, sampling can be used to create a manageable dataset that maintains the statistical properties of the full dataset.

C. Data labeling

Data labeling involves annotating data for supervised learning. It is necessary for training models but does not address the issue of having excess data.

D. Data augmentation

Data augmentation is used to increase the size of the training dataset by creating modified versions of existing data. It is useful when there is insufficient data, not when there is excess data.

Therefore, the correct answer isBbecause data sampling is the most relevant activity when dealing with an excess amount of data for training​​.