ISTQB CT-GenAI: Using Generative AI in Software Testing

Generative AI can assist test teams with requirements analysis, test design, automation scaffolding, defect investigation, and reporting. It can also produce plausible but incorrect assertions, expose confidential data, or persuade testers to trust scripts that have never been executed. The ISTQB Certified Tester – Testing with Generative AI (CT-GenAI) certification examines how to make generative models useful within a disciplined quality-engineering workflow. It does not imply that an AI-generated test case is automatically a reliable test.

ISTQB's latest CT-GenAI syllabus v1.1 is a minor revision of v1.0 that clarifies terminology, evaluation metrics, risk discussions, and uses involving AI-assisted testing and agents. Its main coverage spans generative AI foundations, prompt engineering, evaluating model outputs, managing risks, LLM-powered testing solutions, and organizational adoption. The subject is distinct from CT-AI, which focuses on testing systems that contain AI models. The two certifications can complement each other, but conflating their purposes would lead candidates toward the wrong exercises and syllabus.

Understand what language models can and cannot guarantee

Large language models produce outputs from learned patterns and the context supplied at inference time. They can summarize complex descriptions and draft coherent artifacts quickly, but fluent language is not proof that a technical statement is true or that a generated script behaves as intended. Their behavior may vary with model version, parameters, input phrasing, and available tools. A test team should regard AI output as a candidate work product requiring verification under defined quality criteria.

The value of generative AI depends heavily on the task. Summarizing a meeting transcript may save time if a human can verify decisions and actions. Generating boundary-value test ideas may uncover cases a tester overlooked. Producing an automated assertion from an ambiguous requirement can be dangerous when the model invents expected behavior that no stakeholder authorized. The team should distinguish tasks that are easy to validate from tasks where a wrong output is difficult to detect.

Grounding matters. Providing a curated specification, API contract, code excerpt, risk catalogue, or approved test basis can improve relevance, but the source material must itself be trustworthy and current. Retrieval does not guarantee correctness; a model may misread an excerpt, combine incompatible requirements, or overstate confidence. Every generated artifact should remain traceable to its actual test basis and to the people who approve its use.

Generative AI strategy is useful as general context, yet CT-GenAI requires concrete testing workflows. A team should know which tasks are assisted, which data are permitted, which outputs require review, which tools may act automatically, and what measurements establish whether the approach improves quality rather than merely increasing output volume.

Design prompts around test objectives and source evidence

Prompt engineering begins by defining the work product. A request such as 'write comprehensive tests for this application' lacks scope, expected format, acceptance criteria, and source context. A stronger task specifies the feature, intended users, constraints, approved requirements, relevant risk, output structure, and rules for identifying unknowns. The goal is not a magical wording trick; it is to make an ambiguous test task sufficiently explicit that its output can be evaluated.

Useful prompting approaches include zero-shot instructions, few-shot examples, role or context framing, and prompt chaining. Few-shot examples can communicate desired structure and coverage. Chaining can separate test-condition identification, prioritization, test-case generation, and review so that each stage can be checked before proceeding. These techniques are tools, not quality guarantees. An elaborate prompt may still conceal missing requirements or cause the model to mirror a flawed example.

Generated tests should distinguish inputs, action, expected results, and assumptions. If the requirements do not establish an outcome, the model should flag a question rather than invent an assertion. A tester can ask for negative cases, decision-table combinations, equivalence partitions, boundary values, accessibility considerations, or state transitions, then examine whether those cases actually follow from the product specification.

Prompt management in production generative AI systems offers a related operational concept: versioning prompts and related configuration helps teams understand why an output changed. In testing, the prompt, model version, test basis, input data, generated artifact, reviewer decisions, and execution result should be traceable when those records materially affect the assurance claim.

Use AI for requirements analysis and test design

Requirements often contain missing actors, unclear thresholds, contradictory statements, and unspoken exceptions. A model can highlight possible ambiguities, propose questions to stakeholders, and organize traceability candidates. Yet it may also confidently label an intended feature as a contradiction because it lacks business context. The tester must verify interpretations with authoritative stakeholders and maintain a clear boundary between approved requirements and hypotheses suggested by the model.

For test design, generative AI can propose scenario combinations, alternative input classes, failure paths, and examples written in structured formats such as Given–When–Then. Those outputs should then be assessed for coverage and relevance. A large catalogue of nearly identical tests may be worse than a smaller collection targeting distinct failure modes. Quality comes from demonstrating that meaningful risks and requirements are exercised, not from maximizing the number of lines generated.

Consider a payment refund feature. An AI assistant might propose successful refunds, repeated requests, expired permissions, limits, partial refunds, and concurrent actions. The tester should confirm which behaviors are defined by the product and which require clarification. If the assistant invents an undocumented refund time limit, that rule must not silently become a test oracle. Generative output is an aid to analysis, not an alternative source of business authority.

AI can also help summarize exploratory testing notes and cluster defect symptoms, but those summaries require checks against original evidence. A subtle chronology or reproduction condition may matter more than a polished narrative. Preserve the source observations, versions, environment, and reproduction steps rather than replacing them with an untraceable model-generated summary.

Review generated automation as executable code

Generative AI can draft page objects, API test scaffolds, fixtures, data builders, and assertions. This is useful when reviewers understand the framework and can evaluate the output, but generated code can introduce unsafe dependencies, brittle selectors, timing problems, hidden assumptions, or weak assertions that pass even when the feature is broken. Tests should be executed and inspected, not accepted because the code looks idiomatic.

An automation review should ask whether the test targets the correct environment, creates and cleans up data safely, isolates state, handles asynchronous behavior, checks meaningful expected results, and fails for the intended reasons. A generated test that asserts only a page loaded provides little confidence in a complex purchase flow. Another test might appear sophisticated while its mocked service bypasses the very authorization boundary that needs validation.

Continuous integration for generative AI applications is adjacent to CT-GenAI's automation concerns, although a test team need not use any particular cloud product. The relevant principle is controlled execution: version the generated code, protect credentials, run it in an appropriate environment, collect evidence, and prevent unreviewed AI output from changing production systems or test baselines.

When an AI assistant has tool access, distinguish suggestions from actions. The ability to read a repository, edit files, run scripts, and submit changes increases its usefulness and its potential impact. Apply least privilege, approval gates, isolated environments, audit logs, and defined human responsibility for changes. A tool should not be allowed to delete test assets or publish a release merely because untrusted text suggests that action.

Evaluate outputs and detect plausible mistakes

Evaluating AI-generated testing artifacts requires a task-specific rubric. For a test-case proposal, assess requirement traceability, risk coverage, expected-result correctness, redundancy, clarity, and executability. For generated code, include security, maintainability, framework compatibility, and execution behavior. For a defect summary, check factual fidelity, missing reproduction details, severity rationale, and unwarranted claims about causes.

Repeatability matters. One useful response to a prompt does not establish a reliable process. Teams may sample multiple runs, compare versions against a stable evaluation set, and analyze variation in output quality. Measurements should avoid treating verbosity as completeness or human preference as proof of correctness. Where an evaluation model scores another model's outputs, the evaluator itself requires calibration and human challenge.

Hallucinations can take several forms: invented APIs, nonexistent configuration flags, fabricated documentation, mistaken expected results, or supposed vulnerabilities without evidence. Overconfidence makes such errors especially dangerous because they sound like professional technical advice. Evaluating generative AI outputs is relevant as a supporting technical subject when it explains traceable rubrics and assessment, but any platform-specific procedure must be adapted carefully to the organization's chosen environment.

Useful evaluation compares the assisted workflow with an appropriate baseline. Did testers find distinct defects sooner? Did review effort increase? Were meaningful edge cases added? Did unstable generated scripts create maintenance burdens? A productivity claim should include the cost of review, execution, correction, and ongoing support. Saving drafting time is not a benefit if the generated tests create more false assurance than they remove.

Control privacy, security, bias, and other adoption risks

Test inputs can contain production records, source code, customer information, credentials, contracts, or confidential defect details. Sending them to a generative AI service may introduce processing, retention, access, location, and contractual questions. Data minimization, approved services, retention controls, access restrictions, and suitable synthetic examples should be established before adoption. Masking a few visible names does not necessarily remove all sensitive information from a dataset.

Prompt injection and tool abuse create additional hazards. Text retrieved from an issue tracker, test page, repository file, or external document may attempt to redirect the AI assistant away from its approved task. Prompt injection and LLM application security applies particularly when the model can execute actions or access privileged systems. Untrusted source text should remain evidence to analyze, not authority to override user, organization, or system instructions.

Bias can influence which users or conditions receive testing attention. If a model's training examples favor common web forms, it may overlook assistive technology, localized interfaces, unusual network conditions, or edge-case workflows. Test teams should challenge the distribution of proposed cases and ensure coverage is driven by product risk, not only what the model can easily generate. Environmental impact and resource use should also be considered when adopting expensive inference at scale for tasks that simpler tools can accomplish.

Governance should set boundaries for permitted use. Approved tools, data classes, human review rules, logging, procurement requirements, and incident procedures need identifiable owners. Technical controls should make the safe route easy rather than relying solely on admonitions to 'use AI responsibly.' Staff should be trained to distinguish model suggestions from authoritative requirements and to report problematic outputs without fear of embarrassment.

Integrate LLM solutions into sustainable test operations

An organization can use a direct assistant, a retrieval-augmented application, a tool-using agent, or a more specialized service integrated with test management and CI systems. The architecture changes both opportunities and risks. Retrieval can ground answers in approved documents, but indexing and access control must be designed carefully. Fine-tuning may help particular tasks, yet it adds data provenance, evaluation, and lifecycle responsibilities. Tools or agents require stronger execution boundaries than a read-only conversational assistant.

Adoption should begin with measurable problems. A team struggling to document complex reproduction steps might pilot summarization with mandatory review. Another team could evaluate whether AI-assisted boundary-case generation improves defect discovery on a tightly scoped component. Define baseline effort, quality criteria, costs, risks, review responsibilities, and a stopping condition before the pilot. Expand only when evidence supports the benefit.

Change management matters because testers' skills and work roles evolve. Reviewers need the capacity to detect errors, understand the generated code, and challenge unverified claims. Outsourcing basic thought to a model may weaken expertise if teams stop examining requirements or test oracles themselves. A sustainable approach uses automation to make human judgment more focused, not to erase it.

The organizational roadmap should include model and prompt version management, vendor assessments, data-access controls, incident response, evaluation sets, integration tests, and periodic review of costs and output quality. A governance process should also allow retirement of a workflow that no longer delivers value. An AI tool's popularity is not evidence that it belongs in every testing process.

Prepare for the current CT-GenAI v1.1 exam

ISTQB's current CT-GenAI v1.1 information describes 40 questions, 46 total points, a published passing score of 30 points, and a standard 60-minute examination duration. ISTQB requires candidates for CT-GenAI to already hold the Certified Tester Foundation Level (CTFL) qualification. The exact exam provider and language arrangements must be checked at booking, but the current v1.1 syllabus is the appropriate primary content reference.

Practice with an ordinary testing project. Provide a verified requirement, ask a model for test conditions, identify which are directly supported and which are speculative, refine a prompt, inspect generated automation, execute the tests in an isolated environment, and evaluate whether the output improves quality. Next, introduce deliberately ambiguous input, confidential data, or untrusted repository text, and decide how the system should behave under the organization's guardrails.

A strong candidate understands both where generative AI is helpful and where it must be constrained. CT-GenAI is not a credential for uncritical prompt enthusiasm: it is about incorporating LLM assistance into testing without losing traceability, confidentiality, executable evidence, or human responsibility for quality decisions.