When a generative-AI application gives a wrong answer, the foundation model is only one possible cause. Retrieval may have returned the wrong contract, a prompt revision may have removed an instruction, a guardrail may have blocked a benign task, or an agent may have called a valid tool with incorrect arguments. AWS Certified Generative AI Developer – Professional (AIP-C01) includes testing, validation and troubleshooting because operators must distinguish these failures quickly and recover without worsening customer impact.
Consider an internal compliance assistant that suddenly advises a team to use last year’s retention policy. Users report that the answer sounds confident but cites a current-looking document. An engineer who changes model temperature first may obscure the root cause. A better response traces which source was retrieved, how it was transformed and which model, prompt, index and tool versions produced the answer.
Begin with scope and containment rather than a prompt tweak
Identify affected users, workflows, start time, severity and whether the error could have caused an irreversible action. Suspend sensitive tools or narrow affected traffic if authorization or privacy could be compromised. Preserve relevant immutable release identifiers and incident evidence under approved retention rules. Do not copy raw confidential prompts into an unrestricted debugging channel.
Separate incorrect content from unauthorized action. A bad summary may need correction and customer communication; an unwanted payment could require settlement reconciliation, approval review and formal incident handling. Establish who is authorized to disable the capability, change a feature flag or initiate rollback before the investigation becomes urgent.
Follow a request through five distinct failure layers
Input and preprocessing. Check object identity, MIME validation, extracted text, language, truncation, schema and source authorization. A table parser could have dropped the word “except”; an audio pipeline may have associated a promise with the wrong speaker. Review the multimodal data-quality pipeline when failures correlate with a document type rather than with all users.
Retrieval. Check embedding version, tenant metadata filters, index freshness, query transformation, selected passages and reranking. Verify that the cited passage actually supports the claim, not merely that it has a plausible title. Replay an approved representative query against the recorded index snapshot if possible. If only one tenant sees the error, investigate its visibility filters before changing the global prompt. The RAG evidence path provides the individual checkpoints.
Inference. Record the effective model, Region, inference parameters, content block shape, prompt revision, completion reason and observed latency. A model change could alter formatting or tool selection, while an unsupported field or context limit could cause a request failure. Distinguish transient throttling from invalid authentication or schema errors. A retry policy should address the actual failure class rather than treating every non-success as recoverable.
Safety and authorization. Determine whether input/output filters, denied topics, sensitive-data detection or deterministic permissions rejected the request. A guardrail can legitimately prevent unsafe generation; that is not automatically an outage. Conversely, a safety filter allowing a response does not prove an agent was authorized to execute an external business action. The application security guide provides a separate threat-model view.
Tools and business systems. Check whether the model proposed a call, whether the application validated it, whether the tool actually executed, and whether the downstream system committed a result. A timeout after commitment is especially dangerous: another call may duplicate an action. Reconcile the domain record using a durable operation ID before attempting recovery.
Use observability that reconstructs decisions without leaking secrets
Correlate request identifiers across ingress, retrieval, Bedrock runtime, tool execution and final response. Amazon CloudWatch can support operational metrics and logs; AWS X-Ray can help trace supported application boundaries. Use the organization’s configured instrumentation to associate model, prompt, source, policy and tool revisions, not merely token counts. The detailed GenAI observability design explains useful trace boundaries and privacy safeguards.
Prefer structured fields such as operation ID, tenant-class label, permitted error category, model identifier, prompt version, document snapshot ID and tool outcome. Avoid recording complete unredacted conversations unless there is a documented authorization and retention basis. A privacy incident should not be created by the process intended to debug an earlier error.
Triage with a concise failure matrix
If many requests fail rapidly with access-denied responses, compare current IAM role policies, selected Region, VPC endpoint configuration and client credentials. Retrying every call will not fix missing permission. If latency and throttling rise together, examine concurrency, account limits, model routing, request size and SDK backoff. Shed load or apply bounded retries while preserving the caller’s deadline.
If answers are fluent but cite stale facts, review source revisions, extraction errors, index sync and effective metadata filters. If responses are suddenly overblocked, compare guardrail and application policy versions against an approved false-positive dataset. If tool side effects disagree with the final assistant message, inspect the durable action ledger and backend receipts: the final generated sentence is not an accounting system.
Build minimal reproducible incidents with sanitized fixtures
Keep a regression case containing approved example input, expected source set, model and prompt version references, acceptable outcome and known prohibited behavior. A fixture for the stale-policy bug could include two policy revisions and a strict expected effective date. Test the case before and after the fix. Include a negative query whose right answer is “insufficient evidence,” because an intervention that only improves happy-path accuracy may make unsupported guesses worse.
Measure both statistical and deterministic properties. Retrieval recall and human evaluation help assess answer quality; an authorization control should pass exactly, not “most of the time.” Distinguish candidate evaluation from production performance. The model evaluation article explains how release gates should interpret representative datasets and high-risk cases.
Recover with explicit abort, rollback and verification steps
Rollback a prompt, model configuration, index snapshot or feature only when the previous version remains compatible with current data and business policies. Restoring an old prompt does not reverse an external email or payment already sent. For an agent workflow, disabling the risky tool while maintaining read-only support may be safer than turning off every feature. Define an observation period and a specific success criterion before declaring recovery.
For the retention-policy example, require the corrected source revision to be retrieved under the affected permissions, the response to cite the right effective date, and the original reproduction to fail no longer. Check unaffected contracts so the correction does not create new regressions. Add the sanitized case to CI and record the runbook owner.
A strong AIP-C01 troubleshooting answer names the violated contract, the evidence that locates the failure, the safe containment boundary, the narrowly targeted repair and the test proving the system has recovered. Randomly rewriting the prompt under incident pressure is not a reproducible operational practice.