Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 30

Question 290

An assistant makes confident factual claims when relevant documents cannot be retrieved. Which architecture change should be approved first?

  1. For this production path, instrument no-evidence paths and enforce abstention or human escalation on missing support.
  2. For the intended users, benchmark a more capable model against the same difficult holdout before adopting it.
  3. At this decision point, diff prompt revisions and replay known failing cases using pinned component versions.
  4. As a candidate control, inspect ingestion, indexing and retrieval results before rewriting the system prompt.
  5. For the primary workstream, inspect tool schemas, examples and argument-validation errors at the invocation boundary.

Explanation

Correct Answer

A:

The failure is unsupported synthesis; more assertive wording would worsen the defect; its effect can be measured when empty-retrieval tests yield acknowledged uncertainty rather than fabricated facts.

Incorrect Answers

B:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; this technique fits where a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts, rather than the condition that an assistant makes confident factual claims when relevant documents cannot be retrieved.

C:

Controlled replay distinguishes prompt regression from random variation or data changes; the trigger would be that a production model begins giving different answers after a silent prompt template edit, while the immediate question involves an assistant makes confident factual claims when relevant documents cannot be retrieved.

D:

A time-correlated corpus update points first to source and evidence pipeline changes; the design helps when a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change, but the present constraint is that an assistant makes confident factual claims when relevant documents cannot be retrieved.

E:

Precise parameter contracts control tool execution more directly than general answer-quality tuning; that fits cases where an agent chooses the correct API but sends invalid arguments for a nested object, whereas the current issue is that an assistant makes confident factual claims when relevant documents cannot be retrieved.

Question 291

A task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts. What should be incorporated into the approved design?

  1. In the proposed design, instrument no-evidence paths and enforce abstention or human escalation on missing support.
  2. Under the reviewed approach, inspect tool schemas, examples and argument-validation errors at the invocation boundary.
  3. For this production path, benchmark a more capable model against the same difficult holdout before adopting it.
  4. For the intended users, diff prompt revisions and replay known failing cases using pinned component versions.

Explanation

Correct Answer

C:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; acceptance evidence must confirm that blind comparisons confirm improved constraint satisfaction without unacceptable cost.

Incorrect Answers

A:

The failure is unsupported synthesis; more assertive wording would worsen the defect; the other workload is one where an assistant makes confident factual claims when relevant documents cannot be retrieved, not the risk where a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts.

B:

Precise parameter contracts control tool execution more directly than general answer-quality tuning; this applies when an agent chooses the correct API but sends invalid arguments for a nested object, not this situation where a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts.

D:

Controlled replay distinguishes prompt regression from random variation or data changes; it is justified if a production model begins giving different answers after a silent prompt template edit, but here a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts.

Question 292

A production model begins giving different answers after a silent prompt template edit. Which approach should the design team choose?

  1. As a candidate control, diff prompt revisions and replay known failing cases using pinned component versions.
  2. In the initial solution, benchmark a more capable model against the same difficult holdout before adopting it.
  3. In the proposed design, inspect ingestion, indexing and retrieval results before rewriting the system prompt.
  4. Under the reviewed approach, evaluate grounding and enforce claim-to-source consistency during generation.

Explanation

Correct Answer

A:

Controlled replay distinguishes prompt regression from random variation or data changes; the audit trail can establish that old-version replay restores prior behavior and identifies the problematic instruction change.

Incorrect Answers

B:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; the other case is when a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts, but the actual risk is that a production model begins giving different answers after a silent prompt template edit.

C:

A time-correlated corpus update points first to source and evidence pipeline changes; this makes sense when a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change, but the requested boundary concerns a production model begins giving different answers after a silent prompt template edit.

D:

Retrieval success alone does not guarantee faithful synthesis of the retrieved text; this technique fits where a retrieval bot finds the correct passage but the answer invents numbers not in that passage, not the present case where a production model begins giving different answers after a silent prompt template edit.

Question 293

A retrieval bot finds the correct passage but the answer invents numbers not in that passage. Where should the solution architect focus the implementation?

  1. In the proposed implementation, check schema instructions and output validation before changing model capability tiers.
  2. At this decision point, instrument no-evidence paths and enforce abstention or human escalation on missing support.
  3. As a candidate control, benchmark a more capable model against the same difficult holdout before adopting it.
  4. In the initial solution, evaluate grounding and enforce claim-to-source consistency during generation.

Explanation

Correct Answer

D:

Retrieval success alone does not guarantee faithful synthesis of the retrieved text; the go-live check should establish that numeric-claim checks reject values absent from the cited source span.

Incorrect Answers

A:

A formatting failure may reflect an underspecified interface rather than inadequate reasoning; it serves a setting where a model repeatedly returns valid prose but violates a strict JSON consumer contract, rather than the condition that a retrieval bot finds the correct passage but the answer invents numbers not in that passage.

B:

The failure is unsupported synthesis; more assertive wording would worsen the defect; that is defensible if an assistant makes confident factual claims when relevant documents cannot be retrieved, while the immediate question involves a retrieval bot finds the correct passage but the answer invents numbers not in that passage.

C:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; the other workload is one where a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts, but the present constraint is that a retrieval bot finds the correct passage but the answer invents numbers not in that passage.

Question 294

After a corpus rebuild, retrieval no longer surfaces the right clause although model and prompt versions are unchanged. What result discriminates a working safeguard from a demo?

  1. The reviewer observes that a before-and-after trace finds missing or stale answer-bearing passages.
  2. The release report verifies that schema stress cases isolate missing fields and invalid values reproducibly.
  3. The controlled replay verifies that traces show which required argument or type failed and prevent external side effects.
  4. A reproducible run shows that blind comparisons confirm improved constraint satisfaction without unacceptable cost.

Explanation

Correct Answer

A:

A time-correlated corpus update points first to source and evidence pipeline changes; the condition under test is that a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change.

Incorrect Answers

B:

A formatting failure may reflect an underspecified interface rather than inadequate reasoning; it is relevant if a model repeatedly returns valid prose but violates a strict JSON consumer contract, whereas this scenario involves after a corpus rebuild, retrieval no longer surfaces the right clause although model and prompt versions are unchanged.

C:

Precise parameter contracts control tool execution more directly than general answer-quality tuning; it serves conditions where an agent chooses the correct API but sends invalid arguments for a nested object, but here after a corpus rebuild, retrieval no longer surfaces the right clause although model and prompt versions are unchanged.

D:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; the other case is when a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts, instead of when after a corpus rebuild, retrieval no longer surfaces the right clause although model and prompt versions are unchanged.

Question 295

The structured-output consumer rejects an otherwise helpful explanation because a required JSON field is absent. What outcome must the reviewer see before proceeding?

  1. A production-like test shows that empty-retrieval tests yield acknowledged uncertainty rather than fabricated facts.
  2. The release check confirms that a before-and-after trace finds missing or stale answer-bearing passages.
  3. The reviewer observes that schema stress cases isolate missing fields and invalid values reproducibly.
  4. The release report verifies that old-version replay restores prior behavior and identifies the problematic instruction change.

Explanation

Correct Answer

C:

A formatting failure may reflect an underspecified interface rather than inadequate reasoning; this is relevant for cases where a model repeatedly returns valid prose but violates a strict JSON consumer contract.

Incorrect Answers

A:

The failure is unsupported synthesis; more assertive wording would worsen the defect; this control is needed when an assistant makes confident factual claims when relevant documents cannot be retrieved, but the actual risk is that the structured-output consumer rejects an otherwise helpful explanation because a required JSON field is absent.

B:

A time-correlated corpus update points first to source and evidence pipeline changes; that would be useful if a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change, but the requested boundary concerns the structured-output consumer rejects an otherwise helpful explanation because a required JSON field is absent.

D:

Controlled replay distinguishes prompt regression from random variation or data changes; that is defensible if a production model begins giving different answers after a silent prompt template edit, while this problem involves the structured-output consumer rejects an otherwise helpful explanation because a required JSON field is absent.

Question 296

A tool call passes schema name validation but fails on a nested payload's required data type. Which result should the pilot review prioritize?

  1. The observed outcome is that old-version replay restores prior behavior and identifies the problematic instruction change.
  2. The regression suite verifies that empty-retrieval tests yield acknowledged uncertainty rather than fabricated facts.
  3. A production-like test shows that numeric-claim checks reject values absent from the cited source span.
  4. The release check confirms that traces show which required argument or type failed and prevent external side effects.

Explanation

Correct Answer

D:

Precise parameter contracts control tool execution more directly than general answer-quality tuning; the evidence must be robust when an agent chooses the correct API but sends invalid arguments for a nested object.

Incorrect Answers

A:

Controlled replay distinguishes prompt regression from random variation or data changes; it targets cases where a production model begins giving different answers after a silent prompt template edit, while the immediate question involves a tool call passes schema name validation but fails on a nested payload's required data type.

B:

The failure is unsupported synthesis; more assertive wording would worsen the defect; the approach helps when an assistant makes confident factual claims when relevant documents cannot be retrieved, but the present constraint is that a tool call passes schema name validation but fails on a nested payload's required data type.

C:

Retrieval success alone does not guarantee faithful synthesis of the retrieved text; that belongs to situations where a retrieval bot finds the correct passage but the answer invents numbers not in that passage, whereas the current issue is that a tool call passes schema name validation but fails on a nested payload's required data type.

Question 297

A factual assistant invents authority when its retrieval layer returns no relevant passages. What result would directly validate the necessary safeguard?

  1. The scenario test demonstrates that blind comparisons confirm improved constraint satisfaction without unacceptable cost.
  2. The end-to-end check confirms that schema stress cases isolate missing fields and invalid values reproducibly.
  3. The observed outcome is that a before-and-after trace finds missing or stale answer-bearing passages.
  4. The regression suite verifies that numeric-claim checks reject values absent from the cited source span.
  5. The negative-case evaluation confirms that empty-retrieval tests yield acknowledged uncertainty rather than fabricated facts.

Explanation

Correct Answer

E:

The failure is unsupported synthesis; more assertive wording would worsen the defect; this checks the situation in which an assistant makes confident factual claims when relevant documents cannot be retrieved.

Incorrect Answers

A:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; the design helps when a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts, whereas this scenario involves a factual assistant invents authority when its retrieval layer returns no relevant passages.

B:

A formatting failure may reflect an underspecified interface rather than inadequate reasoning; that fits cases where a model repeatedly returns valid prose but violates a strict JSON consumer contract, but here a factual assistant invents authority when its retrieval layer returns no relevant passages.

C:

A time-correlated corpus update points first to source and evidence pipeline changes; this control is needed when a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change, instead of when a factual assistant invents authority when its retrieval layer returns no relevant passages.

D:

Retrieval success alone does not guarantee faithful synthesis of the retrieved text; that would be useful if a retrieval bot finds the correct passage but the answer invents numbers not in that passage, not the case in which a factual assistant invents authority when its retrieval layer returns no relevant passages.

Question 298

A compact model continues to miss a long dependency chain after prompt wording and retrieval have been corrected. What evidence would reveal whether the control truly works?

  1. A controlled trial finds that traces show which required argument or type failed and prevent external side effects.
  2. The measured evidence confirms that blind comparisons confirm improved constraint satisfaction without unacceptable cost.
  3. The scenario test demonstrates that numeric-claim checks reject values absent from the cited source span.
  4. The end-to-end check confirms that old-version replay restores prior behavior and identifies the problematic instruction change.

Explanation

Correct Answer

B:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; this is relevant for cases where a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts.

Incorrect Answers

A:

Precise parameter contracts control tool execution more directly than general answer-quality tuning; it is justified if an agent chooses the correct API but sends invalid arguments for a nested object, but the requested boundary concerns a compact model continues to miss a long dependency chain after prompt wording and retrieval have been corrected.

C:

Retrieval success alone does not guarantee faithful synthesis of the retrieved text; it targets cases where a retrieval bot finds the correct passage but the answer invents numbers not in that passage, while this problem involves a compact model continues to miss a long dependency chain after prompt wording and retrieval have been corrected.

D:

Controlled replay distinguishes prompt regression from random variation or data changes; the approach helps when a production model begins giving different answers after a silent prompt template edit, and not where a compact model continues to miss a long dependency chain after prompt wording and retrieval have been corrected.

Question 299

A nightly prompt deployment shifts refusal behavior even though no model update was recorded. Which result proves the requested control, not just configuration?

  1. The representative case test finds that schema stress cases isolate missing fields and invalid values reproducibly.
  2. The negative-case evaluation confirms that old-version replay restores prior behavior and identifies the problematic instruction change.
  3. A controlled trial finds that numeric-claim checks reject values absent from the cited source span.
  4. The measured evidence confirms that traces show which required argument or type failed and prevent external side effects.

Explanation

Correct Answer

B:

Controlled replay distinguishes prompt regression from random variation or data changes; the reviewer is concerned with situations where a production model begins giving different answers after a silent prompt template edit.

Incorrect Answers

A:

A formatting failure may reflect an underspecified interface rather than inadequate reasoning; this technique fits where a model repeatedly returns valid prose but violates a strict JSON consumer contract, but the present constraint is that a nightly prompt deployment shifts refusal behavior even though no model update was recorded.

C:

Retrieval success alone does not guarantee faithful synthesis of the retrieved text; the design helps when a retrieval bot finds the correct passage but the answer invents numbers not in that passage, whereas the release criterion addresses a nightly prompt deployment shifts refusal behavior even though no model update was recorded.

D:

Precise parameter contracts control tool execution more directly than general answer-quality tuning; that fits cases where an agent chooses the correct API but sends invalid arguments for a nested object, while the required safeguard concerns a nightly prompt deployment shifts refusal behavior even though no model update was recorded.

Question 300

A deployment has passing prelaunch tests but user complaints rise over time. How should the architect revise the proposed solution?

  1. For this workflow, add behavioral outcome metrics alongside infrastructure availability and timing.
  2. For the affected service, run scheduled regression probes with version tags and groundedness measures.
  3. In the project scope, monitor rate-limit and retry telemetry with saturation and backoff behavior.
  4. Within the rollout design, monitor sampled production correctness and data drift with reviewed feedback queues.

Explanation

Correct Answer

D:

Offline validation must continue as actual inputs and source material change; the intended outcome is that a drift alert links emerging failure categories to newly observed query patterns.

Incorrect Answers

A:

Healthy servers do not prove acceptable task performance or appropriate human review workload; this applies when a support system has a rising escalation rate with no change in average response latency, but here a deployment has passing prelaunch tests but user complaints rise over time.

B:

Longitudinal quality monitoring requires comparable, labeled cases across releases; the alternative helps if a vendor model update changes output style but existing metrics cannot tell whether correctness changed, instead of when a deployment has passing prelaunch tests but user complaints rise over time.

C:

Rate limiting is a capacity interaction that cannot be diagnosed from answer quality alone; it is justified if a model interface experiences intermittent 429 responses during a traffic surge, not the case in which a deployment has passing prelaunch tests but user complaints rise over time.

Claude Certified Architect - Professional (CCAR-P) Practice Tests