Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 26

Question 250

A retrieval assistant cites a source but sometimes attributes the wrong conclusion to it. Which change targets the constraint rather than a proxy metric?

  1. For this workflow, score citation entailment and source correctness separately from answer relevance.
  2. For the affected service, define policy-decision correctness as a primary metric with safety-critical error categories.
  3. In the project scope, track tool-policy violations and approval bypass attempts as independent safety metrics.
  4. For the intended users, stratify evaluation by risk, query type and frequency instead of relying on aggregate accuracy.

Explanation

Correct Answer

A:

The presence of a citation does not prove that the cited text supports a specific claim; its effect can be measured when a reviewer can verify every sampled claim directly against the cited evidence span.

Incorrect Answers

B:

Satisfaction can reward fluency while overlooking harmful violations of binding eligibility criteria; this technique fits where a loan-policy assistant is judged by user satisfaction even when it makes incorrect eligibility statements, rather than the condition that a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it.

C:

Answer correctness cannot substitute for limits on what external side effects the agent may produce; the trigger would be that a tool-using agent makes accurate decisions but sometimes performs unauthorized actions, while the immediate question involves a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it.

D:

A high overall score can conceal failures in infrequent but consequential cases; the design helps when a knowledge assistant is accurate on common questions but misses rare safety procedures, but the present constraint is that a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it.

Question 251

A real-time assistant averages one second but occasionally blocks users for thirty seconds. Which action is justified by the scenario?

  1. At the system boundary, use a weighted evaluation scorecard with explicit hard safety gates for critical errors.
  2. For the primary workstream, set percentile latency and timeout metrics alongside completion and token cost.
  3. For this workflow, stratify evaluation by risk, query type and frequency instead of relying on aggregate accuracy.
  4. Under the reviewed approach, define policy-decision correctness as a primary metric with safety-critical error categories.

Explanation

Correct Answer

B:

Averages conceal long-tail delays that may break user experience obligations; a meaningful test result is that peak-hour monitoring reports p95/p99 latency and failed-request percentages.

Incorrect Answers

A:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; the other workload is one where a stakeholder requests one universal score for a regulated assistant across all tasks, not the risk where a real-time assistant averages one second but occasionally blocks users for thirty seconds.

C:

A high overall score can conceal failures in infrequent but consequential cases; the alternative helps if a knowledge assistant is accurate on common questions but misses rare safety procedures, whereas this scenario involves a real-time assistant averages one second but occasionally blocks users for thirty seconds.

D:

Satisfaction can reward fluency while overlooking harmful violations of binding eligibility criteria; it is justified if a loan-policy assistant is judged by user satisfaction even when it makes incorrect eligibility statements, but here a real-time assistant averages one second but occasionally blocks users for thirty seconds.

Question 252

A stakeholder requests one universal score for a regulated assistant across all tasks. What change directly resolves this production-design issue?

  1. For the intended users, score citation entailment and source correctness separately from answer relevance.
  2. Within the delivery plan, track tool-policy violations and approval bypass attempts as independent safety metrics.
  3. At the system boundary, define policy-decision correctness as a primary metric with safety-critical error categories.
  4. For the primary workstream, measure net handling time, escalation rate and correction effort against the manual baseline.
  5. For this workflow, use a weighted evaluation scorecard with explicit hard safety gates for critical errors.

Explanation

Correct Answer

E:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; the practical acceptance condition is that an item with a safety-gate failure cannot pass release despite strong average quality.

Incorrect Answers

A:

The presence of a citation does not prove that the cited text supports a specific claim; it serves conditions where a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it, but this decision concerns a stakeholder requests one universal score for a regulated assistant across all tasks.

B:

Answer correctness cannot substitute for limits on what external side effects the agent may produce; the other case is when a tool-using agent makes accurate decisions but sometimes performs unauthorized actions, but the actual risk is that a stakeholder requests one universal score for a regulated assistant across all tasks.

C:

Satisfaction can reward fluency while overlooking harmful violations of binding eligibility criteria; this makes sense when a loan-policy assistant is judged by user satisfaction even when it makes incorrect eligibility statements, but the requested boundary concerns a stakeholder requests one universal score for a regulated assistant across all tasks.

D:

Value must include downstream verification and rework, not just generation time; this technique fits where an internal ticket agent claims to save time without counting extra reviewer effort, not the present case where a stakeholder requests one universal score for a regulated assistant across all tasks.

Question 253

Customers praise a benefit advisor's tone while independent reviewers detect material eligibility mistakes. Which finding should the release gate require?

  1. The pilot evaluation finds that a controlled pilot captures total labor minutes per fully resolved ticket.
  2. The comparison run confirms that a labeled case set reports false approvals and false denials separately by severity.
  3. The staged rollout demonstrates that peak-hour monitoring reports p95/p99 latency and failed-request percentages.
  4. The audit sample demonstrates that an item with a safety-gate failure cannot pass release despite strong average quality.

Explanation

Correct Answer

B:

Satisfaction can reward fluency while overlooking harmful violations of binding eligibility criteria; it tests the operating case in which a loan-policy assistant is judged by user satisfaction even when it makes incorrect eligibility statements.

Incorrect Answers

A:

Value must include downstream verification and rework, not just generation time; it serves a setting where an internal ticket agent claims to save time without counting extra reviewer effort, rather than the condition that customers praise a benefit advisor's tone while independent reviewers detect material eligibility mistakes.

C:

Averages conceal long-tail delays that may break user experience obligations; the other workload is one where a real-time assistant averages one second but occasionally blocks users for thirty seconds, but the present constraint is that customers praise a benefit advisor's tone while independent reviewers detect material eligibility mistakes.

D:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; this applies when a stakeholder requests one universal score for a regulated assistant across all tasks, whereas the current issue is that customers praise a benefit advisor's tone while independent reviewers detect material eligibility mistakes.

Question 254

The aggregate evaluation passes even though a rarely used emergency workflow repeatedly gives dangerous advice. Which verified outcome most directly addresses the reported risk?

  1. The sign-off record demonstrates that a labeled case set reports false approvals and false denials separately by severity.
  2. The acceptance evidence shows that a controlled pilot captures total labor minutes per fully resolved ticket.
  3. The pilot evaluation finds that a reviewer can verify every sampled claim directly against the cited evidence span.
  4. The comparison run confirms that release metrics show per-stratum recall and a separately enforced critical-case threshold.
  5. The staged rollout demonstrates that an item with a safety-gate failure cannot pass release despite strong average quality.

Explanation

Correct Answer

D:

A high overall score can conceal failures in infrequent but consequential cases; its finding speaks to the case where a knowledge assistant is accurate on common questions but misses rare safety procedures.

Incorrect Answers

A:

Satisfaction can reward fluency while overlooking harmful violations of binding eligibility criteria; that belongs to situations where a loan-policy assistant is judged by user satisfaction even when it makes incorrect eligibility statements, not this situation where the aggregate evaluation passes even though a rarely used emergency workflow repeatedly gives dangerous advice.

B:

Value must include downstream verification and rework, not just generation time; it is relevant if an internal ticket agent claims to save time without counting extra reviewer effort, whereas this scenario involves the aggregate evaluation passes even though a rarely used emergency workflow repeatedly gives dangerous advice.

C:

The presence of a citation does not prove that the cited text supports a specific claim; it serves conditions where a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it, but here the aggregate evaluation passes even though a rarely used emergency workflow repeatedly gives dangerous advice.

E:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; this makes sense when a stakeholder requests one universal score for a regulated assistant across all tasks, not the case in which the aggregate evaluation passes even though a rarely used emergency workflow repeatedly gives dangerous advice.

Question 255

A support pilot claims efficiency from faster drafting but ignores staff time correcting fabricated references. What independently verifiable observation is needed?

  1. A reproducible run shows that an item with a safety-gate failure cannot pass release despite strong average quality.
  2. The signed test record confirms that a reviewer can verify every sampled claim directly against the cited evidence span.
  3. A production-like test shows that a labeled case set reports false approvals and false denials separately by severity.
  4. The release check confirms that a controlled pilot captures total labor minutes per fully resolved ticket.

Explanation

Correct Answer

D:

Value must include downstream verification and rework, not just generation time; the verification boundary is the one where an internal ticket agent claims to save time without counting extra reviewer effort.

Incorrect Answers

A:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; this control is needed when a stakeholder requests one universal score for a regulated assistant across all tasks, but the actual risk is that a support pilot claims efficiency from faster drafting but ignores staff time correcting fabricated references.

B:

The presence of a citation does not prove that the cited text supports a specific claim; that would be useful if a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it, but the requested boundary concerns a support pilot claims efficiency from faster drafting but ignores staff time correcting fabricated references.

C:

Satisfaction can reward fluency while overlooking harmful violations of binding eligibility criteria; it serves a setting where a loan-policy assistant is judged by user satisfaction even when it makes incorrect eligibility statements, not the present case where a support pilot claims efficiency from faster drafting but ignores staff time correcting fabricated references.

Question 256

An agent's final decisions match the benchmark, while its intermediate tool trace includes prohibited mutations. What outcome would demonstrate this requirement in operation?

  1. The release report verifies that a reviewer can verify every sampled claim directly against the cited evidence span.
  2. The controlled replay verifies that an item with a safety-gate failure cannot pass release despite strong average quality.
  3. A reproducible run shows that release metrics show per-stratum recall and a separately enforced critical-case threshold.
  4. The signed test record confirms that adversarial action tests report zero unauthorized commits under the release gate.

Explanation

Correct Answer

D:

Answer correctness cannot substitute for limits on what external side effects the agent may produce; the evidence must be robust when a tool-using agent makes accurate decisions but sometimes performs unauthorized actions.

Incorrect Answers

A:

The presence of a citation does not prove that the cited text supports a specific claim; it targets cases where a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it, while the immediate question involves an agent's final decisions match the benchmark, while its intermediate tool trace includes prohibited mutations.

B:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; the approach helps when a stakeholder requests one universal score for a regulated assistant across all tasks, but the present constraint is that an agent's final decisions match the benchmark, while its intermediate tool trace includes prohibited mutations.

C:

A high overall score can conceal failures in infrequent but consequential cases; that belongs to situations where a knowledge assistant is accurate on common questions but misses rare safety procedures, whereas the current issue is that an agent's final decisions match the benchmark, while its intermediate tool trace includes prohibited mutations.

Question 257

A citation evaluation marks sources as present without checking whether the cited passage entails the claim. Which observed test outcome would justify sign-off?

  1. The release check confirms that an item with a safety-gate failure cannot pass release despite strong average quality.
  2. The reviewer observes that a reviewer can verify every sampled claim directly against the cited evidence span.
  3. The release report verifies that peak-hour monitoring reports p95/p99 latency and failed-request percentages.
  4. The controlled replay verifies that adversarial action tests report zero unauthorized commits under the release gate.

Explanation

Correct Answer

B:

The presence of a citation does not prove that the cited text supports a specific claim; the test covers the original risk that a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it.

Incorrect Answers

A:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; the design helps when a stakeholder requests one universal score for a regulated assistant across all tasks, whereas this scenario involves a citation evaluation marks sources as present without checking whether the cited passage entails the claim.

C:

Averages conceal long-tail delays that may break user experience obligations; this control is needed when a real-time assistant averages one second but occasionally blocks users for thirty seconds, instead of when a citation evaluation marks sources as present without checking whether the cited passage entails the claim.

D:

Answer correctness cannot substitute for limits on what external side effects the agent may produce; that would be useful if a tool-using agent makes accurate decisions but sometimes performs unauthorized actions, not the case in which a citation evaluation marks sources as present without checking whether the cited passage entails the claim.

Question 258

Average inference time meets the SLA, but a minority of live requests takes far beyond the contract limit. Which sign-off finding best checks the actual boundary?

  1. The regression suite verifies that adversarial action tests report zero unauthorized commits under the release gate.
  2. A production-like test shows that an item with a safety-gate failure cannot pass release despite strong average quality.
  3. The release check confirms that a labeled case set reports false approvals and false denials separately by severity.
  4. The reviewer observes that peak-hour monitoring reports p95/p99 latency and failed-request percentages.

Explanation

Correct Answer

D:

Averages conceal long-tail delays that may break user experience obligations; it is the decisive test when a real-time assistant averages one second but occasionally blocks users for thirty seconds.

Incorrect Answers

A:

Answer correctness cannot substitute for limits on what external side effects the agent may produce; it is justified if a tool-using agent makes accurate decisions but sometimes performs unauthorized actions, but the requested boundary concerns average inference time meets the SLA, but a minority of live requests takes far beyond the contract limit.

B:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; its scope includes cases where a stakeholder requests one universal score for a regulated assistant across all tasks, not the present case where average inference time meets the SLA, but a minority of live requests takes far beyond the contract limit.

C:

Satisfaction can reward fluency while overlooking harmful violations of binding eligibility criteria; it targets cases where a loan-policy assistant is judged by user satisfaction even when it makes incorrect eligibility statements, while this problem involves average inference time meets the SLA, but a minority of live requests takes far beyond the contract limit.

Question 259

A compliance reviewer rejects a single combined score that hides distinct safety, accuracy and latency failures. What validation evidence would address this failure mode?

  1. The end-to-end check confirms that a reviewer can verify every sampled claim directly against the cited evidence span.
  2. The observed outcome is that peak-hour monitoring reports p95/p99 latency and failed-request percentages.
  3. The representative case test finds that adversarial action tests report zero unauthorized commits under the release gate.
  4. The negative-case evaluation confirms that an item with a safety-gate failure cannot pass release despite strong average quality.
  5. A controlled trial finds that a labeled case set reports false approvals and false denials separately by severity.

Explanation

Correct Answer

D:

Multidimensional acceptance stops inexpensive or popular outputs from hiding severe failures; this is a functional check for the case where a stakeholder requests one universal score for a regulated assistant across all tasks.

Incorrect Answers

A:

The presence of a citation does not prove that the cited text supports a specific claim; this technique fits where a retrieval assistant cites a source but sometimes attributes the wrong conclusion to it, but the present constraint is that a compliance reviewer rejects a single combined score that hides distinct safety, accuracy and latency failures.

B:

Averages conceal long-tail delays that may break user experience obligations; the trigger would be that a real-time assistant averages one second but occasionally blocks users for thirty seconds, whereas the current issue is that a compliance reviewer rejects a single combined score that hides distinct safety, accuracy and latency failures.

C:

Answer correctness cannot substitute for limits on what external side effects the agent may produce; the design helps when a tool-using agent makes accurate decisions but sometimes performs unauthorized actions, whereas the release criterion addresses a compliance reviewer rejects a single combined score that hides distinct safety, accuracy and latency failures.

E:

Satisfaction can reward fluency while overlooking harmful violations of binding eligibility criteria; this control is needed when a loan-policy assistant is judged by user satisfaction even when it makes incorrect eligibility statements, not the risk where a compliance reviewer rejects a single combined score that hides distinct safety, accuracy and latency failures.

Question 260

A prototype is evaluated on the same ten easy cases used during prompt development. Which action addresses the root concern without unnecessary scope?

  1. For the current integration, add prompt-injection, exfiltration and malformed-input probes to the evaluation corpus.
  2. Under the system plan, create an adjudicated scoring rubric with examples and inter-rater checks.
  3. As the design priority, build a withheld representative set with routine, edge and adversarial cases.
  4. In the implementation sequence, calibrate model-based grading against expert labels and add deterministic fact checks where possible.
  5. For the operating team, evaluate traces and tool transitions in addition to the final user-facing output.

Explanation

Correct Answer

C:

Training and tuning on the only evaluation examples inflates perceived performance; one operational check should confirm that the final holdout remains unseen during optimization and spans meaningful task classes.

Incorrect Answers

A:

Adversarial cases test the boundary between source content and model/tool authority; this applies when a safety suite has only benign requests although real users supply untrusted retrieved content, but here a prototype is evaluated on the same ten easy cases used during prompt development.

B:

Rubric alignment is necessary before evaluation labels can support confident architectural decisions; the alternative helps if two human reviewers disagree widely on whether an answer is accurate, instead of when a prototype is evaluated on the same ten easy cases used during prompt development.

D:

A judge model can be biased toward style, so its scores need independent validation; its scope includes cases where an LLM grader rewards verbose answers even when specific facts are wrong, whereas the problem presented is that a prototype is evaluated on the same ten easy cases used during prompt development.

E:

A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; it targets cases where an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior, but this decision concerns a prototype is evaluated on the same ten easy cases used during prompt development.

Claude Certified Architect - Professional (CCAR-P) Practice Tests