Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 28

Question 270

A numeric field extractor receives high subjective ratings despite returning wrong amounts. What evidence would reveal whether the control truly works?

  1. A production-like test shows that the final holdout remains unseen during optimization and spans meaningful task classes.
  2. The release check confirms that exact-value scores match curated annotations and escalate unreadable entries.
  3. The reviewer observes that trace assertions reject unauthorized calls even if the final summary looks correct.
  4. The release report verifies that the suite catches attempts to override instructions through retrieved text and tool results.

Explanation

Correct Answer

B:

Machine-checkable targets should not depend on a subjective holistic grading prompt; the test is necessary where a financial extraction system contains fields that can be checked exactly against source forms.

Incorrect Answers

A:

Training and tuning on the only evaluation examples inflates perceived performance; this makes sense when a prototype is evaluated on the same ten easy cases used during prompt development, rather than the risk that a numeric field extractor receives high subjective ratings despite returning wrong amounts.

C:

A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; the trigger would be that an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior, while the immediate question involves a numeric field extractor receives high subjective ratings despite returning wrong amounts.

D:

Adversarial cases test the boundary between source content and model/tool authority; the design helps when a safety suite has only benign requests although real users supply untrusted retrieved content, but the present constraint is that a numeric field extractor receives high subjective ratings despite returning wrong amounts.

Question 271

A multi-step agent gets the right final number after using an unapproved external tool. Which result proves the requested control, not just configuration?

  1. The observed outcome is that freshly adjudicated samples represent current policies without rewriting historical scores.
  2. The regression suite verifies that the final holdout remains unseen during optimization and spans meaningful task classes.
  3. A production-like test shows that independent reviewers achieve acceptable agreement on a calibrated sample.
  4. The release check confirms that trace assertions reject unauthorized calls even if the final summary looks correct.

Explanation

Correct Answer

D:

A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; the test covers the original risk that an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior.

Incorrect Answers

A:

Data drift changes the validity of both expected answers and apparent model quality; the other workload is one where a dataset was collected last year while policy documents and user terminology have changed, not the risk where a multi-step agent gets the right final number after using an unapproved external tool.

B:

Training and tuning on the only evaluation examples inflates perceived performance; this applies when a prototype is evaluated on the same ten easy cases used during prompt development, not this situation where a multi-step agent gets the right final number after using an unapproved external tool.

C:

Rubric alignment is necessary before evaluation labels can support confident architectural decisions; the alternative helps if two human reviewers disagree widely on whether an answer is accurate, whereas this scenario involves a multi-step agent gets the right final number after using an unapproved external tool.

Question 272

A safety regression suite passes until malicious instructions arrive inside retrieved wiki pages. Which acceptance result provides the most specific evidence?

  1. The scenario test demonstrates that the suite catches attempts to override instructions through retrieved text and tool results.
  2. The end-to-end check confirms that disagreement analysis identifies style-driven false passes before using the grader as a gate.
  3. The observed outcome is that independent reviewers achieve acceptable agreement on a calibrated sample.
  4. The regression suite verifies that exact-value scores match curated annotations and escalate unreadable entries.
  5. A production-like test shows that freshly adjudicated samples represent current policies without rewriting historical scores.

Explanation

Correct Answer

A:

Adversarial cases test the boundary between source content and model/tool authority; the control should hold even if a safety suite has only benign requests although real users supply untrusted retrieved content.

Incorrect Answers

B:

A judge model can be biased toward style, so its scores need independent validation; the other case is when an LLM grader rewards verbose answers even when specific facts are wrong, but the actual risk is that a safety regression suite passes until malicious instructions arrive inside retrieved wiki pages.

C:

Rubric alignment is necessary before evaluation labels can support confident architectural decisions; this makes sense when two human reviewers disagree widely on whether an answer is accurate, but the requested boundary concerns a safety regression suite passes until malicious instructions arrive inside retrieved wiki pages.

D:

Machine-checkable targets should not depend on a subjective holistic grading prompt; this technique fits where a financial extraction system contains fields that can be checked exactly against source forms, not the present case where a safety regression suite passes until malicious instructions arrive inside retrieved wiki pages.

E:

Data drift changes the validity of both expected answers and apparent model quality; the trigger would be that a dataset was collected last year while policy documents and user terminology have changed, while this problem involves a safety regression suite passes until malicious instructions arrive inside retrieved wiki pages.

Question 273

A once-representative benchmark contains obsolete terminology and misses new policy scenarios. Which acceptance check evaluates this requirement rather than a proxy?

  1. A controlled trial finds that exact-value scores match curated annotations and escalate unreadable entries.
  2. The measured evidence confirms that the final holdout remains unseen during optimization and spans meaningful task classes.
  3. The scenario test demonstrates that freshly adjudicated samples represent current policies without rewriting historical scores.
  4. The end-to-end check confirms that independent reviewers achieve acceptable agreement on a calibrated sample.

Explanation

Correct Answer

C:

Data drift changes the validity of both expected answers and apparent model quality; the safeguard must work when a dataset was collected last year while policy documents and user terminology have changed.

Incorrect Answers

A:

Machine-checkable targets should not depend on a subjective holistic grading prompt; it serves a setting where a financial extraction system contains fields that can be checked exactly against source forms, rather than the condition that a once-representative benchmark contains obsolete terminology and misses new policy scenarios.

B:

Training and tuning on the only evaluation examples inflates perceived performance; that is defensible if a prototype is evaluated on the same ten easy cases used during prompt development, while the immediate question involves a once-representative benchmark contains obsolete terminology and misses new policy scenarios.

D:

Rubric alignment is necessary before evaluation labels can support confident architectural decisions; this applies when two human reviewers disagree widely on whether an answer is accurate, whereas the current issue is that a once-representative benchmark contains obsolete terminology and misses new policy scenarios.

Question 274

An experiment exposes version B only to expert users, then compares it with version A for everyone else. What intervention best matches the stated boundary?

  1. In the project scope, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
  2. Within the rollout design, canary a small cohort, watch guardrails and preserve a rapid rollback path.
  3. For the next architecture revision, use shadow evaluation on sampled traffic with no user-facing side effects.
  4. For the current integration, randomize eligible traffic into comparable groups before drawing quality conclusions.

Explanation

Correct Answer

D:

Self-selection and role imbalance confound observed differences between variants; validation needs to show that group balance checks verify comparable case mix, user roles and task difficulty.

Incorrect Answers

A:

Confounded component changes prevent attributing a measured difference to the intended treatment; that belongs to situations where an A/B experiment compares outputs from different source-index versions by accident, not this situation where an experiment exposes version B only to expert users, then compares it with version A for everyone else.

B:

Progressive delivery limits blast radius and generates evidence before universal migration; it is relevant if a service rolls a model upgrade to every tenant at once and cannot isolate regressions, whereas this scenario involves an experiment exposes version B only to expert users, then compares it with version A for everyone else.

C:

Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; it serves conditions where a regulated assistant must be assessed on real workflows without showing unreviewed output to users, but here an experiment exposes version B only to expert users, then compares it with version A for everyone else.

Question 275

A new prompt improves accuracy but has markedly slower responses and higher cost. Which implementation decision is the strongest fit?

  1. For this workflow, randomize eligible traffic into comparable groups before drawing quality conclusions.
  2. For the affected service, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
  3. In the project scope, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
  4. Within the rollout design, predefine sample size, guardrail metrics and stopping rules before testing.

Explanation

Correct Answer

C:

An experiment succeeds only if improved correctness justifies its full operational trade-off; production testing should demonstrate that the report shows task gains and service impacts before promotion.

Incorrect Answers

A:

Self-selection and role imbalance confound observed differences between variants; this control is needed when an experiment exposes version B only to expert users, then compares it with version A for everyone else, but the actual risk is that a new prompt improves accuracy but has markedly slower responses and higher cost.

B:

Confounded component changes prevent attributing a measured difference to the intended treatment; that would be useful if an A/B experiment compares outputs from different source-index versions by accident, but the requested boundary concerns a new prompt improves accuracy but has markedly slower responses and higher cost.

D:

Repeated early peeking increases the chance of overstating noisy apparent improvement; that is defensible if a team declares victory after an early positive result on only a handful of questions, while this problem involves a new prompt improves accuracy but has markedly slower responses and higher cost.

Question 276

A regulated assistant must be assessed on real workflows without showing unreviewed output to users. What should be prioritized during architecture review?

  1. At the system boundary, predefine sample size, guardrail metrics and stopping rules before testing.
  2. For the primary workstream, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
  3. For this workflow, use shadow evaluation on sampled traffic with no user-facing side effects.
  4. For the affected service, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.

Explanation

Correct Answer

C:

Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; the go-live check should establish that the shadow variant writes no records and emits only controlled diagnostic artifacts.

Incorrect Answers

A:

Repeated early peeking increases the chance of overstating noisy apparent improvement; it targets cases where a team declares victory after an early positive result on only a handful of questions, while the immediate question involves a regulated assistant must be assessed on real workflows without showing unreviewed output to users.

B:

Confounded component changes prevent attributing a measured difference to the intended treatment; the approach helps when an A/B experiment compares outputs from different source-index versions by accident, but the present constraint is that a regulated assistant must be assessed on real workflows without showing unreviewed output to users.

D:

An experiment succeeds only if improved correctness justifies its full operational trade-off; it is relevant if a new prompt improves accuracy but has markedly slower responses and higher cost, whereas the release criterion addresses a regulated assistant must be assessed on real workflows without showing unreviewed output to users.

Question 277

A team declares victory after an early positive result on only a handful of questions. Which response follows from the actual operational constraint?

  1. For the intended users, predefine sample size, guardrail metrics and stopping rules before testing.
  2. Within the delivery plan, randomize eligible traffic into comparable groups before drawing quality conclusions.
  3. At the system boundary, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
  4. For the primary workstream, use shadow evaluation on sampled traffic with no user-facing side effects.

Explanation

Correct Answer

A:

Repeated early peeking increases the chance of overstating noisy apparent improvement; acceptance evidence must confirm that the completed experiment reports confidence and follows its registered decision criteria.

Incorrect Answers

B:

Self-selection and role imbalance confound observed differences between variants; that fits cases where an experiment exposes version B only to expert users, then compares it with version A for everyone else, but here a team declares victory after an early positive result on only a handful of questions.

C:

Confounded component changes prevent attributing a measured difference to the intended treatment; this control is needed when an A/B experiment compares outputs from different source-index versions by accident, instead of when a team declares victory after an early positive result on only a handful of questions.

D:

Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; that would be useful if a regulated assistant must be assessed on real workflows without showing unreviewed output to users, not the case in which a team declares victory after an early positive result on only a handful of questions.

Question 278

A service rolls a model upgrade to every tenant at once and cannot isolate regressions. Which change targets the constraint rather than a proxy metric?

  1. Under the reviewed approach, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
  2. For this production path, predefine sample size, guardrail metrics and stopping rules before testing.
  3. For the intended users, use shadow evaluation on sampled traffic with no user-facing side effects.
  4. Within the delivery plan, canary a small cohort, watch guardrails and preserve a rapid rollback path.
  5. As a candidate control, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.

Explanation

Correct Answer

D:

Progressive delivery limits blast radius and generates evidence before universal migration; reviewers would verify that a triggered regression returns the cohort to the prior approved version.

Incorrect Answers

A:

An experiment succeeds only if improved correctness justifies its full operational trade-off; it is justified if a new prompt improves accuracy but has markedly slower responses and higher cost, but the requested boundary concerns a service rolls a model upgrade to every tenant at once and cannot isolate regressions.

B:

Repeated early peeking increases the chance of overstating noisy apparent improvement; its scope includes cases where a team declares victory after an early positive result on only a handful of questions, not the present case where a service rolls a model upgrade to every tenant at once and cannot isolate regressions.

C:

Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; it targets cases where a regulated assistant must be assessed on real workflows without showing unreviewed output to users, while this problem involves a service rolls a model upgrade to every tenant at once and cannot isolate regressions.

E:

Confounded component changes prevent attributing a measured difference to the intended treatment; that belongs to situations where an A/B experiment compares outputs from different source-index versions by accident, rather than the risk that a service rolls a model upgrade to every tenant at once and cannot isolate regressions.

Question 279

An A/B experiment compares outputs from different source-index versions by accident. Which action is justified by the scenario?

  1. In the initial solution, randomize eligible traffic into comparable groups before drawing quality conclusions.
  2. In the proposed design, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
  3. Under the reviewed approach, inspect subgroup metrics and require segment-level safety thresholds before expansion.
  4. For this production path, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.

Explanation

Correct Answer

B:

Confounded component changes prevent attributing a measured difference to the intended treatment; the go-live check should establish that experiment metadata confirms equivalent index, tool and routing configuration in both arms.

Incorrect Answers

A:

Self-selection and role imbalance confound observed differences between variants; this technique fits where an experiment exposes version B only to expert users, then compares it with version A for everyone else, but the present constraint is that an A/B experiment compares outputs from different source-index versions by accident.

C:

Global improvement can mask uneven harm when deployment populations differ; the design helps when a successful improvement on common requests harms an underrepresented customer segment, whereas the release criterion addresses an A/B experiment compares outputs from different source-index versions by accident.

D:

An experiment succeeds only if improved correctness justifies its full operational trade-off; that fits cases where a new prompt improves accuracy but has markedly slower responses and higher cost, while the required safeguard concerns an A/B experiment compares outputs from different source-index versions by accident.

Question 280

A successful improvement on common requests harms an underrepresented customer segment. What change directly resolves this production-design issue?

  1. At this decision point, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
  2. As a candidate control, inspect subgroup metrics and require segment-level safety thresholds before expansion.
  3. In the initial solution, canary a small cohort, watch guardrails and preserve a rapid rollback path.
  4. As a scoped first step, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.

Explanation

Correct Answer

B:

Global improvement can mask uneven harm when deployment populations differ; validation needs to show that the rollout gate blocks promotion if a critical subgroup exceeds its failure limit.

Incorrect Answers

A:

An experiment succeeds only if improved correctness justifies its full operational trade-off; this applies when a new prompt improves accuracy but has markedly slower responses and higher cost, but here a successful improvement on common requests harms an underrepresented customer segment.

C:

Progressive delivery limits blast radius and generates evidence before universal migration; it is justified if a service rolls a model upgrade to every tenant at once and cannot isolate regressions, not the case in which a successful improvement on common requests harms an underrepresented customer segment.

D:

Confounded component changes prevent attributing a measured difference to the intended treatment; its scope includes cases where an A/B experiment compares outputs from different source-index versions by accident, whereas the problem presented is that a successful improvement on common requests harms an underrepresented customer segment.

Claude Certified Architect - Professional (CCAR-P) Practice Tests