Anthropic CCAR-P (Claude Certified Architect - Professional) Practice Test 29

Question 280

A successful improvement on common requests harms an underrepresented customer segment. What change directly resolves this production-design issue?

  1. At this decision point, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
  2. As a candidate control, inspect subgroup metrics and require segment-level safety thresholds before expansion.
  3. In the initial solution, canary a small cohort, watch guardrails and preserve a rapid rollback path.
  4. As a scoped first step, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.

Explanation

Correct Answer

B:

Global improvement can mask uneven harm when deployment populations differ; validation needs to show that the rollout gate blocks promotion if a critical subgroup exceeds its failure limit.

Incorrect Answers

A:

An experiment succeeds only if improved correctness justifies its full operational trade-off; this applies when a new prompt improves accuracy but has markedly slower responses and higher cost, but here a successful improvement on common requests harms an underrepresented customer segment.

C:

Progressive delivery limits blast radius and generates evidence before universal migration; it is justified if a service rolls a model upgrade to every tenant at once and cannot isolate regressions, not the case in which a successful improvement on common requests harms an underrepresented customer segment.

D:

Confounded component changes prevent attributing a measured difference to the intended treatment; its scope includes cases where an A/B experiment compares outputs from different source-index versions by accident, whereas the problem presented is that a successful improvement on common requests harms an underrepresented customer segment.

Question 281

A candidate rollout appears better only because it was tested with a more experienced customer cohort. Which finding should the release gate require?

  1. The release report verifies that the report shows task gains and service impacts before promotion.
  2. The controlled replay verifies that a triggered regression returns the cohort to the prior approved version.
  3. A reproducible run shows that group balance checks verify comparable case mix, user roles and task difficulty.
  4. The signed test record confirms that the completed experiment reports confidence and follows its registered decision criteria.

Explanation

Correct Answer

C:

Self-selection and role imbalance confound observed differences between variants; it is the decisive test when an experiment exposes version B only to expert users, then compares it with version A for everyone else.

Incorrect Answers

A:

An experiment succeeds only if improved correctness justifies its full operational trade-off; the other case is when a new prompt improves accuracy but has markedly slower responses and higher cost, not the present case where a candidate rollout appears better only because it was tested with a more experienced customer cohort.

B:

Progressive delivery limits blast radius and generates evidence before universal migration; this makes sense when a service rolls a model upgrade to every tenant at once and cannot isolate regressions, while this problem involves a candidate rollout appears better only because it was tested with a more experienced customer cohort.

D:

Repeated early peeking increases the chance of overstating noisy apparent improvement; the trigger would be that a team declares victory after an early positive result on only a handful of questions, rather than the risk that a candidate rollout appears better only because it was tested with a more experienced customer cohort.

Question 282

An accuracy gain is promoted despite exceeding the agreed cost and response-time ceilings. Which verified outcome most directly addresses the reported risk?

  1. The release check confirms that the rollout gate blocks promotion if a critical subgroup exceeds its failure limit.
  2. The reviewer observes that group balance checks verify comparable case mix, user roles and task difficulty.
  3. The scenario test demonstrates that the report shows task gains and service impacts before promotion.
  4. The controlled replay verifies that the completed experiment reports confidence and follows its registered decision criteria.

Explanation

Correct Answer

C:

An experiment succeeds only if improved correctness justifies its full operational trade-off; this is a functional check for the case where a new prompt improves accuracy but has markedly slower responses and higher cost.

Incorrect Answers

A:

Global improvement can mask uneven harm when deployment populations differ; that is defensible if a successful improvement on common requests harms an underrepresented customer segment, whereas the current issue is that an accuracy gain is promoted despite exceeding the agreed cost and response-time ceilings.

B:

Self-selection and role imbalance confound observed differences between variants; the other workload is one where an experiment exposes version B only to expert users, then compares it with version A for everyone else, whereas the release criterion addresses an accuracy gain is promoted despite exceeding the agreed cost and response-time ceilings.

D:

Repeated early peeking increases the chance of overstating noisy apparent improvement; the alternative helps if a team declares victory after an early positive result on only a handful of questions, not the risk where an accuracy gain is promoted despite exceeding the agreed cost and response-time ceilings.

Question 283

A regulated tool is trialed on operational records, but supervisors cannot expose its draft decisions to clients. What independently verifiable observation is needed?

  1. The regression suite verifies that a triggered regression returns the cohort to the prior approved version.
  2. A production-like test shows that group balance checks verify comparable case mix, user roles and task difficulty.
  3. The release check confirms that the report shows task gains and service impacts before promotion.
  4. The reviewer observes that the shadow variant writes no records and emits only controlled diagnostic artifacts.

Explanation

Correct Answer

D:

Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; the control should hold even if a regulated assistant must be assessed on real workflows without showing unreviewed output to users.

Incorrect Answers

A:

Progressive delivery limits blast radius and generates evidence before universal migration; it is relevant if a service rolls a model upgrade to every tenant at once and cannot isolate regressions, instead of when a regulated tool is trialed on operational records, but supervisors cannot expose its draft decisions to clients.

B:

Self-selection and role imbalance confound observed differences between variants; it serves conditions where an experiment exposes version B only to expert users, then compares it with version A for everyone else, not the case in which a regulated tool is trialed on operational records, but supervisors cannot expose its draft decisions to clients.

C:

An experiment succeeds only if improved correctness justifies its full operational trade-off; the other case is when a new prompt improves accuracy but has markedly slower responses and higher cost, whereas the problem presented is that a regulated tool is trialed on operational records, but supervisors cannot expose its draft decisions to clients.

Question 284

An early split-test result reverses direction after the sample grows and hard cases arrive. What outcome would demonstrate this requirement in operation?

  1. The end-to-end check confirms that experiment metadata confirms equivalent index, tool and routing configuration in both arms.
  2. The observed outcome is that the rollout gate blocks promotion if a critical subgroup exceeds its failure limit.
  3. The regression suite verifies that the report shows task gains and service impacts before promotion.
  4. A production-like test shows that the completed experiment reports confidence and follows its registered decision criteria.

Explanation

Correct Answer

D:

Repeated early peeking increases the chance of overstating noisy apparent improvement; the test is necessary where a team declares victory after an early positive result on only a handful of questions.

Incorrect Answers

A:

Confounded component changes prevent attributing a measured difference to the intended treatment; that would be useful if an A/B experiment compares outputs from different source-index versions by accident, while this problem involves an early split-test result reverses direction after the sample grows and hard cases arrive.

B:

Global improvement can mask uneven harm when deployment populations differ; it serves a setting where a successful improvement on common requests harms an underrepresented customer segment, and not where an early split-test result reverses direction after the sample grows and hard cases arrive.

C:

An experiment succeeds only if improved correctness justifies its full operational trade-off; that is defensible if a new prompt improves accuracy but has markedly slower responses and higher cost, rather than the risk that an early split-test result reverses direction after the sample grows and hard cases arrive.

Question 285

A model change breaks invoice extraction for one tenant as soon as it reaches everyone. Which observed test outcome would justify sign-off?

  1. The measured evidence confirms that group balance checks verify comparable case mix, user roles and task difficulty.
  2. The independent test establishes that the report shows task gains and service impacts before promotion.
  3. The end-to-end check confirms that the completed experiment reports confidence and follows its registered decision criteria.
  4. The observed outcome is that a triggered regression returns the cohort to the prior approved version.
  5. The regression suite verifies that the shadow variant writes no records and emits only controlled diagnostic artifacts.

Explanation

Correct Answer

D:

Progressive delivery limits blast radius and generates evidence before universal migration; the condition under test is that a service rolls a model upgrade to every tenant at once and cannot isolate regressions.

Incorrect Answers

A:

Self-selection and role imbalance confound observed differences between variants; the approach helps when an experiment exposes version B only to expert users, then compares it with version A for everyone else, whereas the release criterion addresses a model change breaks invoice extraction for one tenant as soon as it reaches everyone.

B:

An experiment succeeds only if improved correctness justifies its full operational trade-off; that belongs to situations where a new prompt improves accuracy but has markedly slower responses and higher cost, while the required safeguard concerns a model change breaks invoice extraction for one tenant as soon as it reaches everyone.

C:

Repeated early peeking increases the chance of overstating noisy apparent improvement; it is relevant if a team declares victory after an early positive result on only a handful of questions, not the risk where a model change breaks invoice extraction for one tenant as soon as it reaches everyone.

E:

Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; the other case is when a regulated assistant must be assessed on real workflows without showing unreviewed output to users, whereas this scenario involves a model change breaks invoice extraction for one tenant as soon as it reaches everyone.

Question 286

A prompt trial mixes a new retrieval index with the candidate prompt, obscuring the cause of improved results. Which sign-off finding best checks the actual boundary?

  1. The negative-case evaluation confirms that experiment metadata confirms equivalent index, tool and routing configuration in both arms.
  2. A controlled trial finds that the report shows task gains and service impacts before promotion.
  3. The measured evidence confirms that a triggered regression returns the cohort to the prior approved version.
  4. The scenario test demonstrates that the rollout gate blocks promotion if a critical subgroup exceeds its failure limit.

Explanation

Correct Answer

A:

Confounded component changes prevent attributing a measured difference to the intended treatment; the required safeguard is exercised when an A/B experiment compares outputs from different source-index versions by accident.

Incorrect Answers

B:

An experiment succeeds only if improved correctness justifies its full operational trade-off; this control is needed when a new prompt improves accuracy but has markedly slower responses and higher cost, whereas the problem presented is that a prompt trial mixes a new retrieval index with the candidate prompt, obscuring the cause of improved results.

C:

Progressive delivery limits blast radius and generates evidence before universal migration; that would be useful if a service rolls a model upgrade to every tenant at once and cannot isolate regressions, but this decision concerns a prompt trial mixes a new retrieval index with the candidate prompt, obscuring the cause of improved results.

D:

Global improvement can mask uneven harm when deployment populations differ; it serves a setting where a successful improvement on common requests harms an underrepresented customer segment, but the actual risk is that a prompt trial mixes a new retrieval index with the candidate prompt, obscuring the cause of improved results.

Question 287

A RAG assistant deteriorates immediately after a large document refresh, but prompts did not change. Which option supplies the needed architectural boundary?

  1. For the affected service, instrument no-evidence paths and enforce abstention or human escalation on missing support.
  2. In the project scope, inspect ingestion, indexing and retrieval results before rewriting the system prompt.
  3. Within the rollout design, diff prompt revisions and replay known failing cases using pinned component versions.
  4. For the next architecture revision, benchmark a more capable model against the same difficult holdout before adopting it.

Explanation

Correct Answer

B:

A time-correlated corpus update points first to source and evidence pipeline changes; its effect can be measured when a before-and-after trace finds missing or stale answer-bearing passages.

Incorrect Answers

A:

The failure is unsupported synthesis; more assertive wording would worsen the defect; its scope includes cases where an assistant makes confident factual claims when relevant documents cannot be retrieved, and not where a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change.

C:

Controlled replay distinguishes prompt regression from random variation or data changes; the approach helps when a production model begins giving different answers after a silent prompt template edit, rather than the condition that a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change.

D:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; that belongs to situations where a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts, while the immediate question involves a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change.

Question 288

A model repeatedly returns valid prose but violates a strict JSON consumer contract. Which action addresses the root concern without unnecessary scope?

  1. For the primary workstream, inspect ingestion, indexing and retrieval results before rewriting the system prompt.
  2. For this workflow, benchmark a more capable model against the same difficult holdout before adopting it.
  3. For the affected service, diff prompt revisions and replay known failing cases using pinned component versions.
  4. In the project scope, check schema instructions and output validation before changing model capability tiers.

Explanation

Correct Answer

D:

A formatting failure may reflect an underspecified interface rather than inadequate reasoning; acceptance evidence must confirm that schema stress cases isolate missing fields and invalid values reproducibly.

Incorrect Answers

A:

A time-correlated corpus update points first to source and evidence pipeline changes; the trigger would be that a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change, while the required safeguard concerns a model repeatedly returns valid prose but violates a strict JSON consumer contract.

B:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; the design helps when a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts, not the risk where a model repeatedly returns valid prose but violates a strict JSON consumer contract.

C:

Controlled replay distinguishes prompt regression from random variation or data changes; that fits cases where a production model begins giving different answers after a silent prompt template edit, not this situation where a model repeatedly returns valid prose but violates a strict JSON consumer contract.

Question 289

A Claude-powered tool workflow starts submitting invalid nested JSON just after a template update; its backend API is unchanged. The team needs to distinguish malformed tool argument construction from the untracked prompt change. Which TWO diagnostic actions directly isolate those causes? Choose TWO.

  1. Within the delivery plan, diff prompt revisions and replay known failing cases using pinned component versions.
  2. At the system boundary, inspect ingestion, indexing and retrieval results before rewriting the system prompt.
  3. For the primary workstream, evaluate grounding and enforce claim-to-source consistency during generation.
  4. For this workflow, instrument no-evidence paths and enforce abstention or human escalation on missing support.
  5. For the affected service, inspect tool schemas, examples and argument-validation errors at the invocation boundary.

Explanation

Correct Answers

A:

Controlled replay distinguishes prompt regression from random variation or data changes; the intended outcome is that old-version replay restores prior behavior and identifies the problematic instruction change; another obligation arises when an agent chooses the correct API but sends invalid arguments for a nested object.

E:

Precise parameter contracts control tool execution more directly than general answer-quality tuning; production testing should demonstrate that traces show which required argument or type failed and prevent external side effects; another obligation arises when a production model begins giving different answers after a silent prompt template edit.

Incorrect Answers

B:

A time-correlated corpus update points first to source and evidence pipeline changes; it is justified if a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change, but this decision concerns an agent chooses the correct API but sends invalid arguments for a nested object; another obligation arises when a production model begins giving different answers after a silent prompt template edit.

C:

Retrieval success alone does not guarantee faithful synthesis of the retrieved text; its scope includes cases where a retrieval bot finds the correct passage but the answer invents numbers not in that passage, but the actual risk is that a production model begins giving different answers after a silent prompt template edit; another obligation arises when an agent chooses the correct API but sends invalid arguments for a nested object.

D:

The failure is unsupported synthesis; more assertive wording would worsen the defect; it targets cases where an assistant makes confident factual claims when relevant documents cannot be retrieved, but the requested boundary concerns an agent chooses the correct API but sends invalid arguments for a nested object; another obligation arises when a production model begins giving different answers after a silent prompt template edit.

Question 290

An assistant makes confident factual claims when relevant documents cannot be retrieved. Which architecture change should be approved first?

  1. For this production path, instrument no-evidence paths and enforce abstention or human escalation on missing support.
  2. For the intended users, benchmark a more capable model against the same difficult holdout before adopting it.
  3. At this decision point, diff prompt revisions and replay known failing cases using pinned component versions.
  4. As a candidate control, inspect ingestion, indexing and retrieval results before rewriting the system prompt.
  5. For the primary workstream, inspect tool schemas, examples and argument-validation errors at the invocation boundary.

Explanation

Correct Answer

A:

The failure is unsupported synthesis; more assertive wording would worsen the defect; its effect can be measured when empty-retrieval tests yield acknowledged uncertainty rather than fabricated facts.

Incorrect Answers

B:

Persistent reasoning failure after evidence and prompt checks can be a model-capability mismatch; this technique fits where a task requires complex multi-document reconciliation, and a small model misses constraints despite clean prompts, rather than the condition that an assistant makes confident factual claims when relevant documents cannot be retrieved.

C:

Controlled replay distinguishes prompt regression from random variation or data changes; the trigger would be that a production model begins giving different answers after a silent prompt template edit, while the immediate question involves an assistant makes confident factual claims when relevant documents cannot be retrieved.

D:

A time-correlated corpus update points first to source and evidence pipeline changes; the design helps when a RAG assistant deteriorates immediately after a large document refresh, but prompts did not change, but the present constraint is that an assistant makes confident factual claims when relevant documents cannot be retrieved.

E:

Precise parameter contracts control tool execution more directly than general answer-quality tuning; that fits cases where an agent chooses the correct API but sends invalid arguments for a nested object, whereas the current issue is that an assistant makes confident factual claims when relevant documents cannot be retrieved.

Claude Certified Architect - Professional (CCAR-P) Practice Tests