Question 270
A numeric field extractor receives high subjective ratings despite returning wrong amounts. What evidence would reveal whether the control truly works?
- A production-like test shows that the final holdout remains unseen during optimization and spans meaningful task classes.
- The release check confirms that exact-value scores match curated annotations and escalate unreadable entries.
- The reviewer observes that trace assertions reject unauthorized calls even if the final summary looks correct.
- The release report verifies that the suite catches attempts to override instructions through retrieved text and tool results.
Explanation
Correct Answer
B:Machine-checkable targets should not depend on a subjective holistic grading prompt; the test is necessary where a financial extraction system contains fields that can be checked exactly against source forms.
Incorrect Answers
A:Training and tuning on the only evaluation examples inflates perceived performance; this makes sense when a prototype is evaluated on the same ten easy cases used during prompt development, rather than the risk that a numeric field extractor receives high subjective ratings despite returning wrong amounts.
C:A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; the trigger would be that an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior, while the immediate question involves a numeric field extractor receives high subjective ratings despite returning wrong amounts.
D:Adversarial cases test the boundary between source content and model/tool authority; the design helps when a safety suite has only benign requests although real users supply untrusted retrieved content, but the present constraint is that a numeric field extractor receives high subjective ratings despite returning wrong amounts.
Question 271
A multi-step agent gets the right final number after using an unapproved external tool. Which result proves the requested control, not just configuration?
- The observed outcome is that freshly adjudicated samples represent current policies without rewriting historical scores.
- The regression suite verifies that the final holdout remains unseen during optimization and spans meaningful task classes.
- A production-like test shows that independent reviewers achieve acceptable agreement on a calibrated sample.
- The release check confirms that trace assertions reject unauthorized calls even if the final summary looks correct.
Explanation
Correct Answer
D:A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; the test covers the original risk that an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior.
Incorrect Answers
A:Data drift changes the validity of both expected answers and apparent model quality; the other workload is one where a dataset was collected last year while policy documents and user terminology have changed, not the risk where a multi-step agent gets the right final number after using an unapproved external tool.
B:Training and tuning on the only evaluation examples inflates perceived performance; this applies when a prototype is evaluated on the same ten easy cases used during prompt development, not this situation where a multi-step agent gets the right final number after using an unapproved external tool.
C:Rubric alignment is necessary before evaluation labels can support confident architectural decisions; the alternative helps if two human reviewers disagree widely on whether an answer is accurate, whereas this scenario involves a multi-step agent gets the right final number after using an unapproved external tool.
Question 272
A safety regression suite passes until malicious instructions arrive inside retrieved wiki pages. Which acceptance result provides the most specific evidence?
- The scenario test demonstrates that the suite catches attempts to override instructions through retrieved text and tool results.
- The end-to-end check confirms that disagreement analysis identifies style-driven false passes before using the grader as a gate.
- The observed outcome is that independent reviewers achieve acceptable agreement on a calibrated sample.
- The regression suite verifies that exact-value scores match curated annotations and escalate unreadable entries.
- A production-like test shows that freshly adjudicated samples represent current policies without rewriting historical scores.
Explanation
Correct Answer
A:Adversarial cases test the boundary between source content and model/tool authority; the control should hold even if a safety suite has only benign requests although real users supply untrusted retrieved content.
Incorrect Answers
B:A judge model can be biased toward style, so its scores need independent validation; the other case is when an LLM grader rewards verbose answers even when specific facts are wrong, but the actual risk is that a safety regression suite passes until malicious instructions arrive inside retrieved wiki pages.
C:Rubric alignment is necessary before evaluation labels can support confident architectural decisions; this makes sense when two human reviewers disagree widely on whether an answer is accurate, but the requested boundary concerns a safety regression suite passes until malicious instructions arrive inside retrieved wiki pages.
D:Machine-checkable targets should not depend on a subjective holistic grading prompt; this technique fits where a financial extraction system contains fields that can be checked exactly against source forms, not the present case where a safety regression suite passes until malicious instructions arrive inside retrieved wiki pages.
E:Data drift changes the validity of both expected answers and apparent model quality; the trigger would be that a dataset was collected last year while policy documents and user terminology have changed, while this problem involves a safety regression suite passes until malicious instructions arrive inside retrieved wiki pages.
Question 273
A once-representative benchmark contains obsolete terminology and misses new policy scenarios. Which acceptance check evaluates this requirement rather than a proxy?
- A controlled trial finds that exact-value scores match curated annotations and escalate unreadable entries.
- The measured evidence confirms that the final holdout remains unseen during optimization and spans meaningful task classes.
- The scenario test demonstrates that freshly adjudicated samples represent current policies without rewriting historical scores.
- The end-to-end check confirms that independent reviewers achieve acceptable agreement on a calibrated sample.
Explanation
Correct Answer
C:Data drift changes the validity of both expected answers and apparent model quality; the safeguard must work when a dataset was collected last year while policy documents and user terminology have changed.
Incorrect Answers
A:Machine-checkable targets should not depend on a subjective holistic grading prompt; it serves a setting where a financial extraction system contains fields that can be checked exactly against source forms, rather than the condition that a once-representative benchmark contains obsolete terminology and misses new policy scenarios.
B:Training and tuning on the only evaluation examples inflates perceived performance; that is defensible if a prototype is evaluated on the same ten easy cases used during prompt development, while the immediate question involves a once-representative benchmark contains obsolete terminology and misses new policy scenarios.
D:Rubric alignment is necessary before evaluation labels can support confident architectural decisions; this applies when two human reviewers disagree widely on whether an answer is accurate, whereas the current issue is that a once-representative benchmark contains obsolete terminology and misses new policy scenarios.
Question 274
An experiment exposes version B only to expert users, then compares it with version A for everyone else. What intervention best matches the stated boundary?
- In the project scope, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
- Within the rollout design, canary a small cohort, watch guardrails and preserve a rapid rollback path.
- For the next architecture revision, use shadow evaluation on sampled traffic with no user-facing side effects.
- For the current integration, randomize eligible traffic into comparable groups before drawing quality conclusions.
Explanation
Correct Answer
D:Self-selection and role imbalance confound observed differences between variants; validation needs to show that group balance checks verify comparable case mix, user roles and task difficulty.
Incorrect Answers
A:Confounded component changes prevent attributing a measured difference to the intended treatment; that belongs to situations where an A/B experiment compares outputs from different source-index versions by accident, not this situation where an experiment exposes version B only to expert users, then compares it with version A for everyone else.
B:Progressive delivery limits blast radius and generates evidence before universal migration; it is relevant if a service rolls a model upgrade to every tenant at once and cannot isolate regressions, whereas this scenario involves an experiment exposes version B only to expert users, then compares it with version A for everyone else.
C:Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; it serves conditions where a regulated assistant must be assessed on real workflows without showing unreviewed output to users, but here an experiment exposes version B only to expert users, then compares it with version A for everyone else.
Question 275
A new prompt improves accuracy but has markedly slower responses and higher cost. Which implementation decision is the strongest fit?
- For this workflow, randomize eligible traffic into comparable groups before drawing quality conclusions.
- For the affected service, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
- In the project scope, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
- Within the rollout design, predefine sample size, guardrail metrics and stopping rules before testing.
Explanation
Correct Answer
C:An experiment succeeds only if improved correctness justifies its full operational trade-off; production testing should demonstrate that the report shows task gains and service impacts before promotion.
Incorrect Answers
A:Self-selection and role imbalance confound observed differences between variants; this control is needed when an experiment exposes version B only to expert users, then compares it with version A for everyone else, but the actual risk is that a new prompt improves accuracy but has markedly slower responses and higher cost.
B:Confounded component changes prevent attributing a measured difference to the intended treatment; that would be useful if an A/B experiment compares outputs from different source-index versions by accident, but the requested boundary concerns a new prompt improves accuracy but has markedly slower responses and higher cost.
D:Repeated early peeking increases the chance of overstating noisy apparent improvement; that is defensible if a team declares victory after an early positive result on only a handful of questions, while this problem involves a new prompt improves accuracy but has markedly slower responses and higher cost.
Question 276
A regulated assistant must be assessed on real workflows without showing unreviewed output to users. What should be prioritized during architecture review?
- At the system boundary, predefine sample size, guardrail metrics and stopping rules before testing.
- For the primary workstream, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
- For this workflow, use shadow evaluation on sampled traffic with no user-facing side effects.
- For the affected service, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
Explanation
Correct Answer
C:Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; the go-live check should establish that the shadow variant writes no records and emits only controlled diagnostic artifacts.
Incorrect Answers
A:Repeated early peeking increases the chance of overstating noisy apparent improvement; it targets cases where a team declares victory after an early positive result on only a handful of questions, while the immediate question involves a regulated assistant must be assessed on real workflows without showing unreviewed output to users.
B:Confounded component changes prevent attributing a measured difference to the intended treatment; the approach helps when an A/B experiment compares outputs from different source-index versions by accident, but the present constraint is that a regulated assistant must be assessed on real workflows without showing unreviewed output to users.
D:An experiment succeeds only if improved correctness justifies its full operational trade-off; it is relevant if a new prompt improves accuracy but has markedly slower responses and higher cost, whereas the release criterion addresses a regulated assistant must be assessed on real workflows without showing unreviewed output to users.
Question 277
A team declares victory after an early positive result on only a handful of questions. Which response follows from the actual operational constraint?
- For the intended users, predefine sample size, guardrail metrics and stopping rules before testing.
- Within the delivery plan, randomize eligible traffic into comparable groups before drawing quality conclusions.
- At the system boundary, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
- For the primary workstream, use shadow evaluation on sampled traffic with no user-facing side effects.
Explanation
Correct Answer
A:Repeated early peeking increases the chance of overstating noisy apparent improvement; acceptance evidence must confirm that the completed experiment reports confidence and follows its registered decision criteria.
Incorrect Answers
B:Self-selection and role imbalance confound observed differences between variants; that fits cases where an experiment exposes version B only to expert users, then compares it with version A for everyone else, but here a team declares victory after an early positive result on only a handful of questions.
C:Confounded component changes prevent attributing a measured difference to the intended treatment; this control is needed when an A/B experiment compares outputs from different source-index versions by accident, instead of when a team declares victory after an early positive result on only a handful of questions.
D:Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; that would be useful if a regulated assistant must be assessed on real workflows without showing unreviewed output to users, not the case in which a team declares victory after an early positive result on only a handful of questions.
Question 278
A service rolls a model upgrade to every tenant at once and cannot isolate regressions. Which change targets the constraint rather than a proxy metric?
- Under the reviewed approach, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
- For this production path, predefine sample size, guardrail metrics and stopping rules before testing.
- For the intended users, use shadow evaluation on sampled traffic with no user-facing side effects.
- Within the delivery plan, canary a small cohort, watch guardrails and preserve a rapid rollback path.
- As a candidate control, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
Explanation
Correct Answer
D:Progressive delivery limits blast radius and generates evidence before universal migration; reviewers would verify that a triggered regression returns the cohort to the prior approved version.
Incorrect Answers
A:An experiment succeeds only if improved correctness justifies its full operational trade-off; it is justified if a new prompt improves accuracy but has markedly slower responses and higher cost, but the requested boundary concerns a service rolls a model upgrade to every tenant at once and cannot isolate regressions.
B:Repeated early peeking increases the chance of overstating noisy apparent improvement; its scope includes cases where a team declares victory after an early positive result on only a handful of questions, not the present case where a service rolls a model upgrade to every tenant at once and cannot isolate regressions.
C:Shadow comparisons collect realistic evidence while keeping candidate behavior out of the production decision path; it targets cases where a regulated assistant must be assessed on real workflows without showing unreviewed output to users, while this problem involves a service rolls a model upgrade to every tenant at once and cannot isolate regressions.
E:Confounded component changes prevent attributing a measured difference to the intended treatment; that belongs to situations where an A/B experiment compares outputs from different source-index versions by accident, rather than the risk that a service rolls a model upgrade to every tenant at once and cannot isolate regressions.
Question 279
An A/B experiment compares outputs from different source-index versions by accident. Which action is justified by the scenario?
- In the initial solution, randomize eligible traffic into comparable groups before drawing quality conclusions.
- In the proposed design, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
- Under the reviewed approach, inspect subgroup metrics and require segment-level safety thresholds before expansion.
- For this production path, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
Explanation
Correct Answer
B:Confounded component changes prevent attributing a measured difference to the intended treatment; the go-live check should establish that experiment metadata confirms equivalent index, tool and routing configuration in both arms.
Incorrect Answers
A:Self-selection and role imbalance confound observed differences between variants; this technique fits where an experiment exposes version B only to expert users, then compares it with version A for everyone else, but the present constraint is that an A/B experiment compares outputs from different source-index versions by accident.
C:Global improvement can mask uneven harm when deployment populations differ; the design helps when a successful improvement on common requests harms an underrepresented customer segment, whereas the release criterion addresses an A/B experiment compares outputs from different source-index versions by accident.
D:An experiment succeeds only if improved correctness justifies its full operational trade-off; that fits cases where a new prompt improves accuracy but has markedly slower responses and higher cost, while the required safeguard concerns an A/B experiment compares outputs from different source-index versions by accident.
Question 280
A successful improvement on common requests harms an underrepresented customer segment. What change directly resolves this production-design issue?
- At this decision point, evaluate the change against jointly agreed quality, latency and unit-cost acceptance thresholds.
- As a candidate control, inspect subgroup metrics and require segment-level safety thresholds before expansion.
- In the initial solution, canary a small cohort, watch guardrails and preserve a rapid rollback path.
- As a scoped first step, hold retrieval corpus and evaluation conditions constant when isolating the prompt change.
Explanation
Correct Answer
B:Global improvement can mask uneven harm when deployment populations differ; validation needs to show that the rollout gate blocks promotion if a critical subgroup exceeds its failure limit.
Incorrect Answers
A:An experiment succeeds only if improved correctness justifies its full operational trade-off; this applies when a new prompt improves accuracy but has markedly slower responses and higher cost, but here a successful improvement on common requests harms an underrepresented customer segment.
C:Progressive delivery limits blast radius and generates evidence before universal migration; it is justified if a service rolls a model upgrade to every tenant at once and cannot isolate regressions, not the case in which a successful improvement on common requests harms an underrepresented customer segment.
D:Confounded component changes prevent attributing a measured difference to the intended treatment; its scope includes cases where an A/B experiment compares outputs from different source-index versions by accident, whereas the problem presented is that a successful improvement on common requests harms an underrepresented customer segment.