Question 260
A prototype is evaluated on the same ten easy cases used during prompt development. Which action addresses the root concern without unnecessary scope?
- For the current integration, add prompt-injection, exfiltration and malformed-input probes to the evaluation corpus.
- Under the system plan, create an adjudicated scoring rubric with examples and inter-rater checks.
- As the design priority, build a withheld representative set with routine, edge and adversarial cases.
- In the implementation sequence, calibrate model-based grading against expert labels and add deterministic fact checks where possible.
- For the operating team, evaluate traces and tool transitions in addition to the final user-facing output.
Explanation
Correct Answer
C:Training and tuning on the only evaluation examples inflates perceived performance; one operational check should confirm that the final holdout remains unseen during optimization and spans meaningful task classes.
Incorrect Answers
A:Adversarial cases test the boundary between source content and model/tool authority; this applies when a safety suite has only benign requests although real users supply untrusted retrieved content, but here a prototype is evaluated on the same ten easy cases used during prompt development.
B:Rubric alignment is necessary before evaluation labels can support confident architectural decisions; the alternative helps if two human reviewers disagree widely on whether an answer is accurate, instead of when a prototype is evaluated on the same ten easy cases used during prompt development.
D:A judge model can be biased toward style, so its scores need independent validation; its scope includes cases where an LLM grader rewards verbose answers even when specific facts are wrong, whereas the problem presented is that a prototype is evaluated on the same ten easy cases used during prompt development.
E:A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; it targets cases where an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior, but this decision concerns a prototype is evaluated on the same ten easy cases used during prompt development.
Question 261
Two human reviewers disagree widely on whether an answer is accurate. What is the most appropriate initial architecture decision?
- Within the rollout design, calibrate model-based grading against expert labels and add deterministic fact checks where possible.
- For the next architecture revision, create an adjudicated scoring rubric with examples and inter-rater checks.
- At the system boundary, add prompt-injection, exfiltration and malformed-input probes to the evaluation corpus.
- Under the system plan, version evaluation sets and refresh representative samples through a governed review process.
Explanation
Correct Answer
B:Rubric alignment is necessary before evaluation labels can support confident architectural decisions; the practical acceptance condition is that independent reviewers achieve acceptable agreement on a calibrated sample.
Incorrect Answers
A:A judge model can be biased toward style, so its scores need independent validation; the other case is when an LLM grader rewards verbose answers even when specific facts are wrong, not the present case where two human reviewers disagree widely on whether an answer is accurate.
C:Adversarial cases test the boundary between source content and model/tool authority; this technique fits where a safety suite has only benign requests although real users supply untrusted retrieved content, and not where two human reviewers disagree widely on whether an answer is accurate.
D:Data drift changes the validity of both expected answers and apparent model quality; the trigger would be that a dataset was collected last year while policy documents and user terminology have changed, rather than the risk that two human reviewers disagree widely on whether an answer is accurate.
Question 262
An LLM grader rewards verbose answers even when specific facts are wrong. Which architecture change should be approved first?
- For the affected service, build a withheld representative set with routine, edge and adversarial cases.
- In the project scope, calibrate model-based grading against expert labels and add deterministic fact checks where possible.
- Within the rollout design, version evaluation sets and refresh representative samples through a governed review process.
- For the next architecture revision, evaluate traces and tool transitions in addition to the final user-facing output.
Explanation
Correct Answer
B:A judge model can be biased toward style, so its scores need independent validation; one measurable outcome is that disagreement analysis identifies style-driven false passes before using the grader as a gate.
Incorrect Answers
A:Training and tuning on the only evaluation examples inflates perceived performance; that is defensible if a prototype is evaluated on the same ten easy cases used during prompt development, whereas the current issue is that an LLM grader rewards verbose answers even when specific facts are wrong.
C:Data drift changes the validity of both expected answers and apparent model quality; this applies when a dataset was collected last year while policy documents and user terminology have changed, while the required safeguard concerns an LLM grader rewards verbose answers even when specific facts are wrong.
D:A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; the alternative helps if an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior, not the risk where an LLM grader rewards verbose answers even when specific facts are wrong.
Question 263
A financial extraction system contains fields that can be checked exactly against source forms. What should be incorporated into the approved design?
- For the primary workstream, use deterministic field-level tests for exact values and human review for ambiguous interpretation.
- For this workflow, build a withheld representative set with routine, edge and adversarial cases.
- For the affected service, calibrate model-based grading against expert labels and add deterministic fact checks where possible.
- In the project scope, version evaluation sets and refresh representative samples through a governed review process.
Explanation
Correct Answer
A:Machine-checkable targets should not depend on a subjective holistic grading prompt; validation needs to show that exact-value scores match curated annotations and escalate unreadable entries.
Incorrect Answers
B:Training and tuning on the only evaluation examples inflates perceived performance; it serves conditions where a prototype is evaluated on the same ten easy cases used during prompt development, not the case in which a financial extraction system contains fields that can be checked exactly against source forms.
C:A judge model can be biased toward style, so its scores need independent validation; the other case is when an LLM grader rewards verbose answers even when specific facts are wrong, whereas the problem presented is that a financial extraction system contains fields that can be checked exactly against source forms.
D:Data drift changes the validity of both expected answers and apparent model quality; this makes sense when a dataset was collected last year while policy documents and user terminology have changed, but this decision concerns a financial extraction system contains fields that can be checked exactly against source forms.
Question 264
An agent follows multi-step workflows and a final-answer-only test misses risky tool behavior. Which approach should the design team choose?
- Within the delivery plan, version evaluation sets and refresh representative samples through a governed review process.
- At the system boundary, evaluate traces and tool transitions in addition to the final user-facing output.
- In the initial solution, use deterministic field-level tests for exact values and human review for ambiguous interpretation.
- In the proposed design, build a withheld representative set with routine, edge and adversarial cases.
Explanation
Correct Answer
B:A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; reviewers would verify that trace assertions reject unauthorized calls even if the final summary looks correct.
Incorrect Answers
A:Data drift changes the validity of both expected answers and apparent model quality; that would be useful if a dataset was collected last year while policy documents and user terminology have changed, while this problem involves an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior.
C:Machine-checkable targets should not depend on a subjective holistic grading prompt; that is defensible if a financial extraction system contains fields that can be checked exactly against source forms, rather than the risk that an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior.
D:Training and tuning on the only evaluation examples inflates perceived performance; the other workload is one where a prototype is evaluated on the same ten easy cases used during prompt development, rather than the condition that an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior.
Question 265
A safety suite has only benign requests although real users supply untrusted retrieved content. Where should the solution architect focus the implementation?
- For this production path, use deterministic field-level tests for exact values and human review for ambiguous interpretation.
- For the intended users, add prompt-injection, exfiltration and malformed-input probes to the evaluation corpus.
- Within the delivery plan, build a withheld representative set with routine, edge and adversarial cases.
- At the system boundary, calibrate model-based grading against expert labels and add deterministic fact checks where possible.
- For the primary workstream, create an adjudicated scoring rubric with examples and inter-rater checks.
Explanation
Correct Answer
B:Adversarial cases test the boundary between source content and model/tool authority; the solution is testable when the suite catches attempts to override instructions through retrieved text and tool results.
Incorrect Answers
A:Machine-checkable targets should not depend on a subjective holistic grading prompt; the approach helps when a financial extraction system contains fields that can be checked exactly against source forms, whereas the release criterion addresses a safety suite has only benign requests although real users supply untrusted retrieved content.
C:Training and tuning on the only evaluation examples inflates perceived performance; it is relevant if a prototype is evaluated on the same ten easy cases used during prompt development, not the risk where a safety suite has only benign requests although real users supply untrusted retrieved content.
D:A judge model can be biased toward style, so its scores need independent validation; it serves conditions where an LLM grader rewards verbose answers even when specific facts are wrong, not this situation where a safety suite has only benign requests although real users supply untrusted retrieved content.
E:Rubric alignment is necessary before evaluation labels can support confident architectural decisions; the other case is when two human reviewers disagree widely on whether an answer is accurate, whereas this scenario involves a safety suite has only benign requests although real users supply untrusted retrieved content.
Question 266
A dataset was collected last year while policy documents and user terminology have changed. What revision would make the proposal more defensible?
- As a scoped first step, build a withheld representative set with routine, edge and adversarial cases.
- Under the reviewed approach, version evaluation sets and refresh representative samples through a governed review process.
- For this production path, create an adjudicated scoring rubric with examples and inter-rater checks.
- In the proposed implementation, add prompt-injection, exfiltration and malformed-input probes to the evaluation corpus.
Explanation
Correct Answer
B:Data drift changes the validity of both expected answers and apparent model quality; the intended outcome is that freshly adjudicated samples represent current policies without rewriting historical scores.
Incorrect Answers
A:Training and tuning on the only evaluation examples inflates perceived performance; that fits cases where a prototype is evaluated on the same ten easy cases used during prompt development, not the case in which a dataset was collected last year while policy documents and user terminology have changed.
C:Rubric alignment is necessary before evaluation labels can support confident architectural decisions; that would be useful if two human reviewers disagree widely on whether an answer is accurate, but this decision concerns a dataset was collected last year while policy documents and user terminology have changed.
D:Adversarial cases test the boundary between source content and model/tool authority; it serves a setting where a safety suite has only benign requests although real users supply untrusted retrieved content, but the actual risk is that a dataset was collected last year while policy documents and user terminology have changed.
Question 267
Prompt authors can predict every evaluation case because the test set was copied into examples. What outcome must the reviewer see before proceeding?
- The signed test record confirms that disagreement analysis identifies style-driven false passes before using the grader as a gate.
- The sign-off record demonstrates that freshly adjudicated samples represent current policies without rewriting historical scores.
- The acceptance evidence shows that exact-value scores match curated annotations and escalate unreadable entries.
- The pilot evaluation finds that the suite catches attempts to override instructions through retrieved text and tool results.
- The comparison run confirms that the final holdout remains unseen during optimization and spans meaningful task classes.
Explanation
Correct Answer
E:Training and tuning on the only evaluation examples inflates perceived performance; the safeguard must work when a prototype is evaluated on the same ten easy cases used during prompt development.
Incorrect Answers
A:A judge model can be biased toward style, so its scores need independent validation; its scope includes cases where an LLM grader rewards verbose answers even when specific facts are wrong, and not where prompt authors can predict every evaluation case because the test set was copied into examples.
B:Data drift changes the validity of both expected answers and apparent model quality; it targets cases where a dataset was collected last year while policy documents and user terminology have changed, rather than the risk that prompt authors can predict every evaluation case because the test set was copied into examples.
C:Machine-checkable targets should not depend on a subjective holistic grading prompt; the approach helps when a financial extraction system contains fields that can be checked exactly against source forms, rather than the condition that prompt authors can predict every evaluation case because the test set was copied into examples.
D:Adversarial cases test the boundary between source content and model/tool authority; that belongs to situations where a safety suite has only benign requests although real users supply untrusted retrieved content, while the immediate question involves prompt authors can predict every evaluation case because the test set was copied into examples.
Question 268
Two graders label the same refusal differently because the rubric lacks severity definitions. Which result should the pilot review prioritize?
- The controlled replay verifies that freshly adjudicated samples represent current policies without rewriting historical scores.
- A reproducible run shows that exact-value scores match curated annotations and escalate unreadable entries.
- The signed test record confirms that the suite catches attempts to override instructions through retrieved text and tool results.
- The sign-off record demonstrates that independent reviewers achieve acceptable agreement on a calibrated sample.
Explanation
Correct Answer
D:Rubric alignment is necessary before evaluation labels can support confident architectural decisions; it demonstrates the required control when two human reviewers disagree widely on whether an answer is accurate.
Incorrect Answers
A:Data drift changes the validity of both expected answers and apparent model quality; the trigger would be that a dataset was collected last year while policy documents and user terminology have changed, while the required safeguard concerns two graders label the same refusal differently because the rubric lacks severity definitions.
B:Machine-checkable targets should not depend on a subjective holistic grading prompt; the design helps when a financial extraction system contains fields that can be checked exactly against source forms, not the risk where two graders label the same refusal differently because the rubric lacks severity definitions.
C:Adversarial cases test the boundary between source content and model/tool authority; that fits cases where a safety suite has only benign requests although real users supply untrusted retrieved content, not this situation where two graders label the same refusal differently because the rubric lacks severity definitions.
Question 269
An automatic answer judge gives bonus marks to a polished but factually wrong response. What result would directly validate the necessary safeguard?
- The reviewer observes that the suite catches attempts to override instructions through retrieved text and tool results.
- The release report verifies that freshly adjudicated samples represent current policies without rewriting historical scores.
- The controlled replay verifies that exact-value scores match curated annotations and escalate unreadable entries.
- A reproducible run shows that disagreement analysis identifies style-driven false passes before using the grader as a gate.
Explanation
Correct Answer
D:A judge model can be biased toward style, so its scores need independent validation; a valid sign-off must address the risk that an LLM grader rewards verbose answers even when specific facts are wrong.
Incorrect Answers
A:Adversarial cases test the boundary between source content and model/tool authority; the alternative helps if a safety suite has only benign requests although real users supply untrusted retrieved content, whereas the problem presented is that an automatic answer judge gives bonus marks to a polished but factually wrong response.
B:Data drift changes the validity of both expected answers and apparent model quality; it is justified if a dataset was collected last year while policy documents and user terminology have changed, but this decision concerns an automatic answer judge gives bonus marks to a polished but factually wrong response.
C:Machine-checkable targets should not depend on a subjective holistic grading prompt; its scope includes cases where a financial extraction system contains fields that can be checked exactly against source forms, but the actual risk is that an automatic answer judge gives bonus marks to a polished but factually wrong response.
Question 270
A numeric field extractor receives high subjective ratings despite returning wrong amounts. What evidence would reveal whether the control truly works?
- A production-like test shows that the final holdout remains unseen during optimization and spans meaningful task classes.
- The release check confirms that exact-value scores match curated annotations and escalate unreadable entries.
- The reviewer observes that trace assertions reject unauthorized calls even if the final summary looks correct.
- The release report verifies that the suite catches attempts to override instructions through retrieved text and tool results.
Explanation
Correct Answer
B:Machine-checkable targets should not depend on a subjective holistic grading prompt; the test is necessary where a financial extraction system contains fields that can be checked exactly against source forms.
Incorrect Answers
A:Training and tuning on the only evaluation examples inflates perceived performance; this makes sense when a prototype is evaluated on the same ten easy cases used during prompt development, rather than the risk that a numeric field extractor receives high subjective ratings despite returning wrong amounts.
C:A seemingly correct answer can conceal forbidden steps or harmful intermediate side effects; the trigger would be that an agent follows multi-step workflows and a final-answer-only test misses risky tool behavior, while the immediate question involves a numeric field extractor receives high subjective ratings despite returning wrong amounts.
D:Adversarial cases test the boundary between source content and model/tool authority; the design helps when a safety suite has only benign requests although real users supply untrusted retrieved content, but the present constraint is that a numeric field extractor receives high subjective ratings despite returning wrong amounts.