DESIGNING EVALUATIONS (EVALS)

PRACTICE LAB CONNECTION In the Resume Review Agent lab (Chapter 68), you will build an eval dataset for the agent, and you will include at least one adversarial case, such as a resume file containing a hidden instruction. Write golden answers carefully, label each case as normal, edge, or adversarial, and then test the prompts below against the dataset you built. ARTIFICIAL INTELLIGENCE ASSISTANT PROMPTS [START OF PROMPT] Draft a 10-to-15-item evaluation dataset (test input, golden reference answer, case type: normal/edge/adversarial) for this failure mode and task: task = [TASK DESCRIPTION, e.g. "match resume to job posting"], failure mode to probe = [FAILURE MODE, e.g. "agent fabricates a skill not in the resume" or "agent follows an injected instruction hidden in a resume"]. Use this sample data if relevant: [PASTE LAB SAMPLE RESUMES / JOB POSTINGS HERE]. Output format: a dataset table (ID, Input, Golden Answer, Case Type). [END OF PROMPT] [START OF PROMPT] This agent reached the correct final outcome but the path logged here looks inefficient or risky: [PASTE LAB OR PRODUCTION TRACE SHOWING THE PATH TAKEN HERE]. Design one step-level metric that would catch this problem even when the task-success metric alone would call the run a pass. Output format: metric name, what it measures, pass/fail rule, and a one-line note on how it would have flagged this specific trace. [END OF PROMPT]