THE EVALUATION HARNESS

ARTIFICIAL INTELLIGENCE ASSISTANT PROMPTS These prompts are meant for you to copy into an AI assistant and adapt to your own agent. Replace the text in square brackets with your own details. [START OF PROMPT] Design one evaluation check at each of the four granularity levels (span, trace, session, outcome) for this agent: [AGENT DESCRIPTION, e.g. Resume Review Agent]. For each check, specify exactly what is measured, the pass/fail rule, and what data source (lab trace file, eval set) it reads from. Output format: a table (Granularity, Check Description, Pass/Fail Rule, Data Source). [END OF PROMPT] [START OF PROMPT] Design a combined offline/online evaluation strategy for this agent before and after its next release: [AGENT DESCRIPTION AND RELEASE CHANGE]. Specify what the offline eval suite checks pre-release, what the online evaluation samples and checks post-release, and the specific threshold that would trigger a rollback. Output format: a strategy spec with separate offline and online sections and a rollback-trigger rule. [END OF PROMPT]