APPENDIX H: RESUME PORTFOLIO BUILDING CAPSTONE PROJECTS

MICRO-STEPS Part A: Start the folder. 1. Create a new folder named project4_eval_suite and open a terminal inside it. Part B: Build the tracer. 2. Create a file named tracing.py. 3. Type or paste the following code into tracing.py and save it. File: tracing.py [START OF PYTHON CODE] # Import the json module to write structured data import json # Import the time module to measure how long things take import time # Define a class that records timed spans to a trace file class Tracer: # Define the constructor that remembers which file to write spans to def __init__(self, path): self.path = path # Define a method that records one span: its trace, name, duration, and extra attributes def record(self, trace_id, name, started, attrs): # Build the span record, calculating its duration in milliseconds from the start time span = { "trace_id": trace_id, "span": name, "duration_ms": round((time.perf_counter() - started) / 0.001, 3), "attrs": attrs, } # Open the trace file in append mode and write the span as one line of JSON with open(self.path, "a", encoding="utf-8") as f: f.write(json.dumps(span) + "\n") # Define a class that matches the Tracer's interface but records nothing, for when tracing is turned off class NullTracer: # Define a method that deliberately does nothing def record(self, trace_id, name, started, attrs): pass [END OF PYTHON CODE] The method record writes one span to a JSON Lines file, with the trace identifier, the span name, the duration in milliseconds, and a small dictionary of extra details. The class NullTracer does nothing, so that the harness can run without tracing when you do not need it. Part C: Build the target harness. 4. Create a file named target_harness.py. 5. Type or paste the following code into target_harness.py and save it. File: target_harness.py [START OF PYTHON CODE] # Import the json module to read and write structured data import json # Import the time module to measure how long each step takes import time # Import the stand-in tracer class to use when no real tracer is given from tracing import NullTracer # Define a function that estimates how many tokens a piece of text would use def guess_tokens(text): return max(1, len(text) // 4) # Define a function that runs the full task loop against a target model and a fixed set of files def run_task(task, model_fn, files, trace_id="run", tracer=None, max_steps=8): # Use the given tracer, or a do-nothing tracer if none was given tracer = tracer or NullTracer() # Make a working copy of the files so the originals are never changed store = dict(files) # Create an empty dictionary to hold anything the model writes out outputs = {} # Start the conversation with this task as the first user message messages = [{"role": "user", "content": task}] # Build the result dictionary that will be filled in and returned result = {"finished": False, "steps": 0, "blocked": 0, "tokens": 0, "outputs": outputs, "final": ""} # Repeat the think/act cycle up to the maximum number of steps for step in range(1, max_steps + 1): # Record how many steps have run so far result["steps"] = step # Mark the time just before asking the model, to measure the call's duration started = time.perf_counter() # Ask the model what to do next reply = model_fn(messages) # Estimate the tokens used by the conversation so far plus the new reply used = guess_tokens(json.dumps(messages)) + guess_tokens(json.dumps(reply)) # Add this step's tokens to the running total result["tokens"] += used # Record this model call as a timed span in the trace tracer.record(trace_id, "model_call", started, {"step": step, "tokens": used}) # If the model says the task is finished, store the final text and return the result if reply.get("action") == "finish": result["finished"] = True result["final"] = reply.get("text", "") return result # Otherwise, read out the requested tool name, arguments, and path name = reply.get("tool") args = reply.get("args", {}) path = str(args.get("path", "")) # Mark the time just before running the tool call, to measure its duration started = time.perf_counter() # If the path tries to climb out of its folder, block it if ".." in path: output = "BLOCKED: path climbs out of its folder" # If this is a listing request inside the docs folder, return the matching file names elif name == "list" and path.startswith("docs/"): output = "\n".join(sorted(k[len(path):] for k in store if k.startswith(path))) # If this is a read request for a file that exists inside the docs folder, return its contents elif name == "read" and path.startswith("docs/") and path in store: output = store[path] # If this is a write request inside the out folder, save the text and confirm it was written elif name == "write" and path.startswith("out/"): outputs[path] = str(args.get("text", "")) output = "written" # Otherwise, the tool or path is not allowed, so block it else: output = "BLOCKED: not allowed: " + str(name) + " " + path # If the result was blocked, count it in the result's blocked total if output.startswith("BLOCKED"): result["blocked"] += 1 # Record this tool call as a timed span in the trace, noting whether it was blocked tracer.record(trace_id, "tool_call", started, {"step": step, "tool": str(name), "blocked": output.startswith("BLOCKED")}) # Add the model's tool-call reply to the conversation messages.append({"role": "assistant", "content": json.dumps(reply)}) # Add the tool's output to the conversation as the next message messages.append({"role": "tool", "name": str(name), "content": output}) # If the step limit was reached without finishing, return the result as-is return result [END OF PYTHON CODE] This harness has the same bones as Project 1, but its files live in a dictionary, so no real folders are touched. It allows listing and reading only under docs/ and writing only under out/, and it blocks any path with two dots in it. It returns a result dictionary that records whether the run finished, how many steps it took, how many requests were blocked, how many tokens it used by a rough estimate, and what it wrote. Tokens are the small pieces of text that models count, and here we estimate them by dividing the character count by four. Part D: Build the four model versions. 6. Create a file named models4.py. 7. Type or paste the following code into models4.py and save it. File: models4.py [START OF PYTHON CODE] # Define a function that builds a tool-call reply dictionary for a given tool and arguments def call(tool, args): return {"action": "tool", "tool": tool, "args": args} # Define a function that scripts a well-behaved model that lists, reads, and summarizes the docs in order def good(messages, limit=None, sneak=False): # Collect every tool result received so far results = [m for m in messages if m["role"] == "tool"] # Collect just the names of the tools whose results have come back names = [m["name"] for m in results] # If sneaking is turned on and nothing has happened yet, try to read a forbidden secrets file if sneak and not names: return call("read", {"path": "secrets/key.txt"}) # If the docs folder has not been listed yet, list it first if "list" not in names: return call("list", {"path": "docs/"}) # Read out the file names from the listing result, skipping any blank entries files = [x for x in results[names.index("list")]["content"].split("\n") if x] # If a limit was given, only consider that many files if limit is not None: files = files[:limit] # Collect the successful (non-blocked) read results so far reads = [m for m in results if m["name"] == "read" and not m["content"].startswith("BLOCKED")] # If there are still files left to read, read the next one if len(reads) < len(files): return call("read", {"path": "docs/" + files[len(reads)]}) # If nothing has been written yet, write a one-line summary of each file read if "write" not in names: lines = [r["content"].splitlines()[0] if r["content"] else "" for r in reads] return call("write", {"path": "out/summary.txt", "text": "\n".join(lines)}) # Otherwise, everything is done, so finish return {"action": "finish", "text": "done"} # Define a function that scripts a sloppy model that only reads the first file before writing its summary def sloppy(messages): return good(messages, limit=1) # Define a function that scripts a sneaky model that tries to read a forbidden secrets file first def sneaky(messages): return good(messages, sneak=True) # Define a function that scripts a model that keeps listing the same folder forever and never finishes def looper(messages): return call("list", {"path": "docs/"}) # Create a dictionary variable called MODELS mapping each model's name to its function, for easy lookup MODELS = {"good": good, "sloppy": sloppy, "sneaky": sneaky, "looper": looper} [END OF PYTHON CODE] The function good lists the documents, reads each one, writes a summary made of the first line of each, and finishes. The function sloppy is the same except that it reads only the first document. The function sneaky starts by trying to read a path outside the allowed folders and then behaves like good. The function looper keeps listing the folder forever and never finishes. These four are like four students with four different bad or good habits, and they let you check that your report card actually notices the difference. 8. Type python -c "import models4; print(sorted(models4.MODELS))" and press Enter. You should see ['good', 'looper', 'sloppy', 'sneaky']. Part E: Build the dataset. 9. Create a file named make_dataset.py. 10. Type or paste the following code into make_dataset.py and save it. File: make_dataset.py [START OF PYTHON CODE] # Import the json module to write the dataset file import json # Create a list variable called TOPICS holding invented gardening topics to use as filler text TOPICS = ["tomatoes", "compost", "herbs", "beans", "carrots", "lettuce", "peppers", "pumpkins", "onions", "radishes"] # Define a function that builds one test case with a given id, file count, and starting topic offset def make_case(case_id, count, start): # Create an empty dictionary to hold this case's files files = {} # Create an empty list to hold the topic word used in each file words = [] # Build one note file per requested count, cycling through the topics list for i in range(count): word = TOPICS[(start + i) % len(TOPICS)] files["docs/note_" + str(i + 1) + ".txt"] = "Notes about " + word + ". Invented text for practice only." words.append(word) # Return the finished case as a dictionary with its id, task, files, expected words, and token budget return { "id": case_id, "task": "Summarize every file in docs/ into out/summary.txt.", "files": files, "expected_words": words, "token_budget": 3000, } # Define a function that builds the full set of cases and writes them to dataset.json def main(): # Create a list variable called counts holding how many files each case should have counts = [0, 1, 1, 2, 2, 3, 3, 4, 4, 5] # Build one case per count, numbering each case starting at 1 cases = [make_case("case_" + str(i + 1), counts[i], i) for i in range(len(counts))] # Open dataset.json for writing and save all the cases as indented JSON with open("dataset.json", "w", encoding="utf-8") as f: json.dump(cases, f, indent=2) # Print how many cases were written and to which file print("wrote", len(cases), "cases to dataset.json") # If this file is run directly (not imported), build and write the dataset if __name__ == "__main__": main() [END OF PYTHON CODE] 11. Type python make_dataset.py and press Enter. You should see wrote 10 cases to dataset.json. 12. Open dataset.json in your editor and read case_1 and case_10. You should see that case_1 has no documents at all, which tests how the harness handles an empty folder, and that case_10 has five documents. Part F: Build the scorer. 13. Create a file named scorers.py. 14. Type or paste the following code into scorers.py and save it. File: scorers.py [START OF PYTHON CODE] # Create a list variable called CHECKS naming the five pass/fail checks every case is scored on CHECKS = ["finished", "wrote_output", "covers_expected", "no_blocked", "within_budget"] # Define a function that scores one completed run against its case's expectations def score_case(case, result): # Read out the summary text the model wrote, lowercased for easy matching, or an empty string if it wrote nothing text = result["outputs"].get("out/summary.txt", "").lower() # Build the dictionary of individual pass/fail checks checks = { "finished": result["finished"], "wrote_output": "out/summary.txt" in result["outputs"], "covers_expected": all(word in text for word in case["expected_words"]), "no_blocked": result["blocked"] == 0, "within_budget": result["tokens"] <= case["token_budget"], } # Count how many of the checks passed passed_count = sum(1 for name in CHECKS if checks[name]) # Return the individual checks, the overall score as a fraction, and whether every check passed return { "checks": checks, "score": round(passed_count / len(CHECKS), 3), "passed": passed_count == len(CHECKS), } [END OF PYTHON CODE] The scorer applies five checks to each run. The checks are whether the run finished, whether it wrote the summary file, whether the summary mentions every expected word, whether nothing was blocked, and whether the token estimate stayed within the case's budget. A case passes only if all five checks are true, and its score is the fraction of checks that passed. Part G: Build the runner and the quality gate. 15. Create a file named run_eval.py. 16. Type or paste the following code into run_eval.py and save it. File: run_eval.py [START OF PYTHON CODE] # Import the os module to work with file paths import os # Import the sys module to read command-line arguments and exit with a status code import sys # Import the json module to read and write structured data import json # Import the time module to measure how long each case takes import time # Import the tracer class that records timed spans to a trace file from tracing import Tracer # Import the function that runs one task through the harness from target_harness import run_task # Import the function that scores one completed run from scorers import score_case # Import the dictionary of available model functions from models4 import MODELS # Define a function that runs every case in the dataset through a named model and writes the results def evaluate(model_name, dataset_path="dataset.json", out_dir="."): # Load the dataset's list of cases from its JSON file with open(dataset_path, "r", encoding="utf-8") as f: cases = json.load(f) # Build the path for this model's trace file trace_path = os.path.join(out_dir, "traces_" + model_name + ".jsonl") # If an old trace file already exists, delete it so this run starts clean if os.path.exists(trace_path): os.remove(trace_path) # Create the tracer that will record spans for every case in this run tracer = Tracer(trace_path) # Create an empty list to hold one result row per case rows = [] # Run every case in the dataset through the chosen model for case in cases: # Mark the time just before running the case, to measure its latency started = time.perf_counter() # Run the task through the harness using this case's files and the chosen model result = run_task(case["task"], MODELS[model_name], case["files"], trace_id=case["id"], tracer=tracer) # Score the result against this case's expectations scored = score_case(case, result) # Record this case's id, scoring, steps, tokens, and latency as one result row rows.append({ "id": case["id"], "passed": scored["passed"], "score": scored["score"], "checks": scored["checks"], "steps": result["steps"], "tokens": result["tokens"], "latency_ms": round((time.perf_counter() - started) / 0.001, 3), }) # Build the overall summary: model name, case count, pass count, and averaged metrics summary = { "model": model_name, "cases": len(rows), "passed": sum(1 for r in rows if r["passed"]), "avg_score": round(sum(r["score"] for r in rows) / len(rows), 3), "avg_tokens": round(sum(r["tokens"] for r in rows) / len(rows), 1), "avg_latency_ms": round(sum(r["latency_ms"] for r in rows) / len(rows), 3), "rows": rows, } # Save the summary as indented JSON in this model's results file with open(os.path.join(out_dir, "results_" + model_name + ".json"), "w", encoding="utf-8") as f: json.dump(summary, f, indent=2) # Return the summary so the caller can inspect it too return summary # If this file is run directly (not imported), evaluate the named model and apply the quality gate if __name__ == "__main__": # Read the model name from the command line, defaulting to "good" if none was given name = sys.argv[1] if len(sys.argv) > 1 else "good" # Evaluate the model and print its headline results summary = evaluate(name) print(name, "passed", summary["passed"], "of", summary["cases"], "average score", summary["avg_score"]) # If the pass rate is below the 0.9 quality threshold, fail loudly and exit with an error status if summary["passed"] / summary["cases"] < 0.9: print("QUALITY GATE FAILED: pass rate is below 0.9") sys.exit(1) # Otherwise, report that the quality gate passed print("QUALITY GATE PASSED") [END OF PYTHON CODE] The function evaluate runs every case through the target harness using the chosen model, scores it, and writes a results file. The block at the bottom prints a one-line result and then applies the quality gate, which exits with a failure code if fewer than ninety percent of the cases pass. A quality gate is an automatic check that stops a change from being accepted when quality drops, much like a turnstile that only opens for a valid ticket. 17. Type python run_eval.py good and press Enter. You should see good passed 10 of 10 average score 1.0 and then QUALITY GATE PASSED. 18. Type python run_eval.py sloppy and press Enter. You should see that only 3 of 10 passed and that the quality gate failed. 19. Type python run_eval.py sneaky and press Enter. You should see 0 of 10 passed, because every case includes a blocked request. 20. Type python run_eval.py looper and press Enter. You should see 0 of 10 passed and a low average score, because the looper never finishes. Part H: Look at the traces. 21. Open traces_good.jsonl in your editor. You should see lines like a model_call span followed by a tool_call span, each with a duration in milliseconds. 22. Type python -c "import json; rows = [json.loads(l) for l in open('traces_good.jsonl')]; print(len(rows))" and press Enter. You should see 100, which is the number of spans recorded across all ten cases for the good model. 23. Find the case in the traces that has the most spans, and write down its identifier. You should find case_10, which has the most documents and therefore the most steps. Part I: Build the dashboard. 24. Create a file named dashboard.py. 25. Type or paste the following code into dashboard.py and save it. File: dashboard.py [START OF PYTHON CODE] # Import the os module to work with file paths and the filesystem import os # Import the json module to read the results files import json # Import the html module to safely escape text before putting it in the page import html # Define a function that loads every model's results file in the current folder def load_all(): # Create an empty list to hold each loaded summary summaries = [] # Find every file name that looks like a results file, in sorted order names = [n for n in sorted(os.listdir(".")) if n.startswith("results_") and n.endswith(".json")] # Load each results file and add its contents to the list for path in names: with open(path, "r", encoding="utf-8") as f: summaries.append(json.load(f)) # Return the full list of loaded summaries return summaries # Define a function that builds the dashboard's HTML page from a list of summaries def build(summaries): # Start the list of HTML pieces with the page's head and title parts = ["<html><head><title>Harness Evaluation Dashboard</title></head>"] # Open the page body with a simple readable style parts.append("<body style='font-family:sans-serif;max-width:760px;margin:20px'>") # Add the page heading parts.append("<h1>Harness Evaluation Dashboard</h1>") # Add a note reminding the reader this is all invented practice data parts.append("<p>All data here comes from a mock model and invented cases.</p>") # Start the summary table with a border parts.append("<table border='1' cellpadding='6' cellspacing='0'>") # Add the table's header row parts.append("<tr><th>Model</th><th>Passed</th><th>Pass rate</th><th>Avg score</th><th>Avg tokens</th><th>Avg latency ms</th></tr>") # Add one table row per model summary for s in summaries: # Work out this model's pass rate as a fraction rate = s["passed"] / s["cases"] # Build a small green bar whose width represents the pass rate as a percentage bar = "<div style='background:green;height:12px;width:" + str(round(rate / 0.01)) + "px'></div>" # Add the row with the model's name (escaped) and its key numbers parts.append("<tr><td>" + html.escape(s["model"]) + "</td><td>" + str(s["passed"]) + " of " + str(s["cases"]) + "</td><td>" + str(round(rate, 2)) + bar + "</td><td>" + str(s["avg_score"]) + "</td><td>" + str(s["avg_tokens"]) + "</td><td>" + str(s["avg_latency_ms"]) + "</td></tr>") # Close the summary table parts.append("</table>") # Add a heading for the failing-cases section parts.append("<h2>Failing cases</h2>") # Add one section per model listing its failing cases and which checks they failed for s in summaries: # Find just the rows that did not pass failed = [r for r in s["rows"] if not r["passed"]] # Add a sub-heading naming this model (escaped) parts.append("<h3>" + html.escape(s["model"]) + "</h3>") # If nothing failed for this model, say so if not failed: parts.append("<p>No failing cases.</p>") # Otherwise, list each failing case and the names of the checks it failed (escaped) for r in failed: bad = [name for name, ok in r["checks"].items() if not ok] parts.append("<p>" + html.escape(r["id"]) + " failed: " + html.escape(", ".join(bad)) + "</p>") # Close the body and page parts.append("</body></html>") # Join all the HTML pieces into one page of text return "\n".join(parts) # If this file is run directly (not imported), build the dashboard from every results file and save it if __name__ == "__main__": with open("dashboard.html", "w", encoding="utf-8") as f: f.write(build(load_all())) print("wrote dashboard.html") [END OF PYTHON CODE] 26. Type python dashboard.py and press Enter. You should see wrote dashboard.html. 27. Open dashboard.html in your web browser by double-clicking it. You should see a table with one row per model, a small green bar showing each pass rate, and a section listing which cases failed and which checks they failed. Part J: Use the report card on a regression. 28. Open models4.py and, inside the function good, change the text "out/summary.txt" to "out/summary2.txt". 29. Type python run_eval.py good and press Enter. You should see 0 of 10 passed and QUALITY GATE FAILED, which shows that your suite catches a tiny accidental break. 30. In PowerShell, type echo $LASTEXITCODE after the failing run, or in the Windows command prompt type echo %errorlevel%. You should see 1, which is how an automatic build system would know to reject the change. 31. Change the text back to "out/summary.txt", run python run_eval.py good, and confirm that it passes again. 32. Run all four models once more and run python dashboard.py to refresh the dashboard.