THE EVALUATION HARNESS

OFFLINE EVALUATION Offline evaluation means testing a new version of the harness on a fixed set of test tasks before it is released to real users. The word offline here does not mean without internet. It means away from live traffic, in a safe testing area where mistakes hurt nobody. Offline evaluation is like a dress rehearsal before opening night. Its advantages are that it is safe, repeatable, and fast to run on demand, and that you can compare two versions on exactly the same questions. Its disadvantage is that it can only test what you thought of in advance. ONLINE EVALUATION Online evaluation means continuously scoring a sample of real live traffic after the new version is released. Online means that it happens on the live system. Online evaluation is like a restaurant manager walking the dining room during service, watching real customers, and noticing that people keep pushing away one particular dish. Its advantage is that it reflects what really happens, including surprises that no test designer predicted. Its disadvantages are that problems are discovered after some real users have already been affected, and that you often have no answer key for real requests, so scoring relies on automatic judges or on proxy signals such as whether the user asked the same question again.