COMPUTER-USE AND BROWSER AGENTS

AGENTS THAT CONTROL A SCREEN, MOUSE, AND KEYBOARD A computer-use agent is an agent that operates a computer the way a person does, by looking at the screen and then moving the mouse pointer, clicking, typing, and pressing keys. The basic cycle resembles the agent loop you learned in Part Three, with a visual twist. First, the harness takes a screenshot, which is a picture of what is currently displayed. Second, the model looks at the picture and decides what to do next, such as click the button labeled Save, which sits at a certain position. Third, the harness carries out that action by simulating the mouse click or key press. Fourth, the harness takes a new screenshot and the cycle repeats until the task is done or the agent stops. Positions on a screen are described with coordinates, which are pairs of numbers giving how far across and how far down a spot is, measured in pixels, the tiny dots that make up a display. Computer-use agents are valuable in a few situations. When a program has no API, meaning no official structured way for other programs to talk to it, the screen may be the only way in. Older business software is a common case. When a task spans many different programs, a screen-based agent can move between them like a person. When you need to test software the way a real user experiences it. They also have clear weaknesses. They are slower, because every step involves taking and interpreting a picture. They are more fragile, because a small change in the layout, a pop-up window, or a different screen size can confuse them. They can misread the screen or click the wrong thing, and a wrong click may have real consequences. They typically cost more per task than a structured tool call, since images use many tokens, a cost idea from Chapter 53. For these reasons, a good harness engineer uses a structured API whenever a good one exists, and treats screen control as the tool of last resort.