COMPUTER-USE AND BROWSER AGENTS

CHAPTER SUMMARY Computer-use agents operate a screen, mouse, and keyboard through a loop of screenshot, decision, and simulated action. They are useful when no API exists, but they are slower, costlier, and more fragile than structured tools, so use an API when a good one exists. Browser agents read pages and act on websites, and they must respect site rules and handle pop-ups, logins, and changing layouts. Action-taking agents face indirect prompt injection, irreversible actions, credential exposure, scope creep, misclicks, and data exposure. Containment layers isolation, allow-lists, blocked actions, human approval, logging, budgets, and kill switches. KEY TERMS REVIEW Computer-use agent: an agent that operates a computer by looking at the screen and simulating mouse and keyboard actions. Screenshot: a picture of what is displayed on a screen. Coordinates: numbers giving a position on a screen. Browser agent: an agent that reads and acts on websites through a browser. API: an official structured way for programs to talk to each other. Allow-list: an explicit list of the only things permitted. Virtual machine: a simulated computer running inside a real one. Irreversible action: an action that cannot easily be undone. Indirect prompt injection: harmful instructions hidden in content the agent reads.