> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Computer Use Tools
> Let agents take screenshots and control mouse, keyboard, and scroll — safely gated behind a human approval callback.
Computer Use tools let an agent take screenshots and control the mouse, keyboard, and scroll — gated by a human approval callback so nothing runs without your say-so.
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
subgraph "Computer Use"
A[📋 Agent] --> T[🖱️ Tool Call]
T --> G{🔐 Approval?}
G -->|Yes| B[🖥️ pyautogui]
G -->|No| D[❌ Denied]
B --> R[✅ Screen]
end
classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff
classDef tool fill:#189AB4,stroke:#7C90A0,color:#fff
classDef gate fill:#F59E0B,stroke:#7C90A0,color:#fff
classDef ok fill:#10B981,stroke:#7C90A0,color:#fff
classDef bad fill:#6366F1,stroke:#7C90A0,color:#fff
class A agent
class T,B tool
class G gate
class R ok
class D bad
```
## Quick Start
Read-only tools never require approval, so an observer agent works out of the box.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents.tools import computer_screenshot, computer_screen_size
agent = Agent(
name="observer",
instructions="Describe what's on screen.",
tools=[computer_screenshot, computer_screen_size],
)
agent.start("What size is the screen and what's on it?")
```
Register an approval callback, then add the control tools. Every click, type, key, and scroll passes through the callback first.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents.tools import (
computer_screenshot, computer_click, computer_type,
computer_key, computer_scroll, computer_move,
set_computer_approval,
)
set_computer_approval(lambda action: input(f"Approve {action}? (y/n) ") == "y")
agent = Agent(
name="assistant",
instructions="You can control the computer to help the user.",
tools=[
computer_screenshot, computer_click, computer_type,
computer_key, computer_scroll, computer_move,
],
)
agent.start("Take a screenshot and describe what's on screen")
```
***
## How It Works
```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
participant User
participant Agent
participant Tool as computer_click
participant Approve as Approval Callback
participant PA as pyautogui
User->>Agent: "Click the Submit button"
Agent->>Tool: computer_click(320, 480)
Tool->>Approve: approve("click(320, 480, left)")?
Approve-->>User: (prompts human)
User-->>Approve: y
Approve-->>Tool: True
Tool->>PA: click(x=320, y=480, button="left")
PA-->>Tool: (screen event)
Tool-->>Agent: "Clicked (320, 480) with left button"
Agent-->>User: "Done."
```
| Step | What happens |
| -------------------------- | ---------------------------------------------------------------------- |
| Agent calls a control tool | Tool builds a human-readable action string (`"click(320, 480, left)"`) |
| Approval gate | If no callback is registered, returns `"Action denied: …"` immediately |
| Backend load | `pyautogui` is lazy-imported on first use |
| Missing backend | Returns `"Computer Use backend unavailable: …"` — no exception |
***
## Configuration Options
Every tool returns a string — errors are reported as messages, never raised.
**Tools:**
| Tool | Parameters | Approval | Returns |
| ---------------------- | ------------------------------------------ | :----------------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `computer_screenshot` | `path: str = ""` | Only if `path` set | `"Screenshot captured (WxH)"` (+ `" and saved to PATH"`) |
| `computer_screen_size` | — | Never | `"WIDTHxHEIGHT"` |
| `computer_move` | `x: int, y: int` | Yes | `"Moved to (x, y)"` |
| `computer_click` | `x: int, y: int, button: str = "left"` | Yes | `"Clicked (x, y) with BUTTON button"` |
| `computer_type` | `text: str` | Yes | `"Typed N characters"` or `"Typed N of M characters (non-ASCII skipped)"` |
| `computer_key` | `key: str` | Yes | `"Pressed KEY"` on success, or `"Key press failed: empty or invalid key string 'KEY'"` when `key` is empty / whitespace-only |
| `computer_scroll` | `direction: str = "down", amount: int = 3` | Yes | `"Scrolled DIRECTION by N"` on success, or `"Scroll failed: invalid direction 'DIRECTION' (expected 'up' or 'down')"` when `direction` is not `"up"` or `"down"` |
**Approval Callback:**
| Option | Type | Default | Description |
| ---------- | --------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `callback` | `Optional[Callable[[str], bool]]` | `None` (control denied) | Called with the action string (e.g. `"click(100, 200, left)"`) — return `True` to allow, `False` (or raise) to deny |
`set_computer_approval` sets a process-wide singleton behind a lock. Call it once and every later tool call in the process uses the same gate. Pass `None` to revoke — control actions then default back to deny.
***
## Common Patterns
**Read-only observer** — no approval callback, so the agent can look but not touch.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents.tools import computer_screenshot, computer_screen_size
agent = Agent(
name="observer",
instructions="Report the screen size and describe what's visible.",
tools=[computer_screenshot, computer_screen_size],
)
agent.start("Describe the current screen")
```
**Selective auto-approve** — allow only screenshots-to-disk and cursor moves, deny clicks and typing. Pattern-match on the action string.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents.tools import (
computer_screenshot, computer_move, computer_click,
computer_type, set_computer_approval,
)
def allow_safe(action: str) -> bool:
return action.startswith("screenshot_save(") or action.startswith("move(")
set_computer_approval(allow_safe)
agent = Agent(
name="safe-driver",
instructions="Save screenshots and move the cursor to point things out.",
tools=[computer_screenshot, computer_move, computer_click, computer_type],
)
agent.start("Save a screenshot to /tmp/shot.png and move the cursor to it")
```
**Interactive approval** — prompt a human for every control action during local development.
```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents.tools import (
computer_screenshot, computer_click, computer_type, set_computer_approval,
)
set_computer_approval(lambda action: input(f"Approve {action}? (y/n) ") == "y")
agent = Agent(
name="assistant",
instructions="Help the user by controlling the screen.",
tools=[computer_screenshot, computer_click, computer_type],
)
agent.start("Open the search box and type 'hello'")
```
***
## Best Practices
Call `set_computer_approval(...)` at process start, before you construct the `Agent`. Control tools default to *deny* — an agent that starts without a callback will silently refuse every click, type, key, and scroll.
The callback receives a stable, human-readable action string like `click(320, 480, left)` or `type('hello')`. Pattern-match on this — the tool name is not passed separately.
Emoji, accented letters, and other non-printable characters are dropped by the backend. `computer_type` reports `"Typed N of M characters (non-ASCII skipped)"` when this happens — surface that message to the user rather than assuming the whole string was typed.
`computer_screenshot()` with no `path` is read-only and ungated. `computer_screenshot(path="/tmp/x.png")` writes to disk and goes through the approval gate as `screenshot_save('/tmp/x.png')` — an agent cannot overwrite arbitrary files without approval.
`computer_key("")` returns `"Key press failed: empty or invalid key string ''"` — the backend is never called. `computer_scroll(direction="sideways")` returns `"Scroll failed: invalid direction 'sideways' (expected 'up' or 'down')"` — it does not default to `"down"`. If your callback pattern-matches on the return string, handle these failure prefixes explicitly.
A missing display returns the same `"Computer Use backend unavailable: ..."` message as a missing package. On Linux `pyautogui` needs an X11 display; in CI containers and on headless boxes these tools no-op — don't gate business logic on their success.
***
## Related
Broader human-in-the-loop approval patterns for gating agent tool calls.
Restrict which tools an agent is permitted to call.