# Desktop and computer use

A `TaskCloud` sandbox can run a full Linux desktop. People can watch and control it in the browser, and AI agents can drive it through a screenshot and input API.

The desktop has two paths:

- **People** open a signed stream URL (KasmVNC) in a browser, or in an `<iframe>` inside your product.
- **Agents** take screenshots (`GET /sandboxes/{sandbox_id}/screenshot`) and send mouse and keyboard actions (`POST /sandboxes/{sandbox_id}/input`). Agents don't need the video stream, and screenshots cost much less.

> Private beta. Latency numbers on this page are targets, not guarantees.

## The `desktop` template

| Property | Value |
| --- | --- |
| OS | Ubuntu 24.04 + XFCE |
| Browsers | Chromium, Firefox |
| Streaming | KasmVNC over HTTPS/WebSocket (no UDP, no TURN) |
| Resolution | 1920x1080 |
| Minimum size | `default` (4 vCPU / 8 GB). Image encoding runs on the sandbox's own vCPUs. |
| GPU | None. Rendering is done in software. |

The template is a RAM snapshot of a desktop that is already running, so a new sandbox starts with the screen ready.

**No GPU.** WebGL, 3D maps and video playback run slowly. Office apps, web apps and ordinary browsing work well.

```ts
import { Client } from "@taskcloud/sdk";

const client = new Client(); // reads TASKCLOUD_API_KEY
const sandbox = await client.sandboxes.create({ template: "desktop", size: "default" });
```

```python
from taskcloud import Client

client = Client()
sandbox = client.sandboxes.create(template="desktop", size="default")
```

```bash
taskcloud create --template desktop
```

To add the desktop stack to your own image, set `desktop: true` when you create a template from an image. See [Templates](../concepts/templates.md).

## Watching in a browser: stream URLs

`POST /sandboxes/{sandbox_id}/desktop` returns a signed URL of the form `https://<short_id>-desktop.taskhost.cloud/?token=...`.

| Field | Default | Description |
| --- | --- | --- |
| `ttl_seconds` | `600` | URL lifetime, 60 to 86400. An open session is disconnected when the URL expires. |
| `read_only` | `false` | The viewer can watch but cannot send input. |
| `single_use` | `false` | The URL can be opened only once. |

```bash
curl -X POST https://api.task.cloud/v1/sandboxes/$SANDBOX_ID/desktop \
  -H "Authorization: Bearer $TASKCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"ttl_seconds": 900, "read_only": true}'
```

```json
{
  "id": "dsk_7zab3kq2mxw5",
  "url": "https://a2b3c4d5e6f7-desktop.taskhost.cloud/?token=dt_...",
  "read_only": true,
  "single_use": false,
  "expires_at": "2026-11-09T12:15:00Z",
  "created_at": "2026-11-09T12:00:00Z"
}
```

```ts
const stream = await sandbox.desktop.stream({ ttl_seconds: 900, read_only: true });
console.log(stream.url);

// Later, before it expires:
await sandbox.desktop.revoke(stream.id);
```

```python
stream = sandbox.desktop.stream(ttl_seconds=900, read_only=True)
sandbox.desktop.revoke(stream.id)
```

```bash
taskcloud desktop $SANDBOX_ID --ttl 15m --read-only --open
```

- **List active URLs:** `GET /sandboxes/{sandbox_id}/desktop`.
- **Revoke a URL:** `DELETE /sandboxes/{sandbox_id}/desktop/{stream_id}`. This invalidates the URL and disconnects anyone using it.
- **Embed it** in your product:

  ```html
  <iframe src="https://a2b3c4d5e6f7-desktop.taskhost.cloud/?token=dt_..."
          style="width:100%;aspect-ratio:16/9;border:0" allow="clipboard-read; clipboard-write"></iframe>
  ```

- **Wake on request:** opening the URL of a paused sandbox resumes it. The target is under 2 s from click to desktop.
- **Auto-pause:** an active viewer counts as activity, so the sandbox doesn't pause while someone is watching. A desktop with no viewers and no input pauses normally after its idle timeout. See [Lifecycle](../concepts/lifecycle.md).
- **The session survives a pause.** Pausing snapshots RAM, so windows, open browser tabs and logged-in sessions are all still there after a resume. The viewer reconnects automatically.

## Screenshots

`GET /sandboxes/{sandbox_id}/screenshot`. Target p95 is under 300 ms.

| Query parameter | Default | Description |
| --- | --- | --- |
| `format` | `png` | `png` or `jpeg` |
| `quality` | `80` | JPEG quality, 1 to 100 |
| `scale` | `1` | Downscales the output image, 0.1 to 1 |
| `region` | none | Crop to `x0,y0,x1,y1` in full-screen pixels |
| `include_cursor` | `true` | Draw the cursor |

The response format depends on the `Accept` header:

- **Default:** the raw image, with `X-Screen-Width` and `X-Screen-Height` headers.
- **`Accept: application/json`:** a JSON object `{ data, media_type, width, height, screen_width, screen_height, taken_at }`, where `data` is base64.

```bash
curl -s https://api.task.cloud/v1/sandboxes/$SANDBOX_ID/screenshot \
  -H "Authorization: Bearer $TASKCLOUD_API_KEY" -o shot.png
# or
taskcloud screenshot $SANDBOX_ID -o shot.png
```

```ts
const shot = await sandbox.desktop.screenshot({ format: "png" });
// shot.data is base64; shot.screen_width x shot.screen_height is the coordinate space
```

## Input

`POST /sandboxes/{sandbox_id}/input` runs a batch of actions in order. Action names and fields follow the Anthropic computer-use tool, so a model's tool call can be forwarded as `{ "action": <tool name>, ...input }`.

| `action` | Fields |
| --- | --- |
| `mouse_move` | `coordinate` |
| `left_click`, `right_click`, `middle_click`, `double_click`, `triple_click` | optional `coordinate` (default: current cursor position); optional `text` with held modifiers (`shift`, `ctrl`, `alt`, `super`, joined with `+`) |
| `left_click_drag` | `start_coordinate`, `coordinate` |
| `left_mouse_down`, `left_mouse_up` | none |
| `scroll` | `scroll_direction` (`up`, `down`, `left`, `right`), `scroll_amount` (wheel clicks, default `3`), optional `coordinate`, optional `text` modifiers |
| `type` | `text` (Unicode, up to 65,536 characters) |
| `key` | `text`: a key or chord in xdotool syntax (`Return`, `ctrl+s`, `ctrl+shift+t`); optional `repeat` (1 to 100) |
| `hold_key` | `text`, `duration` (seconds) |
| `wait` | `duration` (seconds, up to 300) |

Coordinates are `[x, y]` in full-screen pixels, with the origin at the top left.

Request options:

- `return_screenshot: true` returns a screenshot taken after the last action. It is taken after `screenshot_delay_ms` (default 300) so the UI can settle.
- `coordinate_scale` divides incoming coordinates. Use it when they come from a screenshot taken with `scale` < 1.

If an action fails, the actions after it are not run and are reported with `status: "skipped"`. The cursor position is available at `GET /sandboxes/{sandbox_id}/cursor`.

```bash
curl -X POST https://api.task.cloud/v1/sandboxes/$SANDBOX_ID/input \
  -H "Authorization: Bearer $TASKCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      {"action": "left_click", "coordinate": [640, 412]},
      {"action": "type", "text": "hello world"},
      {"action": "key", "text": "Return"}
    ],
    "return_screenshot": true
  }'
```

```json
{
  "results": [
    {"index": 0, "action": "left_click", "status": "ok", "error": null},
    {"index": 1, "action": "type", "status": "ok", "error": null},
    {"index": 2, "action": "key", "status": "ok", "error": null}
  ],
  "cursor": {"x": 640, "y": 412},
  "screenshot": {"data": "iVBORw0...", "media_type": "image/png", "width": 1920, "height": 1080, "screen_width": 1920, "screen_height": 1080}
}
```

## Example: a computer-use agent loop with Claude

These examples connect Claude's computer-use toolset (`computer_toolset_20260801`) to a sandbox. The toolset is generally available on the Claude API with no beta header. The tool entry takes no `name` and no display size.

How the toolset maps to the sandbox API:

- **Calls:** each call from Claude is a `tool_use` block with `toolset_name: "computer"`, and the block's `name` is the action (`screenshot`, `zoom`, `left_click`, `type`, `key`, ...). One turn can contain several blocks.
- **Results:** return one `tool_result` per block, all in the next `user` message, and each one must include `toolset_name: "computer"`. Return results for every block:
  - `screenshot` and `zoom` return an image.
  - `cursor_position` returns text such as `X=512, Y=384`.
  - Other actions return `OK`.
  - If an action fails, set `is_error: true` on its result, and answer every later block in that turn with `Not executed: an earlier computer action in this turn failed.`
- **Mapping to the sandbox:**
  - `screenshot` → `sandbox.desktop.screenshot()`
  - `zoom` → a screenshot with `region`
  - `cursor_position` → `sandbox.desktop.cursor()`
  - Everything else → `sandbox.desktop.input({ actions: [{ action: block.name, ...block.input }] })`
- **No scaling needed at 1920x1080.** That resolution fits the model's image limits (up to 2576 px on the long edge), so coordinates pass through unchanged. If you downscale screenshots with `scale`, also send `coordinate_scale` with the same value on input.
- **Optional config:** `configs: { zoom: { enabled: false } }` turns off the zoom action if you don't want it.

### TypeScript

```ts
import Anthropic from "@anthropic-ai/sdk";
import { Client } from "@taskcloud/sdk";

const anthropic = new Anthropic();
const client = new Client();

const sandbox = await client.sandboxes.create({ template: "desktop", size: "default" });
console.log("Watch live:", (await sandbox.desktop.stream({ read_only: true, ttl_seconds: 3600 })).url);

type Block = { type: string; id?: string; name?: string; input?: any; toolset_name?: string };

async function runAction(block: Block) {
  const input = block.input ?? {};
  switch (block.name) {
    case "screenshot": {
      const s = await sandbox.desktop.screenshot({ format: "png" });
      return [{ type: "image", source: { type: "base64", media_type: "image/png", data: s.data } }];
    }
    case "zoom": {
      const [x0, y0, x1, y1] = input.region;
      const s = await sandbox.desktop.screenshot({ format: "png", region: `${x0},${y0},${x1},${y1}` });
      return [{ type: "image", source: { type: "base64", media_type: "image/png", data: s.data } }];
    }
    case "cursor_position": {
      const c = await sandbox.desktop.cursor();
      return [{ type: "text", text: `X=${c.x}, Y=${c.y}` }];
    }
    default: {
      const r = await sandbox.desktop.input({ actions: [{ action: block.name, ...input }] });
      const res = r.results[0];
      if (res.status !== "ok") throw new Error(res.error ?? `${block.name} failed`);
      return [{ type: "text", text: "OK" }];
    }
  }
}

const messages: any[] = [
  { role: "user", content: "Open Firefox, go to news.ycombinator.com and tell me the top story title." },
];

try {
  while (true) {
    const response = await anthropic.messages.create({
      model: "claude-opus-5-5",
      max_tokens: 16000,
      tools: [{ type: "computer_toolset_20260801" } as any],
      messages,
    });
    messages.push({ role: "assistant", content: response.content });

    if (response.stop_reason === "refusal") {
      console.error("Model declined the request:", response.stop_details);
      break;
    }
    const calls = (response.content as Block[]).filter((b) => b.type === "tool_use");
    if (calls.length === 0) {
      for (const b of response.content) if (b.type === "text") console.log(b.text);
      break;
    }

    const results: any[] = [];
    let failed = false;
    for (const call of calls) {
      const base = { type: "tool_result", tool_use_id: call.id, toolset_name: "computer" };
      if (failed) {
        results.push({ ...base, is_error: true, content: "Not executed: an earlier computer action in this turn failed." });
        continue;
      }
      try {
        results.push({ ...base, content: await runAction(call) });
      } catch (err) {
        failed = true;
        results.push({ ...base, is_error: true, content: String(err) });
      }
    }
    messages.push({ role: "user", content: results });
  }
} finally {
  await sandbox.pause(); // $0 compute while paused; the desktop session is kept
}
```

### Python

```python
import anthropic
from taskcloud import Client

claude = anthropic.Anthropic()
client = Client()

sandbox = client.sandboxes.create(template="desktop", size="default")
print("Watch live:", sandbox.desktop.stream(read_only=True, ttl_seconds=3600).url)

SKIPPED = "Not executed: an earlier computer action in this turn failed."


def image(data: str) -> list:
    return [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": data}}]


def run_action(block) -> list:
    name, inp = block.name, dict(block.input or {})
    if name == "screenshot":
        return image(sandbox.desktop.screenshot(format="png").data)
    if name == "zoom":
        x0, y0, x1, y1 = inp["region"]
        return image(sandbox.desktop.screenshot(format="png", region=f"{x0},{y0},{x1},{y1}").data)
    if name == "cursor_position":
        c = sandbox.desktop.cursor()
        return [{"type": "text", "text": f"X={c.x}, Y={c.y}"}]
    result = sandbox.desktop.input(actions=[{"action": name, **inp}]).results[0]
    if result.status != "ok":
        raise RuntimeError(result.error or f"{name} failed")
    return [{"type": "text", "text": "OK"}]


messages = [{"role": "user", "content": "Open Firefox, go to news.ycombinator.com and tell me the top story title."}]

try:
    while True:
        response = claude.messages.create(
            model="claude-opus-5-5",
            max_tokens=16000,
            tools=[{"type": "computer_toolset_20260801"}],
            messages=messages,
        )
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason == "refusal":
            print("Model declined the request:", response.stop_details)
            break
        calls = [b for b in response.content if b.type == "tool_use"]
        if not calls:
            print("".join(b.text for b in response.content if b.type == "text"))
            break

        results, failed = [], False
        for call in calls:
            base = {"type": "tool_result", "tool_use_id": call.id, "toolset_name": "computer"}
            if failed:
                results.append({**base, "is_error": True, "content": SKIPPED})
                continue
            try:
                results.append({**base, "content": run_action(call)})
            except Exception as err:
                failed = True
                results.append({**base, "is_error": True, "content": str(err)})
        messages.append({"role": "user", "content": results})
finally:
    sandbox.pause()  # $0 compute while paused; the desktop session is kept
```

### Tips

- **Batch the non-visual actions** into one `input` call when you control the sequence, and set `return_screenshot: true`. That saves a round trip.
- **Give a person a link to watch the agent.** Use a `read_only` stream URL with a short TTL.
- **Fork a prepared desktop to parallelize work.** For example, log in once and then fork with memory. Each fork has its own browser session. See [Snapshots and fork](../concepts/snapshots-and-fork.md).

## Security

- **Use a sandbox for untrusted browsing.** Each sandbox is a separate microVM with its own kernel. Don't put long-lived credentials on a desktop that an agent drives through untrusted websites.
- **Prompt injection is a risk.** Web pages can contain text that tries to redirect the agent. Restrict what the agent may do: use an API key limited to that sandbox (`restrict.sandbox_ids` or `restrict.labels`), keep a human in the loop for sensitive actions, and review the agent's actions in a recorded or read-only stream.
- **Hand out viewer links carefully.** Use `read_only: true`, a short `ttl_seconds` and `single_use: true` for links you give to end users, and revoke them when you're done. The VNC port is never exposed without a signed token.

See [Security](../security.md) and [Limits](../limits.md).

## Roadmap

- WebRTC streaming at 60 fps, audio, clipboard sync and file upload into the desktop are on the roadmap. They are not part of the MVP.
- No GPU instances are planned for the MVP.