# Computer-use

A first-class engine for driving real desktop sessions, packaged as a sibling of @nikcli-ai/browser-control and @nikcli-ai/terminal-control.

## Overview

`@nikcli-ai/computer-use` is the dedicated engine that drives real desktop sessions. It sits next to `@nikcli-ai/browser-control` (which drives Chromium pages) and `@nikcli-ai/terminal-control` (which drives PTY-backed TUI sessions) — three sibling engines that share the same CLI shape, the same per-workspace background daemon, and the same evidence-bundle output.

The package was extracted from `packages/nikcli/src/computer/` in commit `9b378dac`: the in-tree `computer.ts` shrinks from a 669-line monolith to ~54 lines of glue, while the package gains its own CLI, daemon, host and sandbox backends, recording pipeline, and `bun test` suite.

Relationship to the computer tool

The [computer tool](/docs/tools) exposed to agents, the [computer TUI plugin](/docs/tui), and the `computer-use` CLI all converge on the same package. Use whichever surface fits the workflow; underneath they share one daemon and one registry.

## Why a separate package

Pulling `computer-use` out of `packages/nikcli` accomplishes three things at once:

-   **Sibling composition.** Browser, terminal, and computer automation now share a single CLI grammar (`start / info / click / type / screenshot / recording / bundle`) and a single per-workspace daemon lifecycle. New agents only need to learn one shape.
-   **Independent release.** The package bumps its own semver on the same cadence as nikcli but ships with its own `bun test` suite and its own `package.json` dependency surface (`@ffmpeg-installer/ffmpeg` 1.1.0 for MP4 export, `playwright` 1.61.0 and `chromium-bidi` 0.4.33 for the headless recording layer).
-   **First-class Docker image.** The root `Dockerfile`, `Dockerfile.serve`, and `packages/slack/Dockerfile` now copy the package into the build, and `script/railway-deploy.sh` ships it as part of the minimal upload context.

## Install & skill

The package is published as `@nikcli-ai/computer-use` and exposes the `computer-use` binary. Inside the nikcli monorepo it is a workspace package consumed by `packages/nikcli/src/tool/computer.ts`.

Terminal

bash 

```
# Inside the nikcli monorepo
      bun run --cwd packages/computer-use control
      # Or, after a workspace install, as the binary:
      computer-use --help
```

$ shell · paste into your terminal

The package ships a skill that nikcli discovers automatically when installed. Register it once per workspace, or globally for every nikcli session on the machine:

Terminal

bash 

```
# Per-workspace (default)
      computer-use install-skill

      # Global — installs into ~/.agents/skills so nikcli discovers it everywhere
      computer-use install-skill --global
```

$ shell · paste into your terminal

Daemon lifecycle

Every command except `bundle` and `install-skill` is a thin RPC call against the per-workspace background daemon. The daemon auto-spawns on first use and self-terminates after 10 idle minutes, which is what makes `computer-use start`, then a later `computer-use wait`, then a later `computer-use stop` work as three separate process invocations from shell scripts or CI steps.

## Session lifecycle

A _computer-use session_ is a named, persistent desktop handle. Sessions are held by the per-workspace daemon so they survive across separate CLI invocations; the daemon registers them in `SessionManager` (`packages/computer-use/src/manager.ts`) and exposes them through the `ComputerSession` class (`src/session.ts`).

src/session.ts

### start

Create a named session. `computer-use start [NAME] [--mode sandbox|host] [--width N] [--height N]`. The default `sandbox` mode provisions a Docker-managed Linux desktop served over noVNC; `host` opt-in uses the user's real desktop in real time.

src/session.ts

### info / capabilities

`computer-use info NAME` reports live dimensions, backend, and status. `computer-use capabilities NAME` returns the typed action set supported by the backend (the in-tree computer tool uses this to gate actions).

src/render

### screenshot / screen-size

Capture a single PNG to `--out FILE` or stream it on stdout; query the live screen size to drive coordinate-aware actions.

src/manager.ts

### stop / remove / restart / close-all

`stop` ends the session but keeps the entry queryable so any recorded live URL keeps working until `remove`. `close-all` tears the whole workspace down.

### Action grammar

Every action is invoked the same way: target the session by `NAME`, then a verb. Coordinates are screen pixels unless the backend reports otherwise. Keys use the `translateKey` / `translateKeys` helpers in `src/keys.ts` so chord names match the rest of nikcli.

Terminal

bash 

```
computer-use click NAME X Y [--button left|right|middle] [--double]
      computer-use drag  NAME FROM_X FROM_Y TO_X TO_Y
      computer-use type  NAME TEXT
      computer-use key   NAME COMBO      # e.g. "ctrl+shift+p"
      computer-use scroll NAME X Y (up|down|left|right) [--amount N]
      computer-use wait  NAME (--text VALUE | --stable [--ms N] | --timeout MS)
```

$ shell · paste into your terminal

## Backends

Backends implement the same `SessionBackend` contract from `src/backends/index.ts`. Two are shipped in-tree; new backends (for example a Windows-native or a remote-VNC one) only need to implement the contract.

| Backend | Mode | Source | When to use |
| --- | --- | --- | --- |
| `sandbox` | Default | `src/backends/sandbox.ts` | Isolated, repeatable runs in CI. Provisions a Docker-managed Linux desktop served over noVNC; the daemon teardown cleans up the container. |
| `host` | Opt-in (`--mode host`) | `src/backends/host.ts` | Drives the user's real desktop in real time. Use only on machines you control — permissions are yours, not the orchestrator's. |

host mode is privileged

The `host` backend inherits whatever your shell session can do. Always gate it behind an explicit user approval and prefer the `sandbox` backend for any shared or CI-run workload.

## Recording & evidence

The recording pipeline captures real screenshots at a chosen frame rate and emits a versioned JSON file (`RECORDING_VERSION`) that the renderer can convert into MP4, GIF preview, single-frame PNG, and PR-ready Markdown — all from the same bundle command.

Terminal

bash 

```
# Periodic frame sampling at 1 fps
      computer-use start-recording NAME --fps 1

      computer-use marker       NAME LABEL
      computer-use stop-recording NAME
      computer-use recording-data NAME   # current state, without stopping

      # PR-ready bundle (screenshot, video, GIF, manifest, pr.md)
      computer-use bundle \
        --recording ./recording.json \
        --out ./artifacts/computer-use \
        --link-base artifacts/computer-use \
        --result passed \
        --title "Computer-use verification" \
        --summary "The desktop session completed successfully."
```

$ shell · paste into your terminal

The `bundle` command accepts `--screenshot` for a one-shot image, or `--recording` for a recorded run. With `--recording` the bundle derives the screenshot, the video, and the GIF from the sampled frames; use `--at-marker NAME` or `--at-ms MS` to pick a specific frame. Use `--no-preview` to skip the GIF and `--json` to print the bundle manifest instead of writing it.

Evidence manifest

Every `bundle` writes a `manifest.json` plus a `pr.md` ready to be appended to a pull request description. The format is the same as `@nikcli-ai/browser-control bundle` and `@nikcli-ai/terminal-control bundle`, so PR bots can treat all three the same way.

## TUI integration

The TUI ships a `computer` feature plugin that renders the same sessions, screenshots, and action log inside the terminal. It is a thin client over the package — it talks to the daemon over RPC and surfaces a live preview alongside the transcript.

-   **Sidebar entry.** The `computer` feature plugin is registered in `packages/tui/src/feature-plugins/computer/index.tsx`; the TUI plugin runtime owns its route registration.
-   **Agent-side.** The `computer` tool in `packages/nikcli/src/tool/computer.ts` wraps the package's `ComputerSession` and forwards every permission-aware action through the existing [permission](/docs/permissions) pipeline.
-   **Skill bridge.** The bundled skill (`packages/computer-use/skills/computer-use/SKILL.md`) is registered automatically when the package is loaded.

## Source anchors

| Concern | File |
| --- | --- |
| Public exports | `packages/computer-use/src/index.ts` |
| CLI binary | `packages/computer-use/src/cli.ts` |
| Background daemon | `packages/computer-use/src/daemon.ts` + `daemon-client.ts` |
| Session class | `packages/computer-use/src/session.ts` |
| Session registry | `packages/computer-use/src/manager.ts` |
| Backends | `packages/computer-use/src/backends/host.ts` + `sandbox.ts` + `index.ts` |
| Recording | `packages/computer-use/src/recording.ts` |
| Evidence bundle | `packages/computer-use/src/evidence.ts` |
| Renderers | `packages/computer-use/src/render/json.ts`, `png.ts`, `text.ts`, `video.ts` |
| Key chord helpers | `packages/computer-use/src/keys.ts` |
| Container sandbox | `packages/computer-use/src/sandbox.ts` + `sandbox-image.ts` |
| Tests | `packages/computer-use/test/evidence.test.ts`, `keys.test.ts`, `recording.test.ts`, `session.test.ts` |
| Skill | `packages/computer-use/skills/computer-use/SKILL.md` |
| Agent tool wrapper | `packages/nikcli/src/tool/computer.ts` |
| TUI plugin | `packages/tui/src/feature-plugins/computer/index.tsx` |

---

Source: https://nikcli-ai.dev/docs/computer-use