Computer-use

Overview

@nikcli-ai/computer-use is the dedicated engine that drives real desktop sessions. It sits next to @nikcli-ai/browser-control (which drives Chromium pages) and @nikcli-ai/terminal-control (which drives PTY-backed TUI sessions) — three sibling engines that share the same CLI shape, the same per-workspace background daemon, and the same evidence-bundle output.

The package was extracted from packages/nikcli/src/computer/ in commit 9b378dac: the in-tree computer.ts shrinks from a 669-line monolith to ~54 lines of glue, while the package gains its own CLI, daemon, host and sandbox backends, recording pipeline, and bun test suite.

Why a separate package

Pulling computer-use out of packages/nikcli accomplishes three things at once:

• Sibling composition. Browser, terminal, and computer automation now share a single CLI grammar (start / info / click / type / screenshot / recording / bundle) and a single per-workspace daemon lifecycle. New agents only need to learn one shape.
• Independent release. The package bumps its own semver on the same cadence as nikcli but ships with its own bun test suite and its own package.json dependency surface (@ffmpeg-installer/ffmpeg 1.1.0 for MP4 export, playwright 1.61.0 and chromium-bidi 0.4.33 for the headless recording layer).
• First-class Docker image. The root Dockerfile, Dockerfile.serve, and packages/slack/Dockerfile now copy the package into the build, and script/railway-deploy.sh ships it as part of the minimal upload context.

Install & skill

The package is published as @nikcli-ai/computer-use and exposes the computer-use binary. Inside the nikcli monorepo it is a workspace package consumed by packages/nikcli/src/tool/computer.ts.

# Inside the nikcli monorepo
      bun run --cwd packages/computer-use control
      # Or, after a workspace install, as the binary:
      computer-use --help

The package ships a skill that nikcli discovers automatically when installed. Register it once per workspace, or globally for every nikcli session on the machine:

# Per-workspace (default)
      computer-use install-skill

      # Global — installs into ~/.agents/skills so nikcli discovers it everywhere
      computer-use install-skill --global

Session lifecycle

A computer-use session is a named, persistent desktop handle. Sessions are held by the per-workspace daemon so they survive across separate CLI invocations; the daemon registers them in SessionManager (packages/computer-use/src/manager.ts) and exposes them through the ComputerSession class (src/session.ts).

Action grammar

Every action is invoked the same way: target the session by NAME, then a verb. Coordinates are screen pixels unless the backend reports otherwise. Keys use the translateKey / translateKeys helpers in src/keys.ts so chord names match the rest of nikcli.

computer-use click NAME X Y [--button left|right|middle] [--double]
      computer-use drag  NAME FROM_X FROM_Y TO_X TO_Y
      computer-use type  NAME TEXT
      computer-use key   NAME COMBO      # e.g. "ctrl+shift+p"
      computer-use scroll NAME X Y (up|down|left|right) [--amount N]
      computer-use wait  NAME (--text VALUE | --stable [--ms N] | --timeout MS)

Backends

Backends implement the same SessionBackend contract from src/backends/index.ts. Two are shipped in-tree; new backends (for example a Windows-native or a remote-VNC one) only need to implement the contract.

Recording & evidence

The recording pipeline captures real screenshots at a chosen frame rate and emits a versioned JSON file (RECORDING_VERSION) that the renderer can convert into MP4, GIF preview, single-frame PNG, and PR-ready Markdown — all from the same bundle command.

# Periodic frame sampling at 1 fps
      computer-use start-recording NAME --fps 1

      computer-use marker       NAME LABEL
      computer-use stop-recording NAME
      computer-use recording-data NAME   # current state, without stopping

      # PR-ready bundle (screenshot, video, GIF, manifest, pr.md)
      computer-use bundle \
        --recording ./recording.json \
        --out ./artifacts/computer-use \
        --link-base artifacts/computer-use \
        --result passed \
        --title "Computer-use verification" \
        --summary "The desktop session completed successfully."

The bundle command accepts --screenshot for a one-shot image, or --recording for a recorded run. With --recording the bundle derives the screenshot, the video, and the GIF from the sampled frames; use --at-marker NAME or --at-ms MS to pick a specific frame. Use --no-preview to skip the GIF and --json to print the bundle manifest instead of writing it.

TUI integration

The TUI ships a computer feature plugin that renders the same sessions, screenshots, and action log inside the terminal. It is a thin client over the package — it talks to the daemon over RPC and surfaces a live preview alongside the transcript.

• Sidebar entry. The computer feature plugin is registered in packages/tui/src/feature-plugins/computer/index.tsx; the TUI plugin runtime owns its route registration.
• Agent-side. The computer tool in packages/nikcli/src/tool/computer.ts wraps the package's ComputerSession and forwards every permission-aware action through the existing permission pipeline.
• Skill bridge. The bundled skill (packages/computer-use/skills/computer-use/SKILL.md) is registered automatically when the package is loaded.

Source anchors