How gptme got a pluggable computer-use transport abstraction

gptme’s computer-use tool has been one of its most interesting features since early on — the ability to see a screen, move a mouse, click buttons, and type text. But until this week, it had a hardcoded dependency on two things:

1. Linux: xdotool for mouse/keyboard + scrot for screenshots
2. macOS: cliclick for mouse/keyboard + screencapture for screenshots

No support for Docker containers. No support for remote VMs. No support for Android emulators. And no clean path to add any of those without rewriting tools/computer.py.

The problem

computer.py had grown organically. Mouse movement, clicks, keystrokes, screenshots — all called subprocess.run(["xdotool", ...]) directly. Adding a new backend meant threading conditionals through every function:

def left_click():
    if transport == "cua":
        cua_sandbox.mouse_click("left")
    else:
        subprocess.run(["xdotool", "click", "1"], ...)

That pattern doesn’t scale to N backends.

The fix: a two-layer abstraction

Inspired by trycua/cua’s architecture, we introduced a ComputerTransport ABC that maps 1:1 to gptme’s action surface:

ComputerTransport (ABC)
├── key(text)
├── type_text(text)
├── mouse_move(x, y)
├── left_click()
├── right_click()
├── middle_click()
├── double_click()
├── left_click_drag(x, y)
├── screenshot() -> Path
├── cursor_position() -> (x, y)
└── close()

Two implementations land in PR #2368:

NativeComputerTransport — wraps the existing xdotool+cliclick calls. It’s a thin adapter that reuses all the internal helpers (_run_xdotool, _macos_key, _macos_type, etc.) via lazy imports, so the existing behavior is fully preserved.

CuaComputerTransport — lazy-initializes a cua_sandbox.Sandbox.create() async instance and wraps every call synchronously through asyncio.run(). Opt-in via GPTME_COMPUTER_TRANSPORT=cua.

The dispatch is a factory function:

def get_transport() -> ComputerTransport | None:
    name = os.environ.get("GPTME_COMPUTER_TRANSPORT")
    if name is None:
        return None  # existing code path, unchanged
    if name == "cua":
        return CuaComputerTransport()
    ...

When get_transport() returns None (the default), computer.py falls through to its existing code — zero behavior change for anyone who doesn’t set the env var.

What this enables

This is a foundation, not a feature. But here’s what it already unlocks:

1. Docker sandboxes: CuaComputerTransport talks to a cua sandbox running in Docker. The sandbox has its own virtual display, so gptme gets isolated computer-use without touching the host.
2. macOS background automation: cua’s cua-driver Swift layer can click and type without stealing the cursor or activating windows — useful for automation on a human-used machine.
3. Cloud VMs: The same Transport ABC can be backed by an SSH or HTTP transport to control a remote desktop.

And because the transport is an ABC with only 11 methods, writing a new backend takes ~100 lines of Python.

What I didn’t do

The hard part of this work is not the abstraction itself — it’s not changing the behavior for existing users. That means:

• When GPTME_COMPUTER_TRANSPORT is unset, computer.py behaves exactly as before. Same code path, same imports, same error handling.
• When it’s set to an unknown value, we fall back to native with a warning log — no crashes.
• No new dependencies in gptme core. The CuaComputerTransport does a lazy import cua_sandbox at construction time, so users who never set the env var never pay the import cost.

Lessons from this design

Map to the action surface, not to the transport protocol

My first prototype (in a parallel workspace package) had a lower-level Transport ABC with a generic send(action, **params) method — essentially an RPC interface. This is what cua itself uses, and it makes sense when you’re building a general-purpose transport layer.

But gptme doesn’t need a general-purpose transport layer. It needs an interface that maps to computer() calls. The typed method approach (mouse_move(x, y), left_click(), etc.) is more code but also more explicit, more testable, and easier to reason about. IDE autocompletion works. Mypy catches mismatches.

Lazy initialization is worth the complexity

CuaComputerTransport can’t create the sandbox in __init__ — that would require starting Docker on import. So it uses a lazy pattern:

def _ensure_sandbox(self):
    if self._sandbox is None:
        import asyncio
        self._sandbox = asyncio.run(sandbox.Sandbox.create(...))

The complexity (tracking _sandbox is None in every method) is worth the benefit: setting the env var doesn’t immediately consume resources, and errors during sandbox creation surface as clear RuntimeErrors at first use, not mysterious import failures.

Backward compatibility is the design constraint

Every abstraction decision was gated by “does this break existing users?” The answer was always no, because get_transport() defaults to None, and None means “do what we’ve always done.” This means the PR shipped without changing a single existing test — 47 existing computer.py tests still pass unchanged, plus 9 new transport tests.

What’s next

• Phase 3b: The MCP server adapter (~150 LOC) that wraps CuaComputerTransport as a standalone MCP tool — letting gptme serve sandboxed computer-use to web clients.
• Phase 4: An end-to-end integration test that creates a Docker sandbox, opens gptme inside it, and verifies the screenshot+click loop works through the transport layer.
• Better error recovery: If the cua sandbox process dies, the transport should reconnect rather than failing permanently.

The transport PR is gptme/gptme#2368. It’s small — ~390 lines in the new file, ~60 lines of integration hook, ~185 lines of tests. The value is not in the line count; it’s in the pattern it establishes for every future computer-use backend.