dsh-computer-use-windows

dsh-computer-use

Windows Computer Use tools for DeepSeek Harness. The plugin combines:

• Windows window discovery and foreground input;
• UI Automation element discovery and focus state;
• window screenshots;
• ModLens-compatible 3x3 visual grounding;
• observation-bound element/coordinate actions;
• DeepSeek Harness one-shot approval prompts for risky actions.

Why the grid exists

ModLens deliberately returns semantic evidence rather than invented bounding
boxes. computer_observe therefore produces an annotated A1-C3 screenshot.
For a custom UI, call modlens_read_image on grid_screenshot_path, select the
reported cell with computer_zoom, and repeat until the region is small enough
to click. Standard controls should use the UI Automation element_index path.

Apps without an accessibility tree (WeChat, Electron, …)

Self-drawn UIs such as WeChat expose an empty accessibility tree, so
element_index navigation is unavailable. Use the screenshot path instead:

1. computer_observe returns a grid_screenshot_path. If a window renders
   blank on first capture (it was still activating or was occluded), the native
   helper re-activates it and retries before falling back to a screen copy; a
   still-occluded window now fails with a clear error instead of silently
   returning a screenshot of the wrong window.
2. Call modlens_read_image on the grid, pick an A1-C3 cell, computer_zoom,
   and repeat until the region is small. ModLens can time out or be imprecise:
   retry the read, and cross-check with computer_zoom rather than trusting a
   single grid answer.
3. WeChat hides to tray without minimizing: wake it with computer_hotkey
ctrl+alt+w (or computer_tray) before observing, instead of relying on
   the helper to reveal a blank window.

Install

# 从 npm 安装
npx -y @deepseek-ai/dsh plugin --profile web add dsh-computer-use-windows

# 或从 GitHub 安装
npx -y @deepseek-ai/dsh plugin --profile web add github:nanbbb/dsh-computer-use-windows

Build from source

npm run verify
npm pack
npx -y @deepseek-ai/dsh plugin --profile web add C:\absolute\path\to\dsh-computer-use-windows-0.2.1.tgz

Restart DeepSeek Harness after installation. Keep @liustack/modlens enabled;
this plugin does not duplicate or replace its vision provider.

Example task

Use Computer Use to open the existing Calculator window and calculate 19 × 7.
Do not operate any other application.

The model should call computer_list_windows, choose exactly one returned
window, call computer_observe, act once, and observe again.

Safety model

Risky calls use the Harness approval service. Missing approval support fails
closed. On-screen instructions never count as user authorization. The native
helper refuses invisible windows, invalid handles, out-of-window points,
password controls, terminal windows, authentication UI, and locked desktops.

完整的安全模型见 SECURITY.md。

配置

cordis.patch.yml 里的 config 字段（均可选，有默认值）：

字段	默认	说明
observationTtlMs	120000	观测快照有效期（毫秒），过期后 element_index / 坐标失效
maxAccessibilityElements	300	单次观测返回的 UI Automation 元素上限
screenshotDirectory	auto	截图输出目录；auto = 系统临时目录
requireApprovalForTyping	false	为 true 时所有打字强制走审批
allowedApps	[]	白名单；非空时只允许这些进程名（不含 .exe，如 ["notepad","chrome"]），其余全部拒绝
allowedExternalLabels	[]	允许降级「发送/提交」类标签的名单（子串匹配，不区分大小写，如 ["发送","send"]）。命中时按模型的显式 risk 处理；仅 external 类可放行，删除/支付/登录永远升级

This is an MVP for Windows 10/11 x64 and is fully self-contained — the
bundled native helper ships its own .NET runtime, so no separate install is
required.

Interaction model

Actions activate (bring to foreground) the target window before clicking,
typing, or pressing keys. This is a foreground mode: while the agent operates
a window, that window takes the foreground and the physical mouse moves to the
action point. It does not run in the background. Keep this in mind if you need
to use the machine while the agent is working.