dsh_omnivision

---

English

OmniVision is an OmniParser-powered GUI agent plugin for DeepSeek Harness. It turns the desktop or any image into structured elements (text + icons with pixel coordinates), so the model can see and operate the screen without multimodal vision.

Screenshots

Vision view (智能识别)	Recognition history (识别记录)

Features

Feature	Description
🖥 Screen recognition	Capture the desktop (or parse an image) and get interactive elements with pixel coordinates via OmniParser
🖱 Desktop automation	Click, double-click, right-click, drag, type (unicode-safe), key, hotkey, scroll — by element id or raw x/y
👁 Live vision dock	Real-time recognition view with SOM-annotated overlay, hover highlight, click-to-zoom
🕘 Recognition history	Thumbnails per capture, diff vs latest (added / removed / moved)
📋 One-click summary	Sends the recognized elements to the session for the model to summarize
🖼 File parsing	Parse local images from the dock, bypassing model multimodal limits
📊 Call log	Tracks every gui_* tool call and action for inspection

Tools

The plugin registers these gui_* tools into the shared tools registry:

• gui_capture — capture the desktop screen and run OmniParser to extract interactive elements (text + icons with pixel coordinates), refreshing the shared vision state and saving the SOM-annotated overlay
• gui_act — perform a real mouse/keyboard action on the desktop: click / double-click / right-click / move / type / press / hotkey / scroll / drag
• gui_find — search the elements from the last capture by text or type
• gui_state — show the current vision state without parsing
• gui_verify — re-capture and check whether a text is present or absent, retrying until it settles
• gui_task — execute a scripted multi-step UI plan with re-parse between steps and optional assertions
• gui_open_app — launch an installed desktop app by name via its Windows AUMID
• gui_parse_image — parse a third-party image (from a conversation attachment) into the shared vision state

Installation

dsh plugin --profile web add dsh_omnivision
# or a local path:
dsh plugin --profile web add G:\deepseek\plugins\dsh_omnivision

Restart the web process after installing. The plugin loads as a profile bundle layer:

• Host half registers the gui_* tools into the shared tools registry and serves the browser half over an HTTP route (/dsh-omnivision/<method>)
• Client half mounts the OmniVision dock into shell.overlay and a toggle into the sidebar footer

Prerequisites

• Windows with an OmniParser FastAPI server on http://127.0.0.1:8000 (e.g. via G:\omni\OmniParser\start_server.bat)
• The Python venv needs pyautogui for screenshots and input automation

Usage

Call the gui_* tools from the model side; the OmniVision dock in the browser provides the visual surface:

Control	Description
智能识别	Live element list + SOM-annotated image (hover to highlight, click to zoom)
🖼 解析图片	Pick a local image file to parse (bypasses model multimodal limits)
📋 总结	Send the current recognition info to the session for the model to summarize
识别记录	History with thumbnails + diff vs latest (added / removed / moved)
调用记录	Recent gui_* tool calls and action log
探测	Check OmniParser server connectivity

Development

pnpm build      # tsc (host) + tsc (client) + tsdown (client bundle)
pnpm typecheck

License

MIT

---

中文

OmniVision 是 DeepSeek Harness 的 OmniParser 驱动的 GUI 智能体插件。它把桌面或任意图片变成结构化元素（文本 + 图标 + 像素坐标），让模型无需多模态能力就能「看见」并操作屏幕。

界面截图

智能识别视图	识别记录视图

功能

功能	说明
🖥 屏幕识别	截取桌面（或解析图片），经 OmniParser 提取带像素坐标的可交互元素
🖱 桌面自动化	点击 / 双击 / 右键 / 拖拽 / 输入（Unicode 安全）/ 按键 / 热键 / 滚轮，按元素 id 或原始坐标操作
👁 实时识别 Dock	实时识别视图 + SOM 标注图（悬停高亮、点击放大）
🕘 识别记录	每次识别的缩略图 + 与最新对比（新增 / 移除 / 位移）
📋 一键总结	把当前识别信息发到会话，由模型按提示词总结
🖼 图片解析	从 Dock 选择本地图片解析，绕开模型多模态限制
📊 调用记录	记录每次 gui_* 工具调用与动作，便于排查

工具

插件向共享工具注册表注册以下 gui_* 工具：

• gui_capture — 以原生分辨率截取桌面，运行 OmniParser 提取可交互元素（文本 + 图标 + 像素坐标），刷新共享视觉状态并保存 SOM 标注图
• gui_act — 在桌面上执行真实鼠标/键盘操作：点击 / 双击 / 右键 / 移动 / 输入 / 按键 / 热键 / 滚轮 / 拖拽
• gui_find — 按文本或类型搜索上一次识别的元素
• gui_state — 查看当前视觉状态（不重新解析）
• gui_verify — 重新截屏解析，反复确认某文本出现或消失
• gui_task — 按脚本执行多步 UI 计划，步骤间重新解析并支持断言
• gui_open_app — 通过 Windows AUMID 按名称启动已安装的桌面应用
• gui_parse_image — 解析会话中的第三方图片（附件）进入共享视觉状态

安装

dsh plugin --profile web add dsh_omnivision
# 或本地路径：
dsh plugin --profile web add G:\deepseek\plugins\dsh_omnivision

安装后重启 web 进程。插件作为 profile bundle 层加载：

• Host 半把 gui_* 工具注册进共享 tools 注册表，并通过 HTTP 路由（/dsh-omnivision/<method>）为浏览器半提供数据
• Client 半把 OmniVision Dock 挂到 shell.overlay，侧栏底部挂开关按钮

前置依赖

• Windows + 运行在 http://127.0.0.1:8000 的 OmniParser FastAPI 服务（如 G:\omni\OmniParser\start_server.bat）
• Python venv 需安装 pyautogui（截图与输入自动化）

使用

模型侧直接调用 gui_* 工具；浏览器里的 OmniVision Dock 提供可视化操作：

控件	说明
智能识别	实时元素列表 + SOM 标注图（悬停高亮、点击放大）
🖼 解析图片	选择本地图片文件解析（绕开模型多模态限制）
📋 总结	把当前识别信息发到会话，由模型按提示词总结
识别记录	历史缩略图 + 与最新对比（新增 / 移除 / 位移）
调用记录	最近 gui_* 工具调用与动作日志
探测	检查 OmniParser 服务连通性

开发

pnpm build      # tsc (host) + tsc (client) + tsdown (client bundle)
pnpm typecheck

许可证

MIT