dsh-ds-vision-auto-route

dsh-ds-vision-auto-route

English | 中文

Function plugin that routes image-bearing requests to a configurable image-capable model through the agent loop's agent/request waterfall. It does not change the selected model: every text-only request keeps the selection, and only a request whose turn introduced an image is replaced with the vision fallback. The loop logs the replacement as the request/header, so every routed call stays reconstructable, and the plugin appends a non-surface llm-vision-route/route record naming the turn, step, policy, and fallback route.

Compatibility with the published DeepSeek Harness release

This plugin routes on the agent loop's agent/request waterfall, which runs after the host's prompt-admission step. The image-admission gate lives in the Web and ACP entry layers, not in this plugin.

• In the source harness (running pnpm dsh web from the checkout), that admission consults this plugin's llm-vision-route service before accepting image input, so an image prompt on a text-only main model is admitted and the request is routed to the vision fallback automatically.
• In the published npm release (@deepseek-ai/dsh installed globally, dsh web), the admission still rejects image prompts for a text-only model before this plugin's waterfall can act. Installing this plugin alone does not change that host behavior, so a stock npm host will not auto-route images on a text-only main model — the image is rejected and you would need to switch the model to the vision model manually.

The plugin takes effect out of the box on a source build, or on any host whose admission consults the vision-route service (the service-aware host admission is planned to ship in a future official release). Manual model selection always works either way.

The routing decision is a pure function of the durable log. routePolicy: turn-image (default) inspects only the open turn — events after the latest turn/start — so a text-only follow-up returns to the selected model while the historical image remains in the log; those later requests reach the selected model's ordinary negative-capability projection, which replaces the historical image with a stable placeholder text. any-image routes whenever the log contains any image, and off disables routing.

A request routes only when the selected model's exact metadata does not declare image input; a selected model that already accepts images is never replaced. The fallback route must declare image input, or the request fails with INVALID_REQUEST naming the fallback — a deployment pointing at a broken fallback fails loud instead of silently dropping pixels. A routed request drops reasoningEffort, which the fallback model may not support; sampling values pass through unchanged.

The plugin provides the llm-vision-route service (VISION_ROUTE_SERVICE), whose routesImagesFor(agent) answers whether an image-bearing prompt on one agent reaches an image-capable model, either directly or through routing. The Web prompt admission and the ACP content bridge consult it before accepting image input, so image prompts are admitted exactly when the router will serve them; without the service both surfaces keep their existing text-only rejection. available() answers the same question for initialization-time capability advertising (ACP promptCapabilities.image) and answers false on any unknown fact.

The separately published ./invariant companion checks that every routing record sits inside an open turn and step, names the configured fallback, carries a valid policy, and follows image-bearing input in the inspected range (the open turn under turn-image, the whole prior log under any-image).

- name: 'dsh-ds-vision-auto-route'
  config:
    visionProvider: deepseek-official
    visionModel: deepseek-v4-flash-vision-exp
    routePolicy: turn-image

All three keys are optional: the defaults are the official provider route, the shipped vision model, and turn-image.

Installation

dsh plugin --profile web add github:litianshuo110/dsh-ds-vision-auto-route

dsh plugin add forwards to pnpm, and this package's dsh.bundle.patch wires the llm-vision-route row into the profile automatically. Manual alternative: clone the repository and add dsh-ds-vision-auto-route to the profile, or append the row from cordis.patch.yml to $DSH_HOME/profiles/<profile>/cordis.patch.yml.

Development

The plugin is developed in the DeepSeek Harness repository at packages/llm/llm-vision-route, where its unit, REAL-composition, invariant, and keyless snapshot tests live; this repository mirrors the shipped package. Rebuild the committed lib/ artifacts with pnpm install && pnpm run build.

Model Experience

Vision routing

What the model sees

An image-bearing turn sees the complete image content on the routed vision model, preceded by the stable attachment handle text the provider adapter adds. Text-only turns see no routing artifacts; under turn-image, historical images reach the selected text model as the standard placeholder text instead of pixels. The non-surface routing record is log-only.

Token effect

Routing itself adds no tokens. The routed turn bills under the fallback model's pricing; each request image costs at most 384 tokens after the provider's automatic resize. The two exact-model metadata resolutions per request are catalog lookups, not model calls.

KV Cache effect

The routed request runs on a different model, so its cache identity differs from the selected model's. Under turn-image, alternating between the selected model and the fallback across turns resets the prefix cache at each switch boundary; a session that never sends images keeps one uninterrupted cache prefix.

Known Limitations and Deferred Work

• The selection itself never changes — the Web model selector keeps showing the selected model; a routed turn runs on the fallback without moving the selection. A session that has never picked a model explicitly inherits the last logged header, so after an image turn its next request follows the routed route until a selection is made.
• Turn-local routing projects historical images — after the image turn, later text-model contexts replace the durable image with placeholder text; the pixels return only when the image is attached again (which re-triggers routing) or when a tool reads it on a routed turn.
• Catalog metadata decides capability — a model whose exact metadata omits inputModalities is treated as text-only and gets routed when images appear; declare image input on unlisted pass-through models that accept images to keep them un-routed.
• any-image stays routed — once the log contains an image, every later request whose proposal is not image-capable routes again until compaction or a selection change; choose it when image context should never degrade.
• No provider failover — the fallback must be a registered, image-capable route on the same or another provider; a missing or non-image fallback fails the image-bearing request loudly.