dsh-vision-provider

English | 简体中文

dsh-vision-provider gives
DeepSeek Harness
selectable vision choices under one DeepSeek + Vision provider:

DeepSeek + Vision
  GLM-4.6V-Flash
  Qwen VL Max
  GPT-4.1 mini (Vision)

Select only one combination in Harness. The vision model named in that
selection is used behind DeepSeek:

Text-only message ───────────────────────────────> DeepSeek V4 Flash

Image message ──> private vision sidecar ──> visual description
                                               │
                                               └──> DeepSeek V4 Flash ──> answer

The vision model does not run as the final answer model. Instead, it appears as
part of a selectable DeepSeek combination. DeepSeek still performs reasoning,
tool use, and final response generation.

This is a community project. It is not an official DeepSeek or OpenAI
package.

Why v0.3.0 exists

Version 0.1.0 added a standalone model named vision-openai. DeepSeek
Harness can select only one model for a session, so users had to choose either
DeepSeek or the vision model. The two models could not cooperate.

Version 0.2.0 introduced a runtime composite adapter, but the vision model
remained hidden in environment configuration and Web UI showed only the vague
label DeepSeek V4 Flash + Vision.

Version 0.3.0 brings vision selection into Web UI:

• the plugin reads every model in Settings > Models that advertises
  image input;
• each vision model becomes a separate selectable DeepSeek combination;
• the combination name shows the vision display name, while its description
  starts with the exact model ID and provider route;
• text-only requests go directly to deepseek-official/deepseek-v4-flash;
• image-bearing messages are analyzed by the vision model selected in Web UI;
• the visual analysis replaces the raw image before the request reaches
  DeepSeek;
• DeepSeek remains the model that reasons, uses tools, and writes the final
  answer;
• repeated tool steps reuse cached image analysis in the current process.

This is a two-model bridge, not native pixel input for DeepSeek. The quality of
the final answer depends on both the vision sidecar and DeepSeek.

Requirements

• DeepSeek Harness 0.1.0-rc.5 or a compatible build.
• Node.js >=22.19.0.
• A configured DeepSeek API key for the native deepseek-official provider.
• At least one model in Settings > Models that advertises text and
  image, or a direct OpenAI-compatible vision endpoint.
• pnpm available to dsh plugin.

When upgrading, active routes such as vision-openai are read automatically
and their image-capable models become selectable combinations. A fresh install
also keeps one direct fallback: gpt-4.1-mini at
https://api.openai.com/v1. Any endpoint implementing OpenAI-compatible
/chat/completions image input can replace it.

Install

Harness source checkout

From the DeepSeek Harness repository:

Set-Location D:\deepseek-harness
$env:DSH_HOME = "D:\dsh-home"

pnpm dsh plugin --profile web add github:libinyam/dsh-vision-provider
pnpm dsh web

Installed dsh command

$env:DSH_HOME = "D:\dsh-home"

dsh plugin --profile web add github:libinyam/dsh-vision-provider
dsh web

Always use the same DSH_HOME for plugin management and startup.

Configure keys

The composite model ultimately uses two credentials:

1. DeepSeek key: configure the native DeepSeek provider in
   Settings > Models as usual.
2. Vision key: an existing vision-openai route continues using its own
   Harness configuration. The direct sidecar fallback uses
   VISION_OPENAI_API_KEY by default.

For the current PowerShell window:

$env:VISION_OPENAI_API_KEY = "your-vision-api-key"
pnpm dsh web

To persist it for future PowerShell windows:

[Environment]::SetEnvironmentVariable(
    "VISION_OPENAI_API_KEY",
    "your-vision-api-key",
    "User"
)

Close and reopen PowerShell after setting a persistent user variable.

API keys are never written to this repository or logged by the plugin. The
plugin first asks Harness's credential service for the configured reference,
then falls back to the launching process environment.

Use

1. Start or restart the Web profile.
2. Create a new session.
3. Select DeepSeek + Vision.
4. Select the vision model you want, for example GLM-4.6V-Flash.
5. Paste or drag an image into the composer.
6. Add a question and send it.

Select only one model. Its title is the vision model display name; the second
line starts with the exact API model ID. DeepSeek remains the final-answer
model.

Pure text messages skip the vision endpoint entirely.

Upgrade from v0.1.0

Stop Harness, then run:

Set-Location D:\deepseek-harness
$env:DSH_HOME = "D:\dsh-home"

pnpm dsh plugin --profile web update dsh-vision-provider
pnpm dsh web

If the GitHub dependency does not refresh, perform a clean reinstall:

pnpm dsh plugin --profile web remove dsh-vision-provider
pnpm dsh plugin --profile web add github:libinyam/dsh-vision-provider
pnpm dsh web

Existing vision-openai, GLM, Qwen, and other provider entries remain in
Settings > Models because they are user-owned configuration. Version
0.3.0 reads every model on those routes that advertises image input and
creates the matching combinations.

Do not delete a provider whose vision models you still want to select; those
routes are now the source of the Web combination catalog.

Add a vision model in Web UI

1. Open Settings > Models.
2. Add or edit a third-party provider.
3. Enter its provider ID, display name, protocol, endpoint, and credential
   reference.
4. Add the exact vision model ID and display name.
5. Save, then return to the conversation selector and open
   DeepSeek + Vision.

Models from Harness's built-in catalog already carry their input capabilities,
so known vision models appear automatically. Harness 0.1.0-rc.5 does not
expose modality controls for a hand-declared custom model in its Models page.
For such a model, add input: [text, image] to its settings.yaml entry, or
set the provider's defaultInput to [text, image], then restart Web.

The plugin automatically creates a selectable DeepSeek combination:

Vision Model Display Name
vision-model-id | Provider Display Name (provider-id) | Final answer: DeepSeek-V4-Flash

Declaring image capability is required. A custom model left at Harness's
default input: [text] is intentionally excluded from the vision catalog.

Provider details, credentials, model IDs, and display names can stay in Web UI.
The two modality lines below are the part a custom model may need in
settings.yaml:

llm-pi-ai:
  providers:
    my-vision:
      displayName: My Vision Provider
      apiKeyEnv: MY_VISION_API_KEY
      api: openai-completions
      baseURL: https://gateway.example/v1
      defaultInput: [text, image]
      models:
        - id: vendor-vision-model-id
          name: Vision Model Display Name
          input: [text, image]

Store the API key through the Web credential input. Do not put the secret
itself in settings.yaml.

Advanced: direct vision endpoint

To avoid registering a provider in Settings > Models, set a direct endpoint
before starting Harness:

$env:DSH_VISION_USE_LEGACY = "0"
$env:DSH_VISION_BASE_URL = "https://gateway.example/v1"
$env:DSH_VISION_MODEL = "vendor-vision-model-id"
$env:DSH_VISION_MODEL_NAME = "Vendor Vision Model"
$env:DSH_VISION_API_KEY_ENV = "MY_VISION_GATEWAY_KEY"
$env:MY_VISION_GATEWAY_KEY = "your-api-key"

pnpm dsh web

The direct model also appears as one combination under DeepSeek + Vision.
Because this fallback uses fetch directly, it does not pass through Harness
provider retries, llm/stream middleware, or provider token accounting.
Configure the vision model under Settings > Models when those integrations
are required.

Local endpoint without authentication

Some local OpenAI-compatible servers accept a placeholder Authorization
header:

$env:DSH_VISION_NO_AUTH = "1"
$env:DSH_VISION_BASE_URL = "http://127.0.0.1:11434/v1"
$env:DSH_VISION_MODEL = "your-local-vision-model"

pnpm dsh web

This sends Authorization: Bearer dsh-no-auth. Use it only with a trusted
local endpoint. Do not enable it for a remote service that requires a real
key.

Environment reference

Variable	Purpose	Default
DSH_VISION_DISPLAY_NAME	Composite provider label	DeepSeek + Vision
DSH_VISION_COMPOSITE_MODEL	Backward-compatible preferred combination ID and prefix for additional IDs	deepseek-v4-flash
DSH_VISION_COMPOSITE_NAME	Fallback name when the main model name cannot be read	DeepSeek V4 Flash + Vision
DSH_VISION_MAIN_PROVIDER	Internal text/reasoning provider	deepseek-official
DSH_VISION_MAIN_MODEL	Internal DeepSeek model	deepseek-v4-flash
DSH_VISION_BASE_URL	Vision API root	https://api.openai.com/v1
DSH_VISION_MODEL	Vision model ID for the direct combination	gpt-4.1-mini
DSH_VISION_MODEL_NAME	Vision model display name shown in Web UI	GPT-4.1 mini (Vision)
DSH_VISION_API_KEY_ENV	Vision credential reference	VISION_OPENAI_API_KEY
DSH_VISION_NO_AUTH	Use placeholder auth when set to 1	unset
DSH_VISION_MAX_TOKENS	Maximum vision-analysis output	4096
DSH_VISION_TIMEOUT_MS	Request timeout for direct and registered vision models	120000
DSH_VISION_DETAIL	OpenAI image detail: auto, low, or high	auto
DSH_VISION_USE_LEGACY	Make the configured registered route the preferred combination; set 0 to prefer direct	enabled
DSH_VISION_LEGACY_PROVIDER	Preferred registered vision provider route	vision-openai
DSH_VISION_LEGACY_MODEL	Optional preferred model ID; otherwise use that route's first image model	unset

Data flow and privacy

For a text-only request, no data is sent to the vision endpoint.

For an image-bearing message, the selected sidecar receives:

• the image bytes;
• text in the same image-bearing message;
• a fixed instruction asking for factual visual transcription.

DeepSeek receives the normal conversation plus the generated visual
description. The plugin does not send the entire conversation to the vision
endpoint unless every message in that conversation independently contains an
image.

Review both providers' retention and privacy policies. Image analysis can
incur a separate provider charge in addition to the DeepSeek request.

The process-local cache avoids analyzing the same persisted message on every
tool step. It is cleared when Harness restarts, so old image messages may be
analyzed again after a restart or session resume.

Troubleshooting

Images are still rejected

Create a new session and select a specific combination under
DeepSeek + Vision, not DeepSeek. The native deepseek-official model
intentionally declares text-only input.

A newly added vision model is missing

Check the model's input, or its provider's defaultInput, in
Settings > Models. It must include both text and image. Save, reopen the
model selector, and allow up to 30 seconds for the discovery cache to refresh.
Restart the Web profile if the catalog is still stale after that.

Inspect the composed tree:

pnpm dsh --profile web --dump-config

It should contain a row whose id and name are both
dsh-vision-provider.

The plugin reports MISSING_CREDENTIAL

Set the environment variable named by DSH_VISION_API_KEY_ENV. The default is
VISION_OPENAI_API_KEY. Restart Harness after changing persistent variables.

The vision endpoint returns 401 or 403

Check the sidecar key, Base URL, model ID, and gateway authentication rules.
The DeepSeek key and vision key are separate.

The endpoint says the model does not exist

DSH_VISION_MODEL must be the exact model ID accepted by the configured vision
endpoint. A display name is not an API model ID.

The vision model exhausts its output tokens

Reasoning-capable vision models can spend part of their output budget before
starting the visible description. The default is 4096. If the plugin reports
MAX_TOKENS, increase DSH_VISION_MAX_TOKENS and restart the Web profile.

The old standalone vision model is still visible

That route is user-owned provider configuration; the bundle does not create or
remove it. Its models that advertise image input become selectable
combinations under DeepSeek + Vision. Delete the provider only when none of
its vision models are needed.

DeepSeek answers without using the image

Confirm that the selected entry under DeepSeek + Vision names the intended
vision model. Then test that endpoint or choose another combination directly
from the selector. DeepSeek sees the sidecar's textual description, so omitted
visual details cannot be recovered later.

Update and uninstall

pnpm dsh plugin --profile web update dsh-vision-provider

pnpm dsh plugin --profile web remove dsh-vision-provider

Removing the bundle does not automatically delete user-owned provider settings
or credentials.

Development

npm test
npm pack --dry-run

Install a local checkout:

pnpm dsh plugin --profile web add "C:\path\to\dsh-vision-provider"

The runtime is dependency-free ESM and uses the services already supplied by
Harness: llm for nested DeepSeek routing and attachments for durable image
bytes.

Community acknowledgements

Thanks to the Linux.do community for its discussion,
feedback, and support.

License

MIT