dsh-model-probe

DSH Model Probe

Persistent model-health monitoring for the DeepSeek Harness (DSH) Web UI.

DSH Model Probe discovers the models advertised by active DSH providers, probes them through DSH's own LLM runtime, keeps a persistent Alive / Dead health ledger, observes failures from normal DSH model usage, and automatically rechecks unhealthy models.

This is a third-party DSH plugin maintained by AGSQ11. It is not an official DeepSeek project.

Screenshots

Alive models

Dead models and retry diagnostics

Features

• Native DSH Web UI integration under Settings → Model Probe.
• App-wide runtime picker health badges: Alive models get a green dot and Dead models get a red dot in DSH model-selection lists.
• Health-first model ordering: runtime pickers show Alive models first, unknown/unprobed models next, and Dead models last; each tier is sorted alphabetically A→Z.
• Fast model filtering: the composer Model list gains a case-insensitive prefix filter over model display name and model ID (for example, nem → nemotron-3-ultra). DSH's /model popup keeps its existing native search field.
• Uses DSH semantic --dsw-* theme tokens and follows Light, Dark, and System themes.
• Higher-contrast diagnostic/secondary text for dense health data in both light and dark palettes.
• Enumerates every model advertised by every currently registered DSH provider adapter.
• Alive / Dead tabs with provider, model, latency, TTFT, result/error, attempt count, and next retry time.
• Manual Probe all with configurable timeout and concurrency.
• Persistent host-side health state that survives browser and DSH restarts.
• Runtime monitoring of normal DSH llm/stream calls.
• Two independent health retries after a real model error before declaring the model Dead.
• Dead models are rechecked automatically every 30 minutes.
• Successful normal DSH traffic immediately revives a previously Dead model.
• Explicit user cancellation/abort is not treated as model death.
• Bounded health history (latest 500 outcomes).

Health policy

Manual probe

A manual probe has up to three attempts:

1. Initial minimal probe.
2. Retry #1 if the initial probe fails.
3. Retry #2 if retry #1 also fails.
4. The model is marked Dead only if all three attempts fail.

Failure during normal DSH use

If a normal DSH request ends in a terminal provider/model error:

1. That failed user request counts as the initial failure.
2. Model Probe sends health retry #1.
3. If necessary, it sends health retry #2.
4. The model becomes Dead only if both health retries fail and no successful normal request has revived it in the meantime.

The plugin intentionally does not replay the user's original request. Replaying a partially streamed response could duplicate assistant output or tool calls. Health verification uses a separate, minimal request:

Reply with exactly OK.

with maxTokens=8.

Dead-model recovery

A Dead model receives one background health probe every 30 minutes:

• success → Alive immediately;
• failure → remains Dead and gets another check 30 minutes later.

The scheduler runs in the DSH host process, so the Model Probe page does not need to remain open. If DSH is stopped, overdue checks resume after DSH starts again.

Runtime model-picker integration

Model Probe reuses its persistent provider+model health ledger in DSH's runtime model selectors:

• green dot — the exact provider/model is currently Alive;
• red dot — the exact provider/model is currently Dead;
• no dot — the model has not reached a known Alive/Dead state yet.

The composer Model pane gains a search box above the provider-grouped list. Matching is case-insensitive and prefix-based against both the displayed model name and the provider model ID. Typing nem therefore matches an ID such as nemotron-3-ultra even when its human-readable display name is different. Provider groups with no matches are hidden while the filter is active.

DSH's /model popup already provides its own local search input, so Model Probe does not replace it; it adds health dots to those rows and leaves DSH's native keyboard/search behavior intact.

Health-first ordering

Runtime model lists are reorganized globally by health rather than retaining provider-first visual grouping:

1. Alive models (green dot), A→Z
2. Unknown / not yet probed models (no dot), A→Z
3. Dead models (red dot), A→Z

Provider context remains visible inline, and deterministic provider/model-id tie-breakers are used when display names are identical. The composer and /model picker use the same ordering. Filtering is applied on top of that ordering, so typing a prefix keeps the matching results health-sorted.

Compatibility note

At the time of v0.3.0, DSH's public model-selection client face exposes the shared model directory plus load/select operations, but no third-party per-row decorator/header slot. To avoid replacing or forking DSH's selector, Model Probe's picker enhancer attaches only to accessibility semantics (role=menu, role=group, role=menuitemradio, role=listbox, and role=option) and deliberately does not depend on DSH's generated CSS-module class names. This is substantially less brittle than styling private class names, but a future DSH change to the selector's semantic markup could still require a plugin update.

The provider configuration/editor model rows under Settings → Models are intentionally not treated as runtime health selectors: those rows can describe dormant or not-yet-active routes that cannot truthfully be labeled Alive or Dead.

Persistence

The host stores current health state and a bounded result history in state.json.

Default paths:

Platform	Path
Windows	%LOCALAPPDATA%\\dsh-model-probe\\state.json
macOS	~/Library/Application Support/dsh-model-probe/state.json
Linux/Unix	$XDG_STATE_HOME/dsh-model-probe/state.json or ~/.local/state/dsh-model-probe/state.json

Override the location with:

DSH_MODEL_PROBE_STATE_FILE=/custom/path/state.json

No provider credentials are stored by Model Probe. Probe requests use the provider adapters and credentials already configured in DSH.

Install

Clone or download this repository, then add the local plugin directory to the DSH profile:

dsh plugin --profile default add ./dsh-model-probe

Replace default with the profile actually in use, restart DSH, and open:

Settings → Model Probe

Update

Replace the plugin directory with the newer release/source, keep the same DSH profile entry, and restart DSH. Persistent health state is kept separately from the plugin directory.

Model discovery limitation

DSH provider catalogs are advisory: an adapter may accept model IDs that it does not advertise. Model Probe therefore checks all advertised models, but it cannot infer arbitrary hidden model IDs that no provider exposes through its catalog.

Health is keyed by provider + model, not model name alone. For example, the same model ID can be Alive on one provider and Dead on another.

Development

The distributed package is intentionally small and prebuilt. The browser and host entry points live in both src/ and lib/ so DSH can load the package directly.

Run the test suite with:

npm test

The tests cover failure classification, persistence, manual retry policy, runtime-error verification, dead-model scheduling/recovery behavior, API routes, DSH theme integration, picker health resolution, ambiguity handling, and prefix filtering.

License

MIT © 2026 AGSQ11. See LICENSE.