Blog · Analysis
DeepSeek Harness benchmarked: 60 real tasks, three axes
August 25, 2026 · dshbase · quantitative evaluation for installers
Most agent reviews end at "it worked" or "it didn't." A proper evaluation needs numbers, and the Shanghai AI Laboratory's Sinan (司南) evaluation system published exactly that: a run of 60 long-horizon agent tasks in real environments, scored on three axes — Score (task completion ability), Cost (calling cost), and Time (execution duration). The setup runs on WildClawBench, covering productivity workflows, code intelligence, social interaction, search & retrieval, creative synthesis, and safety alignment, in both Chinese and English, with multimodal tasks included.
The headline: "mid-tier ability, low cost, slow execution"
DeepSeek V4 Pro + DeepSeek Harness scored 45.0 overall. That places it between GPT-5.4 + DSH (53.8) and GLM 5 / Qwen3.5 397B + DSH (39.2) — rank 3 of the seven price-stable DSH configurations compared. The one-line verdict from the report: "中上能力、低成本、较高耗时" — mid-to-upper capability, outstandingly low cost, noticeably longer runtime.
Reference lanes exist only to show the ceiling: GPT-5.6 Sol and Claude Fable 5 (via OpenClaw) cost roughly 9.5× and 14.6× more per task respectively, so they are deliberately not treated as same-price comparisons.
Same model, different harness: DSH halves the bill
The sharpest finding comes from holding the model constant. Compared with OpenClaw on the same V4 Pro, DeepSeek Harness delivered a small score gain while cutting cost by ~50%, at the price of ~4.2% extra time. Scaled to the full 60-task suite: total model-calling cost lands around US$6 versus ~US$12, sequential runtime ~630 minutes versus ~605. The efficiency ratio — points per dollar — roughly doubles from ~218 to ~450 (+106%).
Model dependence: the harness isn't a free upgrade for everyone
Cross-model adaptation is uneven, and that's the honest part of this report:
- GPT-5.4 + DSH — the standout: 53.8 score (rank 2, below Codex), lowest cost at $0.30/task (−9.1%), and the fastest average at 3.98 min/task.
- Qwen3.5 397B + DSH — all three metrics improved versus OpenClaw: rare and genuine synergy.
- DeepSeek V4 Pro + DSH — small score gain, big cost cut, slower runtime; best fit for cost-sensitive, batch-friendly work.
- Grok 4.20 Beta + DSH — the biggest score jump, but at nearly double the time and higher cost.
- GLM 5 and MiniMax M2.7 + DSH — no clear advantage over OpenClaw yet; GLM 5 hits 10.60 min/task, the slowest.
Latency, tool-calling strategy, failure retries, context compression and task-termination judgment all feed Time — and none of them transfer cleanly across models yet.
What an installer should take away
- Cost-sensitive pipeline? V4 Pro + DSH is currently the most efficient price/performance combination measured. Batch or asynchronous workloads fit its ~10.5 min average task time.
- Interactive or latency-bound? Don't anchor on V4 Pro alone — the benchmark shows GPT-5.4 + DSH around 3.98 min with lower cost. Harness choice matters as much as model choice.
- Expect variance: the same harness amplifies some models and does nothing for others. "Everything is a plugin" also means the task planner, prompt template and tool protocol still need per-model tuning.
The report's own roadmap: shorten tool-call chains, cut ineffective loops, improve context compression and task-termination policies. For a developer-preview runtime already in week two, a measured 45.0 with sub-$0.1-per-task economics is a respectable baseline — and the points-per-dollar curve is the metric to re-run after each release.