dsh-eval-harness
Evaluation harness for DSH plugins: YAML cases drive real headless agent runs, assert on tool calls, args, results and token usage, with a baseline gate for CI regression.
DSH 插件评测框架：YAML 用例驱动真实 headless agent，断言工具调用/参数/返回与 token 用量，baseline 门禁做 CI 回归。
How to install
dsh plugin add dsh-eval-harness About
dsh-eval-harness DSH 插件/skill 作者的回归评测门禁：写 yaml 用例 → headless 驱动真实 agent 跑 → 解析 session trace 断言 → 对比 baseline 出 PASS/WARN/FAIL 报告与 CI 退出码。 简介 给 DSH 插件/skill 的回归评测流程提供一个可进 CI 的门禁工具： 1. 用 yaml 写评测用例（prompt + 期望行为断言）； 2. eval_run 逐条 fork dsh --profile headless --patch <overlay> <prompt> 子进程跑真实 agent（overlay 把会话落盘切到隔离目录，每条用例独立 workspace），解析落盘的 session.jsonl / session.jsonl.zstd trace（多帧 zstd 直读），执行断言，写 report.json + report.md； 3. eval_gate 把本次报告与 baseline 报告对比，输出 OVERALL=PASS|WARN|FAIL|N/A 与退出码，供 CI 拦截回归。 安装 已发布到 npm（dsh-eval-harness）： sh dsh plugin --profile headless add dsh-eval-harness 或从 GitHu…
Recommendation signals
Meta
- License
- MIT
- Language
- TypeScript
- GitHub stars
- 13
- mo. downloads
- 480
- Last push
- 2026-09-03
- Created
- 2026-08-13
Links
Basic safety check
- Findings
- None
- Sources
- curated:awesome-dsh-plugin.com, curated:awesome-dsh-plugin/awesome-dsh-plugin
- Topics
- deepseek-harness, dsh, dsh-plugin, evaluation, regression-testing
Related plugins
langfuse
langfuse/langfuse
🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
claude-code-templates
davila7/claude-code-templates
CLI tool for configuring and monitoring Claude Code
semiotic
nteract/semiotic
React data visualization library for streaming, networks, and AI-assisted development
dsh-desktop
dataelement/dsh-desktop
DSHDesktop：DeepSeek Harness Desktop / DeepSeek Harness 桌面版