- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
MedicalHarness is a combination of two (mostly) independent contributions:
- It's a repackaging of existing benchmark tasks into 107 medical tasks
- It's an evaluation of multiple agents/harnesses on those tasks
Critical strengths
The question of which harnesses work better with which models is useful.
Critical weaknesses
Both contributions in this paper are poorly done and add little.
-
It's unclear how the repackaging of existing benchmarks into this dataset helps at all? There are no new tasks being released. If anything, running them in your benchmark is much worse than the original papers because you discarded tons of the existing benchmarks as part of your packaging. It's unclear why you removed tasks.
-
The evaluation is poorly done. It only runs older, weaker models and doesn't evaluate frontier models that would be used in practice. Also, they only run subsets of the original benchmarks, so many of the estimates here are very noisy. There are related issues, like MH-Lab needing to appear in Table 1.
Questions for authors
Why did you only select a subset of tasks from each benchmark? Your numbers already have a lot of noise because of the small sample sizes.
4 comments
Response to Reviewer ibwh [2/3]
W1 and Q1. Rationale for Task Selection and Curation
We thank the reviewer for asking why each environment keeps a subset and what the curation adds. The selection is a quality filter, and it removes noise rather than adding it. Many upstream tasks cannot serve as reliable tests of an agent. Some can be answered without the capability they are meant to test, some cannot be answered from the information given, and some have gold answers or scorers that mark correct work as wrong. On such tasks, a score reflects the flaw of the task rather than the agent or its harness. We therefore keep only tasks that test the capability of their environment, have a gold answer supported by the record and a scorer that separates correct from incorrect answers, and a human expert verified every kept task. Appendix B, Table 3 lists every selection step with the number of tasks it leaves.
Quality Problems in the Source Tasks. Our audit found four kinds of flaws that would distort a harness comparison if the tasks were used as released.
- Leaked answers. In MedMemory, 27 questions with long histories restate their answer in the recent sessions, so any agent answers them without long-range retrieval.
- Unanswerable questions. In MedMemory, 38 temporal questions ask for a date that never appears in the visible record, so no agent can answer them.
- Incorrect gold answers. In MedTool, the audit rejected 53 of the 107 tasks that pass the difficulty screen. The rejected tasks include gold answers with arithmetic contradictions, corrected sodium counted twice and requested outputs that the gold never supplies, so a correct agent would be marked wrong.
- Unreliable scoring. Only 349 of the 1,986 upstream MedMemory questions have answers that can be scored reliably by exact match. The rest require judging free-form text.
Leaked answers push every harness to full credit, unanswerable questions push every harness to zero, incorrect gold answers penalize correct work and free-form judging adds its own noise. Keeping these tasks would therefore shrink the true differences between harnesses and add noise to every estimate. The kept tasks, by contrast, test the intended capability. Answering the kept MedMemory questions without the history, or with only the recent sessions, scores 0 of 23.
Unified Execution and Evaluation. MedicalHarnessBench does not replace its source benchmarks. It asks a different question, how the harness changes outcomes, which requires every model to run under every harness on the same tasks. The four sources come with different task formats, tools and scoring code, and curation runs them under five harnesses with one execution and scoring interface, a shared sandbox and a fixed budget, which no single source provides. The released benchmark also records the selection and verification of every task.
Together, the selection removes tasks that would add noise, and the curation makes the remaining tasks comparable across harnesses. Our response to W2 shows that the harness effects are significant at the resulting task counts.
Response to Reviewer ibwh [3/3]
W2. Model Coverage, Estimate Reliability and MH-Lab Reporting
We thank the reviewer for raising the choice of models, the noise of subset estimates and the placement of MH-Lab. The models are recent, the conclusions hold for a frontier model and the estimates are statistically reliable, as we detail below.
Recent and Large Open-Weight Models. The five open-weight models come from five developers and are recent and large. Four of them were released in 2026, and they range up to 120B parameters with GPT-OSS-120B. We serve them locally for two reasons. A controlled comparison requires every harness to see the same weights and serving stack, which hosted APIs do not guarantee. Open-weight models are also the realistic option for clinical deployments that must keep patient data on premises.
Frontier Model Evaluation (new). We have also added gpt-6-luna under all five harnesses on all 107 tasks (Appendix F.3). It scores higher in every environment, yet the harness effect does not shrink. On MedWeb, ZeroClaw reaches 0.21 while Hermes and Claude Code reach 0.50, a gap as large as under the open-weight models, and the harness order departs from that under the open-weight models.
| Environment | gpt-6-luna | Best open-weight model | Best minus worst harness |
|---|---|---|---|
| MedTool | 0.39 | 0.13 | 0.09 |
| MedMemory | 0.91 | 0.80 | 0.09 |
| MedPlanning | 0.08 | 0.07 | 0.14 |
| MedWeb | 0.42 | 0.40 | 0.29 |
Significance Tests of the Harness Effect (new). The estimates are not dominated by noise. Because each kept task is a valid test (W1), the harness effect stands out clearly from task sampling and seed variation. Every pair runs each task with three seeds, and seed-to-seed variation accounts for only about a tenth of the outcome variance on MedTool, MedMemory and MedWeb (Table 8). A permutation test that shuffles the harness labels within each model, task and seed gives on these three environments, and F-tests against the seed residual are significant on all four (Appendix E).
| Environment | Harness + interaction share | Permutation p | Harness F (p) |
|---|---|---|---|
| MedTool | 14.8% | 5.46 (0.001) | |
| MedMemory | 39.8% | 24.7 () | |
| MedPlanning | 23.7% | 0.11 | 3.01 (0.027) |
| MedWeb | 18.9% | 14.9 () |
In a paired task bootstrap, the best and worst harnesses differ significantly in 15 of the 20 model environment cells, and harness rankings reverse significantly across models. Hermes, for example, is well ahead of ZeroClaw on MedMemory under GPT-OSS but behind it under GLM-4.7-Flash.
MH-Lab in the Main Results (revised). Table 1 now includes the full MH-Lab setting under Qwen3.6 and GPT-OSS, the two models it runs with, and refers to Table 2 for its configuration. With all mechanisms on, MH-Lab stays close to the best product harness under Qwen3.6, and it scores above the product-harness median in every environment under GPT-OSS.
Together, these results show that the harness effects hold across recent open-weight models and a frontier model and are statistically reliable at the current task counts. Table 1 now also places MH-Lab beside the product harnesses.
Dear authors,
Thank for for the responses, but I retain my score as the authors have not addressed my concerns. Adding Luna is not sufficient to represent the performance of frontier models. It's both not frontier level (too weak) and only one model provider.
In particular, I don't think the authors have addressed the ethical problem posed by the authors refusing to disclose that "their" benchmark is actually just a repacking of other people's work in the original paper.
This mistake by the authors irreversibly taints the review process as any reviewer who only looked at the main text (which is an explicitly allowed reviewing policy) will incorrectly come to the conclusion that the tasks were created by the authors.
Response to Reviewer ibwh [1/3]
We sincerely thank the reviewer for recognizing that the question of which harnesses work better with which models is useful. We address each concern below. New experiments are marked (new), and manuscript revisions are shown in blue.
Ethics Concern. Attribution of the Source Benchmarks
We thank the reviewer for asking us to credit the original benchmarks explicitly. All 107 tasks in MedicalHarnessBench are drawn from four public benchmarks, and the submission attributed each source in the appendix, Related Work and the Ethics Statement. We agree that this attribution belongs in the main text, and the revision now places it there.
Attribution in the Submitted Version. Appendix B, Table 3 lists, for every environment, the source benchmark and its license, the upstream pool, each selection step with the number of tasks it leaves, the kept count and the verification. Related Work (Section 2) cites all four benchmarks, and the Ethics Statement names each one with its license. The submission therefore disclosed every source.
Attribution in the Main Text (revised). Section 4.1 now states that every task comes from a public benchmark and names the source of each environment with its citation. The abstract and the contribution list now describe the 107 tasks as curated from four public benchmarks. The mapping is reproduced below.
Scope of the Contribution. We do not claim the original tasks or their data as our contribution. The benchmark contribution is the collection and curation of these tasks. We select the tasks that exercise each harness capability and run all four environments under one execution and scoring interface. The main contribution is the evaluation built on them, a controlled comparison of five models under five harnesses over 8,025 episodes and a mechanism study with MH-Lab. Together, these revisions credit the original authors in the main text and state the scope of our contribution directly.
References
[1] Yakun Zhu et al. MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration. ACL, 2026.
[2] Yihao Wang et al. MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare. arXiv:2605.11814, 2026.
[3] Qianchu Liu et al. HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents. arXiv:2606.31179, 2026.
[4] Suhana Bedi et al. HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks. arXiv:2604.09937, 2026.