acceptodds
Under review as a conference paper at ICLR 2027

MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks

Abstract

LLM agents are increasingly built for medical work and scored on clinical benchmarks. Each such score, however, comes from a model running inside an agent harness, the system that controls the loop between the model and its environment. An agent's score is therefore a property of a model–harness pair. For medical agents, how much outcomes change with the harness has rarely been measured. Measuring this change, and explaining it, raises two challenges. First, a harness comparison must change nothing but the harness and be repeated across models and kinds of task. Second, comparing whole harnesses leaves their mechanisms bundled together, so it cannot show when an individual mechanism helps. To address these challenges, we present MedicalHarness, a controlled study of models and agent harnesses on medical tasks. We first build MedicalHarnessBench to evaluate agents on tasks across four domains that each test a different harness capability. Using this benchmark, we run five open-weight models under five agent harnesses, changing only the harness within a comparison, and analyze both outcomes and execution traces. To study individual mechanisms, we build MH-Lab, a controlled harness that switches off context management, planning or tool exposure one at a time within a shared execution loop. We find that the harness and its interaction with the model account for about a quarter of the outcome variance, and that no single harness is best across models and tasks. Code and data are available at https://anonymous.4open.science/r/MedicalHarness/.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Harness, Medical Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ibwhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

MedicalHarness is a combination of two (mostly) independent contributions:

  • It's a repackaging of existing benchmark tasks into 107 medical tasks
  • It's an evaluation of multiple agents/harnesses on those tasks
Critical strengths

The question of which harnesses work better with which models is useful.

Critical weaknesses

Both contributions in this paper are poorly done and add little.

  1. It's unclear how the repackaging of existing benchmarks into this dataset helps at all? There are no new tasks being released. If anything, running them in your benchmark is much worse than the original papers because you discarded tons of the existing benchmarks as part of your packaging. It's unclear why you removed tasks.

  2. The evaluation is poorly done. It only runs older, weaker models and doesn't evaluate frontier models that would be used in practice. Also, they only run subsets of the original benchmarks, so many of the estimates here are very noisy. There are related issues, like MH-Lab needing to appear in Table 1.

Questions for authors

Why did you only select a subset of tasks from each benchmark? Your numbers already have a lot of noise because of the small sample sizes.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ibwh [1/3]

We sincerely thank the reviewer for recognizing that the question of which harnesses work better with which models is useful. We address each concern below. New experiments are marked (new), and manuscript revisions are shown in blue.

Ethics Concern. Attribution of the Source Benchmarks

We thank the reviewer for asking us to credit the original benchmarks explicitly. All 107 tasks in MedicalHarnessBench are drawn from four public benchmarks, and the submission attributed each source in the appendix, Related Work and the Ethics Statement. We agree that this attribution belongs in the main text, and the revision now places it there.

Attribution in the Submitted Version. Appendix B, Table 3 lists, for every environment, the source benchmark and its license, the upstream pool, each selection step with the number of tasks it leaves, the kept count and the verification. Related Work (Section 2) cites all four benchmarks, and the Ethics Statement names each one with its license. The submission therefore disclosed every source.

Attribution in the Main Text (revised). Section 4.1 now states that every task comes from a public benchmark and names the source of each environment with its citation. The abstract and the contribution list now describe the 107 tasks as curated from four public benchmarks. The mapping is reproduced below.

EnvironmentSource benchmarkUpstream tasksKept
MedTool (clinical calculation)MedMCP-Calc [1]11832
MedMemory (long-context retrieval)MedMemoryBench [2]1,98623
MedPlanning (clinical data workflows)HealthAgentBench [3]5428
MedWeb (browser interaction)HealthAdminBench [4]13524

Scope of the Contribution. We do not claim the original tasks or their data as our contribution. The benchmark contribution is the collection and curation of these tasks. We select the tasks that exercise each harness capability and run all four environments under one execution and scoring interface. The main contribution is the evaluation built on them, a controlled comparison of five models under five harnesses over 8,025 episodes and a mechanism study with MH-Lab. Together, these revisions credit the original authors in the main text and state the scope of our contribution directly.

References

[1] Yakun Zhu et al. MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration. ACL, 2026.

[2] Yihao Wang et al. MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare. arXiv:2605.11814, 2026.

[3] Qianchu Liu et al. HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents. arXiv:2606.31179, 2026.

[4] Suhana Bedi et al. HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks. arXiv:2604.09937, 2026.

AuthorsOpenReview ↗

Response to Reviewer ibwh [2/3]

W1 and Q1. Rationale for Task Selection and Curation

We thank the reviewer for asking why each environment keeps a subset and what the curation adds. The selection is a quality filter, and it removes noise rather than adding it. Many upstream tasks cannot serve as reliable tests of an agent. Some can be answered without the capability they are meant to test, some cannot be answered from the information given, and some have gold answers or scorers that mark correct work as wrong. On such tasks, a score reflects the flaw of the task rather than the agent or its harness. We therefore keep only tasks that test the capability of their environment, have a gold answer supported by the record and a scorer that separates correct from incorrect answers, and a human expert verified every kept task. Appendix B, Table 3 lists every selection step with the number of tasks it leaves.

Quality Problems in the Source Tasks. Our audit found four kinds of flaws that would distort a harness comparison if the tasks were used as released.

  • Leaked answers. In MedMemory, 27 questions with long histories restate their answer in the recent sessions, so any agent answers them without long-range retrieval.
  • Unanswerable questions. In MedMemory, 38 temporal questions ask for a date that never appears in the visible record, so no agent can answer them.
  • Incorrect gold answers. In MedTool, the audit rejected 53 of the 107 tasks that pass the difficulty screen. The rejected tasks include gold answers with arithmetic contradictions, corrected sodium counted twice and requested outputs that the gold never supplies, so a correct agent would be marked wrong.
  • Unreliable scoring. Only 349 of the 1,986 upstream MedMemory questions have answers that can be scored reliably by exact match. The rest require judging free-form text.

Leaked answers push every harness to full credit, unanswerable questions push every harness to zero, incorrect gold answers penalize correct work and free-form judging adds its own noise. Keeping these tasks would therefore shrink the true differences between harnesses and add noise to every estimate. The kept tasks, by contrast, test the intended capability. Answering the kept MedMemory questions without the history, or with only the recent sessions, scores 0 of 23.

Unified Execution and Evaluation. MedicalHarnessBench does not replace its source benchmarks. It asks a different question, how the harness changes outcomes, which requires every model to run under every harness on the same tasks. The four sources come with different task formats, tools and scoring code, and curation runs them under five harnesses with one execution and scoring interface, a shared sandbox and a fixed budget, which no single source provides. The released benchmark also records the selection and verification of every task.

Together, the selection removes tasks that would add noise, and the curation makes the remaining tasks comparable across harnesses. Our response to W2 shows that the harness effects are significant at the resulting task counts.

AuthorsOpenReview ↗

Response to Reviewer ibwh [3/3]

W2. Model Coverage, Estimate Reliability and MH-Lab Reporting

We thank the reviewer for raising the choice of models, the noise of subset estimates and the placement of MH-Lab. The models are recent, the conclusions hold for a frontier model and the estimates are statistically reliable, as we detail below.

Recent and Large Open-Weight Models. The five open-weight models come from five developers and are recent and large. Four of them were released in 2026, and they range up to 120B parameters with GPT-OSS-120B. We serve them locally for two reasons. A controlled comparison requires every harness to see the same weights and serving stack, which hosted APIs do not guarantee. Open-weight models are also the realistic option for clinical deployments that must keep patient data on premises.

Frontier Model Evaluation (new). We have also added gpt-6-luna under all five harnesses on all 107 tasks (Appendix F.3). It scores higher in every environment, yet the harness effect does not shrink. On MedWeb, ZeroClaw reaches 0.21 while Hermes and Claude Code reach 0.50, a gap as large as under the open-weight models, and the harness order departs from that under the open-weight models.

Environmentgpt-6-lunaBest open-weight modelBest minus worst harness
MedTool0.390.130.09
MedMemory0.910.800.09
MedPlanning0.080.070.14
MedWeb0.420.400.29

Significance Tests of the Harness Effect (new). The estimates are not dominated by noise. Because each kept task is a valid test (W1), the harness effect stands out clearly from task sampling and seed variation. Every pair runs each task with three seeds, and seed-to-seed variation accounts for only about a tenth of the outcome variance on MedTool, MedMemory and MedWeb (Table 8). A permutation test that shuffles the harness labels within each model, task and seed gives p≤0.0005p \le 0.0005 on these three environments, and F-tests against the seed residual are significant on all four (Appendix E).

EnvironmentHarness + interaction sharePermutation pHarness F (p)
MedTool14.8%≤0.0005\le 0.00055.46 (0.001)
MedMemory39.8%≤0.0005\le 0.000524.7 (<10−10<10^{-10})
MedPlanning23.7%0.113.01 (0.027)
MedWeb18.9%≤0.0005\le 0.000514.9 (<10−7<10^{-7})

In a paired task bootstrap, the best and worst harnesses differ significantly in 15 of the 20 model ×\times environment cells, and harness rankings reverse significantly across models. Hermes, for example, is well ahead of ZeroClaw on MedMemory under GPT-OSS but behind it under GLM-4.7-Flash.

MH-Lab in the Main Results (revised). Table 1 now includes the full MH-Lab setting under Qwen3.6 and GPT-OSS, the two models it runs with, and refers to Table 2 for its configuration. With all mechanisms on, MH-Lab stays close to the best product harness under Qwen3.6, and it scores above the product-harness median in every environment under GPT-OSS.

Together, these results show that the harness effects hold across recent open-weight models and a frontier model and are statistically reliable at the current task counts. Table 1 now also places MH-Lab beside the product harnesses.

Dear authors,

Thank for for the responses, but I retain my score as the authors have not addressed my concerns. Adding Luna is not sufficient to represent the performance of frontier models. It's both not frontier level (too weak) and only one model provider.

In particular, I don't think the authors have addressed the ethical problem posed by the authors refusing to disclose that "their" benchmark is actually just a repacking of other people's work in the original paper.

This mistake by the authors irreversibly taints the review process as any reviewer who only looked at the main text (which is an explicitly allowed reviewing policy) will incorrectly come to the conclusion that the tasks were created by the authors.

Reviewer_hecdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how agent harnesses influence evaluation outcomes on medical benchmarks. The authors introduce MedicalHarnessBench, which comprises tasks across clinical calculation, long context retrieval, clinical data workflows, and browser interaction. Using five open weight models and five harnesses for a total of 8025 runs, they perform an analysis of variance to separate the contributions of the model, the harness, their interaction, and run to run variation. The study shows that the harness and its interaction with the model account for roughly 24% of outcome variance, that no single harness is universally superior, and that differences in performance frequently stem from answer delivery failures instead of reasoning errors.

Critical strengths

1. The experimental controls are solid. Holding the serving stack, context limits, and execution budgets fixed across more than eight thousand runs gives credible empirical numbers on harness impact. Showing that the harness and its interaction with the model drive about a quarter of outcome variance is an eye opening result that challenges how the community evaluates agents.

 

2. The execution analysis digs into important operational problems. Revealing that hiding tools behind discovery triples token waste because models keep calling tools by name, and showing that harness gaps are often driven by simple file write omissions instead of reasoning failures, offers immediate practical value for practitioners building medical agents.

Critical weaknesses
  1. With only 23 to 32 tasks per domain, the benchmark is underpowered to establish reliable harness rankings. In Table 10, 18 out of 20 leader margins have confidence intervals that cross zero. It is hard to know whether one harness is truly better or if it just got lucky on a handful of sample tasks.

 

  1. The authors note in appendix I that summarization triggers in only two to five percent of episodes under native settings. Artificially shrinking the input budget to sixty five thousand tokens in MH Lab forces the mechanism to fire, but that exaggerates its real world necessity given that modern models natively operate with much larger context windows.

 

  1. MedPlanning suffers from floor effects. Across all setups, models score between 0 and 10%, with 40% of the outcome variance coming from pure run to run noise. Measuring harness differences in a domain where models almost completely fail tells us very little about harness quality.
Questions for authors
  1. Figure 6 reveal that file-only scoring causes Kendall rank correlations to drop as low as 0.4 because models frequently output answers only in dialogue turns. Does this imply that the harness main effect is predominantly a prompt alignment artifact regarding output formatting, instead of a difference in agentic reasoning or tool utilization?

 

  1. Commercial harnesses such as Claude Code and Codex are tuned around specific proprietary frontier models. When deploying them over open weight models on local vLLM instances, did the authors observe systematic interface incompatibilities that might disadvantage them relative to harnesses designed with open weights in mind?

 

  1. In the abstract, the first sentence "XX are increasingly YY" sounds AI-patterned.

 

  1. In the introduction, in the first sentence, "LLM" can be Large language model (LLM).

 

  1. Texts in Figure 1 is too small. Very hard to read.

 

  1. In related works, frameworks like MedAgents (2024) & MDAgents (2024) can be mentioned for medical decision-making.
3 comments
AuthorsOpenReview ↗

Response to Reviewer hecd [1/2]

We sincerely thank the reviewer for recognizing the solid experimental controls across more than eight thousand runs, and the practical value of the execution analysis for practitioners building medical agents. We address each concern below. New experiments are marked (new), and manuscript revisions are shown in blue.

W1. Statistical Power and Ranking Reliability

We thank the reviewer for raising the size of each environment. Each environment keeps only high-quality tasks. Every kept task tests its target capability, has a gold answer supported by the record and a scorer that separates correct from incorrect answers, and the selection removes tasks that are trivial or that cannot be solved from the information given. Each comparison is therefore both challenging and meaningful.

The ranking-stability table asks a stricter question, whether the leading harness beats the runner-up in each cell. Our conclusions do not depend on a single leader. They rest on three tests that these task counts support, which the revision adds to Appendix E.

Significance Tests of the Harness Effect (new). A permutation test shuffles the harness labels within each model, task and seed, so that under the null the harness changes nothing. F-tests compare the harness effect with the seed residual. Both show a clear harness effect on MedTool, MedMemory and MedWeb, and the F-test is significant on all four environments.

EnvironmentHarness + interaction sharePermutation pHarness F (p)
MedTool14.8%≤0.0005\le 0.00055.46 (0.001)
MedMemory39.8%≤0.0005\le 0.000524.7 (<10−10<10^{-10})
MedPlanning23.7%0.113.01 (0.027)
MedWeb18.9%≤0.0005\le 0.000514.9 (<10−7<10^{-7})

Pairwise Comparisons and Rank Reversals (new). A paired task bootstrap finds a significant gap between the best and worst harness in 15 of the 20 model and environment cells. Harness rankings also reverse significantly across models. Hermes, for example, is well ahead of ZeroClaw on MedMemory under GPT-OSS but behind it under GLM-4.7-Flash.

Together, the selected tasks make each comparison demanding, and these tests show that the harness effect and its dependence on the model are detectable at these task counts.

W2. Realism of the 65K Input Budget

We thank the reviewer for asking whether the 65K budget overstates the need for context management. The budget places the mechanism study in a regime that real episodes often reach, as the measurements below show.

Request Length at Native Context Windows (new). We measure the largest request of every main-study episode, served at native context windows far larger than 65K. Of all episodes, 35% send at least one request above 65,536 input tokens, and 48% exceed 49.8K, where MH-Lab starts to summarize (Appendix I). Summarization rarely fires at the native windows because those windows are much larger, not because long requests are rare.

EnvironmentEpisodes above 65,536 tokensMedian peak request
MedWeb66%82.5K
MedPlanning55%71.8K
MedMemory21%39.3K
MedTool6%29.9K

Detectability of Mechanism Effects. A mechanism that fires in a fraction pp of episodes can change the paired outcome by at most pp, so switching it off at the native windows cannot detect its effect (Appendix I). The 65K budget places the ablation where context management acts. At this budget the full MH-Lab stays within 0.03 of the best product harness at the native windows in every environment under Qwen3.6, and above the product-harness median in every environment under GPT-OSS (Tables 1 and 2).

The revised Section 6.3 states that a third of native-window episodes already exceed this budget. Together, these results show that the budget lies within the range of real requests rather than at an artificial extreme.

AuthorsOpenReview ↗

Response to Reviewer hecd [2/2]

W3. Floor Effects on MedPlanning

We thank the reviewer for raising the floor on MedPlanning. The low binary score reflects a strict criterion rather than a lack of signal. A MedPlanning episode scores only if its result files pass the upstream task tests, which check the complete workflow, such as finding all eligible trials among hundreds of candidates. Beneath this all-or-nothing score, the environment records how far each agent gets, and these measures separate the harnesses clearly.

Continuous Process Measures (new). On the 17 MedPlanning tasks with a reference set, we score the returned items with F1, which credits correct items and penalizes both missed and extra ones. On F1, run-to-run variation accounts for about 11% of the variance, compared with 40% for the binary outcome, while the harness and its interaction explain about a third and are significant under both the permutation test and the F-test (Appendix E). Reference-set recall, the Proc. column of Table 1, likewise differs widely between harnesses under the same model.

Partial Progress and Mechanism Effects. MedPlanning is also where partial progress reveals a mechanism effect that the binary score hides. Without context management, binary success in MH-Lab stays rare, but reference-set recall drops markedly for both models (Section 6.3), so summaries preserve partial progress on long workflows.

Robustness of the Variance Decomposition. Without MedPlanning, the harness + interaction share stays at about a quarter of the outcome variance, so the headline result does not depend on this environment.

Together, MedPlanning measures how completely and precisely agents carry out multi-document clinical workflows, a harness difference that a binary score alone would hide.

Q1. File-Only Scoring and Output Formatting

We thank the reviewer for asking whether the harness effect reduces to output formatting. Our main results already credit an answer stated only in the final reply, so a correct answer given in the dialogue is scored. Figure 6 reports file-only scoring as a sensitivity analysis, to show how much the rankings would depend on the delivery channel if only the file counted.

Termination Causes of Unanswered Episodes (new). Under the main scoring, some MedTool and MedMemory episodes still end without an answer. In both environments, about two thirds of them end at the one-hour limit or at a harness guard such as the loop detector (Appendix F.2). These agents never reach an answer, so no formatting instruction would recover them.

Together, these results show that the harness effect on answer rate comes from whether an episode reaches an answer within its budget, not from how a reached answer is formatted.

Q2. Interface Compatibility of Commercial Harnesses

We thank the reviewer for this question on interface fit. We find no systematic disadvantage for Claude Code or Codex.

Tool-Call Validity and Harness Performance. Invalid tool-call JSON is rare, below 0.1% of requests for Gemma4, Nemotron and GLM-4.7-Flash, and a few percent for GPT-OSS, most often under Hermes. Claude Code or Codex has the best mean rank under all five models, and one of the two leads 15 of the 20 model ×\times environment cells.

Q3--Q6. Presentation and Related Work

We thank the reviewer for these suggestions, which we have adopted.

  • Q3. The opening sentence states a trend in a widely used form. We have still rewritten it to open with the premise of the paper, that the score of a medical agent belongs to a model and harness pair.
  • Q4. The introduction now spells out large language models (LLMs) at first use.
  • Q5. We have redrawn Figure 1 with larger text and shorter labels throughout, including the environment steps, the harness and model names and the MH-Lab switches. Its layout and content are otherwise unchanged.
  • Q6. Related Work now discusses MedAgents [1] and MDAgents [2], which coordinate several LLM agents for medical reasoning and decision-making. These systems study collaboration among model roles, while MedicalHarness varies the harness around a single agent.

References

[1] Xiangru Tang et al. MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning. Findings of ACL, 2024.

[2] Yubin Kim et al. MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making. NeurIPS, 2024.

Thank you for the additional analyses. Giving partial credit on MedPlanning makes the differences between harnesses easier to see. I’m still unsure how well the design lessons carry over beyond the two models tested in MH-Lab, so I’m keeping my score.

Reviewer_9mQyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark, MedicalHarnessBench, spanning four categories of clinical calculation, long-context retrieval, clinical data workflows, and browser interaction. It includes a 107 executable medical tasks in total. The paper provides the evaluation of 5 open weight agents ablating the harness used. The paper confirms the known finding in a rigorous evaluation setup. The underlying model matters for performance but its coupling with the harness is also important.

Critical strengths

This study examines the harness of agents in a medical context and validates well-known observations. In this work, an ablative study of harness is more meaningful because only one component changes between the two setups, while everything else remains the same. The paper reports an interesting finding: In calculation and memory tasks, the core difference in performance between harnesses was whether the agent handed in an answer at all. On questions where every harness submitted an answer, performance didn’t differ much. This suggests some failing harnesses mainly fail at reporting answers the model might already have. I am not sure if this is a completely new finding. The paper deals with four categories that address different parts of the harness. I enjoyed reading the paper, and from a medical application domain perspective, it is a nice contribution.

Critical weaknesses

The role of harness in an agent's performance is well known. The paper extends this study in the medical context. Hence, it is more of an application of an observation in a medical context. However, I felt the paper could have gone further by including more detailed, specific medical insights. This seems more like a testbed to showcase the study and the usefulness of harness in agents. Furthermore, all the tasks come from existing tests. Also, the paper tests only small and mid-sized open models. Claude Code and Codex are built on Anthropic's and OpenAI's internal models. Results can vary depending on model size. It is more of an engineering effort built on existing datasets, with few insights specific to the medical setting.

Questions for authors

Could the paper go further by including more detailed, specific medical insights?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9mQy

We sincerely thank the reviewer for recognizing the controlled design that changes only the harness, the finding that harnesses mainly differ in whether an answer is handed in, and the value of the study from a medical application perspective. We address each concern below. New experiments are marked (new), and manuscript revisions are shown in blue.

W1 and Q1. Clinical Insights and Design Implications

We thank the reviewer for this suggestion, and we agree that medical insight is what makes a harness study useful to this domain. Within the scope of a controlled study, we aim to provide such insights together with directions for future medical harnesses. The submission collects them in Appendix A, which turns the task diagnostics into design directions, and the revised Section 6.2 now points to this appendix explicitly. Two of these insights are particularly specific to clinical work.

Trial Eligibility Screening. On the MedPlanning trial tasks, agents return three to seven times more trials than are eligible, and each extra trial is a candidate that a coordinator must rule out by hand. Appendix A proposes plans that track every inclusion and exclusion criterion of each candidate, with its evidence and its status as met, not met or unknown.

Evidence Retention in Long Patient Histories. On MedMemory, questions whose evidence lies further back in the patient history have lower accuracy under every harness, so context management that compresses old sessions loses the facts that long-range clinical questions depend on. Appendix A proposes an evidence record that keeps key facts with their dates and sources, so that a compressed medication entry still shows when it was recorded and whether treatment ended.

Together, these examples show how the study connects harness mechanisms to concrete clinical failure modes and to directions for future medical harnesses.

W2. Benchmark Provenance and Curation

We thank the reviewer for noting that the tasks come from existing benchmarks. We do not claim these tasks as our own creation. All 107 tasks are drawn from four public benchmarks. The submission credits them in Related Work, the Ethics Statement and Appendix B, Table 3, which details how each environment was built, and the revision now also credits them in the abstract, the contribution list and Section 4.1. Our work on them is selection, design and organization. We select the tasks that test each harness capability and arrange them into four environments that run under five harnesses with one execution and scoring interface. These environments then serve our main goal, a controlled evaluation of models and harnesses on medical tasks.

W3. Generalization to Frontier Models

We thank the reviewer for raising model scale and the proprietary models behind Claude Code and Codex. The five open-weight models of the main study are recent, with four released in 2026, and range up to 120B parameters. We serve them locally so that every harness sees the same weights and serving stack, which a controlled comparison requires.

Frontier Model Evaluation (new). We have added gpt-6-luna under all five harnesses on all 107 tasks, with the same tasks, harness settings, scoring and one-hour limit as the main study (Appendix F.3). It scores higher than every open-weight model, yet the harness still matters. On MedWeb, ZeroClaw reaches 0.21 while Hermes and Claude Code reach 0.50. The harness order also departs from that under the open-weight models, and no harness leads in every environment. Even Codex, the harness built for OpenAI models, leads on MedTool and MedPlanning but not on MedMemory or MedWeb.

Environmentgpt-6-lunaBest open-weight modelBest minus worst harness
MedTool0.390.130.09
MedMemory0.910.800.09
MedPlanning0.080.070.14
MedWeb0.420.400.29

Together, the frontier evaluation shows that the harness effect and its dependence on the model persist beyond the open-weight models of the main study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.