- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors present a harness-optimization framework for sequential clinical decision making. A frozen Qwen3.5-9B agent is wrapped in a patient entity graph that coordinates memory retrieval, clinical skills and imaging tools, and an LLM proposer iteratively edits this harness using a Candidate-DAG history and a self-updating library of proposal skills.
Critical strengths
The stage-indexed entity graph is a sensible and inspectable design, and the case-level bookkeeping of corrections and regressions is a reasonable improvement over greedy, score-only harness search.
Critical weaknesses
- Section 3.1.2 states that during testing, skill performance is assessed with the test reference answers and the benchmark evaluator, and that this feedback is passed to the proposer, which decides whether to update Q(s). If test labels influence skill quality, skill eligibility or any later proposer decision, the reported test scores are not held-out estimates. The authors must clarify exactly when test labels are used and whether any test-derived signal reaches a harness that is later evaluated on the same test set. If such a path exists, all headline numbers should be recomputed without it. The leakage audit in step (2) of the evolution loop is carried out by an LLM, so it does not resolve this concern either.
- Gains are confounded with privileged access to training data, and key baselines are missing. The framework builds a global disease graph and a contrastive skill library from 8724 MedChain and 840 ClinicalBench training cases, plus 1,000 and 200 cases for evolution. Most baselines (zero-shot LLMs, MDAgents, MedAgent) appear to have no comparable access to this data. The ablation suggests the data matters more than the optimizer. The variant without the self-adapting proposer already exceeds both harness baselines, so much of the margin seems to come from the initial harness and its training-derived memory rather than from the proposed optimization procedure.
- The central claim about optimization collapse is not tested. The introduction argues that fixed proposers stagnate or regress and that a self-updating proposer avoids this, yet the paper contains no search-dynamics evidence. The single ablation row does not show that the claimed failure mode was present or that it was fixed. Proposer token usage, wall-clock time and monetary cost are also not reported. This matters because the method depends on a frontier proprietary model, and recent work the authors themselves cite stresses matched optimization budgets.
- Generalization is not established. Both benchmarks come from Chinese online or hospital sources. The harness is evolved separately for each benchmark, and memory, skills and evolution cohorts are drawn from the same distribution as the test set. In ClinicalBench, each of the 150 diseases has ten cases, so test diseases are almost certainly represented in the global disease graph, and retrieval may function as disease-level lookup rather than reasoning.
Questions for authors
See weaknesses above.
6 comments
Respond to Reviewer Bh8D (2/3)
W3: Search dynamics and optimization costs
Response: Thank you for this comment. To examine optimization behavior of different methods in our experiments, we report the validation performance across evolution rounds in Table R4. Our proposer distills reusable optimization strategies from historical successes and failures to guide subsequent modifications. The resulting harness performance shows an overall upward trend. In comparison, the two baselines with fixed proposal procedures achieve smaller overall gains and remain at relatively low performance levels for extended periods. Similar conservative reversion has been reported in general agent-harness optimization [1, 2].
Table R4. MedChain validation scores across evolution rounds.
| Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ours | 0.5504 | 0.5557 | 0.5578 | 0.5525 | 0.5637 | 0.5659 | 0.5655 | 0.5718 | 0.5743 | 0.5759 | 0.5816 | 0.5918 | 0.5974 | 0.6138 | 0.6169 |
| Meta-Harness | 0.5405 | 0.5444 | 0.5462 | 0.5451 | 0.5475 | 0.5480 | 0.5532 | 0.5517 | 0.5538 | 0.5576 | 0.5602 | 0.5559 | 0.5597 | 0.5608 | 0.5603 |
| Self-Harness | 0.5410 | 0.5397 | 0.5405 | 0.5402 | 0.5436 | 0.5450 | 0.5558 | 0.5503 | 0.5542 | 0.5581 | 0.5567 | 0.5609 | 0.5592 | 0.5614 | 0.5610 |
Regarding optimization budgets, Table R5 reports the number of candidates and proposer LLM calls, token consumption, execution time per optimization round, and final average score. Our method requires moderately more proposer calls and tokens than Meta-Harness, but obtains a substantially higher score. Compared with Self-Harness, our method uses fewer LLM calls, fewer tokens, and much less execution time while achieving better performance. These results suggest that the proposed self-evolving proposer improves optimization effectiveness without introducing excessive computational overhead.
Table R5. Candidate generation and optimization costs per round.
| Method | Candidates | Proposer LLM calls | Tokens | Execution time | MedChain average score |
|---|---|---|---|---|---|
| Ours | 3 | 3 | 2.866M | 38m 11s | 0.5971 |
| Meta-Harness | 3 | 1 | 2.064M | 34m 30s | 0.5496 |
| Self-Harness | 4 | 5 | 5.127M | 2h 09m 56s | 0.5498 |
[1] Ursekar, V., Shanker, A., Chatrath, V., Xue, Y., and Denton, S. M. (2026). VeRO: A Harness for Agents to Optimize Agents. ICML.
[2] Luo, X., Wang, F., Hu, C., Xue, D., and Deng, Y. (2026). Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity. arXiv:2607.13683v1.
Respond to Reviewer Bh8D (3/3)
W4: Generalization across clinical benchmarks and cross-benchmark adaptation
Response: To further assess generalization beyond separately optimizing the harness for each benchmark, we transfer the candidate with the highest MedChain performance to ClinicalBench. We rebuild its clinical memory and skills using ClinicalBench training cases and perform one additional evolution round on the ClinicalBench validation set to adapt to the target tasks and scoring protocol. We then freeze the adapted harness and evaluate it on the 460 clinicalbench test cases. It achieves an average score of 0.5778 (Table R6). This score exceeds those of Meta-Harness (0.5640) and Self-Harness (0.5537) on ClinicalBench, supporting our framework’s ability to generalize across clinical benchmarks.
Table R6. MedChain-to-ClinicalBench adaptation with one target-side evolution round.
| Framework | Acc | DIFR | PD | DB | DD | FD | PT | TP | Imaging Diagnosis | CDR | Acceptability | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Meta-Harness | 0.7021 | 0.9989 | 0.8456 | 0.4617 | 0.3411 | 0.6391 | 0.3084 | 0.2902 | 0.4885 | 0.4434 | 0.1676 | 0.5640 |
| Self-Harness | 0.6526 | 0.9989 | 0.8401 | 0.4638 | 0.3343 | 0.6081 | 0.3101 | 0.2850 | 0.4908 | 0.3846 | 0.1449 | 0.5537 |
| MedChain → ClinicalBench | 0.7130 | 0.9957 | 0.8652 | 0.4668 | 0.3358 | 0.6891 | 0.3091 | 0.3009 | 0.5242 | 0.4826 | 0.1869 | 0.5778 |
Regarding disease overlap and suggesting broader evaluation of generalization, although training and test cases may share disease categories, patients with the same disease can differ substantially in their symptoms, examination findings, diagnostic evidence, and treatment needs. Making accurate clinical decisions therefore requires integrating patient-specific evidence with relevant medical knowledge, rather than simply retrieving disease-level information. Our graph-based memory is designed to provide structured medical knowledge and associations derived from training cases, enabling the harness to perform case-specific clinical reasoning for diagnosis, differential diagnosis, and treatment planning. We acknowledge that evaluation on clinical datasets from outside China would further strengthen the evidence of generalization and consider this an important direction for future work.
Respond to Reviewer Bh8D (1/3)
We thank the reviewer for the careful assessment and for recognizing the inspectable design of our stage-indexed entity graph and the value in harness optimization.
Response: We apologize for the inaccurate wording and clarify that test set has been strictly held out throughout harness evolution. In Section 3.1.2 line 268-269, "during testing'' refers to testing on validation set in each round of harness evolution, and "test reference answers'' should refer to validation-set reference answers, which provides feedback to determine whether to update Q(s). These evaluations and updates are performed iteratively on the training and validation sets. For clarity, we would like to correct the sentence into: During harness evolution, skill performance is assessed on the validation set using the corresponding validation reference answers and the benchmark evaluator.
As correctly stated in Section 4.1 (Lines 407–408) of our original paper "Graph memory and clinical skills are initialized from training cases, while validation feedback guides candidate evaluation and updates to the Candidate-DAG memory and proposer skill library.'' For our experiments on the two datasets, we strictly split the data into three subsets (Table R1), including a training set to initialize memory and skills, a validation set to provide feedback for iterative harness evolution, and a held-out test set exclusively for final benchmark evaluation.
Table R1. Data splits.
Response: Thank you for raising this point. Regarding training-data access, our comparisons include memory-based agents (MedChain-Agent and MedAgent-Pro) that use the same training cases as our framework, as well as harness-optimization baselines (Meta-Harness and Self-Harness) with access to the same training data. Our method consistently outperforms these baselines on both benchmarks (Table R2), indicating that the gains cannot be attributed solely to privileged access to training data.
Table R2. Average scores of memory-based agents.
Regarding the contribution of harness optimization, we clarify that "w/o Self-Evolving Proposer" still performs harness optimization. This variant removes Candidate-DAG memory and the proposer skill library while retaining a fixed proposer for harness optimization. To disentangle the contributions of initialization and optimization, we additionally report the performance of the initial harness before optimization in Table R3. The overall gain of 0.0593 over initialization (w/o Harness optimization) demonstrates that the improvements cannot be explained by the initial harness and training-derived memory alone.
Table R3. Initialization and optimization on MedChain.