- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
the authors propose a new benchmark for evaluating llm agents on long-horizon clinical data workflows. they also identify the current failures lies in reasoning and execution (instead of clinical modality perception).
Critical strengths
important benchmark, literally what we need in the field as of oct 2026, as it combines both long horizon reasoning and mutimodality for clinical agent evaluation. (this should be by itself a strong enough acceptance strength, in my opinion, but i can add more after discussing with authors further. i like this work but i dont want to just repeat the contributions authors wrote here.)
Critical weaknesses
-
the scale is the main weakness as 126 tasks is modest for a benchmark meant to characterize long-horizon capability.
-
we need clear separation of harness v.s. model capability. i will specify in question 1.
-
Executable tasks need a clinician-validated notion of correctness
Questions for authors
+++more urgent questions+++
-
can we discuss, breakdown or ablate sensitivity of the results so that we know whether the reported failures are due to the agent harness and tool access, prompts, or model capabilities.
-
can we add a section focusing on the clinical realism and validity of task and answers ?
===less urgent questions====
-
i have been doing memory related research recently, and what i observe in my own biased experience is that the 738.72k tokens and 108.03k tokens reported in rq4 is already not so short and most cases under the category of "long context" or "long horizon task". do you think that this is part of why some of the methods fail? (e.g. DataVoyager?)
-
following up to question 3, do you think any sort of memory system comparison or ablation could be helpful, given the long context problem i noticed? of course, the rebuttal timeline is limited, and i totally understand if authors can not afford more experiments. but this may be an important confounding factor.
We thank the reviewer for recognizing the relevance of this benchmark and for the concrete suggestions on system factors, clinical validity, and long-horizon execution.
Q1. Backbone, Harness, Tool, and Prompt Effects
Both backbone models and agent configurations affect performance. We separately compare backbones under a fixed harness and harnesses under a fixed backbone; tools, prompts, and memory remain coupled across different harnesses.
Matched Codex configuration, different backbones. Luna + Codex and Sol + Codex used identical Codex settings, tool permissions, prompts, execution budgets, and the same 126 tasks, changing only the backbone. Their FinalAcc/StrictPass scores are 15.0%/5.5% and 54.0%/29.4%, respectively (Table 1). This matched comparison identifies a substantial backbone-dependent performance difference. Across another five shared harnesses, Gemini obtains 2–6 more correct final answers than Luna, while their StrictPass ordering varies with the harness.
Fixed backbone, different harnesses. Across five shared harnesses, StrictPass outcomes vary for 15/126 tasks with Luna and 13/126 with Gemini. With Luna, ReAct and Claude Code achieve FinalAcc of 17/126 and 27/126, and StrictPass of 7/126 and 10/126, respectively. Because tools, prompts, and execution strategies vary across harnesses, these comparisons measure their combined effects.
Exploratory 19-task comparison. With at most four calls per task, ReAct, ReAct plus a clinical-episode/temporal-window checklist, and the checklist plus model-based verification achieve StrictPass on 2/19, 1/19, and 2/19 tasks. Model-based verification consumes two calls, leaving fewer calls for solving and repair. The results show no consistent gain under this budget and cannot isolate the effect of prompting or verification. We will clarify the controlled conditions and remaining confounds in the revision.
Q2. Clinical Realism and Validity
Independent reference-workflow checks and a six-case clinical review support the task validity: all six reviewed reference answers and success/failure judgments were consistent with the underlying clinical evidence. We randomly selected three successful and three failed cases for review by three researchers with clinical experience. Using original records, they checked patient/admission/ICU/examination linkage, eligibility and temporal windows, reference-answer support, and the original evaluation decisions. All six cases were consistent with the clinical evidence, providing direct qualitative support for these sampled cases.
ClinLens comprises 126 executable tasks across four patient-time strata, five analysis categories, and 20 workflow templates, combining structured EHRs, clinical notes, ECGs, chest radiographs, and echocardiograms. Unlike one-step QA, these tasks require longitudinal evidence linkage, constrained analysis, and verifiable intermediate artifacts. We independently executed reference workflows to inspect cohort eligibility, data use, temporal constraints, intermediate artifacts, and final answers, including T10/T14 reconstruction. The 126 tasks prioritize executable, multi-step workflows with verifiable intermediate outputs. We will document the review procedure and its scope in the revision.
Q3. Could Long Contexts Explain DataVoyager's Failures?
Long contexts may contribute, but token usage does not establish context length as the primary cause. RQ4's 738.72k and 108.03k tokens for Luna + Claude Code and Luna + DataVoyager are mean cumulative input-plus-output tokens per task across calls, not single-call context lengths. The median of each task's longest recorded single-call input is 30.6k tokens for DataVoyager versus 18.9k for ReAct. Although DataVoyager uses more cumulative tokens than ReAct under both backbones, its StrictPass is lower under Luna (3.18% versus 5.52%) but higher under Gemini (7.86% versus 4.74%). Model behavior, tool interactions, and execution strategies may also contribute. We will distinguish cumulative usage from single-call context length in the revision.
Q4. Memory Ablation
Memory management is a potential confound in comparing long-horizon agent harnesses. Agents must retain episode identifiers, temporal windows, evidence sources, and intermediate results across calls. We propose a controlled comparison of no retained intermediate state, full history, and length-limited structured-summary memory under fixed backbones, tools, and budgets, measuring StrictPass, FinalAcc, single-call context length, and repeated tool calls to evaluate memory's effect on evidence reuse and accuracy.