acceptodds
Withdrawn from ICLR 2027

ClinLens: Towards Long-Horizon LLM Agents for Longitudinal Multimodal Clinical Data Science

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Automating clinical data science requires agents to translate patient-centric objectives into executable workflows grounded in longitudinal multimodal evidence. Existing benchmarks provide limited coverage of two essential capabilities: patient-episode grounding and asynchronous multimodal integration. We introduce ClinLens, a patient-centric benchmark for evaluating large language model (LLM) agents on clinical data science. ClinLens comprises 126 executable tasks spanning five data modalities, four analytical scopes, and five analytical capabilities. These tasks assess agents' ability to maintain a coherent patient-specific analytical context across interdependent stages of evidence grounding, multimodal integration, and analytical pipeline generation and execution. We evaluate general-purpose, coding, and biomedical agents. Analyses average 36.93 steps and 8.06 minutes per task. The strongest evaluated agent achieves 86.4% execution success and 54.0% final-answer accuracy, but only a 29.4% strict pass rate. Failure analysis further identifies patient-episode grounding as the most frequently observed failure under the evaluation procedure. By jointly evaluating analytical outcomes and the validity of their supporting clinical context, ClinLens provides a testbed for advancing reliable agents for longitudinal clinical data science.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Multimodal Clinical Data, Longitudinal EHR Analysis, Clinical Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cXh6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the authors propose a new benchmark for evaluating llm agents on long-horizon clinical data workflows. they also identify the current failures lies in reasoning and execution (instead of clinical modality perception).

Critical strengths

important benchmark, literally what we need in the field as of oct 2026, as it combines both long horizon reasoning and mutimodality for clinical agent evaluation. (this should be by itself a strong enough acceptance strength, in my opinion, but i can add more after discussing with authors further. i like this work but i dont want to just repeat the contributions authors wrote here.)

Critical weaknesses
  1. the scale is the main weakness as 126 tasks is modest for a benchmark meant to characterize long-horizon capability.

  2. we need clear separation of harness v.s. model capability. i will specify in question 1.

  3. Executable tasks need a clinician-validated notion of correctness

Questions for authors

+++more urgent questions+++

  1. can we discuss, breakdown or ablate sensitivity of the results so that we know whether the reported failures are due to the agent harness and tool access, prompts, or model capabilities.

  2. can we add a section focusing on the clinical realism and validity of task and answers ?

===less urgent questions====

  1. i have been doing memory related research recently, and what i observe in my own biased experience is that the 738.72k tokens and 108.03k tokens reported in rq4 is already not so short and most cases under the category of "long context" or "long horizon task". do you think that this is part of why some of the methods fail? (e.g. DataVoyager?)

  2. following up to question 3, do you think any sort of memory system comparison or ablation could be helpful, given the long context problem i noticed? of course, the rebuttal timeline is limited, and i totally understand if authors can not afford more experiments. but this may be an important confounding factor.

1 comment

We thank the reviewer for recognizing the relevance of this benchmark and for the concrete suggestions on system factors, clinical validity, and long-horizon execution.

Q1. Backbone, Harness, Tool, and Prompt Effects

Both backbone models and agent configurations affect performance. We separately compare backbones under a fixed harness and harnesses under a fixed backbone; tools, prompts, and memory remain coupled across different harnesses.

Matched Codex configuration, different backbones. Luna + Codex and Sol + Codex used identical Codex settings, tool permissions, prompts, execution budgets, and the same 126 tasks, changing only the backbone. Their FinalAcc/StrictPass scores are 15.0%/5.5% and 54.0%/29.4%, respectively (Table 1). This matched comparison identifies a substantial backbone-dependent performance difference. Across another five shared harnesses, Gemini obtains 2–6 more correct final answers than Luna, while their StrictPass ordering varies with the harness.

Fixed backbone, different harnesses. Across five shared harnesses, StrictPass outcomes vary for 15/126 tasks with Luna and 13/126 with Gemini. With Luna, ReAct and Claude Code achieve FinalAcc of 17/126 and 27/126, and StrictPass of 7/126 and 10/126, respectively. Because tools, prompts, and execution strategies vary across harnesses, these comparisons measure their combined effects.

Exploratory 19-task comparison. With at most four calls per task, ReAct, ReAct plus a clinical-episode/temporal-window checklist, and the checklist plus model-based verification achieve StrictPass on 2/19, 1/19, and 2/19 tasks. Model-based verification consumes two calls, leaving fewer calls for solving and repair. The results show no consistent gain under this budget and cannot isolate the effect of prompting or verification. We will clarify the controlled conditions and remaining confounds in the revision.

Q2. Clinical Realism and Validity

Independent reference-workflow checks and a six-case clinical review support the task validity: all six reviewed reference answers and success/failure judgments were consistent with the underlying clinical evidence. We randomly selected three successful and three failed cases for review by three researchers with clinical experience. Using original records, they checked patient/admission/ICU/examination linkage, eligibility and temporal windows, reference-answer support, and the original evaluation decisions. All six cases were consistent with the clinical evidence, providing direct qualitative support for these sampled cases.

ClinLens comprises 126 executable tasks across four patient-time strata, five analysis categories, and 20 workflow templates, combining structured EHRs, clinical notes, ECGs, chest radiographs, and echocardiograms. Unlike one-step QA, these tasks require longitudinal evidence linkage, constrained analysis, and verifiable intermediate artifacts. We independently executed reference workflows to inspect cohort eligibility, data use, temporal constraints, intermediate artifacts, and final answers, including T10/T14 reconstruction. The 126 tasks prioritize executable, multi-step workflows with verifiable intermediate outputs. We will document the review procedure and its scope in the revision.

Q3. Could Long Contexts Explain DataVoyager's Failures?

Long contexts may contribute, but token usage does not establish context length as the primary cause. RQ4's 738.72k and 108.03k tokens for Luna + Claude Code and Luna + DataVoyager are mean cumulative input-plus-output tokens per task across calls, not single-call context lengths. The median of each task's longest recorded single-call input is 30.6k tokens for DataVoyager versus 18.9k for ReAct. Although DataVoyager uses more cumulative tokens than ReAct under both backbones, its StrictPass is lower under Luna (3.18% versus 5.52%) but higher under Gemini (7.86% versus 4.74%). Model behavior, tool interactions, and execution strategies may also contribute. We will distinguish cumulative usage from single-call context length in the revision.

Q4. Memory Ablation

Memory management is a potential confound in comparing long-horizon agent harnesses. Agents must retain episode identifiers, temporal windows, evidence sources, and intermediate results across calls. We propose a controlled comparison of no retained intermediate state, full history, and length-limited structured-summary memory under fixed backbones, tools, and budgets, measuring StrictPass, FinalAcc, single-call context length, and repeated tool calls to evaluate memory's effect on evidence reuse and accuracy.

Reviewer_zzQ2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ClinLens, an LLM agent benchmark for long-horizon, multimodal clinical data science tasks. It aims to evaluate agents’ ability to handle complex tasks involving patient-episode grounding, temporal alignment across modalities, and clinical data analysis. The benchmark consists of 126 tasks organized into 20 workflow templates. The authors develop an evaluation framework based on executable references, using metrics such as ExecSuccess, FinalAcc, and StrictPass to assess task execution, final-answer accuracy, and strict workflow correctness. Experiments across various LLM agents show that current systems still face significant challenges in completing complex clinical data science tasks.

Critical strengths
  1. The paper addresses a meaningful research problem. It focuses on important aspects of real-world clinical data analysis, including patient-episode grounding, temporal alignment across modalities, and multi-step analysis. This makes the benchmark more representative of real-world clinical data science workflows than benchmarks that focus solely on code generation or final-answer evaluation.

  2. The evaluation framework has several advantages. Rather than assessing only the final results, the authors also examine the analysis process using executable references, distinguishing between execution success, final-answer accuracy, and strict workflow correctness. This approach helps identify errors that may not be apparent from the final results alone.

Critical weaknesses
  1. The benchmark’s ability to distinguish between different agents remains questionable. In Table 1, many systems achieve zero FinalAcc and StrictPass scores in certain patient-time scopes, while their ExecSuccess scores are close to 100%. Figure 6 further shows that, across the ten analyzed configurations, 16 out of 20 task templates did not yield correct final answers. These results suggest a potential floor effect, which may limit the benchmark’s ability to differentiate between agents’ capabilities on these tasks. Moreover, the overall results may be driven largely by the few task templates with non-zero scores. The paper does not sufficiently examine how this issue affects the benchmark’s validity and model rankings.

  2. The representativeness of the tasks and the validity of the evaluation criteria require further investigation. Although the benchmark contains 126 tasks derived from 20 workflow templates, it remains unclear how well these tasks reflect the diversity of real-world clinical data science workflows. In addition, the authors should clarify whether the strict evaluation based on executable references can accommodate alternative but equally valid analytical approaches.

Questions for authors
  1. Given the large number of zero scores in Table 1 and the fact that 16 out of 20 task templates did not yield correct final answers in Figure 6, how do the authors assess the discriminative power of ClinLens? Would the model rankings remain stable if these templates were excluded or assigned different weights?

  2. How does StrictPass avoid incorrectly rejecting solutions that use alternative preprocessing, statistical analysis, or temporal alignment methods but still produce valid results?

2 comments

We thank the reviewer for the constructive questions on benchmark discrimination, ranking robustness, and alternative analytical implementations.

Q1. Floor Effects and Ranking Robustness

Floor effects remain, but a stronger system succeeds on previously unsolved templates. We evaluate ranking sensitivity using the same patient-time-stratified macro-average as Table 1. Across the original ten configurations, FinalAcc and StrictPass are both zero on 16/20 templates. GPT-5.6-Sol + Codex correctly solves 49 tasks from these templates, increasing the number of templates with at least one correct final answer from 4/20 to 12/20; 8/20 remain unsolved.

We recomputed all 22 Overall scores (FinalAcc and StrictPass for 11 configurations) from frozen task-level records; all 22 match Table 1. Table 1 uses patient-time-stratified macro-averaging: the pass rates for four strata containing 32, 32, 31, and 31 tasks are averaged equally. In our sensitivity analysis, each template weight is applied to every task in that template, followed by within-stratum normalization. Let np,tn_{p,t} be the number of tasks from template tt in stratum pp, wtw_t the per-task template weight, and as,p,t(m)a_{s,p,t}^{(m)} the average pass rate for system ss and metric mm. Then

Ss(m)(w)=1∣P(w)∣∑p∈P(w)∑tnp,twtas,p,t(m)∑tnp,twt.S_s^{(m)}(\mathbf{w}) = \frac{1}{|\mathcal{P}(\mathbf{w})|}\sum_{p\in\mathcal{P}(\mathbf{w})}\frac{\sum_t n_{p,t}w_t a_{s,p,t}^{(m)}}{\sum_t n_{p,t}w_t}.

Here P(w)\mathcal{P}(\mathbf{w}) includes strata retaining positive-weight tasks. The full benchmark has four strata; removing the 16 templates leaves the ICU stratum empty, so the remaining three strata are macro-averaged.

For the original ten configurations, halving the weights of these 16 zero-scoring templates yields 0/45 pairwise reversals for both metrics; doubling yields 0/45 for FinalAcc and 4/45 for StrictPass (including a change in the top-ranked system); removing them yields 1/45 and 3/45, respectively. The 45 comparisons are all system pairs among the original ten configurations; tie changes are counted separately. Thus most pairwise rankings remain stable, although some, including the StrictPass leader, are sensitive to weights.

Including Sol + Codex, it leads on the full benchmark. When restricted to the other four templates, Gemini-3.7-Flash + Reflexion instead has higher FinalAcc (94.87% versus 84.62%), while Sol + Codex retains higher StrictPass. We will provide per-template scores, the aggregation formula, and ranking changes in the revision.

Q2. Valid Alternative Analytical Implementations

In the examined cases, StrictPass did not incorrectly reject valid solutions merely for using different implementations. It checks the task-specified analytical units, eligibility, temporal constraints, essential analytical steps, and outputs, rather than code similarity. Independent SQL reproduced T10 admission associations and the 24-hour window; CSV row/column reordering was accepted, but incorrect admission keys were rejected. An independently retrained T14-I05 produced different predictions and held-out metrics yet passed all four checks. Independently auditing 114 T10/T14 failed runs also revealed genuine violations of published rules in every examined run. We will specify the acceptance criteria and their links to the task instructions in the revision.

Thank you for the detailed response and additional analyses. I appreciate the authors' efforts to address my concerns. While the clarifications and new results are helpful, I believe some of my main concerns remain unresolved. Therefore, I will maintain my original rating.

Reviewer_ENEtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ClinLens, a benchmark for evaluating LLM agents on longitudinal clinical data analysis. It contains 126 executable tasks built from five linked MIMIC data sources, covering patient-level analyses such as disease progression, treatment effects, and risk prediction. The benchmark evaluates not only whether agents produce the correct final answer, but also whether they use the right patient records, align data across time, and follow valid analytical procedures. The authors test several general-purpose, coding, and biomedical agents. The strongest system achieves 54.0% final-answer accuracy but only a 29.4% strict pass rate. The failure analysis suggests that identifying and linking the correct clinical episodes remains a major challenge for current agents.

Critical strengths
  1. What I find useful about ClinLens is that it preserves the complexity of real clinical data. A patient may have multiple admissions, and records from different modalities do not always line up neatly in time. Instead of resolving these relationships in advance, the benchmark requires agents to identify the correct patient episodes and connect the relevant information. This makes the tasks more realistic than working with already processed tables.

  2. The gap between execution success (86.4%), final-answer accuracy (54.0%), and strict pass rate (29.4%) is quite striking. It shows that an agent can successfully run its analysis without necessarily following a valid clinical workflow. I think evaluating intermediate steps is important here, especially when a correct-looking answer may come from the wrong records or an incorrect analysis. The executable reference workflows and consistency checks are also useful contributions to the benchmark.

Critical weaknesses
  1. My main concern is the reliability of StrictPass. The evaluation relies on private reference workflows and several intermediate checks, but different analytical approaches can sometimes be equally valid. For example, T10 achieves 91.7% final-answer accuracy but 0% strict accuracy, while 59 out of 60 T14 runs fail the private test-unit coverage check. I cannot tell how many of these failures represent genuine analytical errors and how many might come from valid alternative implementations. The authors provide internal consistency checks, but there is no independent expert validation of the failed cases. Since StrictPass is central to the paper's conclusions, I think this deserves more attention.

  2. I also have concerns about the failure analysis. It excludes the strongest evaluated system, GPT-5.6-Sol with Codex, and focuses on a subset of weaker configurations without prediction tasks. More importantly, the failure categories are determined by the first failed check in a fixed sequence. The first check to fail is not necessarily the actual cause of the problem. The analysis is still useful, but I would be careful about concluding that episode grounding is the main bottleneck for clinical agents in general, especially without results from the strongest system.

Questions for authors
  1. Could the authors provide an independent evaluation of some StrictPass failures, especially for T10 and T14? I would like to know how many failures are genuine analytical errors, and how many might result from alternative valid implementations or unclear task requirements.

  2. How does the evaluator handle different implementations that produce scientifically equivalent analyses? Are all the constraints checked by the private evaluator explicitly specified in the task instructions?

  3. Could the authors provide the failure breakdown for GPT-5.6-Sol + Codex? Does episode grounding remain the main failure point for the strongest system, or does the distribution shift toward other analytical steps?

2 comments

We thank the reviewer for the detailed assessment and questions on evaluation reliability, failure attribution, and data handling.

Q1. Independent Verification of StrictPass Failures

All 114 independently reviewed failed runs (55 T10; 59 T14) contained genuine analytical-unit selection errors; none were false rejections solely due to valid alternatives or ambiguous instructions (0/114). These are failed runs, not distinct tasks. We reconstructed eligible units directly from the agent-visible instructions and source data and compared them with archived submissions, without relying on the evaluator's failure labels.

  • T10 (55/55): Every run included ineligible admissions. Across runs, there were 46,544 erroneous inclusions (repeated selections across runs counted separately): 45,308 (97.3%) violated eligibility rules, 1,176 admission-selection priorities, and 60 involved eight admissions excluded during source-data quality control.
  • T14 (59/59): Every run constructed an incorrect test cohort, with 1,395 ineligible units included and 1,358 eligible units omitted across runs. Every incorrectly included unit violated a published inclusion rule; 58/59 runs retained ineligible units in final feature matrices and prediction tables. Independent reconstruction recovered all 542 eligible test units across six instances.

As a positive control, we independently retrained T14-I05. Despite different predictions and metrics, it passed all four checks of the frozen evaluator.

Q2. Alternative Implementations and Evaluation Criteria

StrictPass checks task-specified analytical units, eligibility, time constraints, required procedures, and outputs—not code-level agreement with a reference implementation. An independent SQL implementation reconstructed T10 admission links and its 24-hour window; CSV row/column reordering was accepted, whereas incorrect admission keys were rejected. For T14, metrics were recomputed on a fixed held-out set, and independently retrained T14-I05 passed all four checks despite different predictions. These checks show that StrictPass accepts the tested valid alternatives while rejecting observed eligibility errors. We will clarify the correspondence between public instructions, acceptance criteria, and numerical tolerances in the revision.

Q3. Failure Breakdown of the Strongest System

Clinical-episode grounding is the most frequent first failed check (37/89, 41.6%), but a first failed check is not necessarily the root cause; downstream failures are also substantial. GPT-5.6-Sol + Codex passes FinalAcc on 68/126 tasks and StrictPass on 37/126. All 25 predictive tasks pass StrictPass; 89/101 non-predictive tasks fail. Their first failed checks are episode grounding (37), cohort construction (23), artifact completeness (15), temporal constraints (7), causal-analysis protocol (6), and source-data usage (1).

Using all non-predictive tasks as the denominator, episode grounding is the first failed check for 37/101 (36.6%) for this system, versus 840/1,010 (83.2%) for the original ten configurations; 37/89 (41.6%) instead uses failed tasks only. Among the 89 failures, 74 have complete four-check records; adding 12 fully recorded passes gives 86/101 with complete records. Within those 74 failures, cohort construction fails 60 times, source-data usage 50, temporal constraints 39, and episode grounding 37. The other 15 failures first fail artifact completeness and lack full four-check records; missing checks were not treated as passes. These findings identify failed evaluation checks, not necessarily the underlying causes.

Ethics and Data Handling

We used model-provider APIs directly, enabled Zero Data Retention (ZDR), and verified the official data-processing terms applicable to the services and endpoints used. Some evaluation requests contained de-identified patient-level MIMIC data, sent without third-party API relays. Under the verified terms and active ZDR settings, request data are not retained by providers, used for model training, or accessed for human review. We will document the data path, privacy settings, and applicable terms in the revision to clarify compliance with PhysioNet's requirements. Prior ICLR studies offer relevant precedents: KARE (Jiang et al., 2025) used Amazon Bedrock for EHR-related LLM interactions; SERSAL (Yan et al., 2025) used Azure OpenAI with human review disabled.

I appreciate the additional analyses, but I will maintain my original rating because the StrictPass validation remains limited to a small subset of tasks, and the failure analysis still does not clearly establish the underlying causes of errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.