acceptodds
Under review as a conference paper at ICLR 2027

ECG-Scroll: A Long-Horizon, Streaming Benchmark and Agent Environment for Interpretation of Ambulatory Electrocardiograms

Abstract

Multimodal large language models (MLLMs) can now interpret a standard ten- second, twelve-lead electrocardiogram (ECG) with clinically grounded, reward- verified reasoning. Real cardiac monitoring is different. Ambulatory (Holter) and telemetry recordings span hours to days and are read as they stream in, and their clinically decisive findings are paroxysmal, brief episodes buried in an otherwise unremarkable trace. Such a recording cannot be held in one context at diagnos- tic resolution, and its future has not yet happened, so a reader must work online, deciding what to measure now, committing evidence to memory as it passes, and reporting events as they occur. We recast long-duration ECG interpretation as a long-horizon, online (streaming, causal) sequential decision process and intro- duce ECG-Scroll. As a benchmark, long ambulatory recordings are streamed to an agent chunk by chunk, and it must localize, quantify, and promptly flag paroxysmal events without access to future signal; because the underlying sig- nal is retained, every answer is checkable against objective ground truth, giv- ing rule-based rather than judge-based rewards, and the streaming formulation adds a metric batch evaluation cannot express, the detection latency between an event’s onset and the moment the agent records it. As an agent environment, it is a fixed, gym-style interaction layer that exercises three competencies single- glance ECG models never touch: Memory, Tool use through signal-grounded measurement rather than reading pixels, and Planning of what to measure now and when to commit. We release 390 whole-recording instances spanning 2,536 hours of two-lead ambulatory ECG and evaluate a signal-threshold rule agent alongside off-the-shelf LLM agents online, characterizing how they use memory, tools, and planning and where the benchmark’s head-room lies.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
ECG, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wm6NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ECG-Scroll, a benchmark and agent environment for interpreting long-duration ECG recordings in a causal, streaming setting. ECG-Scroll implements agents that scan recordings, take measurements, and write findings to memory. The benchmark comprises 390 task instances across 247 recordings, with deterministic evaluation of event localization, burden estimation, and change detection, going beyond mere final answer accuracy or reasoning evaluation. The authors evaluate the same off-the-shelf LLMs in two configurations: Planning agents select measurement windows and record findings autonomously, while Judge agents evaluate fixed chunks supplied by the environment. Experiments include a rule-based baseline and settings with and without diagnostic tools. The authors find that planning agents generally perform worse than Judge agents when diagnostic tools are available; the authors attribute this gap primarily to difficulties with long-horizon planning and committing findings.

Critical strengths
  • Motivation: I fully agree with the motivation, it is well written. Current models either work on short-term windows, or they try to compress data into (recurrent) states. The problem with that is however that the model at the time of perception cannot know what it needs to store for the future. It is very challenging to retain information without knowing what will be relevant in the future. The authors motivate this challenge well and propose an interesting agentic formulation in which the model must actively decide what to measure, what to retain, and when to act.
  • I find the form of evaluation very interesting. Instead of doing final answer classification, the authors define multiple algorithmically verifiable tasks: episode localization, burden estimation, and change detection use explicit, deterministic verifiers, making failures more inspectable than classification or free-text diagnostic evaluation. Generally this is a great environment to assess and benchmark different types of agents or tools.
Critical weaknesses
  • Performance depends on reliability of tools. Agents access the ECG exclusively through tool outputs, yet the paper provides limited validation of their accuracy and robustness across recordings. Errors in beat detection or waveform measurements could propagate into downstream decisions, making it difficult to distinguish agent reasoning failures from unreliable inputs. Independent tool validation and sensitivity analyses are needed to support the conclusions.
  • Comparison of Judge- vs. Planning-agent is confounded. Point 1 above directly propagates to the comparison of the Judge-agent vs. Planning Agent. The Judge receives precomputed measurements or labels for fixed chunks, whereas the Planning agent selects tools and measurement windows itself. Together with different reporting actions, these differences prevent attributing the performance gap solely to planning ability.
Questions for authors
  1. Please elaborate on how the signal and diagnostic tools are implemented and validated? Are diagnostic outputs independently predicted or derived from ground-truth annotations? Can we assess the reliability of the tools alone across the included datasets?
  2. What exact differences exist between the Judge and Planning agents in available measurements, reporting actions, and inference budgets? What evidence supports attributing the performance gap specifically to planning rather than these differences? I find the comparison a bit tricky as this might be confounded.
Reviewer_fb7zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at how to find important events in heart recordings that last for hours, as the data come in rather than after the whole recording is available. The authors build a benchmark using real recordings with known event locations, along with an environment where agents decide what to examine and what findings to save. They evaluate several language models and a rule-based baseline across different detection tasks. The results show that the models often struggle when they need to manage the process themselves, but perform much better when the recording is examined for them and diagnostic labels are provided.

Critical strengths

[S1] This work provides a meaningful and practical benchmark setting. Most existing ECG benchmarks focus on short recordings, so evaluating models on long recordings with sparse events is a useful direction. The benchmark covers several types of events using real annotated recordings, and the evaluation is based on objective measurements rather than model judgments. This makes it a potentially valuable resource for future work.

[S2] The experiments go beyond reporting final accuracy and examine what agents actually do during the process. For example, Figure 3 shows that many models keep scanning but often finish without recording useful findings.

Critical weaknesses

[W1] The experiments do not fully support the main conclusion about planning. The paper argues that planning is the key missing ability, largely based on the gap between the Planning and Judge agents. However, the two settings differ in more than just planning. The Judge agent is given every chunk, receives diagnostic labels in the +dx setting, and is explicitly instructed to trust them. The Planning agent must decide when to call tools and how to record precise event intervals. It is therefore hard to tell how much of the performance gap comes from planning itself, not differences in the information provided and the difficulty of producing the final answer. The strong Judge results may also reflect the quality of the supplied labels.

[W2] An important part of the proposed benchmark is not evaluated. Detection latency is presented as a major advantage of the streaming setting, but the experiments do not report latency results. The authors explain that the Planning agents produce too few valid detections for meaningful latency analysis. However, the Rule and Judge agents achieve reasonable detection performance on several tasks, so their latency results could still help validate this part of the benchmark. Without such evidence, it is difficult to assess whether the proposed setting meaningfully measures timely detection, which is one of its main motivations.

Questions for authors

[Q1] Could the authors compare the Planning and Judge agents with the same available observations and output requirements, changing only who controls the scanning process?

[Q2] How are the diagnostic labels generated, and are they fully independent of the ground-truth annotations used for evaluation?

[Q3] Could the authors report detection latency for the Rule and Judge agents, where enough correct detections are available for meaningful analysis?

Reviewer_ogkhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ECG-Scroll, a benchmark and gym-style environment for interpreting long ambulatory ECG recordings under a streaming, causal constraint. Recordings from public PhysioNet databases are revealed in 30-s chunks. The agent has three actions: SKIM(d) advances the cursor and returns a lossy summary of the newly elapsed span, MEASURE(tool, t0, t1) runs a tool on an elapsed window, and WRITE appends a timestamped finding to a typed memory ledger from which the answer is assembled. Tools are either numeric signal tools (rate/rhythm, P, QRS, ST, T, QT; built on Pan–Tompkins beat detection) or label-returning diagnostic tools (rhythm, beat type, ischemia). The release has 390 instances over 247 recordings (2,536 h) and six tasks: AF episode detection, AF burden, AF change-point detection, ischemic ST-episode detection, PVC burden, and rare-event search. Each task is scored by a rule-based verifier, and a detection-latency metric based on WRITE timestamps is also defined.

The evaluation covers a threshold Rule Agent and seven LLMs (Qwen3 8B/30B/235B and a thinking variant, GLM-4.5-Air, Llama-3.3-70B, DeepSeek-V4.1-Flash), each run with and without diagnostic tools. There are two configurations: a Planning configuration, where the model drives the cursor, and a Judge configuration, where the environment presents every chunk and the model flags events. Planning agents stay near the floor on most tasks. Judge agents with diagnostic labels roughly match the Rule Agent. From this, the authors conclude that autonomous long-horizon planning (when to look, when to commit), not recognition, is the missing competency. Detection latency is not reported.

Critical strengths
  1. A well-motivated, underexplored formulation. The paper treats long-recording ECG interpretation as a causal, streaming decision process, with time-stamped reference annotations as ground truth. This connects ECG-MLLM work, which uses short, fully visible ECGs, with the long-recording annotation ecosystem. It also allows rule-based scoring without an LLM judge, in principle including timeliness. This matters because a carefully executed version would be a genuinely useful testbed for long-horizon agents with verifiable rewards. It is the main reason I find the submission worth engaging with.

  2. A sensible scaffold for attributing failures. The design separates numeric signal tools from label-returning diagnostic tools, contrasts agent-driven with environment-driven scanning, and reports behavioural statistics (Fig. 3: skim/measure/write counts, empty-ledger rates). This is the right kind of design for locating where agents fail, and code and data are released. It shows how the benchmark could become diagnostic, even though, as argued below, the current instantiation does not yet isolate what it claims to.

Critical weaknesses

W1. The Planning/Judge comparison does not isolate planning. Sec. 5 claims perception and knowledge are held fixed, but they are not. The Judge is handed a pre-computed tool output for every chunk, and under +dx it is told to “trust the label and flag the chunk when the label says it is in AF” (App. C), which amounts to label copying. Fig. 3a shows that three of the seven Planning agents average 0 MEASURE calls per recording, so they essentially never see the label at all. Under --dx, every agent, including the Rule reference, is near zero on the detection tasks. The paper therefore offers no evidence that the models can recognize events from the signal.

W2. Most results cannot be distinguished from a default answer. Table 3 is nearly constant across seven very different models: AF-episode F1 is 0.03 for six of them, ischemia F1 is 0.00–0.01 under +dx, and change detection is 0.17 (= 4/23) in most cells. Six of the seven models end 53–82% of recordings with an empty ledger (Fig. 3b). No null baselines or achievable ceilings are reported. At 30-s chunk resolution, AF episodes shorter than 15 s (close to half, given the 16-s median) cannot reach IoU ≥ 0.5 for chunk-level agents. Two tasks have N = 13 and N = 23 and there are no repeated runs, so many reported differences amount to one or two records.

W3. Detection latency, the headline streaming metric, is never reported. This is despite the +dx Rule and Judge agents producing many matched events. The definition is also flawed. An entry written at cursor κ can only cover elapsed signal, so matching at IoU ≥ τ forces latency ≥ τ × event duration. The metric thus measures episode length rather than promptness.

W4. The PVC-burden score compares mismatched units. The ground truth is a beat fraction (App. B), but the prediction is a time fraction computed from ~2-s write intervals (App. D). In the showcased mitdb-118 run, the agent’s own classify_beats output lists at least 12 PVCs, yet the agent writes only 7 intervals and still scores 0.96. Each 2-s window counts as ~2.5 beats, so the metric rewards low recall and penalizes chunk-level outputs. This undermines the “successful planning” evidence in Sec. 5 and App. D.

W5. Key components are unspecified. The paper never says what the diagnostic tools are, or whether they were trained or tuned on these databases. Yet they produce every non-trivial number in Tables 3–4 (Q1). Several other components are also missing: the Rule Agent’s thresholds; the SKIM peak feature; the rewind cost and which variant was run; the ledger-to-answer assembly rules.

Questions for authors

Q1. What exactly are classify_rhythm, classify_beats and flag_ischemia? Were any of them trained, tuned or thresholded on the benchmark databases or their reference annotations? If so, the +dx results are contaminated.

Q2. What do null baselines (empty answer, flag every chunk, constant zero burden) score on each task under the same verifiers? What does a chunk-resolution oracle score? Which Table 3 cells exceed the empty-answer score?

Q3. What cue and threshold does the Rule Agent use per task, and were the thresholds chosen on the evaluation records? The +dx Judge is told to follow the label, so why does it differ from Rule +dx on AF burden (mostly 0.96–0.99 vs 0.83)? How are flagged chunks and written intervals assembled into answers?

Q4. Which rewind variant produced Tables 3–4? What does SKIM’s peak feature compute per task, and what step/token limits applied to the Planning agent? How many runs ended at a limit or with malformed tool calls?

Q5. Can you report detection latency for the +dx Rule and Judge agents? Can you rescore PVC burden with prediction and reference in the same units? Would you revise the latency definition, given that it is bounded below by τ × event duration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.