- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ECG-Scroll, a benchmark and agent environment for interpreting long-duration ECG recordings in a causal, streaming setting. ECG-Scroll implements agents that scan recordings, take measurements, and write findings to memory. The benchmark comprises 390 task instances across 247 recordings, with deterministic evaluation of event localization, burden estimation, and change detection, going beyond mere final answer accuracy or reasoning evaluation. The authors evaluate the same off-the-shelf LLMs in two configurations: Planning agents select measurement windows and record findings autonomously, while Judge agents evaluate fixed chunks supplied by the environment. Experiments include a rule-based baseline and settings with and without diagnostic tools. The authors find that planning agents generally perform worse than Judge agents when diagnostic tools are available; the authors attribute this gap primarily to difficulties with long-horizon planning and committing findings.
Critical strengths
- Motivation: I fully agree with the motivation, it is well written. Current models either work on short-term windows, or they try to compress data into (recurrent) states. The problem with that is however that the model at the time of perception cannot know what it needs to store for the future. It is very challenging to retain information without knowing what will be relevant in the future. The authors motivate this challenge well and propose an interesting agentic formulation in which the model must actively decide what to measure, what to retain, and when to act.
- I find the form of evaluation very interesting. Instead of doing final answer classification, the authors define multiple algorithmically verifiable tasks: episode localization, burden estimation, and change detection use explicit, deterministic verifiers, making failures more inspectable than classification or free-text diagnostic evaluation. Generally this is a great environment to assess and benchmark different types of agents or tools.
Critical weaknesses
- Performance depends on reliability of tools. Agents access the ECG exclusively through tool outputs, yet the paper provides limited validation of their accuracy and robustness across recordings. Errors in beat detection or waveform measurements could propagate into downstream decisions, making it difficult to distinguish agent reasoning failures from unreliable inputs. Independent tool validation and sensitivity analyses are needed to support the conclusions.
- Comparison of Judge- vs. Planning-agent is confounded. Point 1 above directly propagates to the comparison of the Judge-agent vs. Planning Agent. The Judge receives precomputed measurements or labels for fixed chunks, whereas the Planning agent selects tools and measurement windows itself. Together with different reporting actions, these differences prevent attributing the performance gap solely to planning ability.
Questions for authors
- Please elaborate on how the signal and diagnostic tools are implemented and validated? Are diagnostic outputs independently predicted or derived from ground-truth annotations? Can we assess the reliability of the tools alone across the included datasets?
- What exact differences exist between the Judge and Planning agents in available measurements, reporting actions, and inference budgets? What evidence supports attributing the performance gap specifically to planning rather than these differences? I find the comparison a bit tricky as this might be confounded.