- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TRACER is an agent harness using bedside video and EHR data to answer questions about ICU patients. There is a perception stack (VLM with per-minute records from 2 cameras and hourly reports) that give an agent EHR and video tools. Contributions: (i) TRAJECTORY-144, a question set of 144 data points built from two ICU stays. (ii) a perception benchmark with 397 manually labeled samples. (iii) ablations over tool families and agent configurations, including open-weight agents. (iv) a cohort analysis of 280 patients in which delirium patients show significantly less "purposeful activity" (abstract: p < 0.0001). The paper states 84.8% accuracy and an F1-score 68.6 on action detection.
Critical strengths
valuable setting and a serious engineering effort: Multi-day ICU video with EHR from 280 patients. many ablation studies (tool families, 12 agent configurations, open-weight comparisons). cohort analysis that uses per-patient tests. Harness, prompts, taxonomy and evaluation code are released.
Critical weaknesses
trustworthiness of results: TRAJECTORY-144 comes from two stays only (both female). performance for these stays differs substantially with Gemini (88.5 vs 68.9). five gold-answers per question are not independent. On short (<5 min) video-moment questions (n = 33), where perception matters most, TRACER (3.1 Flash Lite) scores 36.4 vs 34.5 for the baseline.
merely an agent harness with questionable scientific contribution. main novelty is the dataset
authors call the results from model predictions on retrospective data hypothesis-generating, but the abstract states the result as a finding and credits their "TRACER".
Questions for authors
Please list every change to the agent prompt, tool set or name_filter rules that was informed by draft benchmark questions, and report results on a held-out stay.
Was the "purposeful activity" defined before looking at the group comparison?
Do the perception errors differ by delirium group?
In the "open-weight" result rows, the minute records and hourly reports are produced by Gemini. What is the number for a fully open-weight pipeline (the Qwen3.5 9B configuration at 57.9?)? Does the on-premise claim hold?
Ethics: is there patient consent or a waiver, how were staff and visitors who appear in the video handled?