acceptodds
Withdrawn from ICLR 2027

TRACER: A Multimodal Harness for Patient Care in Hospitals

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continuous video and clinical records provide complementary information for understanding how patient behavior evolves over a hospital stay. A central challenge is how to represent behavior across the entire stay while preserving access to fine-grained visual and clinical evidence. We introduce TRACER, a training-free multimodal harness that constructs minute-level action records from video and combines them with clinical records into hourly reports. The harness equips an LLM agent with tools to query this hierarchy, inspect relevant video, and retrieve clinical records. To evaluate TRACER, we construct two benchmarks. The first contains 397 annotated one-minute clips spanning 15 atomic actions; the second poses 144 clinician-informed multiple-choice questions, grounded in 264 hours of video and clinical records from two ICU patients. TRACER achieves up to 84.8% accuracy for action detection and 61.0% QA accuracy versus 49.3% for the strongest baseline using the same backbone. Across 280 patients spanning approximately 60,000 camera-hours, TRACER identifies significantly less purposeful activity in patients with a positive delirium assessment during their stay than in those without (). Our results show that combining continuous video with clinical records can support understanding of patient behavior over time and across patient cohorts without training on protected health data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-form video understanding, multimodal agents, LLM agent harness, tool use, electronic health records, hospital, healthcare

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sSEzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

TRACER is an agent harness using bedside video and EHR data to answer questions about ICU patients. There is a perception stack (VLM with per-minute records from 2 cameras and hourly reports) that give an agent EHR and video tools. Contributions: (i) TRAJECTORY-144, a question set of 144 data points built from two ICU stays. (ii) a perception benchmark with 397 manually labeled samples. (iii) ablations over tool families and agent configurations, including open-weight agents. (iv) a cohort analysis of 280 patients in which delirium patients show significantly less "purposeful activity" (abstract: p < 0.0001). The paper states 84.8% accuracy and an F1-score 68.6 on action detection.

Critical strengths

valuable setting and a serious engineering effort: Multi-day ICU video with EHR from 280 patients. many ablation studies (tool families, 12 agent configurations, open-weight comparisons). cohort analysis that uses per-patient tests. Harness, prompts, taxonomy and evaluation code are released.

Critical weaknesses

trustworthiness of results: TRAJECTORY-144 comes from two stays only (both female). performance for these stays differs substantially with Gemini (88.5 vs 68.9). five gold-answers per question are not independent. On short (<5 min) video-moment questions (n = 33), where perception matters most, TRACER (3.1 Flash Lite) scores 36.4 vs 34.5 for the baseline.

merely an agent harness with questionable scientific contribution. main novelty is the dataset

authors call the results from model predictions on retrospective data hypothesis-generating, but the abstract states the result as a finding and credits their "TRACER".

Questions for authors

Please list every change to the agent prompt, tool set or name_filter rules that was informed by draft benchmark questions, and report results on a held-out stay.

Was the "purposeful activity" defined before looking at the group comparison?

Do the perception errors differ by delirium group?

In the "open-weight" result rows, the minute records and hourly reports are produced by Gemini. What is the number for a fully open-weight pipeline (the Qwen3.5 9B configuration at 57.9?)? Does the on-premise claim hold?

Ethics: is there patient consent or a waiver, how were staff and visitors who appear in the video handled?

Reviewer_sxweOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents TRACER, a training-free multimodal harness for understanding multi-day ICU patient trajectories. It works in two stages: 1) Background abstraction. A VLM labels each minute of dual-view video zero-shot for 15 clinically relevant atomic actions and writes a short caption. An LLM then summarizes each hour of minute records, together with the medication events from that hour, into a six-section report. 2) Question-time retrieval. An LLM agent gathers evidence through timeline, report, EHR, and video tools before answering. The evaluation has three parts: atomic action detection on 397 one-minute clips; TRAJECTORY-144, a set of 144 multiple-choice questions over 264 hours of video and records from two stays; and a delirium-related behavioral analysis of 280 patients covering about 60,000 camera-hours.

Critical strengths
  1. The deployment scale is substantial. Continuous ICU video aligned with EHR data is rare in the literature, and running the pipeline over 60,000 camera-hours is a nontrivial achievement.
  2. The training-free design, which keeps all problem-specific components shareable, is practically meaningful in a privacy-constrained setting.
  3. The experimental analysis is careful: a)tool-family ablations; b)comparisons across 12 agent configurations; c)a fixed-perception experiment that separates perception from reasoning; d)in the cohort analysis, multiple-comparison correction, patient-level bootstrap intervals, and an equal-weight sensitivity analysis.
Critical weaknesses
  1. Methodological novelty is limited. The overall architecture has direct precedents in long-video understanding. It consists of minute-level captions, hourly summaries, and an agent that retrieves across the hierarchy and re-inspects raw clips: a) EgoLife, Ego-R1, and VideoLucy use hierarchical memory with retrieval; b)VideoAgent and Deep Video Discovery use tool-calling agents that re-inspect video segments. The atomic action representation follows the AVA formulation, although the 15 ICU-specific categories were defined by the authors together with clinicians. The main technical addition is EHR integration with query tools, which largely amounts to time-windowed queries over structured records. The application value is clear, but the methodological contribution is modest.
  2. No comparable agentic baseline is included, so the source of the gains is unclear. a) Because the contribution lies in the harness design rather than the backbone, the most informative comparison would pit different agent architectures against each other under the same backbone, the same video and EHR access, and a similar tool-call budget. The current baselines are limited to long-context VLMs and Socratic pipelines; no agentic long-video method is evaluated, with or without the same EHR tools. b) Under the cumulative ordering used in Table 3, adding EHR tools yields a much larger average gain (+17.4) than adding the video tool last (+4.2). This suggests that a substantial part of the overall improvement may come from making the structured EHR queryable, rather than from the long-video representation itself. c) On video questions spanning less than five minutes, TRACER reaches 36.4% with the Gemini 3.1 Flash Lite backbone.
  3. The QA benchmark is small and cannot be released. a) TRAJECTORY-144 contains 144 questions but only two independent patient trajectories. The authors acknowledge it is “deep rather than broad”, and it cannot test generalization across patients. b) The data contain protected health information, so the community cannot use or reproduce the benchmark. c) Language priors are strong. Gemini 3.5 Flash reaches 39.3% average accuracy without any patient data, and 25 questions (17%) are answered correctly at least 80% of the time by a text-only model. d) The final questions were not directly used for prompt development. However, the agent prompt was developed on earlier question drafts from the same two stays, which raises concerns about benchmark-specific adaptation.
  4. The delirium cohort analysis supports only patient-level retrospective associations. a) The delirium group has longer stays (median 4.62 vs 3.09 days) and a higher share of deep-sedation assessments (5.0% vs 2.0%), yet no confounder-adjusted analysis is reported. b) The label is “any positive CAM-ICU screen during the stay”, while behavior frequencies are pooled over the entire stay. Behavior before the onset of delirium is therefore attributed to the delirium group. This grouping is legitimate for patient-level association, but it cannot show that the behavioral changes coincide with delirium episodes. c) The perception model used for this analysis has a micro-F1 of 62.2. If detection errors differ between groups (for example, through more occlusion or restraint in the delirium group), these errors alone could contribute to the observed differences.
  5. The faithfulness of the hourly reports is insufficiently validated. The agent relies on hourly reports to localize evidence, yet only ten reports were audited, with an 84% support rate. The audit found false statements that no medication was given, and these errors propagated into the activity–medication section. The agent can re-inspect raw video and EHR, but that does not guarantee such errors are caught. A larger audit across more patients and activity levels is needed.
Questions for authors
  1. Can you add at least one agentic long-video baseline (for example VideoAgent or Deep Video Discovery) with the same backbone, the same EHR tools, and a comparable tool-call budget?
  2. Can you report a delirium analysis adjusted for length of stay, sedation depth, and mechanical ventilation? Where feasible, an analysis aligned with CAM-ICU assessment times would also help, for example windows adjacent to positive versus negative screens.
  3. On the annotated benchmark, does per-action detection performance differ between patients with and without delirium?
  4. Do you plan to release any evaluation resources, such as de-identified minute-level records or synthetic data, so that the community can reproduce and compare against your results?
Reviewer_3EFJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses reasoning over multi-day ICU patient trajectories that combine continuous bedside video with electronic health records, under the constraint that no model can be trained on protected health information and that raw video far exceeds any current context window. The proposed system, TRACER, is a training-free harness that converts each one-minute video clip into a structured atomic-action vector plus caption, fuses these with medication events into hourly reports, and equips a tool-using agent LLM to query this hierarchy and the raw evidence on demand. Evaluated on a perception benchmark and a newly introduced longitudinal QA benchmark, TRACER substantially outperforms long-context and Socratic baselines sharing the same backbone, and applying the perception pipeline to a 280-patient cohort surfaces a statistically significant association between reduced purposeful activity and delirium.

Critical strengths

The baseline comparison is credible since the long-context and Socratic baselines are matched to the same backbone and shown to use roughly three times TRACER's tokens yet still underperform, ruling out the trivial explanation that TRACER wins on context budget alone. The ablation design is a genuine strength because decomposing gains by tool family and question time-scale shows the hierarchy's components specialize rather than being redundant, which directly substantiates the multi-scale motivation. The cohort-level delirium analysis is unusually careful, since it includes bootstrap intervals, two weighting schemes, multiple-testing correction, and a faithfulness audit, which makes the clinical claim considerably more defensible than a typical single-cohort correlation.

Critical weaknesses

The hierarchical compression has a structural chicken-and-egg problem that the paper's design does not address. The agent decides whether to drop down to raw video based on what the hourly report (itself built from per-minute atomic vectors) suggests is worth investigating. If a clinically salient event is subtle enough that it was never captured in the 15-dimension atomic vector or its caption in the first place — rather than merely summarized away at the hourly level — then nothing in the hourly report will ever flag it as worth checking, and the agent has no mechanism to go looking for an event it has no reason to suspect exists. This means the system's recall for rare but important events (a fall, a self-extubation attempt, an assault) depends entirely on whether the fixed 15-dimension taxonomy was designed with clinical completeness in mind, not just accuracy on the categories it already defines. The paper validates perception accuracy within its chosen taxonomy but does not validate the taxonomy's coverage against real clinical information needs; a targeted test of recall specifically on rare safety-critical events, independent of the categories used for the main 397-clip perception benchmark, would be the way to check this.

Related to this, nothing in the pipeline carries calibrated uncertainty from the perception layer up to the agent's reasoning layer. A per-minute atomic vector that is wrong (misclassified, occluded, ambiguous) is folded into the hourly report by the summarizing LM with the same apparent authority as a correct one, and the agent has no signal telling it "this report segment was built from low-confidence perception, verify against raw video before trusting it." Table 3's toolset ablation shows that having access to raw video helps overall, but it does not show whether the agent's actual choice of when to invoke the raw-video tool is correlated with where perception is weak — if the agent mostly asks for raw video on already-easy segments and trusts the report exactly where perception is shakiest, the tool's availability would not actually be buying the robustness the paper implies. Reporting the correlation between tool-invocation rate and per-category perception accuracy would test this directly.

The fusion of medication events into hourly-granularity reports creates a specific temporal-alignment risk that the paper does not discuss: a medication administered near an hour boundary, or a behavioral change that actually precedes a dose by a few minutes but gets bucketed into the same hourly window as the dose, could be misattributed by the summarizing LM as co-occurring or causally linked when the finer-grained ordering says otherwise. For a system whose main downstream use case is reasoning about temporal association between medication and behavior, this is a design choice that trades resolution for compression at exactly the junction most load-bearing for the kind of question the system is meant to answer, and a sensitivity check — perturbing event timestamps near hour boundaries and checking whether the agent's answers change — would clarify how much this matters in practice.

The architecture itself — hierarchical temporal summarization exposed to a tool-using agent — closely follows existing long-video and life-logging agent designs, as the related work section states. The genuinely new piece is the clinical atomic-action taxonomy and the second asynchronous modality (EHR), but no experiment isolates how much of the reported gain comes from this clinical-specific design versus from the general hierarchical-agent recipe already validated elsewhere. Swapping in free-text captions in place of the structured atomic vectors, with everything else held fixed, would show whether the taxonomy itself is pulling weight or whether a generic captioning hierarchy would get most of the way there.

The longitudinal benchmark is built from only two patient stays, and the fixed agent prompt contains working rules naming medications that appear in the example questions, with the paper stating these rules were developed against earlier drafts of the benchmark questions from the same two stays. This creates a real risk that part of the reported accuracy reflects the prompt being shaped around these two records' specific vocabulary rather than a general longitudinal-reasoning capability, and the cleanest check is accuracy on a stay that played no role in prompt iteration.

Questions for authors

How would the system behave if a clinically important event falls entirely outside the 15-dimension atomic taxonomy — is there any fallback mechanism for the agent to suspect something happened that the structured representation cannot express? Could the authors report whether raw-video tool invocation is correlated with per-category perception accuracy, to establish whether the agent actually checks itself where it is weakest? Could the authors isolate the gain attributable to the clinical taxonomy specifically, versus a generic captioning hierarchy with the same agent scaffold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.