acceptodds
Under review as a conference paper at ICLR 2027

ViSTA: A Simple Bridge Extends Visual Alignment to Clinical Time-Series Understanding in Multimodal LLMs

Abstract

Clinical prediction models estimate risk from patient measurements, while large language models support medical text understanding and question answering. Yet their language capabilities do not ensure accurate prediction from structured, high-dimensional clinical time series. Improving this ability would connect risk estimation with flexible questions about a patient's evolving condition. We introduce , a compact adapter that incorporates irregular numerical measurements into a pretrained vision-language model's chart representations. It learns corrections to visual tokens while leaving all pretrained parameters unchanged. On MIMIC-IV, has the highest mean scores among the compared adaptations on all four metrics for acute kidney injury and mortality prediction across models with 2-9 billion parameters. With 0.516 million trainable parameters, the 2-billion-parameter model reaches an area under the ROC curve of 0.7376 for acute kidney injury, compared with GPT-5.6 Sol's 0.7380 with text input and high reasoning effort. Training for temporal question answering yields 69.27% accuracy at 4 billion parameters with over 90% fewer trainable parameters than low-rank adaptation using charts or numerical text, at a 2.82-4.88 percentage-point accuracy gap. extends pretrained language models to numerical prediction and temporal questions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Clinical Time Series, Visual Alignment, Latent Communication, Parameter-Efficient Adaptation, Electronic Health Records

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ki91OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ViSTA augments a frozen vision-language model with numerical information from irregular clinical time series. An event encoder pools measurements, timestamps, observation gaps, and variable identities, and a cross-attention bridge adds zero-initialized, RMS-scaled residuals at the original chart-token positions. The pretrained decoder and language output head remain unchanged. Evaluation uses MIMIC-IV for creatinine-defined AKI during ICU hours (24,48], in-hospital mortality, and eleven categories of automatically generated temporal multiple-choice QA, across Qwen3.5-2B/4B/9B. Outcome prediction and supervised QA use separately trained adapters. ViSTA-Joint achieves the highest reported mean AUROC/AP among the compared Qwen adaptations at every scale. Supervised QA improves substantially over unadapted models but remains below LoRA, several specialized clinical predictors also outperform ViSTA on prediction.

Critical strengths
  1. A favorable trainable-parameter tradeoff for outcome prediction. At 4B, ViSTA uses 0.582M trainable parameters, versus 6.193M for Chart LoRA and 7.032M for Text LoRA. Its AKI AP is 0.4650 versus the strongest competing adaptation's 0.4508, and mortality AP is 0.2651 versus 0.2464. ViSTA-Joint leads the reported prediction means across all three backbone sizes. This consistency makes the compact interface promising, although trainable parameter count alone does not establish lower runtime or memory use.

  2. QA gains span several operations, with promising transfer after outcome training. At 4B, ViSTA-Separate achieves 69.27% accuracy, compared with 39.40% for unadapted text and 59.18% for the strongest of ChatTS, MLLM4TS, ITFormer, and OpenTSLM. Removing numerical summaries still leaves 66.45% versus 59.25% for the category-wise best of those four baselines. This supports usefulness beyond one especially favorable category. Chart/Text LoRA remain stronger at 72.09%/74.15%, making this a promising parameter-accuracy tradeoff. At 9B, outcome-only training yields 51.00% QA accuracy for ViSTA-Joint, versus 47.00% for unadapted text and 43.67% for outcome-trained Text LoRA. This suggests useful cross-task behavior, although its robustness requires repeated-seed validation.

Critical weaknesses
  1. The distinctive benefit of the visual residual interface is not isolated. Prior MLLM4TS already combines numerical and plot features, while ITFormer connects temporal encoders to frozen language models. I therefore view ViSTA's incremental contribution as correction at native visual positions while preserving the pretrained output interface. Table 8 removes visual tokens or numerical residuals only at inference. These interventions change the input distribution, and the residual-only branch still depends on visual features through its queries and RMS scaling. They do not establish an advantage over a comparably trained numerical interface. The initialization/scaling ablations are useful, and Appendix F correctly acknowledges the limitation, but matched retraining controls are central to judging the proposed mechanism and its novelty.

  2. The comparative advantage needs stronger robustness and implementation evidence. The 4B AP gains over the strongest adaptations are 0.0142 for AKI and 0.0187 for mortality. The reported bootstrap SDs quantify patient-sampling uncertainty for one trained seed, training variability and paired uncertainty for these outcome differences remain unreported. Overlapping marginal SDs alone establish neither significance nor its absence. Table 8 also reports the zero-initialization and RMS-scaling training ablations without uncertainty estimates, leaving the robustness of these design-specific gains unclear. Moreover, Section 4 and Appendix D omit baseline search ranges and selected configurations, including LoRA ranks/target modules and details of the modified interfaces and output heads. This is particularly consequential for understanding how an outcome-trained, task-head-based baseline such as MLLM4TS is evaluated on QA. These gaps make the claimed performance and parameter comparisons difficult to independently assess.

Questions for authors
  1. Controlled experiment: With the same numerical encoder, backbone, and comparable trainable-parameter/token budgets, how does ViSTA compare with a numerical-only interface and a model using fixed, patient-independent chart tokens, each trained under its own input condition? These controls would determine whether the proposed visual pathway contributes beyond numerical adaptation.

  2. Clarification and robustness: Can the authors provide the selected baseline architectures, output-head handling for outcome-to-QA evaluation, LoRA settings, and hyperparameter search ranges? Please report repeated-seed results and paired patient-bootstrap intervals for the principal 4B prediction comparisons and for the zero-initialization and RMS-scaling training ablations. This would materially affect my confidence in the reported ranking and design-specific gains.

4 comments

Dear Reviewer, thank you again for the detailed review. As the discussion period ends on November 18, we wanted to check whether our response addresses your questions. Following your suggestions, we added matched controls trained under their own input conditions (numerical-only interfaces and fixed, patient-independent chart tokens), three-seed results with paired bootstrap intervals for the principal 4B comparisons and the design ablations, and complete baseline implementation details (Appendices L–N). We would be glad to clarify anything further before the deadline.

Dear Reviewer ki91, a brief follow-up: the discussion period closes this Wednesday, November 18 (AoE). We would be grateful for any feedback on our response and are happy to clarify anything before then. Thank you for your time.

Dear Reviewer ki91. We understand this is a busy period and appreciate your time. As the author–reviewer discussion closes tomorrow, November 18 (AoE), we would be grateful to know whether our response addresses your concerns. We can still clarify anything before the deadline and we look forward to discussing. Thank you again for your review.

Dear Reviewer, today is the last day of the author–reviewer discussion (November 18, AoE). Since November 9, we have provided all the analyses you requested. These are matched controls trained under their own input conditions, including fixed, patient-independent chart tokens; three-seed results with paired bootstrap intervals for the principal comparisons and the design ablations; and full baseline implementation details (Appendices L–N). We would be grateful if you could take them into account in your final assessment, and we remain available for any questions until the deadline.

Reviewer_gGdXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that LLMs struggle with numerical time series such as mortality risk. It proposes ViSTA, a lightweight adapter that uses pretrained visual representations as the foundation and refines them with numerical measurements through residual corrections. By keeping the pretrained VLM frozen and reusing its original language output interface, the method supports both clinical prediction and temporal QA without task-specific output heads. Experiments on MIMIC-IV show competitive prediction performance and strong parameter efficiency.

Critical strengths

Parameter-efficient and practical architecture. The paper proposes a lightweight way to incorporate numerical time-series information into pretrained VLMs without updating the backbone or introducing task-specific output heads. The idea of using existing visual representations as the foundation and refining them with numerical information is interesting. The method achieves competitive performance with substantially fewer trainable parameters.

Extensive empirical evaluation. The paper compares several existing approaches across three backbone sizes, clinical prediction tasks, and different temporal QA categories and shows competitive results.

Critical weaknesses

The motivation behind the proposed bridge is not entirely clear. I can understand the general idea of combining visual representations with numerical measurements, and the experiments show that the approach works well. However, I am still not fully clear on how the authors arrived at this particular design. The paper discusses limitations of existing approaches, but the connection between these limitations and the specific architectural choices could be better explained. This would help clarify the main methodological insight.

Questions for authors

Could the authors explain more clearly why the proposed residual bridge is needed compared with existing approaches? How does the proposed bridge address the limitations in the prior work? I think a clearer explanation of the reasoning behind this design would help me better understand the contribution and could positively influence my rating.

6 comments

Dear Reviewer, thank you again for your thoughtful review. You noted that a clearer explanation of the design reasoning could inform your assessment. Section 3 now opens with this rationale, and Appendix K traces each component to a specific limitation of prior interfaces, together with a step-by-step design-path experiment. We hope this makes the contribution clearer, and we are happy to answer any remaining questions before the discussion closes on November 18.

Dear Reviewer gGdX18 , a brief follow-up: the discussion period closes this Wednesday, November 18 (AoE). We would be grateful for any feedback on our response and are happy to clarify anything before then. Thank you for your time.

Dear Reviewer gGdX. We understand this is a busy period and appreciate your time. As the author–reviewer discussion closes tomorrow, November 18 (AoE), we would be grateful to know whether our response addresses your concerns. We can still clarify anything before the deadline and we look forward to discussing. Thank you again for your review.

Reviewer_7PycOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors proposed a compact adapter that enables pretrained vision-language models to perform accurate prediction from structured, high-dimensional clinical time series, bridging traditional risk estimation with flexible questions about a patient’s evolving condition. The adapter injects irregular numerical measurements into the model’s chart representations by learning corrections to visual tokens while keeping all pretrained parameters frozen. Evaluated on MIMIC-IV across models ranging from 2 to 9 billion parameters, it achieves the highest mean scores among compared adaptation methods on all four metrics for acute kidney injury and mortality prediction. Notably, with only 0.516 million trainable parameters, the 2-billion-parameter model reaches an AUROC of 0.7376 for acute kidney injury, nearly matching GPT-5.6 Sol’s 0.7380 obtained with text input and high reasoning effort. For temporal question answering, the 4B model attains 69.27% accuracy using over 90% fewer trainable parameters than low-rank adaptation with charts or numerical text.

Critical strengths
  • The method description is very clear. The method also stays simple when a lot of papers stack complicated designs. The event features (Eq. 7), pooling, position-conditioned queries, zero-init and RMS scaling are well motivated and symbols stay consistent across the paper. All terms are motivated and introduced before being used. This makes the paper easy to reproduce.

  • The experiments covers a wide range of baselines, datasets, alongside complete ablations. The main table gives very comprehensive results with confidence intervals. Figure 3 also shows the performance improvements are persistent across model sizes.

Critical weaknesses
  • A lot of the designs in the method description has been widely used in past works. For example, the embedding design and attention pooling is similar to STraTS https://arxiv.org/abs/2107.14293, and the learned query resampler design with a frozen LLM is similar to Flamingo (https://arxiv.org/abs/2204.14198). The paper would benefit from a clear comparison of methods in general domain, not limited to medical.

  • The paper did a detailed ablation study on the design and comparison on model size. However, it seems that not enough ablation has been done on the modality side. Table 8 did seem to provide some ablation on the settings where only V/R is provided. However, the experiments are inference time only. I would be interested to see what kind of unique information the combination of two modalities add, which are not visible with the training of only one modalities, and what kind of benefits it brings.

Questions for authors

Please see weaknesses.

5 comments

Dear Reviewer, thank you again for your review. In response to your comments, we added a comparison with general-domain interfaces under a fixed encoder, and training-time modality ablations that show what each modality contributes (Appendices K–L). We would welcome any further feedback before the discussion period ends on November 18.

Dear Reviewer 7Pyc, a brief follow-up: the discussion period closes this Wednesday, November 18 (AoE). We would be grateful for any feedback on our response and are happy to clarify anything before then. Thank you for your time.

Dear Reviewer 7Pyc. We understand this is a busy period and appreciate your time. As the author–reviewer discussion closes tomorrow, November 18 (AoE), we would be grateful to know whether our response addresses your concerns. We can still clarify anything before the deadline and we look forward to discussing. Thank you again for your review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.