acceptodds
Under review as a conference paper at ICLR 2027

MedJEPA: Latent Cross-Modal Prediction for Medical Vision-Language Reasoning under Missing Images

Abstract

Vision Language Models (VLMs) have become effective tools for reasoning over medical images, yet their multimodal reasoning pathways typically assume that the image is available at inference. In many real-world clinical scenarios, images may be unavailable because they are not transmitted across clinical workflows, are withheld due to privacy or data-sharing constraints, are corrupted during transmission, or have not yet been acquired. Existing missing-modality methods rely on prompt adaptation, retrieval, or modality reconstruction, but these strategies either work around the missing input or attempt to recreate the observation itself, rather than directly recovering the latent visual representation that the downstream VLM was trained to reason over. To overcome these limitations, we propose MedJEPA, a latent cross-modal prediction framework inspired by the Joint-Embedding Predictive Architecture (JEPA). Given only the clinical text and the corresponding question, MedJEPA predicts the patch-level visual representations that a frozen medical VLM would extract from the missing image, without synthesizing pixels.The predictor is trained with a JEPA-style masked latent objective in CLIP feature space, using the frozen image representation as a stop-gradient target. A masking curriculum spans partial to complete visual absence, with patch annealing progressively removing real visual tokens and iterative self-refinement improving the latent prediction from previous estimates. At inference time, the predicted visual tokens are projected through the frozen multimodal projector of the VLM and injected into its visual stream. A confidence-gated cascade then progressively evaluates sparse and dense visual-token injection, reverting to text-only reasoning when the predicted visual signal fails to improve the model’s confidence. Across PMC-VQA and ScienceQA, MedJEPA consistently improves image-absent accuracy, from 70.8% to 73.4% on PMC-VQA and from 68.8% to 72.3% on ScienceQA, yielding gains of 2.6 and 3.5 percentage points, respectively. Ablations confirm the benefits of latent prediction, anti-collapse regularization, and self-refinement. These results show that latent cross-modal prediction enables medical VLMs to reason more reliably when images are unavailable.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
JEPA, Missing Modality, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_MwtPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses medical visual question answering when the image is unavailable at inference. MedJepa is an 85M-parameter transformer (text encoder over frozen PMC-Llama token embeddings, decoder with 576 learnable spatial queries) that, from the caption/hint and question alone, regresses the 576x1024 Clip-Vit-L/14 at 336 patch features the missing image would have produced. The paper formulates missing-image medical VQA as predicting a VLM's latent visual input rather than pixel reconstruction.

Critical strengths

Predicting the frozen VLM's pre-projector CLIP tokens means the VLM is left untouched, a single predictor can serve any model sharing the encoder, and the confidence gate provides a text-only fallback. The reported gains are positive across all six backbone x dataset cells, including the two non-primary backbones. This is good because it is the one piece of evidence that the approach does something real across models.

There are candid diagnostics that correctly identify what is learned. The retrieval analysis and limitations state plainly that the predictor recovers imaging modality rather than the specific image, and that confidence is not correctness. The eight-model table is also a useful reference for how informative captions alone are for current VLMs on PMC-VQA: caption-only accuracy matches or exceeds the full-image column for most models (though, with the columns undefined, it does not establish that adding the image to the caption does not help). This does not offset the soundness problems, since the claims in the abstract and 2.4 are not revised in light of the paper's own diagnosis.

Critical weaknesses

Figures are definitely not a strength of this paper. Figures 1 and 3 are central to the paper's story but do not have publication quality. Figures 2 and 4 are almost uninformative as they are quite small and difficult to read.

As far as I understand from the ICLR 2027 submission guidelines, the "AI Use Statement" should be placed in the main text, right before the references, but this is not the case in this paper. Indeed, there is a "Use of Large Language Models" section in the Appendix, so it is not a fatal omission, but it conflicts with the guidelines.

The source of the gain is unestablished, and the paper's own evidence points away from visual recovery. Sec. 2.4 defines the output as an estimate of the representation the VLM would have received from the missing image. Yet, the retrieval diagnostic shows the predictor recovers the imaging modality rather than the image.

Table 1's columns are not defined: if "Full-image" includes the caption, the image lowers accuracy on the primary backbone (69.69 vs. 70.8), and the ground-truth-token ceiling contradicts it; if it excludes the caption, the column is inconsistent with the protocol described in A.9, and the degradation/recovery narrative holds only once that is stated. Either way, the sole negative control is random visual tokens (distribution unspecified) that collapse the model to chance and therefore bind nothing.

Without structured-but-wrong controls (tokens of another image, a same-modality retrieved image, a modality-mean grid, predictions from shuffled captions) and a no-gate/oracle-gate ablation, the +2.6/+3.5 pp cannot be distinguished from restoring an in-distribution input format or from confidence-based selection among three candidate answers. This is decisive because contribution 1 (latent recovery rather than reconstruction) and the paper's framing rest on it. What survives without it is a small, unexplained multiple-choice gain.

The experimental record is internally inconsistent and omits the comparison that would situate the work. The default configuration in every sensitivity analysis and the v2 recipe score below the 70.8 text-only figure in Table 1, and the paper does not say which test subset they use or what the text-only scores on that subset are.

Table 3 attributes a +2.74 jump to the addition of auxiliary losses, while Tables 6-7 show that zeroing λalign\lambda_{align} or λsiglip\lambda_{siglip} changes accuracy by less than 0.1 pp. It is difficult to believe they are describing the same model, and the paper does not specify which configuration Tables 5-7 evaluate. Two incompatible sets of default loss weights are given. No validation protocol is described, sensitivity sweeps are reported on test examples, no seeds or intervals are given, and every hyperparameter-sweep difference and most Table 4 rows fall below the ~0.65 pp sampling error of a 5,000-example subset, yet are narrated as findings. Therefore, a reader cannot identify which configuration produced Table 1 or whether any proposed component matters.

At the same time, the closest prior method, the Latent Imagination Module (Kim et al., ICLR 2026, https://arxiv.org/pdf/2605.12517) -- one learnable query per visual token, cross-attention to text, injection into a frozen VLM -- is cited to an unrelated paper (GeoWorld-VLM), is said to differ from MedJEPA on patch-level prediction and preservation of the frozen pathway, both of which it shares, and is never compared against. The only distinction I can identify, prediction of pre-projector rather than post-projector features, is unverified and modest. This is decisive because, even taking the headline numbers at face value, the paper does not show that its additions (curriculum, refinement, and gating) improve on the pre-existing architecture, and its novelty positioning is incorrect.

Questions for authors

Does the "Full-image" column include the caption (PMC-VQA) or hint (ScienceQA)? A.9 describes text-only as the full-image setting with the visual input removed, which implies it does; if so, why does injecting ground-truth CLIP tokens not reproduce the full-image result when both should traverse the same frozen encoder -> projector -> LLM path? If it does not, please relabel and state the ceiling for ScienceQA as well.

Which model is the final model? State the configuration behind each of 73.4, 73.2 (v3), 70.46 (v2), and 69.82: loss weights (Eq. 6 or sec. 3.3/Table 6), cascade on or off, number of training steps (Table 4 gives 20,000 for the ablations only), test subset and its size, and the text-only accuracy on that subset. Define v2, v3, L1-L3, R1, T2, B3, B4.

Adding SigLIP, alignment, and VICReg jointly is reported as +2.74, while λalign\lambda_{align} or λsiglip\lambda_{siglip} changes accuracy by <0.1 pp. Do Tables 6-7 evaluate a different configuration or subset from Table 3, or does v3 differ from v2 in something other than loss weights?

Sec. 3.1 can be read as saying the predictor was trained on ScienceQA images, whereas secs. 3.2 and 4 describe the result as out-of-distribution / task transfer. Which is it? Which subset was used (image-bearing questions only? questions with non-empty hints?), and how were questions with 2, 3, or 5 choices handled, given that the text states four?

Image-absent accuracy when injecting (a) ground-truth CLIP tokens of a random other test image, (b) tokens of a same-modality image retrieved from the training set by caption, (c) a modality-mean token grid, (d) MedJEPA's prediction from a shuffled caption. Also, what distribution was used for the "random visual tokens" baseline (B4)? If any of (a)-(d) approaches 73.4, the gain is not image-specific.

Provide seeds or confidence intervals for Tables 1 and 3

The LIM citation points to Gu et al. (GeoWorld-VLM), which concerns geometry distillation. The LIM I could locate is Kim et al., ICLR 2026 (https://arxiv.org/pdf/2605.12517), and uses one learnable query per visual token, cross-attends to text, and injects into a frozen VLM. Please correct the citation, state precisely what MedJEPA adds (does LIM predict pre- or post-projector features?), and, if possible, report LIM on PMC-VQA with the same backbone.

Figure 3's loss decomposition (L_rec, L_sg, L_ap, L_neg, L_cop), masking range U(0,1), top-K range 20-50, and confidence ranges differ from the text. Which is implemented? How do dynamic masking, patch annealing, and the refinement pass compose within a single training step? Which pass does L_pred supervise? Are injected tokens L2-normalized or restored to CLIP's native scale before the projector?

Which checkpoint is MedLLaVA-7B (LLaVA-Med v1.5 / Mistral-7B? How were 576 predicted tokens injected into Phi-3.5-Vision's multi-crop pipeline? How are 144 sparse tokens presented to a model trained on 576-token grids?

Sec. 3.1 describes question-only and caption+question baselines; only one text-only column is reported. Where is caption+question? Please report question-only accuracy so the caption's contribution to 70.8 can be quantified.

Reviewer_mHukOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MedJEPA, a latent cross-modal prediction framework for medical VQA under missing-image conditions. It predicts visual representations from clinical text using JEPA-style learning, incorporating progressive masking, latent self-refinement, and confidence-gated visual token injection. Experiments on PMC-VQA and ScienceQA demonstrate improvements over text-only inference.

Critical strengths
  1. Interesting approach to missing-modality reasoning. The paper explores predicting latent visual representations instead of reconstructing missing images. This provides an interesting direction for maintaining multimodal reasoning capabilities when visual inputs are unavailable.

  2. Practical integration with existing VLMs. MedJEPA integrates predicted visual tokens into frozen VLMs without modifying their underlying architectures. The experiments show consistent improvements over text-only inference across several backbones.

Critical weaknesses
  1. Insufficient evidence of meaningful visual information recovery. The model uses image-derived captions as textual inputs, which may already contain substantial visual information. Moreover, text-only inference outperforms full-image inference in several settings, and the retrieval analysis mainly demonstrates modality-level correspondence.

  2. Limited comparisons and component validation. The experiments primarily compare MedJEPA against text-only inference, without sufficient comparisons to simpler latent prediction or text-only adaptation strategies. Additionally, the confidence-gated injection mechanism lacks dedicated ablation and calibration analysis.

Questions for authors
  1. Can the authors evaluate MedJEPA with incomplete or less informative textual context to determine whether it recovers information beyond what is already contained in image-derived captions?

  2. Can the authors compare against simpler text-to-visual embedding predictors and parameter-matched text-only adaptation baselines?

  3. Can the authors provide ablations of the confidence-gated injection mechanism and evaluate whether increased confidence corresponds to improved answer correctness?

  4. Can the authors clarify the inconsistencies in reported performance gains and loss-weight configurations?

Comments on the paper

1 comment

The authors did not respond and address the concerns. I will keep original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.