- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper addresses medical visual question answering when the image is unavailable at inference. MedJepa is an 85M-parameter transformer (text encoder over frozen PMC-Llama token embeddings, decoder with 576 learnable spatial queries) that, from the caption/hint and question alone, regresses the 576x1024 Clip-Vit-L/14 at 336 patch features the missing image would have produced. The paper formulates missing-image medical VQA as predicting a VLM's latent visual input rather than pixel reconstruction.
Critical strengths
Predicting the frozen VLM's pre-projector CLIP tokens means the VLM is left untouched, a single predictor can serve any model sharing the encoder, and the confidence gate provides a text-only fallback. The reported gains are positive across all six backbone x dataset cells, including the two non-primary backbones. This is good because it is the one piece of evidence that the approach does something real across models.
There are candid diagnostics that correctly identify what is learned. The retrieval analysis and limitations state plainly that the predictor recovers imaging modality rather than the specific image, and that confidence is not correctness. The eight-model table is also a useful reference for how informative captions alone are for current VLMs on PMC-VQA: caption-only accuracy matches or exceeds the full-image column for most models (though, with the columns undefined, it does not establish that adding the image to the caption does not help). This does not offset the soundness problems, since the claims in the abstract and 2.4 are not revised in light of the paper's own diagnosis.
Critical weaknesses
Figures are definitely not a strength of this paper. Figures 1 and 3 are central to the paper's story but do not have publication quality. Figures 2 and 4 are almost uninformative as they are quite small and difficult to read.
As far as I understand from the ICLR 2027 submission guidelines, the "AI Use Statement" should be placed in the main text, right before the references, but this is not the case in this paper. Indeed, there is a "Use of Large Language Models" section in the Appendix, so it is not a fatal omission, but it conflicts with the guidelines.
The source of the gain is unestablished, and the paper's own evidence points away from visual recovery. Sec. 2.4 defines the output as an estimate of the representation the VLM would have received from the missing image. Yet, the retrieval diagnostic shows the predictor recovers the imaging modality rather than the image.
Table 1's columns are not defined: if "Full-image" includes the caption, the image lowers accuracy on the primary backbone (69.69 vs. 70.8), and the ground-truth-token ceiling contradicts it; if it excludes the caption, the column is inconsistent with the protocol described in A.9, and the degradation/recovery narrative holds only once that is stated. Either way, the sole negative control is random visual tokens (distribution unspecified) that collapse the model to chance and therefore bind nothing.
Without structured-but-wrong controls (tokens of another image, a same-modality retrieved image, a modality-mean grid, predictions from shuffled captions) and a no-gate/oracle-gate ablation, the +2.6/+3.5 pp cannot be distinguished from restoring an in-distribution input format or from confidence-based selection among three candidate answers. This is decisive because contribution 1 (latent recovery rather than reconstruction) and the paper's framing rest on it. What survives without it is a small, unexplained multiple-choice gain.
The experimental record is internally inconsistent and omits the comparison that would situate the work. The default configuration in every sensitivity analysis and the v2 recipe score below the 70.8 text-only figure in Table 1, and the paper does not say which test subset they use or what the text-only scores on that subset are.
Table 3 attributes a +2.74 jump to the addition of auxiliary losses, while Tables 6-7 show that zeroing or changes accuracy by less than 0.1 pp. It is difficult to believe they are describing the same model, and the paper does not specify which configuration Tables 5-7 evaluate. Two incompatible sets of default loss weights are given. No validation protocol is described, sensitivity sweeps are reported on test examples, no seeds or intervals are given, and every hyperparameter-sweep difference and most Table 4 rows fall below the ~0.65 pp sampling error of a 5,000-example subset, yet are narrated as findings. Therefore, a reader cannot identify which configuration produced Table 1 or whether any proposed component matters.
At the same time, the closest prior method, the Latent Imagination Module (Kim et al., ICLR 2026, https://arxiv.org/pdf/2605.12517) -- one learnable query per visual token, cross-attention to text, injection into a frozen VLM -- is cited to an unrelated paper (GeoWorld-VLM), is said to differ from MedJEPA on patch-level prediction and preservation of the frozen pathway, both of which it shares, and is never compared against. The only distinction I can identify, prediction of pre-projector rather than post-projector features, is unverified and modest. This is decisive because, even taking the headline numbers at face value, the paper does not show that its additions (curriculum, refinement, and gating) improve on the pre-existing architecture, and its novelty positioning is incorrect.
Questions for authors
Does the "Full-image" column include the caption (PMC-VQA) or hint (ScienceQA)? A.9 describes text-only as the full-image setting with the visual input removed, which implies it does; if so, why does injecting ground-truth CLIP tokens not reproduce the full-image result when both should traverse the same frozen encoder -> projector -> LLM path? If it does not, please relabel and state the ceiling for ScienceQA as well.
Which model is the final model? State the configuration behind each of 73.4, 73.2 (v3), 70.46 (v2), and 69.82: loss weights (Eq. 6 or sec. 3.3/Table 6), cascade on or off, number of training steps (Table 4 gives 20,000 for the ablations only), test subset and its size, and the text-only accuracy on that subset. Define v2, v3, L1-L3, R1, T2, B3, B4.
Adding SigLIP, alignment, and VICReg jointly is reported as +2.74, while or changes accuracy by <0.1 pp. Do Tables 6-7 evaluate a different configuration or subset from Table 3, or does v3 differ from v2 in something other than loss weights?
Sec. 3.1 can be read as saying the predictor was trained on ScienceQA images, whereas secs. 3.2 and 4 describe the result as out-of-distribution / task transfer. Which is it? Which subset was used (image-bearing questions only? questions with non-empty hints?), and how were questions with 2, 3, or 5 choices handled, given that the text states four?
Image-absent accuracy when injecting (a) ground-truth CLIP tokens of a random other test image, (b) tokens of a same-modality image retrieved from the training set by caption, (c) a modality-mean token grid, (d) MedJEPA's prediction from a shuffled caption. Also, what distribution was used for the "random visual tokens" baseline (B4)? If any of (a)-(d) approaches 73.4, the gain is not image-specific.
Provide seeds or confidence intervals for Tables 1 and 3
The LIM citation points to Gu et al. (GeoWorld-VLM), which concerns geometry distillation. The LIM I could locate is Kim et al., ICLR 2026 (https://arxiv.org/pdf/2605.12517), and uses one learnable query per visual token, cross-attends to text, and injects into a frozen VLM. Please correct the citation, state precisely what MedJEPA adds (does LIM predict pre- or post-projector features?), and, if possible, report LIM on PMC-VQA with the same backbone.
Figure 3's loss decomposition (L_rec, L_sg, L_ap, L_neg, L_cop), masking range U(0,1), top-K range 20-50, and confidence ranges differ from the text. Which is implemented? How do dynamic masking, patch annealing, and the refinement pass compose within a single training step? Which pass does L_pred supervise? Are injected tokens L2-normalized or restored to CLIP's native scale before the projector?
Which checkpoint is MedLLaVA-7B (LLaVA-Med v1.5 / Mistral-7B? How were 576 predicted tokens injected into Phi-3.5-Vision's multi-crop pipeline? How are 144 sparse tokens presented to a model trained on 576-token grids?
Sec. 3.1 describes question-only and caption+question baselines; only one text-only column is reported. Where is caption+question? Please report question-only accuracy so the caption's contribution to 70.8 can be quantified.
The authors did not respond and address the concerns. I will keep original rating.