acceptodds
Under review as a conference paper at ICLR 2027

Q-JEPA: Predicting Question-Relevant Visual Representations

Abstract

Acquiring visual representations that capture question-relevant regions is central to visual understanding. Existing approaches obtain such representations through explicit image manipulation, attention-guided visual selection, or autoregressive latent generation, requiring additional visual processing or intermediate generation within the language model. As the transition from a full image to a question-relevant local view can be considered as a zoom-in prediction in video, we propose Q-JEPA, which draws on V-JEPA’s video-pretrained spatiotemporal representations and predictive priors to infer question-relevant target-view representations directly from the full-image context. These representations are predicted independently of the LLM’s generation process, providing an effective and flexible visual interface with the potential to reduce inference overhead. Specifically, Q-JEPA introduces a Question-Guided Predictor that builds on the pretrained predictor with global cross-attention module to incorporate the full question context and focus cross-attention module to further associate target-view features with object- and relation-bearing contents in the question, thereby guiding prediction toward the relevant visual evidence. In addition, a three-stage training strategy progressively enables the predictor with the ability to produce question-relevant visual representations and subsequently aligns these representations with a downstream LLM for visual understanding. Evaluations show that Q-JEPA achieves 64.10% on GQA and 79.03% on PhysObjects, outperforming Qwen2.5-VL-7B by 3.72 and 19.58 percentage points, respectively, while reducing image inference latency by 22.2%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal large language models, visual question answering, visual focusing, V-JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2AMXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Q-JEPA predicts question-relevant target-view features from a full-image representation and a question, then passes the predicted features through a projector to a frozen language model. It adapts a video-pretrained predictor using global and focus cross-attention, token-routing gates, and a three-stage training procedure. The paper reports 64.10% accuracy on GQA test-dev, 79.03% on a constructed five-property, full-scene PhysObjects protocol, and video results on a four-frame STAR Interaction T1 subset. It also reports reductions in visual-module parameters and image inference time. These results make the interface promising, although supervision, task adaptation, and measurement conventions still limit attribution of the gains.

Critical strengths
  • The design addresses a clear interface problem: different questions about the same image require different visual evidence. Performing this transformation before language generation and providing a fixed 144-token visual interface gives the method a coherent modular structure. The appendix's clarification that the target position is a virtual readout slot also helps explain what target-view prediction means.
  • The ablations provide meaningful component evidence. Under the corresponding configurations, predictor pretraining improves accuracy by 1.59 percentage points, focus cross-attention adds 1.55 points, and the auxiliary focus loss adds .33 points. The pretrained-versus-random comparison is particularly useful because it supports a specific design choice more directly than comparisons between heterogeneous complete systems.
  • The reported QA improvements are practically interesting. Relative to Qwen2.5-VL-7B, accuracy increases by 3.72 points on GQA and 19.58 points on the constructed PhysObjects protocol. The video subset also shows gains under both sampling strategies. These findings extend beyond the selected qualitative examples.
  • The appendices clarify several important experimental distinctions: region pairs versus unique images, the five-property PhysObjects scope, the STAR subset and video-disjoint roles, and the use of ground-truth crop dimensions when displaying predicted features. These details make the scope of the results easier to assess.
Critical weaknesses
  • The main comparisons do not adequately separate architecture from additional training. Q-JEPA receives Flickr region-pair training and downstream supervision, but the corresponding data, region/token annotations, adaptation budgets, and input configurations for competitors are not clearly specified. The Stage 1 ablation also adds data and training, so it does not isolate the benefit of curriculum order. This is not evidence that the comparison is necessarily unfair, but it currently favors a system-level interpretation of the gains.
  • Several training definitions are important yet incomplete. The paper should clarify how target regions are selected for multi-entity and relational questions, how PhysObjects and STAR targets are constructed, and whether the original pretrained predictor is updated. Feature-loss reduction, stage lengths, and the relationship between task-specific training and checkpoints are also unclear. These choices affect target semantics, effective loss weights, and ablation budgets.
  • The efficiency measurements need a more precise scope. The reported 22.2% time reduction and 37.8% visual-parameter reduction are arithmetically correct, but timing boundaries, inference batch size and precision, inclusion of text encoding and language-model or candidate scoring, and counted modules are not fully defined. Since the visual backbone and token interface also change, the entire speed difference cannot be attributed to predictive focusing.
  • Some mechanistic descriptions exceed the available visualization and ablation evidence. The .33-point gain lacks run-to-run variability, and a few successful images do not establish general localization accuracy. Figure 5 calls its spatial map cross-attention, whereas the appendix defines target-to-source self-attention; predicted-feature displays also obtain their dimensions from the ground-truth crop. These descriptions should distinguish feature prediction, text routing, and box localization more clearly.
Questions for authors
  • What downstream data, region/token supervision, input resolutions, and visual-token settings did each Table 1 method actually use? What stage data and training budgets correspond to the Table 2 configurations?
  • Which parts of the original predictor, added attention modules, FocusGate, text projection, and final projector are updated at each stage? Please also clarify feature-loss reduction, task/checkpoint relationships, and the existing evidence-region construction rules for each dataset.
  • What exact operations are included in Table 1's Time, with what inference batch size, precision, and aggregation? Which modules are included in the 0.421B parameter count?
  • Should Figure 5's spatial map be labeled target-to-source self-attention, and are predicted-feature display dimensions taken entirely from the true crop? Do existing records provide variability information for the .33-point gain?
Reviewer_KPUAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to predict question-relevant visual representations from a full image and a question and then provide them to a language model. Q-JEPA formulates visual focusing as target-view prediction in latent space, combining a pretrained V-JEPA predictor, Global/Focus cross-attention, and a three-stage training strategy. It reports accuracy improvements on GQA and the processed PhysObjects protocol and extends the approach to the four-frame STAR Interaction T1 setting. Page references below refer to PDF page numbers, which coincide with the printed page numbers.

Critical strengths

S1. Formulating the acquisition of question-relevant visual evidence as target-view representation prediction is a practically meaningful research direction. The interface uses full-image context to produce focused representations, with evidence crops serving as training supervision and no additional evidence boxes or focus labels required at inference. This provides a concrete approach to selecting visual information for multimodal question answering (pp. 4–6; Appendix B.2, p. 14). S2. The paper reports empirically meaningful accuracy gains. In Table 1 on p. 8, Q-JEPA achieves 64.10% on GQA and 79.03% under the PhysObjects protocol used in the paper, improving over ViCrop, the best-performing competing method in that table, by 3.10 and 12.79 percentage points, respectively. Page 9 also reports that, in the four-frame STAR-Raw setting without additional temporal annotations, accuracy increases from 67.96% for Qwen2.5-VL-7B to 75.25%. These results support the practical value of the interface under the evaluated protocols. S3. The ablation studies provide evidence for several design choices within the tested configurations. Section 4.3 and Table 2 on pp. 8–9 show that, under the corresponding control settings, pretrained predictor initialization, FCA, and Stage 1 improve GQA accuracy by 1.59, 1.55, and 1.35 percentage points, respectively. These results support the roles of pretraining transfer, question conditioning, and explicitly guided warmup in the evaluated configurations.

Critical weaknesses

w1. Insufficient reporting of baseline task adaptation and evaluation configurations makes the comparison conditions underlying the accuracy gains difficult to verify. Section 4.2 on p. 7 states that publicly released code and checkpoints were used where available and discloses that the authors implemented the Qwen2.5-VL-7B version of AttWarp. The same page also makes clear that Q-JEPA underwent second- and third-stage training on GQA and PhysObjects. However, the paper does not provide per-method checkpoint versions, task-adaptation and model-selection rules, or the actual image-input and scoring-pipeline configurations. In addition, Appendix B.3 on p. 15 gives the PhysObjects split sizes but does not specify whether the splits were made by image, object, or question, or whether there is overlap across splits. These gaps make it difficult for me to assess the conditions under which the reported accuracy gains were obtained and the generalization conclusions they support. I need this information to assess the scope of the performance advantage; missing reporting alone does not establish an unfair comparison or data leakage. w2. The construction of evidence targets and the way target objects are specified in full-scene inputs remain unclear, limiting my understanding of the training supervision and task scope. Appendix B.2 on p. 14 states that GQA evidence crops are derived from training annotations and used only as supervision, but does not explain how evidence regions are determined for questions involving multiple objects or relations, or how multiple candidate regions are handled. Appendix B.3 on p. 15 specifies that PhysObjects uses 868 questions over 417 full-scene images, but does not show how the input identifies the target object whose physical properties are to be assessed. These details determine what target-view representations the model learns and how physical-property predictions correspond to particular objects, directly affecting the assessment of question-relevant representation learning, the supervision design, and the accuracy results. The Flickr30k construction procedure, the handling of missing focus labels, and the absence of evidence boxes at inference have been described, but the downstream construction and input rules above are still insufficiently specified for reproduction. w3. The processing boundaries and execution configurations of the efficiency measurements are insufficiently clear, leaving the scope of the latency gains to be verified. Table 1 on p. 8 reports a latency of 0.133 seconds for Q-JEPA, a 22.2% reduction from 0.171 seconds for Qwen2.5-VL-7B, and 0.421 B visual-module parameters, a 37.8% reduction from 0.677 B. These calculations are correct, and the table caption explicitly excludes the downstream language model from the parameter counts. Although Section 4.1 on p. 6 gives the hardware, input resolution, and batch size for the overall experiments, it remains unclear which task was timed in Table 1, whether those settings were retained, whether execution used a single GPU or multiple GPUs, and what timing boundaries, numerical precision, caching, and repeated-measurement procedures were used. In particular, the paper does not specify whether text encoding and answer generation are included; the components making up the 0.421 B visual-module parameters also require further clarification. I therefore cannot confirm that the timing scope is consistent across methods or directly interpret the reported numbers as end-to-end latency gains. The claims of lower inference latency and fewer visual-module parameters require clear measurement boundaries and component-accounting conventions; the available information is insufficient to determine whether any costs were actually omitted.

Questions for authors

Q1 (w1). Could you provide the code and checkpoint versions, task-adaptation and model-selection rules, actual visual-input configurations, and implementation of the common scoring rules for each method in Table 1? What is the splitting unit for PhysObjects, and do the sets overlap at the image or object level? These details would clarify the comparison conditions and scope of generalization. If there are substantive configuration differences, please report the results of the corresponding controlled comparisons. Q2 (w2). What rules are used to select, merge, or expand evidence targets for GQA questions involving multiple objects or relations? Could you provide representative construction examples and the corresponding rules or code? For PhysObjects, could you show examples of the actual image and text inputs supplied to the model, explaining how the target object is identified in a full scene and whether all compared methods receive equivalent information identifying that object? Q3 (w3). Which task is timed in Table 1, what are the input and output sizes, and which processing stages are included? Please provide the timing boundaries, hardware and execution configurations, caching and warm-up strategies, repeated-measurement results, and a breakdown of the 0.421 B visual-module parameters. If the timing scopes differ across methods, please provide results under common processing boundaries to determine whether the reported latency advantage still holds.

Reviewer_ipxMOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Q-JEPA, which provides MLLMs with question-relevant visual features by framing the transition from a full image to a question-relevant region as a “zoom-in” prediction, analogous to predicting a future view in video. Built on V-JEPA 2.1, it keeps the encoder frozen and fine-tunes the pretrained predictor (the Question-Guided Predictor) to predict the encoded features of a question-relevant crop from the full-image features. The predictor uses two cross-attention modules: global cross-attention over the whole question, and focus cross-attention with a learned FocusGate that emphasizes object- and relation-bearing tokens. Training proceeds in three stages: explicit focus-phrase warmup on Flickr30k Entities, question-conditioned training with FocusGate supervision on GQA and PhysObjects, and alignment to the frozen language model of Qwen2.5-VL-7B through a projector. The method reports 64.10% on GQA and 79.03% on PhysObjects (vs. 60.38 / 59.45 for Qwen2.5-VL-7B), with fewer visual parameters and 22.2% lower visual inference latency. It also includes an extension to a subset of STAR for video QA.

Critical strengths
  • Reusing a video-pretrained JEPA predictor to produce question-conditioned visual features is an interesting idea. It avoids external tool calls, multiple MLLM forward passes and attention-map extraction, and the ablation shows that initializing from the pretrained predictor helps.

  • The resulting interface is lightweight. With fewer visual parameters (0.421B vs. 0.677B) and lower latency than both the base model and the region-focusing baselines, it could be practically useful if the accuracy gains hold under fair comparison.

Critical weaknesses
  • The main results lack a controlled comparison. Q-JEPA is trained on the GQA and PhysObjects training sets, and on PhysObjects the training and evaluation data come from the same prepared protocol. The baselines are evaluated with released checkpoints. Whether or not Qwen2.5-VL has seen GQA, it is unclear how much of the gain (especially +19.6 on PhysObjects) comes from the proposed design rather than from targeted training. Two controls are needed, both trained on the same data: the same frozen LLM with Qwen2.5-VL’s own visual encoder and projector, and V-JEPA full-image features with a projector but no predictor. The latter is necessary to show that predicting a target view helps at all.

  • The “zoom-in prediction in video” framing is not directly validated. Initializing from the pretrained V-JEPA predictor helps (+1.59 on GQA), but this does not show that the video or zoom aspect matters: the gain could come from generic feature-prediction priors. Appendix A also states that the second temporal position is only a readout slot. A comparison with an image-pretrained predictor (e.g., I-JEPA), or with target queries placed at the same temporal position, would clarify this.

Questions for authors
  • Were any baselines fine-tuned on the GQA / PhysObjects training data, and how were they evaluated on PhysObjects (e.g., was the same log-likelihood scoring over fixed candidates applied to reasoning models such as Thyme, which scores below the base model)? Could the authors report the two controlled baselines described above? This is the main factor in my assessment.

  • Does the LLM receive only the predicted target-view tokens, or also the full-image tokens? If only the former, could the authors report GQA accuracy by question type, especially for questions that require global context?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.