- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Q-JEPA predicts question-relevant target-view features from a full-image representation and a question, then passes the predicted features through a projector to a frozen language model. It adapts a video-pretrained predictor using global and focus cross-attention, token-routing gates, and a three-stage training procedure. The paper reports 64.10% accuracy on GQA test-dev, 79.03% on a constructed five-property, full-scene PhysObjects protocol, and video results on a four-frame STAR Interaction T1 subset. It also reports reductions in visual-module parameters and image inference time. These results make the interface promising, although supervision, task adaptation, and measurement conventions still limit attribution of the gains.
Critical strengths
- The design addresses a clear interface problem: different questions about the same image require different visual evidence. Performing this transformation before language generation and providing a fixed 144-token visual interface gives the method a coherent modular structure. The appendix's clarification that the target position is a virtual readout slot also helps explain what target-view prediction means.
- The ablations provide meaningful component evidence. Under the corresponding configurations, predictor pretraining improves accuracy by 1.59 percentage points, focus cross-attention adds 1.55 points, and the auxiliary focus loss adds .33 points. The pretrained-versus-random comparison is particularly useful because it supports a specific design choice more directly than comparisons between heterogeneous complete systems.
- The reported QA improvements are practically interesting. Relative to Qwen2.5-VL-7B, accuracy increases by 3.72 points on GQA and 19.58 points on the constructed PhysObjects protocol. The video subset also shows gains under both sampling strategies. These findings extend beyond the selected qualitative examples.
- The appendices clarify several important experimental distinctions: region pairs versus unique images, the five-property PhysObjects scope, the STAR subset and video-disjoint roles, and the use of ground-truth crop dimensions when displaying predicted features. These details make the scope of the results easier to assess.
Critical weaknesses
- The main comparisons do not adequately separate architecture from additional training. Q-JEPA receives Flickr region-pair training and downstream supervision, but the corresponding data, region/token annotations, adaptation budgets, and input configurations for competitors are not clearly specified. The Stage 1 ablation also adds data and training, so it does not isolate the benefit of curriculum order. This is not evidence that the comparison is necessarily unfair, but it currently favors a system-level interpretation of the gains.
- Several training definitions are important yet incomplete. The paper should clarify how target regions are selected for multi-entity and relational questions, how PhysObjects and STAR targets are constructed, and whether the original pretrained predictor is updated. Feature-loss reduction, stage lengths, and the relationship between task-specific training and checkpoints are also unclear. These choices affect target semantics, effective loss weights, and ablation budgets.
- The efficiency measurements need a more precise scope. The reported 22.2% time reduction and 37.8% visual-parameter reduction are arithmetically correct, but timing boundaries, inference batch size and precision, inclusion of text encoding and language-model or candidate scoring, and counted modules are not fully defined. Since the visual backbone and token interface also change, the entire speed difference cannot be attributed to predictive focusing.
- Some mechanistic descriptions exceed the available visualization and ablation evidence. The .33-point gain lacks run-to-run variability, and a few successful images do not establish general localization accuracy. Figure 5 calls its spatial map cross-attention, whereas the appendix defines target-to-source self-attention; predicted-feature displays also obtain their dimensions from the ground-truth crop. These descriptions should distinguish feature prediction, text routing, and box localization more clearly.
Questions for authors
- What downstream data, region/token supervision, input resolutions, and visual-token settings did each Table 1 method actually use? What stage data and training budgets correspond to the Table 2 configurations?
- Which parts of the original predictor, added attention modules, FocusGate, text projection, and final projector are updated at each stage? Please also clarify feature-loss reduction, task/checkpoint relationships, and the existing evidence-region construction rules for each dataset.
- What exact operations are included in Table 1's Time, with what inference batch size, precision, and aggregation? Which modules are included in the 0.421B parameter count?
- Should Figure 5's spatial map be labeled target-to-source self-attention, and are predicted-feature display dimensions taken entirely from the true crop? Do existing records provide variability information for the .33-point gain?