acceptodds
Under review as a conference paper at ICLR 2027

Q-PhysJEPA: Query-Conditioned Physical Abstraction for JEPA World Models

Abstract

Embodied world models should preserve the physical evidence needed for the task at hand, yet accurate latent prediction and physical-property decoding do not by themselves establish decision relevance. We introduce Q-PhysJEPA, a query-conditioned joint-embedding predictive architecture that incorporates task queries before low-dimensional history compression, rather than only during candidate evaluation. The model aggregates visual, proprioceptive, and action histories into shared evidence slots, reweights these slots through query-conditioned gating, and compresses them into a task-dependent representation. It then predicts future observation representations for candidate control settings and estimates their utilities from the predicted futures. Future prediction, utility regression, and candidate ranking are jointly trained using observed executions. We examine query-relative physical content through property readouts, candidate-future predictions, and utility estimates under matched query switches and physical interventions. On ManiSkill manipulation tests with physical parameters outside the training ranges, Q-PhysJEPA achieves 56.29% top-1 candidate-selection accuracy, averaged equally across four task objectives, compared with 46.83% for a query-independent representation baseline, an improvement of 9.46 percentage points. Controlled analyses further suggest that greater query-dependent physical selectivity does not necessarily translate into better decisions. These findings support task-conditioned history compression while highlighting the distinction between accessible physical information and its usefulness for candidate selection. Codes are available at https://anonymous.4open.science/r/Q-PhysJEPA-30A7\faGithub.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p9QfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Q-PhysJEPA, which conditions history compression on task queries and combines latent future prediction with utility learning. A direct-readout component study reports improved candidate selection over a query-independent representation on ManiSkill. Separate diagnostics investigate physical-property retention and its relationship to decision quality.

Critical strengths

1.Conditioning compression on the task objective is well motivated. The structured experiments provide evidence: query conditioning reduces regret relative to late fusion at both tested bottleneck widths. 2.The paper distinguishes accessible physical information from its demonstrated use in decisions and transparently reports negative controls and limitations. This is a useful evaluation perspective for physical world models.

Critical weaknesses

1.Appendix A.4 states that the headline results use a direct candidate scorer with utility-projection targets, rather than the future-mediated architecture and observation-embedding prediction objective in Section 3. The structured diagnostics also use separate decision heads. Although explicitly acknowledged, this leaves the central combination of future prediction and decision supervision without direct empirical validation. 2.In Table A3, the query-only baseline has higher reported accuracy than Q-PhysJEPA (DR) on all four OOD objectives. Its average is 58.73%, calculated from the table, compared with 56.29% for Q-PhysJEPA. Moreover, all 192 rapid-completion histories share the same first optimal candidate, while balanced efficient-transport accuracy is nearly identical between the methods. Thus, the improvement over the no-query representation does not demonstrate that the model extracts useful physical evidence from history.

Questions for authors

1.Can you provide a matched evaluation of the complete future-mediated model against the direct-readout variant, establishing whether future prediction contributes to candidate-selection quality? 2.Given Table A3, what evidence supports a decision advantage from history beyond query-and-candidate priors? Does the advantage persist when evaluated on balanced candidate-label distributions?

Reviewer_cZYeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Q-PhysJEPA, a query-conditioned Joint-Embedding Predictive Architecture (JEPA) world model for embodied decision-making (specifically, selecting among a discrete set of N=9N=9 predefined feedback controller parameterizations). The core hypothesis is that task queries should gate interaction history representations before low-dimensional bottleneck compression (via learnable evidence slots and a sigmoid query gate), rather than conditioning only after a query-agnostic representation is formed. Methodologically, Section 3 presents a future-mediated pipeline that predicts phase-aligned latent rollouts via a GRU and evaluates them using a process–terminal utility head trained with latent prediction, utility regression, and pairwise ranking losses. Empirically, the paper evaluates several disconnected tracks: a direct-readout (DR) visual candidate-selection experiment on ManiSkill3 (PushCube-v1 and PickCube-v1), single-seed structured-observation ablations, frozen state/action readouts, and grounding/intervention controls.

Critical strengths
  1. Well-Motivated Conceptual Problem: The distinction between physical property decodability, average latent prediction error, and actual decision relevance/functional use in embodied world models is conceptually sound and well-articulated. Asking how task objectives should shape history compression is a meaningful direction.

  2. Transparency in Negative Results and Diagnostic Controls: The paper is unusually candid in reporting negative findings, failure cases, and diagnostic controls in the appendix (e.g., Query-only baselines, class-balanced evaluations, and temporal/RGB masking interventions), which clearly expose dataset shortcuts and lack of visual grounding.

  3. Insight on Selectivity vs. Capacity: The structured diagnostic showing that higher physical selectivity (suppressing task-irrelevant properties) does not monotonically lead to lower decision regret when bottleneck capacity increases (d=8→16d=8 \to 16) is a useful empirical observation.

Critical weaknesses
  1. Severe Disconnect Between the Proposed Method (Section 3) and the Evaluated Model (Section 4): The main methodological contribution in Section 3 (Figure 2, Sections 3.4–3.5) is a future-mediated JEPA world model where a GRU predicts candidate-conditioned latent rollouts (Y^i\hat{Y}i), supervised by future observation embeddings (Lpred\mathcal{L}{\text{pred}}), and scored strictly through predicted futures (Uη([fi;q])U_\eta([f_i; q])). However, the primary ManiSkill visual experiment (Table 1, reporting 56.29% vs. 46.83%) does not evaluate this architecture at all. Instead, as disclosed in Appendix A.4, it evaluates a Direct-Readout (DR) variant (Dξ([zq;q;ci])D_\xi([z_q; q; c_i])) trained with an auxiliary utility-projection target rather than future-observation prediction Lpred\mathcal{L}_{\text{pred}}. Thus, the core JEPA world model proposed in the paper is never actually validated on the main benchmark.

  2. The Main Empirical Gain (+9.46 pp) Collapses Under Grounding Controls (Shortcut Learning): While Table 1 highlights a +9.46 pp OOD top-1 accuracy gain over the No-query baseline, Appendix B.4 (Table A3) and Appendix C.1 (Table A4) reveal that this gain is entirely an artifact of label prior shortcuts:

  • The improvement comes exclusively from two objectives (Efficient transport and Rapid completion), whereas performance on Precise positioning (-0.35 pp) and Stable holding (-0.87 pp) slightly degrades.

  • A Query-only baseline (which completely ignores the interaction history and only sees the query qq and candidate parameters cic_i) achieves 78.65% on Efficient and 100.00% on Rapid (because candidate index 2 is always optimal across all 192 OOD episodes), outperforming Q-PhysJEPA (DR) (76.22% and 93.23%).

  1. Failure to Establish Visual/Physical Grounding: The paper's central premise is extracting task-relevant physical evidence from interaction histories. Yet, in the RGB-window masking intervention (Section 4.4, Appendix C.2), masking out historical RGB frames while keeping the current observation fixed changes 0 out of 36,864 candidate selections. Temporal all-input masking also fails to meet the preregistered 0.02 threshold. This indicates the visual model is not actually using historical visual dynamics to infer physical parameters for decision-making.

  2. Instead of a unified evaluation, the paper stitches together multiple incompatible tracks (DR visual scorer, single-seed 14D structured state probes, synthetic analytic trajectories, and separate continuous-action models) with different architectures, losses, and seeds.

  3. In the frozen-representation readout benchmark (Appendix B.1, Table A1), the state readout heads for Q-PhysJEPA received extra hyperparameter tuning and up to 4,0004,000 updates, whereas the ACT and ACT-JEPA baselines were kept at their original 500500-update schedules.

Questions for authors
  1. Why was the complete future-mediated architecture (Section 3.4–3.5, with Lpred\mathcal{L}_{\text{pred}} on future observation embeddings and GRU rollout prediction) not evaluated in the main ManiSkill visual experiments (Table 1)? How does the full model perform compared to the DR variant?

  2. Given that Query-only outperforms Q-PhysJEPA (DR) on OOD Efficient and Rapid tasks and that RGB history masking changes 0 out of 36,864 decisions, how can we conclude that Q-PhysJEPA learns meaningful physical abstractions from visual interaction history rather than simply fitting query-candidate priors?

  3. Why were the ACT and ACT-JEPA baselines in Table A1 not evaluated under the same 2,000–4,000 update tuning budget used for Q-PhysJEPA?

Reviewer_vnGhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies task-dependent history representations for embodied decision-making. Given visual, proprioceptive and action histories, a task query, and a set of candidate controllers, the goal is to select the candidate with the highest task utility. Q-PhysJEPA uses the query to reweight evidence slots before compressing them into a low-dimensional representation. The proposed full model then predicts candidate futures and estimates their utilities from the predicted embeddings. Experiments use ManiSkill pushing and picking tasks with four objectives: precise positioning, efficient transport, stable holding and rapid completion. On out-of-distribution physical parameters, a direct-readout variant achieves 56.29% candidate-selection accuracy, compared with 46.83% for query-independent compression. These results test the compression component, rather than the complete future-based scoring model.

Critical strengths
  • The motivation is clear. Using the query before a small bottleneck makes sense.
  • The authors clearly report limitations and negative results.
  • The structured study gives a useful example that higher physical selectivity does not always mean better decisions.
Critical weaknesses
  • The complete proposed model is not validated. The tested DR version directly scores candidates. Its auxiliary target is projected utility, rather than future observations. So the experiments do not show whether the future-based scoring path actually helps (Appendix A.4–A.5).

  • The main improvement does not show an advantage over ignoring history. On the natural OOD test, query-only is higher on all four objectives (Table A3). Its average is about 58.73%, compared with 56.29% for DR. For rapid completion, all 192 histories have the same first best candidate. It is therefore unclear how much the result needs physical history.

  • The masking results do not resolve this concern. With the current observation fixed, RGB-window masking changes no selections in 36,864 records pooled across both methods and ID/OOD tests. These are not independent scenes. This does not prove history is unused, but it does not show that the masked evidence matters for selection.

  • Training robustness needs more explanation. The three paired gains are +22.53, +7.03 and −1.17 percentage points. The bootstrap intervals keep the trained models fixed, so they do not cover retraining variation.

Questions for authors
  • Can the complete model and DR be compared under the same training budget?
  • Can you test cases where different physical histories need different candidates, including query-only and current-observation-only controls?
  • Can you explain the seed differences and match the state-readout tuning budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.