acceptodds
Withdrawn from ICLR 2027

PRISM: Probing Physical Consequence Signals In World Model Predictions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Action-conditioned world models predict the consequence of each candidate action, yet standard planners reduce each prediction to a single feature distance to a goal image. This scalar readout couples task progress with obstacle safety in one number, and the progress signal dominates: consequence information that the model faithfully predicts is lost in the aggregation. We expose this readout bottleneck through a controlled comparison. A frozen text-image projection, aligned once on robot data without collision labels, reads named object relations from the same predictions. Where feature-distance planning fails on obstacle-dense pick-and-place sequences, this relational readout achieves 71.7%, revealing that the learned representations already encode physical consequences. Three isolating interventions confirm the information is per-candidate and prediction-dependent: shuffling action-score pairings eliminates every gain, reading from current observations instead of predictions yields only partial benefit, and goal-image contrast rules out directional matching. This is a representation-level finding: the consequence information exists regardless of how downstream planning uses it. PRISM exploits this by decoupling direction from selection: geometry handles progress toward the goal, and two sequential consequence gates read VLM-written relations from each candidate’s predicted scene, filtering harmful candidates before geometry picks among survivors. On 450 transport trials, consequence gating raises completion from 62.2% to 86.7% and cuts obstacle displacement by 63%, lifting safe completion from 33.3% to 55.6%. The mechanism transfers to pick-and-place sequences and to a real Franka arm without retraining

Primary area:
applications to robotics, autonomy, planning
Keywords:
JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oLf2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper has a good structure. PRISM investigated whether action-conditional world model predictions include information about physical consequences that planning processes fail to utilize based on the distances between features in target image. This method separates geometric progress from the acceptability of candidate actions. RGB-D anchor points provide geometric scoring, while a projected module trained to align V-JEPA features with SigLIP embeddings and remain frozen during planning scores predictions based on the expected and unexpected object relationships generated by the VLM. However, some ideas remain limited novelty besides (i) extending the “readout bottleneck” argument from reachability, as studied in prior work, to side effects on surrounding objects, and (ii) a relative gating scheme in which a frozen projection, trained without collision labels, scores candidates against VLM-generated negative relations to remove harmful ones before geometry selects among the survivors, which avoids calibrating geometric and semantic scores on a common scale.

Critical strengths

The results show that compared with the matched geometric baseline, there are substantial improvements in both the completion rate and perturbation aspects. Table 1 provides substantive evidence for the usefulness of the combined method within the tested transport setting. The simultaneous improvement in completion and disturbance is more informative than task success alone and provides an acceptance-relevant contribution even if the mechanistic explanation needs revision. Also, joint score shuffling provides genuine evidence that action–score alignment matters.

Critical weaknesses

The goal-image comparison does not isolate a readout-only explanation of the performance gap. The main evidence was the pick-and-place comparison: the target image planning was successful 0/180 and PRISM was successful 129/180. This comparison appears in the abstract, Figure 1, the first contribution, and Section 4.2. Figure 1 describes the difference between the two as "purely in how information is read". However, the two methods differ in more than the readout. PRISM’s final action is selected by an RGB-D geometric progress term G, which goal-image planning lacks. The G500 baseline illustrates the weight of this item: it has already reached 115/180 using only geometric selection and no relational gating. Therefore, among the 129 successes of PRISM, only the additional 14 could be attributed to the relationship readout, approximately 7.8 percentage points. The remaining differences come from geometric guidance rather than the way they are read out. Furthermore, 0/180 itself contradicts the explanation in the paper. The 180 trials included scenarios with direct paths without obstacles (Sec 4.4), and the target image planning was also zero in this case. When there are no obstacles, there are no consequences to be drowned out, so it is more likely that this baseline cannot achieve the basic goal of reaching the target under the current settings. The paper also does not explain how the target image is constructed. Hence, this comparison does not establish the readout bottleneck. If the authors cannot clarify this, the score may be maintained.

Questions for authors

Q1. Can the readout comparison be made without simultaneously changing geometric guidance? Please elaborate how goal images are constructed and whether the goal-image policy uses the same anchors, motion eligibility, and geometric selection available to PRISM. Q2. What varies independently across the ten repetitions, and how stable are the paired effects across layouts and seeds?

Reviewer_CtWPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates whether goal-image feature distance obscures safety-relevant information in action-conditioned world-model predictions. The central hypothesis is that progress toward the goal dominates this scalar readout, making local physical consequences, such as disturbing nearby objects, difficult to recover. PRISM addresses this by using a VLM to generate desired and undesired spatial relations. For each candidate action, V-JEPA predicts a future representation, which a frozen projection maps into SigLIP space to compute contrastive relation scores. Two sequential gates filter candidates based on these scores; an RGB-D geometric progress measure then selects the action to execute. Experiments in LIBERO report improvements in task completion and reduced protected-asset disturbance over geometric selection on transport tasks. Additional pick-and-place experiments report zero success for a goal-image planning baseline, compared with 71.7% for PRISM. The paper also presents preliminary results on a Franka robot.

Critical strengths
  1. A useful distinction between goal progress and physical consequences. The paper identifies a real limitation of goal-image planning: a single feature-distance objective need not reflect local physical consequences, particularly when progress toward the goal dominates the score. Separating consequence assessment from geometric progress is a worthwhile direction, and the motivating hypothesis is compelling.

  2. Task-conditioned generation of relational safety criteria. Using a VLM to formulate desired and undesired relations from the current scene and task is an appealing design choice. It avoids writing a separate semantic cost for every scene, and the projection head is trained without collision-specific labels. The relative gating formulation is also a reasonable way to avoid combining geometric distances and semantic similarities with incompatible scales.

Critical weaknesses
  1. Insufficient probing of the proposed bottleneck. The paper positions itself as a probing study and declines a system-level comparison on that basis, yet its evidence is almost entirely downstream planning outcomes, which measure how the whole pipeline performs rather than characterizing the mechanism. The ablations suggest the scores are useful, but not what information they capture, when goal-distance readout fails, or under which conditions the approach repairs it. Section 4.2 asserts that the consequence signal occupies a low-variance subspace invisible to feature distance, while no variance or subspace analysis appears anywhere. I would expect diagnostic measurements: the relationship between predicted scores and realized physical outcomes across candidates from a fixed state, reported alongside the same relationship for goal-image distance and a simple clearance measure. As it stands, the experiments demonstrate practical utility but offer limited insight into the mechanism the paper sets out to investigate.

  2. Limited methodological justification and reliance on heuristics. The proposed approach largely consists of several heuristic design choices whose individual contributions and underlying motivations are not sufficiently examined: geometric anchors lifted from depth, the assumption that the held object displaces exactly as the end-effector is commanded, grasp transitions and motion eligibility triggered on fixed distances, and two sequential gates defined relative to the candidate set. The paper says little about why this formulation was adopted or which components are essential, and while the ablations separate the gates from each other and from geometry, they neither isolate the geometric construction nor test simpler alternatives such as a conservatism rule on action magnitude or a clearance-based filter. Both are natural comparisons, given that the paper's own diagnostic shows the method preferring a shorter advance than geometric maximization. What is missing is a justification linking these particular heuristics to the bottleneck the paper is about.

Questions for authors
  1. How well do the relation margins predict realized physical outcomes when candidates are executed independently from identical simulator states, compared with goal-image distance and a simple clearance measure? (It would also help to see representative relation statements and the correlation between the two margins.)

  2. How are grasp transitions, terminal conditions, and geometric information handled in the goal-image baseline, and what explains its complete failure?

  3. What is the independent unit of evaluation in the simulation trials, and how is variability across layouts and seeds reflected in the reported intervals? For the real robot, what are the trial counts, success rates, per-decision latency, and achieved control frequency?

Reviewer_CxuwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether action-conditioned world models contain information useful for safe manipulation that goal-image distance fails to make full use of. PRISM separates progress toward the goal from action filtering: it uses RGB-D geometry to measure progress, maps V-JEPA 2's predicted representations for candidate actions into the SigLIP space, and filters actions through two successive gates based on positive and negative relation descriptions generated by a VLM. The projection head is trained to align features on robot data without collision labels and stays frozen during planning. Across 450 held-object transport trials, PRISM improves completion from 62.2% to 86.7% and safe completion from 33.3% to 55.6% over the geometric baseline, while reducing the mean maximum displacement of protected objects from 52.9 mm to 19.7 mm. On full pick-and-place tasks, success improves from 63.9% to 71.7%. The paper also presents demonstrations on a real Franka arm, using a separate projection head trained on DROID data. The authors interpret these results as evidence of a bottleneck in reading physical consequence information from world models. What the experiments directly establish is that relation-based filtering improves planning on the tested tasks; the proposed explanation still needs to be separated from other possible sources of improvement.

Critical strengths
  1. The method meaningfully improves both task completion and protection of nearby objects. In Table 1, PRISM improves completion over G500 by 24.5 percentage points and safe completion by 22.3 points, while reducing displacement by about 63%. Taken together, these metrics show that the gains do not simply come from taking fewer actions or sacrificing task completion for safety. This is a meaningful empirical result for the specific problem of safe transport and an important reason to consider acceptance.

  2. The paper offers a simple relation-based filtering method that needs no collision labels, with ablations that examine the role of its components. Sections 3.3–3.4 and Appendix A clearly specify the projection training objective and gating rules. Table 1 shows that the full method outperforms either gate alone, while shuffling the action-score correspondence substantially hurts performance. This supports the practical value of candidate-specific relation scores and makes the approach promising for use and further study in other planning systems.

Critical weaknesses
  1. The current controls do not sufficiently isolate the paper's central claim of a “physical consequence readout bottleneck.” This is a concrete issue with the interpretation of the experiments, with two related aspects. First, the current-observation prompting control changes only the text descriptions and still uses the same action-conditioned future predictions. The increase from 71.1% to 86.7% therefore shows that prompt content matters, but does not isolate the contribution of forward prediction as claimed in Section 4.2, lines 402–408. Similarly, score shuffling shows that the action-score correspondence matters, but does not distinguish physical consequence information from information about relative position after an action. Second, the goal-image control selects actions directly by their predicted feature distance to a goal image, whereas PRISM selects by the geometric score G after filtering. This comparison therefore changes more than the readout. G500 already reaches 63.9% in Table 2, which further shows why the entire gap from 0% to 71.7% cannot be attributed to consequence readout. These issues directly affect the paper's main contribution as a finding about representations, though they do not invalidate the performance improvements in Table 1.
Questions for authors
  1. Is there evidence that separates “predicting the physical effects on obstacles” from “recognizing favorable action directions or relative positions”? For example, among candidate actions that make similar geometric progress, do the relation scores still predict the object displacement or tilting that actually occurs? A targeted control that preserves static spatial information while removing predictions of contact consequences could also help. Evidence of this distinction would substantially strengthen the central claim about the representations. Without it, I would favor narrowing the conclusion to the effectiveness of relation-guided action filtering.

  2. What specifically causes the goal-image baseline's 0/180 result, and how is the difference in geometric guidance accounted for? Please explain how the goal images are constructed and distinguish failures to reach the goal in obstacle-free scenes from collision failures in scenes with obstacles. If goal-image scoring still performs substantially worse than relation scoring when both retain the same geometric progress mechanism, that would more directly support the readout bottleneck. If the baseline mainly fails at basic goal reaching, this result would not establish that it loses safety-related information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.