- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper asks whether we can evaluate how well a world model predicts the effects of different actions without using a simulator. The authors use simulators as ground truth and test several existing metrics on models from robotics, control tasks, and Atari. They also create corrupted models by reversing or rotating their predicted action effects. They show that several metrics fail to detect these errors, and action retrieval sometimes even prefers the corrupted models. They propose a new metric, the matched probe, which compares similar recorded states with different actions. It agrees better with the simulator in many settings. They also use it as a training loss, with mixed results.
Critical strengths
I like the idea of testing whether evaluation metrics actually measure what they claim. The corrupted models are a simple but convincing experiment, and the theoretical results support the findings. The evaluation is broad, including models released by other groups, and the direct comparison with simulator ground truth is particularly useful. The paper is well written and clear about its limitations.
Critical weaknesses
- The matched probe relies on finding similar states with different actions. Its accuracy drops as states become less similar, and it is unclear how well it works with less diverse datasets. Atari uses a random policy, which makes matching easier.
- The inverse model direction check performs similarly on LIBERO and TD-MPC2 and can also evaluate actions outside the dataset. The advantage of the matched probe is mainly on Atari, where the inverse model performs poorly.
- The corrupted model experiments are convincing, but the results are somewhat expected by construction. On original models, the matched probe is not significantly better than retrieval overall. Its clearest advantage comes from a smaller LIBERO comparison.
- The connection to planning is still weak. No probe significantly predicts planning performance among the original models, and improving the matched probe during training does not improve control.
Questions for authors
- How fast does the probe degrade as action diversity decreases?
- With a strong inverse dynamics model available, is there a reason to use the matched probe?
- Can it predict planning performance among original models on harder tasks?
- Could the lack of control gains from the loss reflect planning tasks that are too easy?
4 comments
Response to reviewer 4Wkj (2/2)
W4 (Regarding the connection to planning)
Errors of direction are what break planning and the existing probes miss them. In Table 1 a reflected response makes model predictive control miss a goal 4 cm away by 32.0 cm and a rotated one by 24.9 cm, while the trained models miss by 0.7 cm. Retrieval rates both as more faithful than the trained models. On a new harder task the matched probe predicts planning success across 90 original models and retrieval does not (please see Q3). We agree that matched swap alignment does not improve control (Q4).
Q1 (how fast the probe degrades as action diversity decreases)
Slowly. We sorted the same 1800 pairs by how much the two 8 step action sequences differ (in normalized units) and cut them into 5 groups of 360.
Table R2. Matched probe on our proprioceptive LIBERO models by action difference between the two segments of a pair. "Estimate" is the mean cosine between a pair's logged estimate and what the simulator says. The last two columns are the Spearman correlation between the matched probe and the simulator's direction, both computed on that group only. Higher is better in the last three columns.
| group | median action difference | estimate | 30 models | 92 paper models |
|---|---|---|---|---|
| 1 (most similar actions) | 1.9 | 0.70 | 0.76 | 0.97 |
| 2 | 3.2 | 0.74 | 0.89 | 0.99 |
| 3 | 4.4 | 0.83 | 0.97 | 0.99 |
| 4 | 5.8 | 0.82 | 0.88 | 0.98 |
| 5 (most different actions) | 8.2 | 0.83 | 0.69 | 0.97 |
With the most similar actions the estimate is less accurate (0.70 against 0.83) but the probe still ranks the 30 models at 0.76 and the 92 models at 0.97. The 30 model score moves between 0.69 and 0.97 with no downward trend as the actions get more similar.
Q2 (is there a reason to use the matched probe when a strong inverse model is available?)
For actions in the data and with an accurate inverse model there is little reason. On our proprioceptive LIBERO models the two probes are about equal. The reason to prefer the matched probe is that a decent fit of the inverse model does not guarantee that the inverse check works. On the image feature models the fit is R² 0.67 but the check is at 0.09, 0.02 and 0.07 (W2). The matched probe needs no inverse model.
Q3 (planning performance among original models on harder tasks)
New test: we ran model predictive control with 90 of our LIBERO models as trained (no transforms) on a harder task. The goal is 10 cm away instead of 4, an episode has 40 steps and the planner executes 4 actions before it replans (in the paper it replans after every action). Every model starts from the same 24 states (2 in each of 12 tasks) which gives 2160 episodes.
Table R3. Spearman correlation across models between each measure and the success rate on the harder task, with 95% intervals. Success means the gripper ends within 2 cm of the goal. Higher is better. "90 models" are the 30 of Table 9 plus the 60 models of our training study. "55 models" leave out the objectives that lower D_logged by more than 0.1 (retrieval's contrastive loss, inverse dynamics at weight 1, shuffled targets).
| measure | 90 models | 55 models |
|---|---|---|
| true direction on logged swaps (D_logged, needs the simulator) | 0.83 [0.57, 0.90] | 0.34 [−0.03, 0.55] |
| matched probe (ours) | 0.76 [0.44, 0.88] | 0.07 [−0.32, 0.38] |
| validation error (lower error counted as better) | 0.90 [0.70, 0.94] | 0.64 [0.12, 0.87] |
| retrieval | −0.37 [−0.78, 0.17] | −0.06 [−0.61, 0.47] |
Across the 90 models the matched probe predicts success and retrieval does not. Among the 55 models even the simulator's true direction does not predict success significantly, so a direction probe cannot be expected to. There validation error predicts best. We therefore recommend reporting validation error together with a direction probe. Validation error cannot see a response that is reflected around the logged action (Proposition 4) but a direction probe can.
Q4 (could the lack of control gains reflect planning tasks that are too easy?)
We tested this with the harder task of Q3 and it did not show a clear gain from matched swap alignment either. At the weight we fixed in advance (0.01) neither task shows a significant gain. At weight 0.001 the success rate on the harder task goes from 0.68 to 0.78 (uncorrected p = 0.007) but the final distance to the goal does not change, and the gain does not survive a Holm correction over our 12 control comparisons of matched swap alignment. So we do not claim a control benefit.
We hope this addresses your concerns.
Thank you so much for the work you have done.
The analysis on degradation answer my question about the matching. Also the image feature results answer my questions about the inverse check. The comparison on model is much broader within LIBERO.
My concerns have been addressed, I increased the soundness score and kept my positive review score.
Thank You
Dear Reviewer 4Wkj,
Thank you for your supportive review and for raising the soundness score. We are glad our responses addressed your concerns.
Best regards, The authors
Response to reviewer 4Wkj (1/2)
We thank the reviewer for the supportive review and the specific questions. We ran new experiments, including 152 new LIBERO models trained on image features and a released robot video world model (A2World).
Note: For each model, the probe produces one score while the simulator gives its true direction on the logged swaps (D_logged in the paper). Unless stated otherwise the numbers below are the Spearman correlation between the two across models. A value of 1 means the probe and the simulator put the models in the same order. Intervals are 95% bootstrap intervals over training objectives.
W1 (Regarding similar states and less diverse data)
The matched probe degrades slowly when the pairs get worse. We ran new checks on our proprioceptive LIBERO models ("30 models" are the 6 objectives x 5 seeds of Table 9).
Less similar states: we cut the 1800 pairs that the paper also ran in the simulator (the 5 nearest partners of each of the 360 positions) into 5 groups by the distance between the two start states. The estimate's cosine with the simulator drops from 0.90 in the closest group to 0.70 in the farthest. But computed on the farthest group alone the matched probe still ranks the 30 models at 0.84 and the 92 paper models at 0.98.
Less diverse data: with only 8, 4 or 2 other demos per task to match against (normally about 15) it ranks the 30 models at 0.89, 0.85 and 0.80. Retrieval gets 0.14 with all the data.
These are LIBERO's human teleoperated demos (50 per task) so the results do not depend on a random logging policy as in Atari. For less diverse actions please see our response to Q1.
W2 (Regarding the inverse check)
Both direction probes are ours. On our proprioceptive LIBERO models they are about equal (0.89 and 0.92, Table 9) and for actions outside the data we recommend the inverse check. New test: at the 360 held out positions we ran the simulator under 3 translation-only action sequences each, sampled the way our planner samples its candidates. Computed on these sequences the inverse check predicts the true direction at 0.68 across 65 of our LIBERO models (the 30 of Table 9 plus the 35 models of the Table 13 training study that use no contrastive loss). Retrieval gets 0.38.
Regarding the advantage of the matched probe mainly on Atari, we trained new LIBERO models on image features. They use the same transformer as in the paper but the state is 1536 DINOv2 ViT-S/14 features of LIBERO's two cameras (we rendered all demos and all simulator branches ourselves). There the inverse model has a held-out R² of 0.67 but the inverse check only gets 0.09, 0.02 and 0.07 at the three model sizes. The matched probe gets 0.90, 0.87 and 0.86 (Table R1).
W3 (Regarding the transform test and the advantage on original models)
The true direction under each transform is known by construction and that is what makes it a test. Whether a probe follows it is not known in advance and the existing probes do not. Retrieval even rates the reflected and rotated responses as more faithful than the originals on our LIBERO models, the released TD-MPC2 models and the released Atari models (Tables 1, 4 and 6). We found the same on A2World [1], a released robot video world model with 2.5B parameters. At 90 held out positions retrieval ranks the logged action chunk first at all 90 for the reflected copy and for the rotated copy (69 of 90 for the released model).
[1] Learning Transferable Dynamics Priors from Action to World Modeling, ECCV 2026
We agree that the transform test is necessary but not sufficient (Section 4 of the paper). So we compared the probes on many more original models, including the image feature models at three sizes.
Table R1. Spearman correlation with D_logged on models as trained (no transformed copies). Higher is better. "30" are the 6 objectives x 5 seeds of Table 9. "90" add the 60 models of our training study (Table 13 plus three new contrastive settings) and "65" are the 90 without the 25 contrastive models. The last column is matched probe minus retrieval with its 95% interval.
The matched probe leads retrieval significantly in all seven sets (the first row is Table 9 of the paper), so the advantage on original models is not limited to one small comparison. On the released TD-MPC2 and Atari models as trained the two probes remain tied as the paper reports.