- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Boundary-FidelityBench, a benchmark for evaluating first-crossing boundary fidelity as a surrogate for ranking world models beyond visual resemblance. This was accompanied by FCRG, a learning-based detector that detects boundaries from model's predicted trajectories. The experiments evaluate state-based world models based off this benchmark, and reaches findings such as leveraging boundary fidelity can help with planning within the world model.
Critical strengths
None
Critical weaknesses
- I find the problem motivation quite weak. It is well known that visual fidelity does not correlate to better planning/model-based reinforcement learning performance. But using boundary fidelity as a evaluation metric lacks strong incentives, which in turn yields limited applications. Especially since the authors criticize visual fidelity based methods for evaluating world models, however basing the whole experiment section around state-based world models rather than pixel-space world models is very confusing. This makes my assessment on the paper's significance as well as novelty quite negative.
Questions for authors
- FCRG achieves 80.76% exact-bin accuracy even on simulator results. This makes the measurement error too large to ignore, I wonder if the authors could provide additional experiments supporting the validity of measurements, i.e. being attributed to world model errors rather than FCRG error.
- Could the authors provide justification on why pixel-based world models are absent from experiments? It would also be helpful to provide explanation on how this limited-scope experiment both in terms of methods as well as evaluated environments, can have downstream significance.
4 comments
Response to Reviewer 3Vs3 (Part 2 of 3)
Q1. Measurement validity and readout error
Continued from Part 1.
2.Readout improvement and sensitivity analysis. FCRG is a new evaluation method proposed in this paper, and we thank the reviewer for focusing attention on its measurement accuracy. Given the 80.76% Exact-bin accuracy on simulator reference responses, we tested whether an improved readout could achieve higher accuracy. These supplementary results suggest that the method has room for further development, and we intend to continue improving its accuracy. We retained the original evaluator-development data splits and tested counterfactual supervision based on matched continuation controls. On BouncingBall, Earlier/Later Exact-bin accuracy increased from 76.38%/71.35% to 86.98%/90.55%. Across the other six physical-event systems, macro-average Earlier/Later Exact-bin accuracy increased from 53.63%/57.92% to 63.18%/66.36%, respectively. Within each tested calibration strategy, the point-estimate ranking of seven-system macro-average metrics also remained unchanged. These results indicate that model comparisons have some robustness across the tested readout variants.
In addition, readout correction improved the validity of FCRG scores as measures of model performance. In the known-generator experiments, the corrected scores showed strong overall agreement with true rankings (mean Spearman's rank correlation = 0.944, compared with 0.923 for the matched control). Among comparisons for which both rankings were strictly ordered, the reversal rate decreased from 5.05% to 3.75%. For the more challenging Earlier/Later generators, 73.33% of strictly ordered comparisons retained the correct direction, and the reversal rate decreased from 34.88% to 26.67%. When comparisons were further limited to a true-score difference of at least one standard deviation of the system's Earlier/Later true scores, the observed reversal rate declined from 20.83% to 13.04%.
Response to Reviewer 3Vs3 (Part 3 of 3)
Q2. Visual world models and downstream relevance
Thank you for highlighting the importance of evaluating visual world models and downstream applicability. To examine generalization, we extended FCRG evaluation to multiple predictors on PushT and MuJoCo/OGBench Cube. The results support extending FCRG to more complex interactions and visual settings.
We evaluated FCRG on publicly available LeWM, DINO-WM, and JEPA-WM checkpoints for PushT, and on LeWM and GA-WM predictors for MuJoCo/OGBench Cube. In the Cube contact tests, Exact-bin accuracy on encoded ground-truth/predicted responses was 74.83%/49.66% for LeWM and 78.23%/44.22% for GA-WM. In the visual pushing tests on PushT, the corresponding Exact-bin accuracies for LeWM, DINO-WM, and JEPA-WM were 26.76%/23.94%, 39.44%/38.03%, and 52.11%/40.85%, respectively. These results show that the protocol can be applied to three-dimensional contact settings and visual models, and that a shared frozen readout reveals reproducible diagnostic differences between predicted and ground-truth responses. We also examined FCRG's validity as an indicator of model performance by applying it to three model types on PushT and using planning success as a criterion. Exact-bin accuracy showed a moderate positive correlation with planning success (Pearson r = 0.500; Spearman ρ = 0.4352). This result supports using FCRG metrics to evaluate model performance.
Table. Original AP and Exact-bin metrics for Cube and PushT.
| Setting | Model | GT AP | Pred. AP | GT Exact | Pred. Exact | ΔExact 95% CI (pp) |
|---|---|---|---|---|---|---|
| Cube | LeWM | .9822 | .7502 | .7483 | .4966 | [−35.42, −14.96] |
| Cube | GA-WM | .9832 | .8382 | .7823 | .4422 | [−45.83, −21.91] |
| PushT | LeWM | .9047 | .8924 | .2676 | .2394 | [−8.77, 2.63] |
| PushT | DINO-WM | .8358 | .8530 | .3944 | .3803 | [−11.77, 8.54] |
| PushT | JEPA-WM | .9205 | .9147 | .5211 | .4085 | [−21.43, −1.33] |
We agree that the benefits of using FCRG for guidance remain limited in the current experiments. At the same time, supplementary analyses indicate that FCRG's boundary metrics can evaluate model performance, with AP and Exact both strongly correlated with task-planning success. Across the tested controlled generators and environments, FCRG's overall ranking agreed closely with the true ranking (Pearson r = 0.958; Spearman's rank correlation = 0.930). On PushT, across three model types, Exact was moderately positively correlated with planning success (Pearson r = 0.500; Spearman ρ = 0.4352). We therefore see potential for FCRG to provide an effective guidance signal. In future work, we will refine the boundary-guided search strategy to make more effective use of FCRG. Given the space constraints of the present conference paper, we plan to investigate this direction more fully in an extended journal version.
Table. Original generator ranking results by system and comparison scope.
| System / scope | Reversed / comparable | Reversal rate | Kendall τ-b |
|---|---|---|---|
| BouncingBall | 0/46 | 0.0% | .979 |
| HardStop | 2/42 | 4.8% | .845 |
| PiecewiseAffine | 3/48 | 6.3% | .875 |
| PuckWall | 4/45 | 8.9% | .796 |
| SaturationKink | 2/48 | 4.2% | .917 |
| StickSlip | 1/47 | 2.1% | .947 |
| TwoBody | 3/45 | 6.7% | .848 |
| All strictly ordered pairs | 15/321 | 4.67% | .887 |
| Earlier/Later | 13/46 | 28.26% | — |
| Earlier/Later, true gap ≥ 1 SD | 5/26 | 19.23% | — |
We thank the reviewer again for the thoughtful feedback, which has helped us clarify and strengthen our work. If our responses and additional evidence have adequately addressed your concerns, we would be grateful if you would consider revising your score upward.
Dear Reviewer,
Thank you very much for the time and effort you have devoted to reviewing our paper. We sincerely appreciate your helpful comments and suggestions.
We have carefully responded to your comments and provided additional experimental results and clarifications in our rebuttal. As the discussion deadline approaches, we would be very grateful if you could review our responses and let us know whether any concerns remain. We would welcome the opportunity to discuss these further and provide any additional clarification that may be helpful.
Thank you again for your time and consideration.
Best regards,
The authors
Response to Reviewer 3Vs3 (Part 1 of 3)
We thank the reviewer for the thoughtful feedback. Our point-by-point responses are provided below.
Q1. Measurement validity and readout error
1.Known-generator experiments show the limited impact of FCRG measurement errors on the agreement between measured scores and true performance rankings. We replaced the world models with four controlled generators with known internal physical-mode signatures across 64 parameter environments in eight systems, obtaining each generator's actual boundary label (M) in addition to the environmental reference label (E) and readout prediction (F). This allows us to distinguish, at the discrete-bin level, generator-only errors, readout-only errors, and joint errors, including error cancellation. On the macro average across seven physical-event systems, the readout error rate was 7.88% under Identity (M = E, so only readout errors are possible). Under Earlier/Later (M shifts earlier/later relative to E), generator-only errors accounted for 12.28%/11.40%, readout-only errors for 15.67%/15.38%, and joint errors for 7.87%/8.63%, respectively. Thus, observed boundary disagreements in the original evaluation may arise from generator deviations, readout errors, or both.
To assess the practical effect of readout errors, we examined agreement between the generators' true performance rankings and FCRG score rankings. Across the tested controlled generators and environments, FCRG's overall ranking agreed closely with the true ranking (Spearman's rank correlation = 0.930). Among comparisons for which both rankings were strictly ordered, the reversal rate was 4.67%. For the harder-to-distinguish Earlier/Later generators, 71.74% of strictly ordered comparisons preserved the correct direction and 28.26% were reversed. Restricting comparisons to true-score differences of at least one standard deviation of the system's Earlier/Later true scores reduced the observed reversal rate to 19.23%. These results support FCRG's overall discriminative ability on the tested generators.
Table. Original generator ranking results by system and comparison scope.
This response to Q1 continues in Part 2 below.