acceptodds
Under review as a conference paper at ICLR 2027

Boundary-FidelityBench: Probing First-Crossing Fidelity in Action-Conditioned World Models

Abstract

Action-conditioned world models support planning by predicting the consequences of candidate actions, yet accurate predictions do not guarantee that a model preserves the action conditions under which physical behavior changes. Existing evaluations assess physical consistency and action response, but do not explicitly characterize where the first physical-mode change occurs along controlled action perturbations. We therefore formulate first-crossing boundary fidelity as an evaluation target. We operationalize it through Boundary-FidelityBench, which combines controlled action probing with independent physical references, together with FCRG, a learned readout that uses scale-aware residual evidence for fine-grained estimates of boundary presence, first-crossing location, and supported transition types. Across seven mode-switching systems, 5,629 fresh evaluation rays, and six world-model outputs, boundary-resolved evaluation reveals differences not captured by predictive-quality and behavioral-outcome measures. FCRG achieves 80.76% exact-bin accuracy on fresh simulator responses and separates missed boundaries from incorrect first-interval assignments. First-crossing estimates can guide action-query allocation under a fixed world-model query budget, yielding gains in tested settings while showing that better boundary coverage does not always translate into better decisions. These results establish boundary fidelity as a fine-grained complement to world-model evaluation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Action-Conditioned World Models, Boundary Fidelity, First-Crossing Detection, World Model Benchmarking, Physical Consistency Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3Vs3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Boundary-FidelityBench, a benchmark for evaluating first-crossing boundary fidelity as a surrogate for ranking world models beyond visual resemblance. This was accompanied by FCRG, a learning-based detector that detects boundaries from model's predicted trajectories. The experiments evaluate state-based world models based off this benchmark, and reaches findings such as leveraging boundary fidelity can help with planning within the world model.

Critical strengths

None

Critical weaknesses
  1. I find the problem motivation quite weak. It is well known that visual fidelity does not correlate to better planning/model-based reinforcement learning performance. But using boundary fidelity as a evaluation metric lacks strong incentives, which in turn yields limited applications. Especially since the authors criticize visual fidelity based methods for evaluating world models, however basing the whole experiment section around state-based world models rather than pixel-space world models is very confusing. This makes my assessment on the paper's significance as well as novelty quite negative.
Questions for authors
  1. FCRG achieves 80.76% exact-bin accuracy even on simulator results. This makes the measurement error too large to ignore, I wonder if the authors could provide additional experiments supporting the validity of measurements, i.e. being attributed to world model errors rather than FCRG error.
  2. Could the authors provide justification on why pixel-based world models are absent from experiments? It would also be helpful to provide explanation on how this limited-scope experiment both in terms of methods as well as evaluated environments, can have downstream significance.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3Vs3 (Part 1 of 3)

We thank the reviewer for the thoughtful feedback. Our point-by-point responses are provided below.

Q1. Measurement validity and readout error

1.Known-generator experiments show the limited impact of FCRG measurement errors on the agreement between measured scores and true performance rankings. We replaced the world models with four controlled generators with known internal physical-mode signatures across 64 parameter environments in eight systems, obtaining each generator's actual boundary label (M) in addition to the environmental reference label (E) and readout prediction (F). This allows us to distinguish, at the discrete-bin level, generator-only errors, readout-only errors, and joint errors, including error cancellation. On the macro average across seven physical-event systems, the readout error rate was 7.88% under Identity (M = E, so only readout errors are possible). Under Earlier/Later (M shifts earlier/later relative to E), generator-only errors accounted for 12.28%/11.40%, readout-only errors for 15.67%/15.38%, and joint errors for 7.87%/8.63%, respectively. Thus, observed boundary disagreements in the original evaluation may arise from generator deviations, readout errors, or both.

To assess the practical effect of readout errors, we examined agreement between the generators' true performance rankings and FCRG score rankings. Across the tested controlled generators and environments, FCRG's overall ranking agreed closely with the true ranking (Spearman's rank correlation = 0.930). Among comparisons for which both rankings were strictly ordered, the reversal rate was 4.67%. For the harder-to-distinguish Earlier/Later generators, 71.74% of strictly ordered comparisons preserved the correct direction and 28.26% were reversed. Restricting comparisons to true-score differences of at least one standard deviation of the system's Earlier/Later true scores reduced the observed reversal rate to 19.23%. These results support FCRG's overall discriminative ability on the tested generators.

Table. Original generator ranking results by system and comparison scope.

System / scopeReversed / comparableReversal rateKendall τ-b
BouncingBall0/460.0%.979
HardStop2/424.8%.845
PiecewiseAffine3/486.3%.875
PuckWall4/458.9%.796
SaturationKink2/484.2%.917
StickSlip1/472.1%.947
TwoBody3/456.7%.848
All strictly ordered pairs15/3214.67%.887
Earlier/Later13/4628.26%—
Earlier/Later, true gap ≥ 1 SD5/2619.23%—

This response to Q1 continues in Part 2 below.

AuthorsOpenReview ↗

Response to Reviewer 3Vs3 (Part 2 of 3)

Q1. Measurement validity and readout error

Continued from Part 1.

2.Readout improvement and sensitivity analysis. FCRG is a new evaluation method proposed in this paper, and we thank the reviewer for focusing attention on its measurement accuracy. Given the 80.76% Exact-bin accuracy on simulator reference responses, we tested whether an improved readout could achieve higher accuracy. These supplementary results suggest that the method has room for further development, and we intend to continue improving its accuracy. We retained the original evaluator-development data splits and tested counterfactual supervision based on matched continuation controls. On BouncingBall, Earlier/Later Exact-bin accuracy increased from 76.38%/71.35% to 86.98%/90.55%. Across the other six physical-event systems, macro-average Earlier/Later Exact-bin accuracy increased from 53.63%/57.92% to 63.18%/66.36%, respectively. Within each tested calibration strategy, the point-estimate ranking of seven-system macro-average metrics also remained unchanged. These results indicate that model comparisons have some robustness across the tested readout variants.

In addition, readout correction improved the validity of FCRG scores as measures of model performance. In the known-generator experiments, the corrected scores showed strong overall agreement with true rankings (mean Spearman's rank correlation = 0.944, compared with 0.923 for the matched control). Among comparisons for which both rankings were strictly ordered, the reversal rate decreased from 5.05% to 3.75%. For the more challenging Earlier/Later generators, 73.33% of strictly ordered comparisons retained the correct direction, and the reversal rate decreased from 34.88% to 26.67%. When comparisons were further limited to a true-score difference of at least one standard deviation of the system's Earlier/Later true scores, the observed reversal rate declined from 20.83% to 13.04%.

AuthorsOpenReview ↗

Response to Reviewer 3Vs3 (Part 3 of 3)

Q2. Visual world models and downstream relevance

Thank you for highlighting the importance of evaluating visual world models and downstream applicability. To examine generalization, we extended FCRG evaluation to multiple predictors on PushT and MuJoCo/OGBench Cube. The results support extending FCRG to more complex interactions and visual settings.

We evaluated FCRG on publicly available LeWM, DINO-WM, and JEPA-WM checkpoints for PushT, and on LeWM and GA-WM predictors for MuJoCo/OGBench Cube. In the Cube contact tests, Exact-bin accuracy on encoded ground-truth/predicted responses was 74.83%/49.66% for LeWM and 78.23%/44.22% for GA-WM. In the visual pushing tests on PushT, the corresponding Exact-bin accuracies for LeWM, DINO-WM, and JEPA-WM were 26.76%/23.94%, 39.44%/38.03%, and 52.11%/40.85%, respectively. These results show that the protocol can be applied to three-dimensional contact settings and visual models, and that a shared frozen readout reveals reproducible diagnostic differences between predicted and ground-truth responses. We also examined FCRG's validity as an indicator of model performance by applying it to three model types on PushT and using planning success as a criterion. Exact-bin accuracy showed a moderate positive correlation with planning success (Pearson r = 0.500; Spearman ρ = 0.4352). This result supports using FCRG metrics to evaluate model performance.

Table. Original AP and Exact-bin metrics for Cube and PushT.

SettingModelGT APPred. APGT ExactPred. ExactΔExact 95% CI (pp)
CubeLeWM.9822.7502.7483.4966[−35.42, −14.96]
CubeGA-WM.9832.8382.7823.4422[−45.83, −21.91]
PushTLeWM.9047.8924.2676.2394[−8.77, 2.63]
PushTDINO-WM.8358.8530.3944.3803[−11.77, 8.54]
PushTJEPA-WM.9205.9147.5211.4085[−21.43, −1.33]

We agree that the benefits of using FCRG for guidance remain limited in the current experiments. At the same time, supplementary analyses indicate that FCRG's boundary metrics can evaluate model performance, with AP and Exact both strongly correlated with task-planning success. Across the tested controlled generators and environments, FCRG's overall ranking agreed closely with the true ranking (Pearson r = 0.958; Spearman's rank correlation = 0.930). On PushT, across three model types, Exact was moderately positively correlated with planning success (Pearson r = 0.500; Spearman ρ = 0.4352). We therefore see potential for FCRG to provide an effective guidance signal. In future work, we will refine the boundary-guided search strategy to make more effective use of FCRG. Given the space constraints of the present conference paper, we plan to investigate this direction more fully in an extended journal version.

Table. Original generator ranking results by system and comparison scope.

System / scopeReversed / comparableReversal rateKendall τ-b
BouncingBall0/460.0%.979
HardStop2/424.8%.845
PiecewiseAffine3/486.3%.875
PuckWall4/458.9%.796
SaturationKink2/484.2%.917
StickSlip1/472.1%.947
TwoBody3/456.7%.848
All strictly ordered pairs15/3214.67%.887
Earlier/Later13/4628.26%—
Earlier/Later, true gap ≥ 1 SD5/2619.23%—

We thank the reviewer again for the thoughtful feedback, which has helped us clarify and strengthen our work. If our responses and additional evidence have adequately addressed your concerns, we would be grateful if you would consider revising your score upward.

Dear Reviewer,

Thank you very much for the time and effort you have devoted to reviewing our paper. We sincerely appreciate your helpful comments and suggestions.

We have carefully responded to your comments and provided additional experimental results and clarifications in our rebuttal. As the discussion deadline approaches, we would be very grateful if you could review our responses and let us know whether any concerns remain. We would welcome the opportunity to discuss these further and provide any additional clarification that may be helpful.

Thank you again for your time and consideration.

Best regards,
The authors

Reviewer_zuQBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies action-conditioned world models and their ability to capture transitions between dynamics modes induced by actions. The authors propose an evaluation methodology for this capability, built on a formalization of boundary crossing and an associated metric. The metric is validated on state-based world models and complemented by a planning study on two tasks.

Critical strengths
  • Evaluating whether world models accurately capture mode-switching dynamics is an important problem, as many real-world systems exhibit such behavior. Robotics tasks, for instance, involve contact interactions that induce frequent contact-mode switches.
  • The paper proposes a clean formalization for measuring how well world models detect mode transitions.
Critical weaknesses
  • The empirical evaluation is limited: only two models and two planning tasks are considered, and a planning gain appears on only one of the two tasks. It therefore remains unclear whether the boundary-crossing metric provides meaningful insight into planning performance. The experiments are restricted to simple examples and do not include realistic dynamics with contact-mode switches, such as robotics scenarios. Experiments could, for instance, be conducted in a simulator like MuJoCo, which exposes full contact information.
  • More generally, the scope is narrow, as only state-based world models are considered. It would be valuable to conduct the same study on publicly available checkpoints of vision-based world models (e.g., LeWM, DINO-WM). The world models are trained on small-scale datasets (512 trajectories), which is orders of magnitude smaller than the data used to train models such as LeWM or DINO-WM (tens of millions of transitions).
  • The approach applies a learned classifier to the states predicted by the world model, yet this classifier is frequently wrong. It should be possible to rely on simulator routines to detect dynamics modes instead.
  • The appendix spans roughly 120 pages, which cannot reasonably be reviewed. This makes it difficult to assess the validity of claims that rely on the appendix for support.
Questions for authors
  • Does the boundary-crossing metric correlate with planning performance?
  • Why use a learned classifier rather than simulator routines to detect modes from the world model's predictions?
  • Why not consider more challenging scenarios, such as contact-rich MuJoCo tasks? Locomotion and manipulation tasks typically involve complex mode switches, with contacts frequently transitioning between sticking, sliding, and breaking.
4 comments
AuthorsOpenReview ↗

Response to Reviewer zuQB (Part 1 of 3)

We thank the reviewer for the thoughtful feedback. Our point-by-point responses are provided below.

Q1. Correlation with planning performance

  1. We replaced the world models with four controlled generators with known internal physical-mode signatures across 64 parameter environments in eight systems. Each generator has a predefined boundary error relative to the environmental reference (no error, earlier, later, or no boundary). Across the tested controlled generators and environments, FCRG's overall ranking closely matched the generators' true performance ranking (Pearson r = 0.958, descriptive 95% interval [0.944, 0.971]; Spearman ρ = 0.930, descriptive 95% interval [0.908, 0.952]). Among comparisons for which both rankings were strictly ordered, the reversal rate was 4.67%. For the harder-to-distinguish Earlier/Later generators (whose predicted boundaries shift earlier/later relative to the true environmental boundary), 71.74% of strictly ordered comparisons retained the correct direction, while 28.26% were reversed. When we additionally required a true-score difference of at least one standard deviation of the system's Earlier/Later true scores, the observed reversal rate fell to 19.23%. These findings support FCRG's overall discriminative ability on the tested generators.

Table. Original generator ranking results by system and comparison scope.

System / scopeReversed / comparableReversal rateKendall τ-b
BouncingBall0/460.0%.979
HardStop2/424.8%.845
PiecewiseAffine3/486.3%.875
PuckWall4/458.9%.796
SaturationKink2/484.2%.917
StickSlip1/472.1%.947
TwoBody3/456.7%.848
All strictly ordered pairs15/3214.67%.887
Earlier/Later13/4628.26%—
Earlier/Later, true gap ≥ 1 SD5/2619.23%—
  1. We evaluated FCRG's boundary-crossing metrics using publicly available LeWM, DINO-WM, and JEPA-WM checkpoints on PushT, with each model's task-planning success rate as the criterion. Exact-bin accuracy showed a moderate positive correlation with planning success (Pearson r = 0.500; Spearman ρ = 0.4352). This result supports the use of FCRG metrics to assess model performance.
AuthorsOpenReview ↗

Response to Reviewer zuQB (Part 2 of 3)

Q2. Learned readout versus simulator routines

Thank you for this constructive suggestion. We use a learned readout to jointly estimate boundary presence, the first-crossing interval, and supported transition types from the ordered, sparse predicted responses along an entire action ray. The central challenge is distinguishing smooth within-mode variation and prediction noise from genuine mode changes. FCRG combines Local Continuation Residuals (LCR), a scale-aware envelope learned from mode-preserving training responses, and raw responses to learn how these signals relate to boundary labels. In the original state-based model experiments, the trained and calibrated readout was frozen within each system and applied to different evaluated world models, providing a consistent measurement procedure.

Existing experiments provide empirical support for this design. Under matched noise augmentation, FCRG achieved 70.76% Exact-bin accuracy, compared with 65.98% for Raw and 65.87% for MLP32. Its joint NLL was 0.3386, compared with 0.4263 and 0.3511, respectively (original paper, Section 5.5 and Appendix F). In this experiment, FCRG's structured evidence improved first-interval localization and probabilistic prediction beyond the benefits of simply using a learned classifier or noise augmentation. Moreover, in the prescribed physical-boundary shift control from 0.3 to 0.6, all three readout seeds recovered the target interval in all 32 paired blocks, supporting the method's ability to track the tested displacement (Appendix D.1).

We agree with the reviewer that directly observable modes should be verified using state-based rules. A learned readout is also useful when model outputs cannot be directly evaluated by simulator routines. For example, the LeWM/Cube interface in our supplementary experiment outputs a 192-dimensional latent representation for which we do not yet have a validated state or image decoder. We therefore cannot directly apply finger-pad–cube contact rules to this output. FCRG can instead be trained on encoded ground-truth responses and then, once frozen, applied to model predictions in the same representation space. Across 147 resolved contact-boundary-positive rays, Exact-bin accuracy was 74.83% on encoded ground-truth responses and 49.66% on predicted responses. Thus, when direct physical-state interfaces are unavailable, a learned readout can still evaluate predicted representations against an independent simulator-based boundary reference.

Table. Original Cube contact-boundary readout metrics.

Model / responseAPRecallFPRExactnMAE
LeWM encoded ground truth.9822.9320.0511.7483.1582
LeWM prediction.7502.9524.4148.4966.2009
GA-WM encoded ground truth.9832.9524.0682.7823.1318
GA-WM prediction.8382.8503.3011.4422.2928
AuthorsOpenReview ↗

Response to Reviewer zuQB (Part 3 of 3)

Q3. More challenging contact-rich scenarios

Thank you for suggesting validation on more challenging interaction tasks. To examine generalization, we extended FCRG evaluation to multiple predictors on PushT and MuJoCo/OGBench Cube. The results support applying FCRG to more complex interaction processes and visual settings.

We extended FCRG to publicly available LeWM, DINO-WM, and JEPA-WM checkpoints on PushT, as well as LeWM and GA-WM predictors on MuJoCo/OGBench Cube. In the Cube contact tests, Exact-bin accuracy on encoded ground-truth/predicted responses was 74.83%/49.66% for LeWM and 78.23%/44.22% for GA-WM; the initial-state-paired intervals for predicted minus ground-truth accuracy were [−35.42, −14.96] and [−45.83, −21.91] percentage points, respectively. In the visual pushing tests on PushT, the corresponding Exact-bin accuracies for LeWM, DINO-WM, and JEPA-WM were 26.76%/23.94%, 39.44%/38.03%, and 52.11%/40.85%, with initial-state-paired intervals of [−8.77, 2.63], [−11.77, 8.54], and [−21.43, −1.33] percentage points. These results show that the protocol can be applied to three-dimensional contact settings and visual models, with reproducible diagnostic differences between predicted and ground-truth responses under the same frozen readout. We also examined the validity of FCRG as a model-performance indicator by applying it to three model types on PushT and using task-planning success as an external criterion. Exact-bin accuracy showed a positive correlation with planning success (Pearson r = 0.500; Spearman ρ = 0.4352), supporting its use in model evaluation.

Table. Original AP and Exact-bin metrics for Cube and PushT.

SettingModelGT APPred. APGT ExactPred. ExactΔExact 95% CI (pp)
CubeLeWM.9822.7502.7483.4966[−35.42, −14.96]
CubeGA-WM.9832.8382.7823.4422[−45.83, −21.91]
PushTLeWM.9047.8924.2676.2394[−8.77, 2.63]
PushTDINO-WM.8358.8530.3944.3803[−11.77, 8.54]
PushTJEPA-WM.9205.9147.5211.4085[−21.43, −1.33]

We thank the reviewer again for the thoughtful feedback, which has helped us clarify and strengthen our work. If our responses and additional evidence have adequately addressed your concerns, we would be grateful if you would consider revising your score upward.

Dear Reviewer,

Thank you very much for the time and effort you have devoted to reviewing our paper. We sincerely appreciate your helpful comments and suggestions.

We have carefully responded to your comments and provided additional experimental results and clarifications in our rebuttal. As the discussion deadline approaches, we would be very grateful if you could review our responses and let us know whether any concerns remain. We would welcome the opportunity to discuss these further and provide any additional clarification that may be helpful.

Thank you again for your time and consideration.

Best regards,
The authors

Reviewer_4zShOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces first-crossing boundary fidelity for action-conditioned world models: whether predicted responses preserve where a physical mode first changes along a controlled action perturbation. Boundary-FidelityBench constructs independent simulator-based references, while FCRG estimates boundary presence, location, and supported transition types from sparse response sequences using local continuation residuals and scale-aware evidence. Experiments across seven mode-switching systems and six model outputs demonstrate complementary information beyond aggregate prediction error and task outcomes. Additional experiments examine measurement accuracy, robustness, component contributions, and fixed-budget action search.

Critical strengths
  1. The paper is well-written. The motivation is clear. The figures help explain the evaluation protocol and FCRG architecture.
  2. The research question is useful. The paper asks whether world models preserve the action conditions that cause physical behavior to change. This provides information beyond average prediction error and task success.
  3. Boundary-FidelityBench separates simulator-defined reference boundaries from the responses used to estimate them. FCRG combines local continuation residuals with scale-aware evidence to produce structured boundary predictions, enabling interpretable diagnosis of boundary presence, first-crossing location, and uncertainty.
  4. Experiments across seven mode-switching systems and six model outputs demonstrate both diagnostic and downstream utility.
Critical weaknesses
  1. Errors in the evaluator complicate model assessment. FCRG achieves 80.76% Exact-bin accuracy even on true simulator responses. When it reports an incorrect boundary on a world-model output, the discrepancy may therefore come from the world model, FCRG, or both. These error sources are not fully separated.
  2. The experiments evaluate new inputs within fixed, low-dimensional parameter environments. They do not establish whether the approach generalize to unseen physical parameters, more complex interactions, or visual observations.
  3. It seems that the downstream benefits are task-dependent. Boundary-guided search improves task success on StickSlipBlock but provides no comparable improvement on PuckWall2D. Thus, better boundary information does not consistently lead to better action selection in the tested settings.
  4. The source code provides core FCRG components but omits the complete data-generation, training, and evaluation pipeline, preventing inspection of the implementation and reproduction for future research.
Questions for authors

Please see the previous Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4zSh (Part 1 of 4)

W1. Evaluator error and error attribution

Thank you for highlighting the important issue of error attribution. We agree that an incorrect FCRG judgment cannot be uniquely attributed to the world model. To address this concern, we conducted several supplementary analyses to distinguish different error sources. The results show that both types of error occur. Due to limitations in observability, we cannot fully separate world-model errors from FCRG errors; however, the additional experiments also indicate that errors in the FCRG readout can be reduced.

1.An independent audit of model outputs identifies genuine prediction failures. To directly assess whether learned world models exhibit genuine output deficiencies, we conducted an independent state audit that does not rely on FCRG. Using the original physical systems' geometry, contact, and velocity criteria, we constructed deterministic state observers, applied them separately to simulator responses and world-model responses, and compared each response with its corresponding center-action response. None of the three DreamerV3 checkpoints preserved the sampled terminal-mode changes observed in the reference simulator across several tasks; TrajWorld similarly failed to preserve these changes on BouncingBall. These findings provide direct evidence, independent of FCRG, that some learned world models fail to preserve specific observable changes in physical responses.

2.Known-generator controls reveal FCRG measurement errors, with limited impact on the agreement between measured scores and true performance rankings. We replaced the world models with four controlled generators whose internal physical-mode signatures are known, across 64 parameter environments in eight systems. This provides each generator's actual boundary label (M), in addition to the environmental reference label (E) and the readout prediction (F). At the discrete-bin level, we can therefore distinguish generator-only errors, readout-only errors, and joint errors, including error cancellation. On the macro average across the seven physical-event systems, the readout error rate is 7.88% in the Identity condition (M = E, so only readout errors are possible). Under the Earlier/Later conditions (M shifts earlier/later relative to E), generator-only errors account for 12.28%/11.40%, readout-only errors for 15.67%/15.38%, and joint errors for 7.87%/8.63%, respectively. These results show that the boundary disagreements observed in the original evaluation may arise from generator deviations, readout errors, or both.

To further examine the effect of readout errors on the usefulness of the method, we evaluated the agreement between the generators' true performance rankings and their FCRG score rankings. Across the tested controlled generators and environments, the overall FCRG ranking agrees closely with the true ranking (Spearman's rank correlation = 0.930; descriptive 95% interval [0.908, 0.952]). Among comparisons for which both rankings were strictly ordered, the reversal rate was 4.67%. For the harder-to-distinguish Earlier/Later generators, 71.74% of strictly ordered comparisons retained the correct direction, while 28.26% were reversed. When we further required the true-score difference to be at least one standard deviation of the system's Earlier/Later true scores, the observed reversal rate fell to 19.23%. These results support FCRG's overall discriminative ability on the tested generators.

Table. Original generator ranking results by system and comparison scope.

System / scopeReversed / comparableReversal rateKendall τ-b
BouncingBall0/460.0%.979
HardStop2/424.8%.845
PiecewiseAffine3/486.3%.875
PuckWall4/458.9%.796
SaturationKink2/484.2%.917
StickSlip1/472.1%.947
TwoBody3/456.7%.848
All strictly ordered pairs15/3214.67%.887
Earlier/Later13/4628.26%—
Earlier/Later, true gap ≥ 1 SD5/2619.23%—

This response to W1 continues in Part 2 below.

AuthorsOpenReview ↗

Response to Reviewer 4zSh (Part 2 of 4)

W1. Evaluator error and error attribution

Continued from Part 1.

  1. Readout improvement and sensitivity analysis. FCRG is a new evaluation method proposed in this paper, and we appreciate the reviewer's attention to its measurement accuracy. In light of the 80.76% Exact-bin accuracy on simulator reference responses, we additionally tested whether improving the readout could increase accuracy. These supplementary findings suggest that FCRG has scope for further development, and we intend to continue improving its accuracy in future work. Using the original evaluator-development data splits, we tested counterfactual supervision based on matched continuation controls. On BouncingBall, Earlier/Later Exact-bin accuracy improved from 76.38%/71.35% to 86.98%/90.55%. Across the other six physical-event systems, the macro-average Earlier/Later Exact-bin accuracy increased from 53.63%/57.92% to 63.18%/66.36%, respectively. Within each tested calibration strategy, the point-estimate rankings of the seven-system macro-average metrics were also unchanged. These results indicate some robustness of the model comparisons across the tested readout variants.

In addition, readout correction improved the validity of FCRG scores as indicators of model performance. In the known-generator experiments, the corrected FCRG scores showed strong overall agreement with the true rankings (mean Spearman's rank correlation = 0.944, compared with 0.923 for the matched control). Among comparisons for which both rankings were strictly ordered, the reversal rate decreased from 5.05% to 3.75%. For the more difficult Earlier/Later generators, 73.33% of strictly ordered comparisons preserved the correct direction, and the reversal rate declined from 34.88% to 26.67%. When the true-score difference was further required to be at least one standard deviation of the system's Earlier/Later true scores, the observed reversal rate decreased from 20.83% to 13.04%.

Table. Per-system ranking validity under matched control and corrected readout.

System / scopeControl reversalsCorrected reversalsControl ρCorrected ρ
HardStop2/422/42.909.910
PiecewiseAffine4/474/48.894.900
PuckWall3/440/43.885.968
SaturationKink2/482/48.950.950
StickSlip1/471/47.969.969
TwoBody4/442/45.865.944
BouncingBall0/451/47.987.969
Overall16/317 (5.05%)12/320 (3.75%).923.944
Earlier/Later15/43 (34.88%)12/45 (26.67%)——
Earlier/Later, true gap ≥ 1 SD5/24 (20.83%)3/23 (13.04%)——
AuthorsOpenReview ↗

Response to Reviewer 4zSh (Part 3 of 4)

W2. Generalization beyond fixed parameter settings

Thank you for raising the question of generalization. To evaluate this aspect, we extended FCRG to multiple predictors on PushT and MuJoCo/OGBench Cube and conducted additional tests under physical-parameter changes. The findings support extending FCRG to more complex interactions and visual settings and indicate robustness to the tested parameter perturbations.

  1. Generalization to more complex interactions and visual observations. We extended FCRG evaluation to publicly available LeWM, DINO-WM, and JEPA-WM checkpoints on PushT, as well as LeWM and GA-WM predictors on MuJoCo/OGBench Cube. In the Cube contact tests, Exact-bin accuracy on encoded ground-truth responses versus predicted responses was 74.83%/49.66% for LeWM and 78.23%/44.22% for GA-WM; the initial-state-paired intervals for predicted minus ground-truth accuracy were [−35.42, −14.96] and [−45.83, −21.91] percentage points, respectively. In the visual pushing tests on PushT, the corresponding Exact-bin accuracies for LeWM, DINO-WM, and JEPA-WM were 26.76%/23.94%, 39.44%/38.03%, and 52.11%/40.85%; the initial-state-paired intervals for predicted minus ground-truth accuracy were [−8.77, 2.63], [−11.77, 8.54], and [−21.43, −1.33] percentage points, respectively. These results show that the protocol can be applied to three-dimensional contact settings and visual models, and that a shared frozen readout reveals reproducible diagnostic differences between predicted and ground-truth responses.

Table. Original AP and Exact-bin metrics for Cube and PushT.

SettingModelGT APPred. APGT ExactPred. ExactΔExact 95% CI (pp)
CubeLeWM.9822.7502.7483.4966[−35.42, −14.96]
CubeGA-WM.9832.8382.7823.4422[−45.83, −21.91]
PushTLeWM.9047.8924.2676.2394[−8.77, 2.63]
PushTDINO-WM.8358.8530.3944.3803[−11.77, 8.54]
PushTJEPA-WM.9205.9147.5211.4085[−21.43, −1.33]
  1. Tests under parameter changes. On PushT, we froze both LeWM and the readout and varied friction, brightness, and object geometry. FCRG showed relatively stable performance across the tested conditions: AP on encoded ground-truth responses ranged from .8586 to .9070 (baseline setting: .9047), and AP on predicted responses ranged from .8393 to .9122 (baseline: .8924). For three mild perturbations that did not change interval-level physical labels (friction 0.3 and 0.6, and RGB brightness −24), deviations from the source were small: encoded-response AP shifted by no more than .003 and predicted-response AP by no more than .004. Exact-bin accuracy remained within .2394–.2958 (baseline: .2676) and .2394–.2535 (baseline: .2394), respectively. Under two geometric perturbations that changed the labels (T scale 24/36), Exact-bin accuracy was .4483/.3067 for encoded ground-truth responses and .3966/.2267 for predicted responses. We therefore find that, for the tested range of parameter and appearance perturbations, the frozen FCRG pipeline can provide stable boundary readouts without recalibration.

Table. PushT parameter and appearance changes with frozen LeWM and readout.

ConditionPositive raysLabel changes / 272Encoded GT APPred. APEncoded GT ExactPred. Exact
Source environment (baseline)71—.9047.8924.2676.2394
Friction 0.3710.9042.8924.2535.2394
Friction 0.6710.9070.8924.2394.2394
RGB brightness −24710.9062.8961.2958.2535
T scale 245823.9070.9122.4483.3966
T scale 367518.8586.8393.3067.2267

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.