- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a method for predicting the occurrence, time and mode of failure of a stack of objects given a candidate motion of its supporting surface. The authors compare predictions over the complete motion to predictions restarted from an intermediate state without a new observation. Differences are used to select simulator tests, and outcomes supervise predictions on both sides of the comparison. A fixed physical monitor maps the predicted responses into a failure decision. Experiments use simulations constructed from PLP and ShapeStacks, and an industrial experiment to evaluate the learned representation for predicting collapse.
Critical strengths
- This training method accounts for the possibility that the prediction paths agree but are physically wrong. Both paths are trained on the same simulator response, and half of the post-seed testing budget is used for independent coverage. This seems more convincing than just testing for consistency, which can miss cases where both paths have the same prediction error.
- Experiments contain solid controls examining the effect of continuation supervision. For the representation ablations, the authors use 5 paired seeds, capacity matching, and shared simulator forks. As shown in Table 9, verified continuation supervision has a lower split-response nRMSE than the one-future and raw-consistency controls (0.103 vs. 0.162), yet direct-response accuracy is similar. The main results also show lower missed-failure and false-alarm rates than the Direct baseline across both simulation benchmarks.
Critical weaknesses
- My main concern is the novelty of the proposed approach. Deep-OSG already addressed direct-composed consistency by supervising both paths to have the same observed outcome [1]. Prior work has noted that agreement alone does not ensure physical correctness [2]. While StackProbe extends these ideas to visual states, controllable motion, and failure-guided testing, the paper does not provide a strong rationale for how the continuation-training procedure contributes beyond existing methods. The raw-consistency ablation does not convincingly show an advantage over existing supervised temporal-composition objectives, since this control replaces split physical supervision with latent agreement rather than implementing such an objective.
- For the acquisition experiment, it would have been good to introduce an ensemble baseline, as well as some ablations on the fixed 50 per cent coverage fraction. It remains unclear how the method compares with ensemble-based acquisition or how sensitive the prediction gains are to the chosen coverage fraction under the same simulator budget. Additionally, the authors could expand the discussion of how their approach relates to existing work on disagreement-driven exploration and active failure discovery [3–5].
- The paper details simulations with fixed density and contact coefficients and centered centers of mass in Appendix B.4, and uses the same simulator and material parameters for both benchmarks. While the test motion trajectories were not part of the training set, no experiments measure the effect of unseen friction, density, or mass distributions, leaving the behavior of the method under unmodeled physics unquantified.
- The industrial experiment is limited to feature transfer for collapse classification, not motion prediction. Features are extracted from StackProbe and combined with operational logs to train a classifier on real-world collapse and noncollapse episodes. This result shows the value of the learned representations, but does not establish that the system can predict physical trajectories, failure times, failure modes, or failure radii in the real world.
References
- Chen, Junfeng, and Kailiang Wu. "Deep-OSG: Deep learning of operators in semigroup." Journal of Computational Physics 493 (2023): 112498
- Shikhman, Lennon J. "Semigroup Consistency as a Diagnostic for Learned Physics Simulators." arXiv preprint arXiv:2605.26324 (2026).
- Sekar, Ramanan, et al. "Planning to explore via self-supervised world models." International conference on machine learning. PMLR, 2020.
- Sagar, Som, Aditya Taparia, and Ransalu Senanayake. "Failures are fated, but can be faded: Characterizing and mitigating unwanted behaviors in large-scale vision and language models." arXiv preprint arXiv:2406.07145 (2024).
- Lee, Ritchie, et al. "Adaptive stress testing: Finding likely failure events with reinforcement learning." Journal of Artificial Intelligence Research 69 (2020): 1165-1201.
Questions for authors
- How does performance change under unseen friction, density, and center-of-mass variations, and can controlled real-world carrier-motion experiments validate the predicted failure times and modes?
6 comments
Reviewer d8pi — Comment 2
2. Ensemble selection and the 50% coverage choice. We test whether our selector chooses more useful training data than ensemble disagreement, and whether its gains depend on the coverage share. Shikhman [3] uses direct–composed discrepancy to diagnose learned dynamics. Unlike Plan2Explore [4], which uses disagreement among predictive models, our selector compares two prediction routes within one model. FATED [5] uses reinforcement learning and human feedback to discover and mitigate failures of pretrained vision and language models. Adaptive Stress Testing [6] searches for likely failure paths, including cases where a system fails and another does not. In contrast, StackProbe selects carrier motions near a reference plan where its two routes disagree about failure, then learns from the simulator's response. Section 2 clarifies these differences (revised lines 126–133).
Table 17 in Appendix C.9 compares coverage alone, our failure-set selector at 25%/50%/75% coverage, and a 5-member ensemble at 50% (revised lines 1695–1724). All settings use the same initial data and 64k total simulator calls, counting invalid attempts. Final predictors share the architecture, training schedule and held-out tests, learning from their selector's data. The ensemble trains and queries 5 predictors to select motions on which they disagree about continuation failure. Evaluation uses one member chosen in advance. We measure split false-safe, the fraction of simulated failures predicted safe after continuation.
Table 17: selection at 64k simulator calls on PLP. FS denotes continuation false-safe and FA denotes false-alarm, both in percent. Brackets give 95% confidence intervals for FS.
| Selector | Coverage share | FS [95% CI] | FA |
|---|---|---|---|
| Coverage only | 100% | 12.57 [11.54, 13.57] | 11.06 |
| Failure-set | 25% | 8.19 [7.29, 9.21] | 9.47 |
| Failure-set | 50% | 8.32 [7.50, 9.27] | 8.88 |
| Failure-set | 75% | 9.04 [8.12, 10.07] | 8.71 |
| 5-member ensemble | 50% | 9.61 [8.67, 10.61] | 9.02 |
At 50% coverage, our selector reduces FS by 1.29 percentage points [0.34, 2.24] relative to the ensemble, with similar false alarms. Its gains over coverage alone also persist across the three coverage shares.
3. Prediction under unseen material properties. We test missed failures under hidden changes in friction, density or mass distribution, keeping each stack and motion fixed. We added this evaluation in Appendix C.6, Table 15 (revised lines 1531–1605).
We compare Direct, the full-response baseline, with StackProbe on 1,000 motion queries per simulation benchmark (PLP and ShapeStacks) from 200 independent stacks that are initially stable in every setting. Both models stay frozen, with the same inputs and decision thresholds. We vary object–object friction (0.30–0.70), object–carrier friction (0.40–0.90), density (125–500 kg/m³), and internal mass distribution separately. The mass-distribution change preserves total mass and shifts a box's center of mass by 10% of its width. The changed properties are not given to either model.
False-safe rates (%) under hidden material changes. False-safe is the fraction of simulated failures predicted safe using the full-motion prediction route.
| Hidden physical condition | PLP Direct | PLP StackProbe | ShapeStacks Direct | ShapeStacks StackProbe |
|---|---|---|---|---|
| Nominal | 11.83 | 7.89 | 13.10 | 8.90 |
| Object–object friction | 12.37 | 8.64 | 13.93 | 9.93 |
| Object–carrier friction | 13.39 | 9.69 | 14.19 | 10.25 |
| Per-object density | 13.33 | 9.52 | 14.67 | 10.76 |
| Internal mass distribution | 14.36 | 10.85 | 16.03 | 12.25 |
Each material shift increases missed failures, while StackProbe retains lower rates on both benchmarks.
Reviewer d8pi — Comment 3
4. Real responses beyond collapse classification. We test whether the simulation-trained state predicts collapse and parcel movement under real carrier motion. StackProbe and Direct receive the same pre-prediction images, available records and recovered carrier motion, without retraining on these videos. We added this evaluation in Section 6 and Appendix C.8 (revised lines 514–520 and 1634–1693). Here we assess visible collapse and its onset. Detailed failure modes are evaluated in simulation using contact and support information (Appendix C.3). Failure radii are also evaluated in simulation, using alternative motions from comparable starting stacks (Appendix C.4).
Collecting data in this large-scale industrial logistics deployment (Figure 6), especially for manual-shaking trials, requires additional equipment, dedicated staff, and company approval. We use calibrated video to recover carrier motion as input and independently reconstruct parcel trajectories for evaluation. We evaluate 120 independent stacks after routine placement: 100 remain stable on a stationary carrier and 20 are manually shaken, with 10 collapses and 10 noncollapses in the common evaluation window. Collapse time is the onset of the fall or toppling that leaves the stack collapsed at the window's end.
Real-motion results from the added Table 16.
| Evaluation measure | Direct | StackProbe |
|---|---|---|
| Correct stability decisions, stationary carrier | 93/100 | 96/100 |
| Correct collapse decisions, manual shaking | 7/10 | 9/10 |
| Correct stability decisions, manual shaking | 8/10 | 9/10 |
| Displacement error, shaken noncollapses (cm) | 3.6 | 2.5 |
| Orientation error, shaken noncollapses (degrees) | 6.3 | 4.6 |
| Collapse-time MAE (s) | 0.29 | 0.21 |
Motion errors are averaged over the 10 shaken noncollapses. Time MAE uses the same 7 collapses detected by both models. StackProbe improves both collapse decisions and parcel-motion prediction.
We hope these comparisons clarify the contribution and its practical value. We welcome further questions and will follow up promptly.
References
[1] Junfeng Chen and Kailiang Wu. “Deep-OSG: Deep learning of operators in semigroup.” Journal of Computational Physics, 493:112498, 2023. doi:10.1016/j.jcp.2023.112498.
[2] Andy Zeyi Liu, Haoran Sun, Lucas Baker, Randall Balestriero, and John Sous. “Semigroup-JEPA: Latent Dynamics Consistency for Zero-Shot Physics Generalization.” arXiv:2609.10464, 2026. arXiv:2609.10464.
[3] Lennon J. Shikhman. “Semigroup Consistency as a Diagnostic for Learned Physics Simulators.” AI4Physics Workshop at ICML, 2026. arXiv:2605.26324. doi:10.48550/arXiv.2605.26324.
[4] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. “Planning to Explore via Self-Supervised World Models.” In Proceedings of the 37th International Conference on Machine Learning, PMLR 119, pp. 8583–8592, 2020. Official proceedings.
[5] Som Sagar, Aditya Taparia, and Ransalu Senanayake. “Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models.” In Proceedings of the 41st International Conference on Machine Learning, PMLR 235, pp. 42999–43023, 2024. Official proceedings.
[6] Ritchie Lee, Ole J. Mengshoel, Anshu Saksena, Ryan W. Gardner, Daniel Genin, Joshua Silbermann, Michael Owen, and Mykel J. Kochenderfer. “Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning.” Journal of Artificial Intelligence Research, 69:1165–1201, 2020. doi:10.1613/jair.1.12190.
Reviewer d8pi — Comment 1
We thank the reviewer for the careful review and positive assessment of our physical supervision and matched controls. We appreciate your focus on what StackProbe adds beyond prior work, which helps clarify the contribution and its supporting evidence. For your concerns, we have revised the paper as follows. (1) We clarified what StackProbe's state design adds beyond Deep-OSG's shared-target supervision [1]. The original matched comparison shows improved continuation with full physical supervision held fixed. (2) We added an ensemble comparison and varied the share of new simulator calls reserved for independently sampled motions (coverage). Models trained on our tests miss fewer failures than those using ensemble-selected tests at 50% coverage or independent samples. (3) We added tests with changes in friction, density and mass distribution. StackProbe misses fewer failures than Direct, our matched baseline that predicts the response to the full motion without resuming from an intermediate state. (4) We added a video-based evaluation for real carrier motion. StackProbe predicts collapse and its timing more accurately than Direct, and reduces parcel-motion error in stacks that remain standing.
1. What the state design adds beyond shared supervision. Deep-OSG [1] trains direct and composed predictions to match the same observed outcome. Unlike Deep-OSG, which evolves a supplied physical state, StackProbe learns a state from visual evidence using this same supervision. Compared with SG-JEPA [2], which learns visual features through recursive latent prediction, StackProbe trains a dedicated state (the core) to predict physical responses to alternative carrier motions, keeping fixed object information and failure-check history separate. The core can be updated through a motion segment and then used to evaluate different continuations without another observation. We clarified these differences in the Introduction and Section 2 (revised lines 94–104 and 118–124).
To test the core's contribution, we compare a model that uses image-encoder features as its state with a model that converts them into a learned core. Both states have 512 dimensions. Each model updates its state for an initial motion segment, then predicts how the stack responds to the rest. Both use the same training data and schedule, with trainable parameter counts outside the shared backbone matched within 5%. Both are trained to match simulated parcel movements and contacts for the full motion and its continuation. We clarified this comparison in Section 6 and Appendix C.2 (revised lines 454–461 and 1224–1233).
Original matched results (Table 10, revised lines 1386–1402). nRMSE is normalized response-prediction error. Lower is better.
The paired continuation-error reduction is 0.029 (95% CI [0.019, 0.037]), while full-motion errors remain similar. The matched comparison supports the dedicated state's value for continuation under full physical supervision. Raw consistency tests removal of continuation physical targets, so it is not a baseline for Deep-OSG's supervised objective.