acceptodds
Under review as a conference paper at ICLR 2027

Learning Stack Stability through Physical Perturbation Tests

Abstract

Stack stability concerns whether an arrangement of physical objects will remain intact. Physics-based simulations require detailed scene parameters, such as friction coefficients which are often unavailable, while visual classifiers predict stability labels without understanding how failure develops. We propose StackProbe, which learns a state representation from images and available records, by observing how stacks respond to various physical perturbations. Then we can assess stack stability based on the learned state representation. During training, we test the learned state representation under different physical perturbations and refine it using responses from high-fidelity simulations. At inference, we use the learned state representation to predict whether, when, and how a stack may fail, without running a simulator. We evaluate StackProbe on two simulation benchmarks and data from a deployed industrial palletizing system. Empirical results show that StackProbe reduces missed-failure rates by over 32% compared with the strongest learned baseline, while its representation improves average precision for industrial collapse prediction by at least 6.0 percentage points. Code is available at https://anonymous.4open.science/r/StackProbe/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
physical reasoning, visual prediction, world models, joint-embedding predictive architectures, failure prediction, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_d8piOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for predicting the occurrence, time and mode of failure of a stack of objects given a candidate motion of its supporting surface. The authors compare predictions over the complete motion to predictions restarted from an intermediate state without a new observation. Differences are used to select simulator tests, and outcomes supervise predictions on both sides of the comparison. A fixed physical monitor maps the predicted responses into a failure decision. Experiments use simulations constructed from PLP and ShapeStacks, and an industrial experiment to evaluate the learned representation for predicting collapse.

Critical strengths
  1. This training method accounts for the possibility that the prediction paths agree but are physically wrong. Both paths are trained on the same simulator response, and half of the post-seed testing budget is used for independent coverage. This seems more convincing than just testing for consistency, which can miss cases where both paths have the same prediction error.
  2. Experiments contain solid controls examining the effect of continuation supervision. For the representation ablations, the authors use 5 paired seeds, capacity matching, and shared simulator forks. As shown in Table 9, verified continuation supervision has a lower split-response nRMSE than the one-future and raw-consistency controls (0.103 vs. 0.162), yet direct-response accuracy is similar. The main results also show lower missed-failure and false-alarm rates than the Direct baseline across both simulation benchmarks.
Critical weaknesses
  1. My main concern is the novelty of the proposed approach. Deep-OSG already addressed direct-composed consistency by supervising both paths to have the same observed outcome [1]. Prior work has noted that agreement alone does not ensure physical correctness [2]. While StackProbe extends these ideas to visual states, controllable motion, and failure-guided testing, the paper does not provide a strong rationale for how the continuation-training procedure contributes beyond existing methods. The raw-consistency ablation does not convincingly show an advantage over existing supervised temporal-composition objectives, since this control replaces split physical supervision with latent agreement rather than implementing such an objective.
  2. For the acquisition experiment, it would have been good to introduce an ensemble baseline, as well as some ablations on the fixed 50 per cent coverage fraction. It remains unclear how the method compares with ensemble-based acquisition or how sensitive the prediction gains are to the chosen coverage fraction under the same simulator budget. Additionally, the authors could expand the discussion of how their approach relates to existing work on disagreement-driven exploration and active failure discovery [3–5].
  3. The paper details simulations with fixed density and contact coefficients and centered centers of mass in Appendix B.4, and uses the same simulator and material parameters for both benchmarks. While the test motion trajectories were not part of the training set, no experiments measure the effect of unseen friction, density, or mass distributions, leaving the behavior of the method under unmodeled physics unquantified.
  4. The industrial experiment is limited to feature transfer for collapse classification, not motion prediction. Features are extracted from StackProbe and combined with operational logs to train a classifier on real-world collapse and noncollapse episodes. This result shows the value of the learned representations, but does not establish that the system can predict physical trajectories, failure times, failure modes, or failure radii in the real world.

References

  1. Chen, Junfeng, and Kailiang Wu. "Deep-OSG: Deep learning of operators in semigroup." Journal of Computational Physics 493 (2023): 112498
  2. Shikhman, Lennon J. "Semigroup Consistency as a Diagnostic for Learned Physics Simulators." arXiv preprint arXiv:2605.26324 (2026).
  3. Sekar, Ramanan, et al. "Planning to explore via self-supervised world models." International conference on machine learning. PMLR, 2020.
  4. Sagar, Som, Aditya Taparia, and Ransalu Senanayake. "Failures are fated, but can be faded: Characterizing and mitigating unwanted behaviors in large-scale vision and language models." arXiv preprint arXiv:2406.07145 (2024).
  5. Lee, Ritchie, et al. "Adaptive stress testing: Finding likely failure events with reinforcement learning." Journal of Artificial Intelligence Research 69 (2020): 1165-1201.
Questions for authors
  1. How does performance change under unseen friction, density, and center-of-mass variations, and can controlled real-world carrier-motion experiments validate the predicted failure times and modes?
6 comments
AuthorsOpenReview ↗

Reviewer d8pi — Comment 1

We thank the reviewer for the careful review and positive assessment of our physical supervision and matched controls. We appreciate your focus on what StackProbe adds beyond prior work, which helps clarify the contribution and its supporting evidence. For your concerns, we have revised the paper as follows. (1) We clarified what StackProbe's state design adds beyond Deep-OSG's shared-target supervision [1]. The original matched comparison shows improved continuation with full physical supervision held fixed. (2) We added an ensemble comparison and varied the share of new simulator calls reserved for independently sampled motions (coverage). Models trained on our tests miss fewer failures than those using ensemble-selected tests at 50% coverage or independent samples. (3) We added tests with changes in friction, density and mass distribution. StackProbe misses fewer failures than Direct, our matched baseline that predicts the response to the full motion without resuming from an intermediate state. (4) We added a video-based evaluation for real carrier motion. StackProbe predicts collapse and its timing more accurately than Direct, and reduces parcel-motion error in stacks that remain standing.

1. What the state design adds beyond shared supervision. Deep-OSG [1] trains direct and composed predictions to match the same observed outcome. Unlike Deep-OSG, which evolves a supplied physical state, StackProbe learns a state from visual evidence using this same supervision. Compared with SG-JEPA [2], which learns visual features through recursive latent prediction, StackProbe trains a dedicated state (the core) to predict physical responses to alternative carrier motions, keeping fixed object information and failure-check history separate. The core can be updated through a motion segment and then used to evaluate different continuations without another observation. We clarified these differences in the Introduction and Section 2 (revised lines 94–104 and 118–124).

To test the core's contribution, we compare a model that uses image-encoder features as its state with a model that converts them into a learned core. Both states have 512 dimensions. Each model updates its state for an initial motion segment, then predicts how the stack responds to the rest. Both use the same training data and schedule, with trainable parameter counts outside the shared backbone matched within 5%. Both are trained to match simulated parcel movements and contacts for the full motion and its continuation. We clarified this comparison in Section 6 and Appendix C.2 (revised lines 454–461 and 1224–1233).

Original matched results (Table 10, revised lines 1386–1402). nRMSE is normalized response-prediction error. Lower is better.

Propagated stateFull-motion nRMSEContinuation nRMSE
Image-encoder features0.0840.127
Learned core0.0820.098

The paired continuation-error reduction is 0.029 (95% CI [0.019, 0.037]), while full-motion errors remain similar. The matched comparison supports the dedicated state's value for continuation under full physical supervision. Raw consistency tests removal of continuation physical targets, so it is not a baseline for Deep-OSG's supervised objective.

AuthorsOpenReview ↗

Reviewer d8pi — Comment 2

2. Ensemble selection and the 50% coverage choice. We test whether our selector chooses more useful training data than ensemble disagreement, and whether its gains depend on the coverage share. Shikhman [3] uses direct–composed discrepancy to diagnose learned dynamics. Unlike Plan2Explore [4], which uses disagreement among predictive models, our selector compares two prediction routes within one model. FATED [5] uses reinforcement learning and human feedback to discover and mitigate failures of pretrained vision and language models. Adaptive Stress Testing [6] searches for likely failure paths, including cases where a system fails and another does not. In contrast, StackProbe selects carrier motions near a reference plan where its two routes disagree about failure, then learns from the simulator's response. Section 2 clarifies these differences (revised lines 126–133).

Table 17 in Appendix C.9 compares coverage alone, our failure-set selector at 25%/50%/75% coverage, and a 5-member ensemble at 50% (revised lines 1695–1724). All settings use the same initial data and 64k total simulator calls, counting invalid attempts. Final predictors share the architecture, training schedule and held-out tests, learning from their selector's data. The ensemble trains and queries 5 predictors to select motions on which they disagree about continuation failure. Evaluation uses one member chosen in advance. We measure split false-safe, the fraction of simulated failures predicted safe after continuation.

Table 17: selection at 64k simulator calls on PLP. FS denotes continuation false-safe and FA denotes false-alarm, both in percent. Brackets give 95% confidence intervals for FS.

SelectorCoverage shareFS [95% CI]FA
Coverage only100%12.57 [11.54, 13.57]11.06
Failure-set25%8.19 [7.29, 9.21]9.47
Failure-set50%8.32 [7.50, 9.27]8.88
Failure-set75%9.04 [8.12, 10.07]8.71
5-member ensemble50%9.61 [8.67, 10.61]9.02

At 50% coverage, our selector reduces FS by 1.29 percentage points [0.34, 2.24] relative to the ensemble, with similar false alarms. Its gains over coverage alone also persist across the three coverage shares.

3. Prediction under unseen material properties. We test missed failures under hidden changes in friction, density or mass distribution, keeping each stack and motion fixed. We added this evaluation in Appendix C.6, Table 15 (revised lines 1531–1605).

We compare Direct, the full-response baseline, with StackProbe on 1,000 motion queries per simulation benchmark (PLP and ShapeStacks) from 200 independent stacks that are initially stable in every setting. Both models stay frozen, with the same inputs and decision thresholds. We vary object–object friction (0.30–0.70), object–carrier friction (0.40–0.90), density (125–500 kg/m³), and internal mass distribution separately. The mass-distribution change preserves total mass and shifts a box's center of mass by 10% of its width. The changed properties are not given to either model.

False-safe rates (%) under hidden material changes. False-safe is the fraction of simulated failures predicted safe using the full-motion prediction route.

Hidden physical conditionPLP DirectPLP StackProbeShapeStacks DirectShapeStacks StackProbe
Nominal11.837.8913.108.90
Object–object friction12.378.6413.939.93
Object–carrier friction13.399.6914.1910.25
Per-object density13.339.5214.6710.76
Internal mass distribution14.3610.8516.0312.25

Each material shift increases missed failures, while StackProbe retains lower rates on both benchmarks.

AuthorsOpenReview ↗

Reviewer d8pi — Comment 3

4. Real responses beyond collapse classification. We test whether the simulation-trained state predicts collapse and parcel movement under real carrier motion. StackProbe and Direct receive the same pre-prediction images, available records and recovered carrier motion, without retraining on these videos. We added this evaluation in Section 6 and Appendix C.8 (revised lines 514–520 and 1634–1693). Here we assess visible collapse and its onset. Detailed failure modes are evaluated in simulation using contact and support information (Appendix C.3). Failure radii are also evaluated in simulation, using alternative motions from comparable starting stacks (Appendix C.4).

Collecting data in this large-scale industrial logistics deployment (Figure 6), especially for manual-shaking trials, requires additional equipment, dedicated staff, and company approval. We use calibrated video to recover carrier motion as input and independently reconstruct parcel trajectories for evaluation. We evaluate 120 independent stacks after routine placement: 100 remain stable on a stationary carrier and 20 are manually shaken, with 10 collapses and 10 noncollapses in the common evaluation window. Collapse time is the onset of the fall or toppling that leaves the stack collapsed at the window's end.

Real-motion results from the added Table 16.

Evaluation measureDirectStackProbe
Correct stability decisions, stationary carrier93/10096/100
Correct collapse decisions, manual shaking7/109/10
Correct stability decisions, manual shaking8/109/10
Displacement error, shaken noncollapses (cm)3.62.5
Orientation error, shaken noncollapses (degrees)6.34.6
Collapse-time MAE (s)0.290.21

Motion errors are averaged over the 10 shaken noncollapses. Time MAE uses the same 7 collapses detected by both models. StackProbe improves both collapse decisions and parcel-motion prediction.

We hope these comparisons clarify the contribution and its practical value. We welcome further questions and will follow up promptly.

References

[1] Junfeng Chen and Kailiang Wu. “Deep-OSG: Deep learning of operators in semigroup.” Journal of Computational Physics, 493:112498, 2023. doi:10.1016/j.jcp.2023.112498.

[2] Andy Zeyi Liu, Haoran Sun, Lucas Baker, Randall Balestriero, and John Sous. “Semigroup-JEPA: Latent Dynamics Consistency for Zero-Shot Physics Generalization.” arXiv:2609.10464, 2026. arXiv:2609.10464.

[3] Lennon J. Shikhman. “Semigroup Consistency as a Diagnostic for Learned Physics Simulators.” AI4Physics Workshop at ICML, 2026. arXiv:2605.26324. doi:10.48550/arXiv.2605.26324.

[4] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. “Planning to Explore via Self-Supervised World Models.” In Proceedings of the 37th International Conference on Machine Learning, PMLR 119, pp. 8583–8592, 2020. Official proceedings.

[5] Som Sagar, Aditya Taparia, and Ransalu Senanayake. “Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models.” In Proceedings of the 41st International Conference on Machine Learning, PMLR 235, pp. 42999–43023, 2024. Official proceedings.

[6] Ritchie Lee, Ole J. Mengshoel, Anshu Saksena, Ryan W. Gardner, Daniel Genin, Joshua Silbermann, Michael Owen, and Mykel J. Kochenderfer. “Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning.” Journal of Artificial Intelligence Research, 69:1165–1201, 2020. doi:10.1613/jair.1.12190.

Reviewer_mgN9OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes StackProbe, a method for learning a state representation of stacked objects from visual observations and available records. Instead of directly predicting a stable/unstable label, StackProbe predicts how a stack responds to a planned physical perturbation. The key idea is continuation testing: the model predicts a motion response directly and also predicts it by advancing to an intermediate state and continuing from there. Disagreement between these two routes is used to select informative simulator tests, and simulator responses supervise both routes.

The paper also defines a failure radius around a motion plan and provides theoretical results connecting continuation-prediction error to failure decisions and local failure-radius error.

The method is evaluated on PLP and ShapeStacks using MuJoCo-generated perturbations. StackProbe improves response prediction, false-safe rate, false-alarm rate, failure timing, failure mode prediction, and finite-bank failure-radius estimation over the learned baselines. The authors also perform ablations on continuation supervision and acquisition.

Finally, the learned representation is evaluated on 1,562 industrial pallet-building episodes. Adding frozen StackProbe features to operating records increases AUPRC for terminal collapse prediction from 0.558 to 0.732.

Critical strengths
  1. The continuation-testing idea is novel and well validated. The strongest contribution is not simply the architecture. It is the idea of testing whether a learned physical state remains useful under alternative continuations. The ablation results support this mechanism. On held-out PLP data, verified continuation supervision reduces split nRMSE from 0.162 for the one-future control to 0.103, while direct errors remain similar. Nulling or parent-shuffling the propagated core strongly degrades performance. These experiments provide evidence that the model actually learns information that is useful for state propagation rather than only improving direct prediction. This is consequential because it directly tests the paper's main methodological claim.

  2. The experimental methodology is unusually careful. The evaluation is much stronger than a simple benchmark comparison. The paper separates continuation learning from acquisition, includes multiple baselines, reports confidence intervals and invalid cases, uses held-out motion families, and provides detailed simulation and data-generation protocols. The gains against the Direct baseline are also meaningful because that baseline shares much of the architecture and training setup. On PLP, for example, false-safe decreases from 11.72% for Direct to 7.80% for StackProbe with acquisition; ShapeStacks shows a similar improvement from 12.96% to 8.79%.

Critical weaknesses

The real-world experiment does not validate the central StackProbe capability:

The industrial evaluation predicts terminal pallet collapse after placement 10 using operating records combined with frozen learned features. It shows that StackProbe features transfer better than frozen V-JEPA and Direct features. This is useful evidence for representation transfer.

However, it does not demonstrate in the real world that StackProbe can predict the response to a planned perturbation. There is no real-world evaluation of alternative carrier motions, continuation prediction, failure time, failure mode, or failure radius. These are the capabilities that distinguish StackProbe from a conventional learned stability representation.

Therefore the statement that the work demonstrates real-world transfer is reasonable at the representation level, but the experiment provides only limited validation of the complete proposed method.

This matters because the paper motivates StackProbe as an alternative to physics simulation when physical parameters are unavailable. That claim would be substantially stronger with even a smaller real-world perturbation experiment.

Questions for authors
  1. Real-world validation The industrial experiment evaluates terminal collapse prediction rather than the motion-conditioned task used throughout the paper. Can the authors provide any real-world evaluation where a known carrier perturbation is applied and StackProbe predicts the resulting response or failure? Even a smaller experiment reporting failure/no-failure and failure time under several physical perturbations would materially change my assessment of the method's real-world validity.

  2. Physical parameter shift The motivation stresses that detailed physical parameters such as friction may be unavailable, while the main simulations use fixed material parameters. How does StackProbe perform when test-time friction, density, centre of mass, or contact parameters differ from the simulation settings used during training? In particular, I would like to see the false-safe rate under such shifts. This would help establish whether StackProbe learns transferable physical state information or partly learns the simulator's fixed material regime.

4 comments
AuthorsOpenReview ↗

Reviewer mgN9 — Comment 1

We thank the reviewer for the careful review and positive assessment of our continuation tests and matched controls. We appreciate your focus on real-response prediction and reliability under material changes, which help assess the learned state's value outside simulation. For your concerns, we have revised the paper as follows. (1) We added a video-based evaluation for collapse decisions and parcel movement under real carrier motion. StackProbe makes more correct decisions and predicts collapse time and parcel movement more accurately than Direct, our matched baseline that predicts the response to the full motion without resuming from an intermediate state. (2) We added tests with changes in friction, density and mass distribution. Errors increase under these changes, while StackProbe still misses fewer failures than Direct.

1. Predicting real responses and collapse. We test whether the simulation-trained state predicts collapse and parcel movement under real carrier motion. StackProbe and Direct receive the same pre-prediction images, available records and recovered carrier motion, with all weights frozen. We added this evaluation in Section 6 and Appendix C.8 (revised lines 514–520 and 1634–1693).

Collecting data in this large-scale industrial logistics deployment (Figure 6), especially for manual-shaking trials, requires additional equipment, dedicated staff, and company approval. We use calibrated video to recover carrier motion as input and independently reconstruct parcel trajectories for evaluation. We evaluate 120 independent stacks after routine placement: 100 remain stable on a stationary carrier and 20 are manually shaken, with 10 collapses and 10 noncollapses in the common evaluation window. Collapse time is the onset of the fall or toppling that leaves the stack collapsed at the window's end.

Real-motion results from the added Table 16.

Evaluation measureDirectStackProbe
Correct stability decisions, stationary carrier93/10096/100
Correct collapse decisions, manual shaking7/109/10
Correct stability decisions, manual shaking8/109/10
Displacement error, shaken noncollapses (cm)3.62.5
Orientation error, shaken noncollapses (degrees)6.34.6
Collapse-time MAE (s)0.290.21

Motion errors are averaged over the 10 shaken noncollapses. Time MAE uses the same 7 collapses detected by both models. StackProbe improves both collapse decisions and parcel-motion prediction.

AuthorsOpenReview ↗

Reviewer mgN9 — Comment 2

2. Performance under unseen material properties. We added a comparison that holds each stack and motion fixed while changing hidden physical properties (Appendix C.6, Table 15, revised lines 1531–1605). It uses 1,000 motion queries per benchmark from 200 independent stacks initially stable in every setting. Both models, inputs and thresholds stay fixed. Object–object friction (0.30–0.70), object–carrier friction (0.40–0.90), density (125–500 kg/m³), and internal mass distribution change separately. The mass-distribution change preserves total mass and shifts a box's center of mass by 10% of its width. The changed properties are not supplied to the models.

False-safe rates (%) under hidden material changes. False-safe is the fraction of simulated failures predicted safe using the full-motion prediction route.

Hidden physical conditionPLP DirectPLP StackProbeShapeStacks DirectShapeStacks StackProbe
Nominal11.837.8913.108.90
Object–object friction12.378.6413.939.93
Object–carrier friction13.399.6914.1910.25
Per-object density13.339.5214.6710.76
Internal mass distribution14.3610.8516.0312.25

Each material shift increases missed failures, while StackProbe retains lower rates on both benchmarks.

To examine the fixed material setting, we also compare changes in material properties and placement under the same carrier motion (Appendix C.6, Table 14, revised lines 1531–1570). Placement varies by up to 20% of the object's shorter side and ±10° in yaw, with the material ranges given above.

Changes in simulated failure labels (%). This measures how often the physical outcome changes between paired simulations.

Changed factorPLPShapeStacks
Object–object friction2.83.5
Object–carrier friction4.14.8
Per-object density5.56.7
Placement21.023.8

Within these ranges, the larger effect of placement supports studying stack geometry under controlled material settings. We also test the added value of measured mass for industrial collapse prediction by removing this input while keeping frozen StackProbe features, other records and readout training fixed (Table 3 and Appendix C.5, revised lines 494–510 and 1521–1529).

Mass input with frozen StackProbe featuresReal-task AUPRC [95% CI]
With mass0.732 [0.684, 0.778]
Without mass0.729 [0.681, 0.776]

The paired mass benefit is 0.003 [−0.006, 0.012], indicating little additional value given the visual features and other records.

We hope these results address your questions about real responses and material changes. We welcome further discussion and will follow up promptly.

Thank you for the added real world response and the additional test with changed material properties. They address my concerns. I will change my score to clear acceptance and keep my confidence.

Thank you for taking the time to review our response. We're glad the additional experiments addressed your concerns.

Reviewer_CcjtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presented StackProbe which is a model predicting how stacked objects respond to planned carrier motions and when they fail instead of just stable/unstable labeling. Its key trick is to split a movement in two and check whether predicting the whole motion matches predicting the second half from the predicted midpoint. When those disagree, the system runs a physics simulation to find out the truth and learn from it. This reduced missed collapses by over 32% in tests and improved real warehouse palletizing predictions.

Critical strengths

Its main strength is thus physically grounded, failure-aware state learning, not just better prediction accuracy. Its continuation-testing mechanism, it is stronger than generic latent-consistency training because the signal is tied to physics. outcomes, failure bits, monitor margins, and eventually failure time/mode/radius. The paper also backs this with theory connecting response error to failure decisions and finite-bank radius error, empirically, it lowers missed failures by over 32% versus learned baselines and improves real palletizing AUPRC from .558 to .732.

Critical weaknesses

The main weakness is the challenge of sim-to-real gaps caused by the simulator verification. Strong feedback requirements comparing to simple labels might be more susceptible to sim-to-real gap.

Questions for authors

The authors should explain how to handle sim-to-real gap or robustness with respect to noisy labels.

1 comment
AuthorsOpenReview ↗

Reviewer Ccjt — Comment 1

We thank the reviewer for the careful review and positive assessment of our physically grounded state learning and continuation supervision. We appreciate your question about whether detailed feedback is more vulnerable to the sim-to-real gap than simple labels, a key test of its value for real prediction. For your concerns, we have revised the paper as follows. (1) We compared detailed, noisy detailed and binary simulator feedback to test their value for real collapse prediction. The results show little loss from added noise and an advantage over simple labels. (2) We clarified how simulation-trained features are reused with ordinary real collapse labels. The original industrial results show that these frozen features improve prediction.

1. Does detailed feedback remain useful when simulation is imperfect? We test whether errors in simulator feedback reduce the learned features' value for real collapse prediction.

We added this comparison in Appendix C.7 (revised lines 1607–1630). The three representations learn from the same images and independently sampled motions, using detailed physical feedback, the same feedback with small, time-correlated errors, or binary failure labels from the original simulator records. Detailed feedback includes poses, velocities, contacts and the history used to check for failure. For scale, position noise has a standard deviation of 2% of the carrier's reference length. All three retain continuation training and use the same data, representation network and training schedule, with prediction-head capacity matched. We then freeze each representation and train a collapse predictor using the same real data and procedure.

Results of the added supervision comparison.

Simulator supervisionReal-task AUPRC [95% CI]
Detailed physical feedback0.719 [0.669, 0.767]
Noisy detailed feedback0.713 [0.662, 0.762]
Binary failure labels0.681 [0.630, 0.730]

In this comparison, noise reduces AUPRC by 0.006 (95% CI [−0.003, 0.015]), while noisy detailed feedback exceeds failure labels by 0.032 [0.012, 0.053]. These results support the value of detailed feedback despite the specified supervision errors.

2. How the simulated responses help the real task. We freeze the simulation-trained representation and train a collapse predictor with its features and operating records as inputs and ordinary collapse labels as targets. We clarified these steps in Section 6 and Appendix C.5 (revised lines 486–507 and 1514–1515).

Original industrial results on 1,562 episodes (Table 3). The models in point 1 are separately trained on independently sampled motions. All rows use the same predictor architecture and training procedure. Direct is our matched baseline that predicts the full response without resuming from an intermediate state. V-JEPA denotes pretrained V-JEPA 2.1 video features.

Real-task inputsAUPRC [95% CI]
Records alone0.558 [0.507, 0.614]
Records + frozen V-JEPA0.644 [0.592, 0.696]
Records + frozen Direct0.672 [0.621, 0.721]
Records + frozen StackProbe0.732 [0.684, 0.778]

These original results show that features learned in simulation improve real collapse prediction.

We hope these comparisons clarify the value of detailed feedback for real prediction despite supervision errors. We welcome further questions and will follow up promptly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment

We thank all three reviewers for their careful reading and positive assessments of our physical supervision, continuation tests, matched controls and representation transfer. We appreciate your focus on the contribution beyond prior work, the assumptions behind simulator-based learning, and validation of physical predictions in the real world. For your concerns, we have revised the paper as follows, adding 6 evaluations and 4 new tables.

1. Contribution beyond prior work (Reviewer d8pi). We revised the contribution statement in Section 1 (page 2) and the discussion of prior work in Section 2 (page 3) to explain what our state design and test selection add to established supervision methods. We clarified the state design in Section 4.1 (page 4) and its original matched evaluation in Section 6 (page 9), and added comparisons with ensemble selection and independent sampling in Appendix C.9 (page 32). The dedicated state improves continuation prediction under matched physical supervision. Test selection reduces missed failures at a fixed simulator budget across different mixes of selected and independently sampled tests.

2. Simulator assumptions and physical generalization (Reviewers mgN9 and d8pi). In Appendix C.6 (pages 29–30), we added comparisons of how material properties and placement affect stability, and tested prediction under unseen friction, density and mass distributions. StackProbe still misses fewer failures than the matched baseline under these shifts. We also added a mass ablation in Section 6 (page 10), showing little change in real collapse prediction when measured mass is removed.

3. The sim-to-real gap and representation transfer (Reviewer Ccjt). We added a comparison in Appendix C.7 (pages 30–31) to test whether detailed simulator feedback remains useful for real collapse prediction when it contains errors. Features learned from noisy detailed feedback outperform those learned from binary labels. We also clarified how frozen simulation-trained features support a collapse predictor trained on real labels in Appendix C.5 (pages 28–29), and moved the original industrial comparison into Section 6 (page 10).

4. Real-world physical validity (Reviewers mgN9 and d8pi). We added an evaluation of 120 independent stacks at the operational industrial site, reported in Appendix C.8 (pages 31–32) and summarized in Section 6 (page 10). Using carrier motion recovered from video as input, frozen StackProbe predicts parcel movement, collapse and collapse timing more accurately than the matched baseline. This extends the real-world evidence from feature transfer to physical response prediction under carrier motion.

The revised paper makes the methodological contribution clearer and strengthens the evidence for its robustness and real-world value. We welcome further discussion and will follow up promptly.

Area_Chair_dZg8OpenReview ↗

please engage

Dear reviewers,

Please join the discussion with the authors.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.