- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes STORC, a residual correction framework for robot video prediction that uses newly observed ground-truth frames to improve cached future predictions without additional backbone inference or online gradient updates. The method consists of three main components: (1) Local Analytic Residual Modeling (LARM), which estimates local relationships between prediction changes and residual changes through spatially aggregated ridge regression; (2) Causal Predictive Shrinkage (CPS), which evaluates the reliability of residual propagation against residual persistence using a held-out historical transition and adaptively adjusts the propagation strength; and (3) a dual-resolution correction mechanism that combines regional and native-resolution residual estimates to balance estimation stability and spatial detail preservation. The authors evaluate STORC across five prediction backbones and nine test splits from BridgeData, RoboNet, and BAIR. The results show improvements in prediction accuracy and computational efficiency over the evaluated test-time adaptation baselines, with an average MSE reduction of 34.8% relative to the strongest evaluated adaptation baseline in each setting.
Critical strengths
Strength 1: Well-motivated and efficient feedback-based correction The paper addresses a practical problem in robot video prediction: efficiently improving cached forecasts using newly observed feedback. The proposed framework is conceptually simple and well-motivated, combining local residual modeling, historical reliability assessment, and dual-resolution correction in a coherent manner. Importantly, STORC requires neither additional backbone inference nor online gradient updates, making it a computationally efficient approach to prediction correction. This simplicity and efficiency are valuable for applications where repeated model inference is computationally expensive.
Strength 2: Extensive empirical evaluation and consistent performance improvements The paper provides an extensive evaluation across five prediction backbones and nine test splits from three robot video datasets, covering both standard and distribution-shifted conditions. STORC consistently outperforms the evaluated parameter-adaptation baselines in prediction accuracy while maintaining low computational overhead. Furthermore, the component ablations and mechanism analyses provide evidence that LARM and CPS offer additional improvements beyond basic residual correction. These results support the effectiveness and practical applicability of the overall framework across different prediction architectures and evaluation conditions.
Critical weaknesses
Weaknesses 1: Insufficient Comparisons with Existing Residual Correction Methods The main concern is the lack of comparisons with representative existing residual correction methods. The paper identifies limitations of residual persistence and temporal extrapolation, and proposes LARM and CPS to address them. However, the main experiments in Table 1 primarily compare STORC with test-time adaptation methods rather than closely related residual correction approaches. Consequently, the reported 34.8% MSE reduction demonstrates the effectiveness of the overall framework against the evaluated adaptation baselines but does not establish its relative advantage within the residual correction literature. Furthermore, Table 3 shows that the residual-persistence-based variant with regional fusion already achieves most of the overall MSE improvement, while LARM and CPS provide relatively modest additional gains (1.3%–5.8%) across the eight evaluated settings. Although these results support the effectiveness of the proposed components, comparisons with representative existing residual correction methods under matched evaluation conditions are needed to better assess the significance of their incremental contributions.
Weaknesses 2: Contribution of Prediction-Driven Residual Propagation A key motivation for LARM is that cached future predictions provide useful information for estimating residual evolution. However, the ablation study in Table 3 primarily evaluates LARM against regional residual persistence, which does not model temporal residual changes. As a result, it remains unclear whether the observed improvements specifically arise from exploiting future prediction trajectories or from temporal residual extrapolation more generally. A comparison with a simple residual-only temporal extrapolation method, under the same spatial correction and feedback protocol, would provide more direct evidence for the distinctive contribution of prediction-driven residual modeling.
Weaknesses 3: Limited Justification for the CPS Shrinkage Strategy The specific shrinkage formulation used in CPS lacks sufficient justification. While the proposed strategy improves performance under distribution shifts, Table 3 shows slight MSE degradation on the standard splits, indicating that its benefits are not consistent across all evaluation conditions. Although the paper provides evidence that historical confidence is associated with future propagation benefits, it remains unclear whether the proposed shrinkage formulation is preferable to simpler propagation control strategies. Comparisons with alternatives such as fixed shrinkage or binary gating based on the same historical validation would help establish the necessity and effectiveness of the specific CPS design.
Questions for authors
Question 1: Comparison with Existing Residual Correction Methods (Primary Concern) The paper aims to address limitations of existing residual correction approaches, yet the main experiments primarily compare STORC with test-time adaptation methods. Could the authors provide comparisons with representative existing residual correction methods under the same causal feedback protocol, or explain why such comparisons are not feasible? In particular, how does STORC compare with these methods in terms of prediction accuracy and computational efficiency? This is my primary concern and the main factor influencing my current rating of 2 (Weak Rejection). If the authors can convincingly address this concern and provide sufficient evidence to establish the incremental contribution of STORC over closely related methods, I would consider increasing my rating to 3 (Weak Acceptance).
Question 2: Contribution of Prediction-Driven Residual Propagation Could the authors compare LARM with a simple temporal residual extrapolation baseline that relies only on historical residual changes, without using future prediction trajectories? A comparison under identical spatial correction and feedback settings would help clarify whether the observed improvements specifically result from exploiting future prediction information rather than temporal extrapolation itself.
Question 3: Justification of the CPS Shrinkage Strategy What motivates the specific shrinkage formulation adopted in CPS? Could the authors compare it with simpler alternatives, such as fixed shrinkage or binary gating based on historical validation? Given the mixed effects of CPS across standard and distribution-shifted settings in Table 3, these comparisons would help clarify whether the proposed adaptive shrinkage strategy provides meaningful advantages over simpler propagation control mechanisms.
2 comments
Q3. Motivation and Comparison of CPS Shrinkage
CPS compares propagation with residual persistence on the same held-out transition (Eq. (4)) and converts their relative error reduction into a continuous shrinkage weight (Eq. (5), Section 3.3). A small historical advantage therefore produces a small correction rather than full propagation, while the observed residual anchor is preserved. We added Binary Gating, which uses identical LARM errors and the same 16×16 grid, but enables propagation only where E_prop < E_hold. Fixed coefficients 0.3/0.5/0.7 and global adaptive shrinkage are also included in Appendix A.11, Table 16; the key comparison is in main-text Table 5.
On shifted RoboNet splits, CPS lowers mean MSE by 6.67–7.23% versus the best tested fixed coefficient and 3.08–4.63% versus global shrinkage. It improves over binary gating by 0.39–4.58% across all four splits. Unshrunk LARM remains best on Standard, and binary gating is slightly better at steps 5–6 there. The results support continuous validation-based shrinkage particularly under shifts, with the Standard trade-off retained in the reported results.
| RoboNet split | LARM (no shrinkage) | Binary Gating | CPS |
|---|---|---|---|
| Standard | 58.38 | 58.92 | 58.69 |
| Dynamics | 115.17 | 111.22 | 108.49 |
| Scene | 108.16 | 101.64 | 96.99 |
| Joint | 306.92 | 291.40 | 280.07 |
Same fixed-prefix protocol; fixed-coefficient and global-shrinkage results are in Table 16.
References
[1] Im and Kwon. COSA: Context-aware Output-Space Adapter for Test-Time Adaptation in Time Series Forecasting. ICLR, 2026.
[2] Dai et al. Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Models. NeurIPS, 2026.
Thank you for recognizing the motivation and efficiency of our framework and for suggesting targeted comparisons. We have added ORCA, residual-only extrapolation, and binary-gating experiments to address your questions.
Q1. Comparison with Existing Residual Correction Methods
We now include ORCA (NeurIPS 2026) [2] in main-text Table 1, alongside COSA (ICLR 2026) [1]. Both perform output-space residual correction: COSA uses a gated linear adapter, and ORCA learns errors conditioned on historical inputs and base forecasts. STORC uses the latest observed residual as an explicit anchor, analytically propagates residual changes along cached predictions, and shrinks only the propagation increment, without online gradient updates. Native and Regional Persistence are internal ablations of this framework; their strong performance establishes the value of basic correction, while LARM/CPS comparisons isolate additional gains.
With causal feedback at p=2,4,6,8, STORC has lower MSE and LPIPS than ORCA in all 45 backbone–split combinations. Average relative reductions are approximately 34.00% MSE and 35.92% LPIPS, computed per setting from Table 1’s displayed values and then averaged equally. Appendix A.2, Table 6 documents the official-core implementation, online updates, replay, state resets, and global configuration selection. ORCA requires online adapter updates, replay, and routing, whereas STORC uses analytic correction without online gradient updates.
Q2. Prediction-Driven versus Residual-Only Extrapolation
We added Res-Extrap and Res+CPS to main-text Table 5, with definitions in Appendix A.11. Res-Extrap anchors at the latest residual and linearly extrapolates the mean historical residual increment, without future prediction trajectories. It retains the same regional/native fusion, clipping, frozen TAU, and 256 clips per RoboNet split. All methods use only the prefix through p=4 and are evaluated over h=5:10 without later feedback. Res+CPS computes its own held-out propagation error using the same validation split, rather than borrowing LARM’s confidence.
LARM reduces suffix-average MSE by 4.71–13.04% versus Res-Extrap across the four splits. With CPS applied to both, STORC improves over Res+CPS by 6.01–14.50%. These comparisons support prediction-driven propagation over the specified residual-only extrapolator; all individual results appear in Table 5.
Mean MSE over h=5–10; compare Res-Extrap with LARM, and Res+CPS with STORC.