- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates conformal prediction for trajectory world models in settings where both model inputs and calibration targets are corrupted by temporally correlated observation noise. Such noise can partially cancel within the observed prediction residuals, leading standard conformal prediction to underestimate errors relative to the true future states. To address this issue, the authors propose Noise-Informed Conformal Prediction (NICP), which propagates state-estimator covariance information and leverages it to augment the calibration residuals. The paper establishes finite-sample marginal coverage under stated assumptions and evaluates the method on three trajectory world models, simulated humanoid and quadruped tasks, and a real-world quadruped dataset.
Critical strengths
-
The paper addresses an important problem with a well-motivated methodological response. It tackles calibration against latent true states when clean calibration targets are unavailable. The cancellation mechanism in Equations (10)–(11) concretely explains why calibrating on observed residuals can be misleading. NICP makes effective use of estimator-provided covariance information without requiring changes to the predictor’s architecture or training procedure.
-
The empirical support is broad and informative. The evaluation spans three trajectory world models, multiple simulated tasks, and real robot data. In Table 1, NICP achieves average coverage close to the 90% target while producing substantially smaller prediction sets than the Robust CP baseline. The covariance-mapping, prediction-horizon, and estimator-consistency studies further help explain the method’s behavior and limitations. These results provide meaningful evidence that the approach is useful beyond a single model or task.
Critical weaknesses
-
The link between the coverage theorem and the practical algorithm requires clearer establishment. Theorem 1 establishes covariance dominance, whereas Theorem 2 additionally requires centered Gaussian error marginals; similarly, the extension in Appendix B requires a shared centered elliptical family. These distributional properties are not implied solely by conservative estimator covariances or by Gaussian augmentation conditioned on a randomly constructed covariance matrix, particularly under the default K=1 implementation. Figure 9 offers useful empirical support for the weaker coverage condition on Humanoid Crouch, but its scope remains limited. Since coverage guarantees are central to the contribution, a clearer account of which practical settings satisfy the required conditions would materially strengthen the paper.
-
Aggregate coverage can obscure failures in individual state quantities. The main results average coverage across state dimensions and time steps. However, Table 8 reports only 87.18 ± 0.34% coverage for the x-axis linear velocity, despite an aggregate result of 92.30 ± 0.15% for the corresponding task and model. Although the authors explicitly acknowledge this failure, it nonetheless illustrates why aggregate estimator consistency and average coverage cannot guarantee reliable coverage for each state quantity. This distinction is important for the motivating control applications, in which a particular state variable may be critical. The main claims should more explicitly acknowledge this limitation.
Questions for authors
-
For the default K=1 implementation, which properties support the required dominance of true-error over augmented-error scores beyond covariance dominance alone? Please clarify which theoretical conditions are expected to hold in the evaluated settings and which remain empirical approximations, including the implications of the random auxiliary covariance for the marginal error distribution.
-
How prevalent is the dimension-wise undercoverage illustrated in Table 8 across the remaining tasks, particularly the real-world dataset? Could the authors provide the corresponding dimension-wise coverage results and explain what, if any, diagnostics can identify unreliable estimator covariances when ground-truth states are unavailable? These answers would directly inform my assessment of the practical scope of the coverage claims.
4 comments
W2/Q2.1: About a more comprehensive presentation and discussion of the undercoverage observed in individual state dimensions.
A2.1: Table 2 below summarizes the number of NICP undercoverage cases across all tasks, models, and state dimensions, together with the average coverage across these cases and whether the state estimator underestimates uncertainty in the corresponding dimensions. Such underestimation is indicated by an average NEES greater than 1 in the corresponding dimension. We observe that every individual dimension with undercoverage also exhibits uncertainty underestimation by the state estimator, consistent with our discussion of the results in Table 8 of Appendix E. Furthermore, across these undercoverage cases, Vanilla CP achieved average coverage rates of 66.46%, 74.81%, and 28.55% on the three datasets, respectively, all below the corresponding NICP results. This indicates that imperfect covariance information can also help improve coverage.
In addition, as a commonly used metric for assessing estimator consistency, an average NEES below the reference value for the full state vector does not imply that the dimension-wise average NEES values are all below their respective reference values, and estimator consistency can indeed vary across dimensions. In this work, we did not specifically focus on calibrating the estimator to achieve consistency in each dimension. Our primary focus was on evaluating NICP when the estimator behaves as expected and examining the effects of changes in estimator consistency. As discussed in Appendix E, the dimension-wise and aggregate results exhibit a consistent pattern with respect to estimator consistency. We will clarify this point more explicitly in the Conclusion and Limitations section of the submission.
Table 2: Summary of dimension-wise NICP undercoverage across all evaluated tasks and models. The target coverage is 90%. Average NICP coverage is reported over the undercoverage cases only. Average NEES values above 1 indicate that the estimator-reported covariances underestimate uncertainty.
| Dataset | Total Cases | Undercoverage Cases | Avg. NEES > 1 | Avg. NICP Coverage |
|---|---|---|---|---|
| Humanoid | 24/108 | 24/24 | 86.57% | |
| Quadruped | 10/108 | 10/10 | 87.83% | |
| Go2 | 9/36 | 9/9 | 80.15% |
Q2.2: If any diagnostics can identify unreliable estimator covariances when ground-truth states are unavailable?
A2.2: Our framework builds on state estimation systems that already exist in robotic systems and have been extensively studied. Many existing studies [1–4] focus on assessing or improving the consistency of state estimation systems without ground truth and can serve as useful references. Compared with vanilla CP, which requires ground-truth state values for calibration, our framework offers an alternative source of information. Furthermore, even when the covariance estimates underestimate the actual uncertainty, prediction sets constructed using our framework can still achieve higher coverage than those obtained by applying vanilla CP directly to noisy observations without accounting for observation noise. This is supported by the results reported in A2.1.
[1] Griebel T, et al. Self-Assessment and Monitoring Module for Tracking Algorithms in the Stone Soup Framework. IEEE MFI, 2025.
[2] Chen Z, et al. Kalman filter auto-tuning with consistent and robust Bayesian optimization. IEEE TAES, 2024.
[3] Piché R. Online tests of Kalman filter consistency. Int. J. Adapt. Control Signal Process., 2016.
[4] Vega-Brown W, Roy N. CELLO-EM: Adaptive sensor models without ground truth. IEEE/RSJ IROS, 2013.
Thank you for your reply and clarification. This has resolved most of my questions, so I will keep my positive rating.
Dear Reviewer 87eR,
Thank you for your positive feedback on our response. We have incorporated the discussion into the latest version of our manuscript for your reference. We sincerely appreciate your constructive suggestions, which have helped us improve our work. Thank you again!
Dear Reviewer 87eR,
Thank you for your careful review and thoughtful suggestions. We have carefully addressed your comments in our responses and appreciate your feedback in helping us improve our work. We would be happy to discuss any remaining questions and welcome any further suggestions you may have.
W1/Q1: About the additional conditions needed beyond covariance dominance to support the required dominance of true-error scores over augmented-error scores.
A1: In the evaluated settings, the fundamental condition for achieving the required score dominance in a specific dimension is Δ:=supr≥0[HA(r)−HT(ηr)]≤0, where HT(r)=P(∣etrue∣/Var(etrue)≤r), HA(r)=P(∣eaug∣/Var(eaug)≤r) and η=Var(eaug)/Var(etrue). We can clearly see that covariance dominance, which can be expressed as η≥1, is not the only determining factor; the error distributions also need to be considered. The distributional conditions described in Theorem 2 and Corollary 1 allow η≥1 to directly imply Δ≤0. Outside these conditions, this implication is not guaranteed, and Δ>0 may occur even when η>1. However, in such cases, we can still provide a lower bound on the coverage probability considered in Theorem 2, namely 1−α−Δ. This follows from Equations (49)–(54) in Appendix B.
The distributions of etrue and eobs depend on the task, but we can adjust the distribution of the augmentation term ξ to help the distributions of etrue and eaug=eobs+ξ satisfy the required distributional conditions or become more similar in shape, potentially reducing Δ. As described in lines 219–220, Gaussian sampling is not essential to the NICP framework; we only require ξ to be drawn from a zero-mean distribution with the constructed covariance. For example, we can replace the current Gaussian sampling in Algorithm 1 with ξi=Sidi, where Si is sampled uniformly from {+1,−1} and di is the change in the model output induced by the input perturbation in the preceding step. In this case, if etrue, eobs, and di each have centered Gaussian distributions, and di is independent of eobs, then the distributional conditions of Theorem 2 are satisfied. If these conditions hold after factoring out a common random factor that is independent of the remaining components, then the distributional conditions of Corollary 1 are satisfied.
We conducted an additional ablation study of different sampling schemes for the augmentation term in Algorithm 1 on the Humanoid Crouch task across three different models. The results are reported in Table 1 below. The different sampling schemes produce similar experimental results, supporting the flexibility of our method in the choice of sampling scheme. In addition, the sampling scheme alone does not automatically ensure that these distributional properties hold; whether they hold also depends on the actual error distributions, and the corresponding assumptions remain empirical approximations in the evaluated settings. The earlier analysis can serve as a reference regarding the impact of these approximations on coverage. We will add the related discussion and experimental results to our submission and, in future work, investigate how to select suitable sampling schemes for different settings to help satisfy these conditions.
Table 1: Experimental results for NICP using different augmentation sampling schemes on the Humanoid Crouch task across three trajectory world models (mean ± std.).