acceptodds
Under review as a conference paper at ICLR 2027

Noise-Informed Conformal Prediction for Trajectory World Models

Abstract

Trajectory world models are predictive models that learn complex dynamics of diverse robotic systems from low-level states. Quantifying the uncertainty of trajectory world models is critical to ensuring the safety and reliability of downstream control and planning tasks. Conformal prediction (CP) is a model-agnostic uncertainty quantification framework with formal calibration guarantees, making it widely applicable to various learning-based dynamics models. However, trajectory data collected from robotic systems are inevitably affected by unbounded and temporally correlated observation noise, leading to an underestimation of the model’s true prediction error. In such scenarios, existing methods cannot provide calibrated prediction sets for the system’s true future states. To address this challenge, we propose Noise-Informed Conformal Prediction (NICP), which constructs an auxiliary covariance matrix by applying a covariance mapping to the noise information provided by the state estimator and then draws samples according to it to augment the observed prediction errors, thereby mitigating the underestimation of the model's true prediction error. Moreover, under certain conditions, we prove that the proposed method provides prediction sets with finite-sample marginal coverage guarantees. Experiments across different trajectory world models and both simulated and real-world robotic tasks show that our method can provide calibrated uncertainty estimates under noisy observations, while existing CP methods suffer from significant underestimation or overestimation in this setting.

Primary area:
applications to robotics, autonomy, planning
Keywords:
uncertainty quantification, conformal prediction, world model, robotics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_87eROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates conformal prediction for trajectory world models in settings where both model inputs and calibration targets are corrupted by temporally correlated observation noise. Such noise can partially cancel within the observed prediction residuals, leading standard conformal prediction to underestimate errors relative to the true future states. To address this issue, the authors propose Noise-Informed Conformal Prediction (NICP), which propagates state-estimator covariance information and leverages it to augment the calibration residuals. The paper establishes finite-sample marginal coverage under stated assumptions and evaluates the method on three trajectory world models, simulated humanoid and quadruped tasks, and a real-world quadruped dataset.

Critical strengths
  1. The paper addresses an important problem with a well-motivated methodological response. It tackles calibration against latent true states when clean calibration targets are unavailable. The cancellation mechanism in Equations (10)–(11) concretely explains why calibrating on observed residuals can be misleading. NICP makes effective use of estimator-provided covariance information without requiring changes to the predictor’s architecture or training procedure.

  2. The empirical support is broad and informative. The evaluation spans three trajectory world models, multiple simulated tasks, and real robot data. In Table 1, NICP achieves average coverage close to the 90% target while producing substantially smaller prediction sets than the Robust CP baseline. The covariance-mapping, prediction-horizon, and estimator-consistency studies further help explain the method’s behavior and limitations. These results provide meaningful evidence that the approach is useful beyond a single model or task.

Critical weaknesses
  1. The link between the coverage theorem and the practical algorithm requires clearer establishment. Theorem 1 establishes covariance dominance, whereas Theorem 2 additionally requires centered Gaussian error marginals; similarly, the extension in Appendix B requires a shared centered elliptical family. These distributional properties are not implied solely by conservative estimator covariances or by Gaussian augmentation conditioned on a randomly constructed covariance matrix, particularly under the default K=1 implementation. Figure 9 offers useful empirical support for the weaker coverage condition on Humanoid Crouch, but its scope remains limited. Since coverage guarantees are central to the contribution, a clearer account of which practical settings satisfy the required conditions would materially strengthen the paper.

  2. Aggregate coverage can obscure failures in individual state quantities. The main results average coverage across state dimensions and time steps. However, Table 8 reports only 87.18 ± 0.34% coverage for the x-axis linear velocity, despite an aggregate result of 92.30 ± 0.15% for the corresponding task and model. Although the authors explicitly acknowledge this failure, it nonetheless illustrates why aggregate estimator consistency and average coverage cannot guarantee reliable coverage for each state quantity. This distinction is important for the motivating control applications, in which a particular state variable may be critical. The main claims should more explicitly acknowledge this limitation.

Questions for authors
  1. For the default K=1 implementation, which properties support the required dominance of true-error over augmented-error scores beyond covariance dominance alone? Please clarify which theoretical conditions are expected to hold in the evaluated settings and which remain empirical approximations, including the implications of the random auxiliary covariance for the marginal error distribution.

  2. How prevalent is the dimension-wise undercoverage illustrated in Table 8 across the remaining tasks, particularly the real-world dataset? Could the authors provide the corresponding dimension-wise coverage results and explain what, if any, diagnostics can identify unreliable estimator covariances when ground-truth states are unavailable? These answers would directly inform my assessment of the practical scope of the coverage claims.

4 comments

Dear Reviewer 87eR,

Thank you for your careful review and thoughtful suggestions. We have carefully addressed your comments in our responses and appreciate your feedback in helping us improve our work. We would be happy to discuss any remaining questions and welcome any further suggestions you may have.

W1/Q1: About the additional conditions needed beyond covariance dominance to support the required dominance of true-error scores over augmented-error scores.

A1: In the evaluated settings, the fundamental condition for achieving the required score dominance in a specific dimension is Δ:=sup⁡r≥0[HA(r)−HT(ηr)]≤0\Delta:=\sup_{r\ge0}\left[H_A(r)-H_T(\sqrt{\eta} r)\right]\leq0, where HT(r)=P(∣etrue∣/Var⁡(etrue)≤r)H_T(r)=P\left(|e^{\mathrm{true}}|/\sqrt{\operatorname{Var}(e^{\mathrm{true}})}\le r\right), HA(r)=P(∣eaug∣/Var⁡(eaug)≤r)H_A(r)=P\left(|e^{\mathrm{aug}}|/\sqrt{\operatorname{Var}(e^{\mathrm{aug}})}\le r\right) and η=Var⁡(eaug)/Var⁡(etrue)\eta=\operatorname{Var}(e^{\mathrm{aug}})/\operatorname{Var}(e^{\mathrm{true}}). We can clearly see that covariance dominance, which can be expressed as η≥1\eta\ge1, is not the only determining factor; the error distributions also need to be considered. The distributional conditions described in Theorem 2 and Corollary 1 allow η≥1\eta\ge1 to directly imply Δ≤0\Delta\leq0. Outside these conditions, this implication is not guaranteed, and Δ>0\Delta>0 may occur even when η>1\eta>1. However, in such cases, we can still provide a lower bound on the coverage probability considered in Theorem 2, namely 1−α−Δ1-\alpha-\Delta. This follows from Equations (49)–(54) in Appendix B.

The distributions of etruee^{\mathrm{true}} and eobse^{\mathrm{obs}} depend on the task, but we can adjust the distribution of the augmentation term ξ\xi to help the distributions of etruee^{\mathrm{true}} and eaug=eobs+ξe^{\mathrm{aug}}=e^{\mathrm{obs}}+\xi satisfy the required distributional conditions or become more similar in shape, potentially reducing Δ\Delta. As described in lines 219–220, Gaussian sampling is not essential to the NICP framework; we only require ξ\xi to be drawn from a zero-mean distribution with the constructed covariance. For example, we can replace the current Gaussian sampling in Algorithm 1 with ξi=Sidi\xi_i=S_id_i, where SiS_i is sampled uniformly from {+1,−1}\lbrace +1,-1\rbrace and did_i is the change in the model output induced by the input perturbation in the preceding step. In this case, if etruee^{\mathrm{true}}, eobse^{\mathrm{obs}}, and did_i each have centered Gaussian distributions, and did_i is independent of eobse^{\mathrm{obs}}, then the distributional conditions of Theorem 2 are satisfied. If these conditions hold after factoring out a common random factor that is independent of the remaining components, then the distributional conditions of Corollary 1 are satisfied.

We conducted an additional ablation study of different sampling schemes for the augmentation term in Algorithm 1 on the Humanoid Crouch task across three different models. The results are reported in Table 1 below. The different sampling schemes produce similar experimental results, supporting the flexibility of our method in the choice of sampling scheme. In addition, the sampling scheme alone does not automatically ensure that these distributional properties hold; whether they hold also depends on the actual error distributions, and the corresponding assumptions remain empirical approximations in the evaluated settings. The earlier analysis can serve as a reference regarding the impact of these approximations on coverage. We will add the related discussion and experimental results to our submission and, in future work, investigate how to select suitable sampling schemes for different settings to help satisfy these conditions.

Table 1: Experimental results for NICP using different augmentation sampling schemes on the Humanoid Crouch task across three trajectory world models (mean ± std.).

ModelMethodGaussian SamplingRandom Sign Sampling
Coverage (Target 90%)Set SizeCoverage (Target 90%)Set Size
TDMNICP91.69 ± 0.611.144 ± 0.02792.03 ± 0.601.160 ± 0.023
TrajWorldNICP91.94 ± 0.811.214 ± 0.03292.20 ± 0.781.225 ± 0.029
WestWorldNICP91.88 ± 0.791.198 ± 0.03292.13 ± 0.771.206 ± 0.028

W2/Q2.1: About a more comprehensive presentation and discussion of the undercoverage observed in individual state dimensions.

A2.1: Table 2 below summarizes the number of NICP undercoverage cases across all tasks, models, and state dimensions, together with the average coverage across these cases and whether the state estimator underestimates uncertainty in the corresponding dimensions. Such underestimation is indicated by an average NEES greater than 1 in the corresponding dimension. We observe that every individual dimension with undercoverage also exhibits uncertainty underestimation by the state estimator, consistent with our discussion of the results in Table 8 of Appendix E. Furthermore, across these undercoverage cases, Vanilla CP achieved average coverage rates of 66.46%, 74.81%, and 28.55% on the three datasets, respectively, all below the corresponding NICP results. This indicates that imperfect covariance information can also help improve coverage.

In addition, as a commonly used metric for assessing estimator consistency, an average NEES below the reference value for the full state vector does not imply that the dimension-wise average NEES values are all below their respective reference values, and estimator consistency can indeed vary across dimensions. In this work, we did not specifically focus on calibrating the estimator to achieve consistency in each dimension. Our primary focus was on evaluating NICP when the estimator behaves as expected and examining the effects of changes in estimator consistency. As discussed in Appendix E, the dimension-wise and aggregate results exhibit a consistent pattern with respect to estimator consistency. We will clarify this point more explicitly in the Conclusion and Limitations section of the submission.

Table 2: Summary of dimension-wise NICP undercoverage across all evaluated tasks and models. The target coverage is 90%. Average NICP coverage is reported over the undercoverage cases only. Average NEES values above 1 indicate that the estimator-reported covariances underestimate uncertainty.

DatasetTotal CasesUndercoverage CasesAvg. NEES > 1Avg. NICP Coverage
Humanoid3×3×123\times3\times1224/10824/2486.57%
Quadruped3×3×123\times3\times1210/10810/1087.83%
Go23×123\times129/369/980.15%

Q2.2: If any diagnostics can identify unreliable estimator covariances when ground-truth states are unavailable?

A2.2: Our framework builds on state estimation systems that already exist in robotic systems and have been extensively studied. Many existing studies [1–4] focus on assessing or improving the consistency of state estimation systems without ground truth and can serve as useful references. Compared with vanilla CP, which requires ground-truth state values for calibration, our framework offers an alternative source of information. Furthermore, even when the covariance estimates underestimate the actual uncertainty, prediction sets constructed using our framework can still achieve higher coverage than those obtained by applying vanilla CP directly to noisy observations without accounting for observation noise. This is supported by the results reported in A2.1.

[1] Griebel T, et al. Self-Assessment and Monitoring Module for Tracking Algorithms in the Stone Soup Framework. IEEE MFI, 2025.

[2] Chen Z, et al. Kalman filter auto-tuning with consistent and robust Bayesian optimization. IEEE TAES, 2024.

[3] Piché R. Online tests of Kalman filter consistency. Int. J. Adapt. Control Signal Process., 2016.

[4] Vega-Brown W, Roy N. CELLO-EM: Adaptive sensor models without ground truth. IEEE/RSJ IROS, 2013.

Thank you for your reply and clarification. This has resolved most of my questions, so I will keep my positive rating.

Dear Reviewer 87eR,

Thank you for your positive feedback on our response. We have incorporated the discussion into the latest version of our manuscript for your reference. We sincerely appreciate your constructive suggestions, which have helped us improve our work. Thank you again!

Reviewer_feXtOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Noise-Informed Conformal Prediction (NICP) for trajectory models calibrated using noisy state estimates. Correlation between input and output observation noise can make observed prediction errors underestimate errors relative to true states. NICP propagates estimator uncertainty through the predictor and adds sampled perturbations to calibration residuals. The paper establishes covariance dominance under specified assumptions and finite-sample marginal coverage under additional distributional conditions. Experiments cover three trajectory models, six simulated tasks, and one real-world task.

Critical strengths

S1: The distinction between observed and true prediction errors is important and clearly formulated. S2: The covariance argument is sound under its assumptions. Appendix A also gives explicit constructions, including a Monte Carlo construction. S3: The experiments show substantial improvements over vanilla CP and smaller sets than the specified Robust CP baseline.

Critical weaknesses

The practical algorithm’s coverage requires more than accurate estimator covariance. Theorem 2 assumes both true and augmented errors have centered Gaussian marginal distributions. Then, algorithm 1 instead samples from a random covariance. even with an affine predictor and exact Gaussian estimator uncertainty, its K=1 augmented errors can be products of Gaussians. An independent scalar check satisfying the covariance assumptions produced approximately 88.87% coverage at a 90% target with N=50. This does not contradict Theorem 2, whose Gaussian assumption fails. It does expose a practical limitation that should be made explicit alongside the coverage claims.

Questions for authors

Across the existing tasks, how often do individual state dimensions miss the nominal coverage? Do the scalar-score versions of Figure 9 explain those failures?

4 comments

Dear Reviewer feXt,

Thank you for reviewing our submission and for your constructive comments. We have carefully addressed the points you raised in our responses. Please let us know if further clarification would be helpful. We appreciate your time and valuable feedback.

W1: About the additional distributional conditions required for coverage beyond estimator consistency.

A: Our coverage guarantees are established under the distributional conditions described in Theorem 2 and Corollary 1, and at the same time our framework has the potential to make these conditions easier to satisfy in different settings by flexibly adjusting the sampling scheme for the augmentation term ξ\xi. As described in lines 219–220, Gaussian sampling is not essential to the NICP framework; we only require ξ\xi to be drawn from a zero-mean distribution with the constructed covariance. For example, we can replace the current Gaussian sampling in Algorithm 1 with ξi=Sidi\xi_i=S_id_i, where SiS_i is sampled uniformly from {+1,−1}\{+1,-1\} and did_i is the change in the model output induced by the input perturbation in the preceding step. In the reviewer’s example, this satisfies the required distributional assumptions and yields a finite-sample marginal coverage of approximately 90.2%90.2\%, thereby resolving the identified assumption violation. In future work, we will further investigate suitable sampling schemes for ξ\xi that help satisfy the distributional conditions in different settings. We will also make this point more explicit in our submission.

Q1: Across the existing tasks, how often do individual state dimensions miss the nominal coverage? Do the scalar-score versions of Figure 9 explain those failures?

A: Table 1 below summarizes the number of NICP undercoverage cases across all tasks, models, and state dimensions, together with whether these cases are accompanied by failures of CDF dominance and uncertainty underestimation by the state estimator. The latter is indicated by an average NEES greater than 1 in the corresponding dimension. We observe that NICP exhibits undercoverage in some individual dimensions and that the CDF dominance condition fails in all these cases. In Appendix B, we identify the expected-CDF inequality in Equation (54) as the key condition underlying the coverage guarantee, and Figure 9 presents its empirical evaluation for the full error vector. These scalar-score results for individual dimensions further support the analysis in our work. Furthermore, consistent with the findings reported in Table 8 of our submission, all dimensions with undercoverage also exhibit uncertainty underestimation by the estimator, which can substantially affect whether the CDF dominance condition holds.

In this work, we did not specifically focus on calibrating the estimator to achieve consistency in each dimension. Our primary focus was on evaluating NICP when the estimator behaves as expected and examining the effects of changes in estimator consistency. Across all dimensions where NICP fell below the target coverage, it nevertheless achieved an average coverage of 85.52%, compared with 60.47% for Vanilla CP on the same dimensions. These results indicate that even when the uncertainty reported by the estimator does not fully meet expectations, using its covariance information to compensate observed prediction errors can still improve coverage over directly calibrating on those errors while ignoring observation noise.

Table 1: Dimension-wise undercoverage of NICP and the corresponding CDF and state-estimator consistency diagnostics across all evaluated tasks and models. The target coverage is 90%. Average NEES values above 1 indicate that the estimator-reported covariances underestimate uncertainty.

DatasetTotal CasesUndercoverage CasesCDF Dominance FailuresAvg. NEES > 1
Humanoid3×3×123\times3\times1224/10824/2424/24
Quadruped3×3×123\times3\times1210/10810/1010/10
Go23×123\times129/369/99/9

Dear Reviewer feXt,

Thank you for taking the time to review our submission and for your insightful comments. We have revised our manuscript in response to your feedback, and the updated version is available for your reference. We would be happy to answer any further questions you may have. Thank you very much!

Reviewer_feXtOpenReview ↗

Thanks for the discussion

I appreciate the thorough discussion you further provided. I will consider changing the score accordingly.

Dear Reviewer feXt,

Thank you so much for your positive assessment of our responses and revised manuscript. We sincerely appreciate your constructive feedback, which has helped us improve our work. Thank you again!

Reviewer_q7rAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Noise-Informed Conformal Prediction (NICP) for uncertainty quantification in learned robotic dynamics models when calibration trajectories contain correlated observation noise. Its central observation is that input-induced prediction errors and future-state observation errors can cancel, making observed residuals underestimate errors relative to the true physical state.

NICP propagates estimator-reported covariance through the dynamics model, samples artificial noise from the resulting auxiliary covariance, and adds it to calibration residuals before applying conformal prediction. The authors establish covariance dominance under specified assumptions and derive finite-sample marginal coverage under additional distributional conditions. Experiments evaluate three dynamics models on six simulated tasks and one real-robot task.

Critical strengths
  • The paper identifies a practically relevant failure mode. Correlated input and target observation errors can conceal substantial prediction errors.
  • The method makes practical use of uncertainty information already produced by many state estimators. This avoids requiring clean calibration labels or changes to the dynamics model. The approach supports both sampling-based and Jacobian-based implementations.
  • The theoretical decomposition is useful and appears correct under its assumptions. Theorem 1 identifies sufficient conditions for covariance dominance, while Theorem 2 adds distributional conditions that allow coverage to transfer from augmented errors to true errors.
  • The empirical improvements are substantial across the tested settings. NICP improves coverage over vanilla CP while producing smaller prediction sets than the robust baseline. The authors also provide a real-robot evaluation, covariance-scaling study, and supplementary probability-ordering analysis.
Critical weaknesses
  • There is a gap between the coverage theorem and the main implemented algorithm. Theorem 2 assumes centered Gaussian distributions for both true and augmented errors. The algorithm, however, samples Gaussian noise using a randomly estimated covariance; mixing Gaussian distributions with different covariances generally produces a non-Gaussian distribution, so the augmentation procedure does not itself establish the theorem’s distributional assumptions. Section 4.3 explains why the covariance result may survive imperfect uncertainty estimates or error correlations, but this does not resolve the separate distributional requirement for coverage – even when the covariance assumptions hold exactly. As a simple example, an identity model with perfectly correlated input and output observation noise and an exact sensor variance estimate can produce augmented errors distributed as abs(eta) * Z under the main K=1 procedure, where eta and Z are independent standard Gaussians. The true errors are standard Gaussian. Both distributions have variance 1, but nominal 90% intervals achieve only about 88.9% coverage. This does not contradict Theorem 2, but it shows that exact sensor covariance is insufficient to ensure coverage for the implemented algorithm.
  • The practical requirements on estimator uncertainty need clearer justification. A state estimator can provide covariance estimates without knowing its realized error, but those estimates may not be reliable. NICP requires sufficiently conservative covariance propagation, and its sampling-based justification additionally assumes conditional calibration and Gaussianity. These conditions may be difficult to assess when ground truth is unavailable. The covariance-scaling experiment demonstrates sensitivity to uncertainty magnitude, but does not establish robustness to systematic bias, incorrect covariance structure, or changing operating conditions. It would help to explain when practitioners can reasonably expect the assumptions to hold and how they might assess them.
  • Theorem 1’s uncorrelatedness assumption could benefit from further discussion. The model’s clean-input prediction error and the estimator’s future observation error can share causes, such as difficult environmental or motion conditions; coming from different systems does not make them uncorrelated. Section 4.3 notes that some correlation can be tolerated, but it remains unclear whether this holds in the tested settings. Measuring the relevant correlation and its effect on coverage would help assess the assumption’s practical significance.
  • The safety motivation would benefit from closed-loop evaluation. The experiments demonstrate improvements in prediction coverage and interval size, but do not show whether those improvements translate into safer decisions (e.g. fewer collisions). A closed-loop experiment measuring collisions or constraint violations alongside task performance would strengthen the practical contribution.
Questions for authors
  1. When does the main algorithm satisfy the distributional conditions needed for coverage? How do you reconcile Theorem 2’s assumptions with the Gaussian mixture produced by the random auxiliary covariance?
  2. How can practitioners assess whether estimator uncertainty is sufficiently reliable without ground truth? Beyond uniformly scaling covariance estimates, how sensitive is NICP to systematic bias, errors in covariance structure, or changing operating conditions?
  3. How much correlation between model and observation errors can the method tolerate in practice? Have you measured the correlation between clean-input model errors and future observation errors in your experiments, and examined whether it affects coverage?
  4. Do the improved prediction intervals translate into safer control? Could you provide a closed-loop evaluation measuring collisions or constraint violations alongside task performance, or clarify what additional steps are needed to obtain such benefits?
6 comments

Dear Reviewer q7rA,

Thank you for taking the time to review our submission and for your feedback. We have carefully addressed your comments and suggestions in our responses, which also include new results from a closed-loop control experiment. Please let us know if you have any remaining concerns or would like further clarification. We appreciate the opportunity to improve our work.

W1/Q1: About the potential impact of the Gaussian mixture induced by Gaussian sampling with a random covariance on the validity of the error distribution assumptions underlying Theorem 2.

A1: As described in lines 219–220, Gaussian sampling is not essential to the NICP framework, and we can flexibly adjust the sampling scheme to make the distributional assumptions easier to satisfy. In fact, the sampling distribution of the augmentation term ξ\xi only needs to satisfy the corresponding mean and covariance conditions. For example, in Algorithm 1, we can set ξi=Sidi\xi_i=S_i d_i, where SiS_i is sampled uniformly from {+1,−1}\{+1,-1\} and did_i is the change in the model output induced by the input perturbation in the preceding step. This can eliminate the Gaussian mixing effect caused by Gaussian sampling with a random covariance and resolve the assumption violation issues in the reviewer's example. More generally, if etruee^{\mathrm{true}}, eobse^{\mathrm{obs}}, and did_i each have centered Gaussian distributions, and did_i is independent of eobse^{\mathrm{obs}}, then the distributional conditions of Theorem 2 are satisfied. If these conditions hold after factoring out a common random factor that is independent of the remaining components, then the distributional conditions of Corollary 1 are satisfied.

We also conducted an additional ablation study of different sampling schemes for the augmentation term in Algorithm 1 on the Humanoid Crouch task across three different models. The results are reported in Table 1 below. The different sampling schemes produce similar experimental results, supporting the flexibility of our method in the choice of sampling scheme. In addition, the augmentation sampling scheme indeed affects the distribution of the augmented prediction errors, but does not by itself determine whether the error-distribution assumptions are satisfied, as these assumptions also depend on the distribution of the true prediction errors. We will make this point more explicit in our revised submission and, in future work, investigate how to select suitable sampling schemes for different settings to help satisfy these assumptions.

Table 1: Experimental results for NICP using different augmentation sampling schemes on the Humanoid Crouch task across three trajectory world models (mean ± std.).

ModelMethodGaussian SamplingRandom Sign Sampling
Coverage (Target 90%)Set SizeCoverage (Target 90%)Set Size
TDMNICP91.69 ± 0.611.144 ± 0.02792.03 ± 0.601.160 ± 0.023
TrajWorldNICP91.94 ± 0.811.214 ± 0.03292.20 ± 0.781.225 ± 0.029
WestWorldNICP91.88 ± 0.791.198 ± 0.03292.13 ± 0.771.206 ± 0.028

W2/Q2.1: How can practitioners assess whether estimator uncertainty is sufficiently reliable without ground truth?

A2.1: Our framework builds on state estimation systems that already exist in robotic systems and have been extensively studied. Many existing studies [1–4] focus on assessing or improving the consistency of state estimation systems without ground truth and can serve as useful references. Compared with vanilla CP, which requires ground-truth state values for calibration, our framework offers an alternative source of information. Furthermore, even when the covariance estimates are not sufficiently reliable, using this information to partially compensate for the underestimation of true prediction errors can still yield higher coverage than applying vanilla CP directly to noisy observations without accounting for observation noise. This is supported by a comparison of the results in Table 1 and Figure 4 of our submission: on Humanoid Crouch using TDM, NICP achieves an average coverage of 87.60% even when the covariance estimates are scaled by 0.25, compared with 78.04% for Vanilla CP.

[1] Griebel T, et al. Self-Assessment and Monitoring Module for Tracking Algorithms in the Stone Soup Framework. IEEE MFI, 2025.

[2] Chen Z, et al. Kalman filter auto-tuning with consistent and robust Bayesian optimization. IEEE TAES, 2024.

[3] Piché R. Online tests of Kalman filter consistency. Int. J. Adapt. Control Signal Process., 2016.

[4] Vega-Brown W, Roy N. CELLO-EM: Adaptive sensor models without ground truth. IEEE/RSJ IROS, 2013.

W2/Q2.2: Beyond uniformly scaling covariance estimates, how sensitive is NICP to systematic bias, errors in covariance structure, or changing operating conditions?

A2.2: We conducted additional ablations on these conditions using TDM on the Humanoid Crouch task. We first added fixed biases to the noisy trajectories in both the calibration and test sets while keeping their covariance estimates unchanged. The bias in each dimension was a multiple of the corresponding observation-error standard deviation estimated from the training set. We tested different multiples and report the average coverage results in Table 2 below. As the bias magnitude increases, the average coverage of NICP gradually decreases. To better understand this sensitivity to bias magnitude, we computed the corresponding average NEES using the biased observation errors and the unchanged covariance estimates. A comparison with the results in Figure 4 of our submission shows that, when assessed in terms of NEES, NICP exhibits similar sensitivity to systematic bias and uniform scaling of covariance estimates, with undercoverage observed in both cases when the average NEES exceeds 20.

Table 2: Empirical coverage of NICP and average state-estimator NEES under different systematic observation biases on the Humanoid Crouch task using the TDM model. Bias magnitudes are expressed as multiples of the observation-error standard deviation in each dimension. Average NEES values above 12 indicate that the estimator-reported covariances underestimate uncertainty.

Added Bias (×σ)(\times \sigma)0−0.25+0.25−0.50+0.50−1.00+1.00
NICP Coverage (Target 90%)91.69%91.50%91.27%90.62%90.22%86.73%85.89%
Avg. NEES12.07913.83313.95219.21719.45040.88141.317

We also introduced covariance-structure errors by removing off-diagonal entries. We found that even removing all off-diagonal entries changed NICP’s average coverage by only 0.31 percentage points. This is mainly because the ratio of the mean absolute off-diagonal value to the mean diagonal value averages only 4.24% across covariance matrices. This also suggests that the coverage changes observed in Figure 4 are primarily driven by scaling the diagonal entries. In addition, changing operating conditions can affect estimation-error statistics and uncertainty calibration. Their potential adverse effects on the estimator may manifest as uncertainty underestimation, systematic bias, or covariance-structure errors. These types of changes are examined in Section 5.3 and the additional experiments above.

W3/Q3.1: How much correlation between model and observation errors can the method tolerate in practice?

A3.1: Based on Equation (47) in Appendix B, we computed the correlation tolerance bounds for three models on three Humanoid tasks. As discussed in Section 4.3, the other two terms in Equation (47) can provide positive semidefinite slack, allowing the covariance-dominance condition to tolerate some correlation between model and observation errors. The first source of slack arises from the inherent positive semidefiniteness of Cov⁡(u−v)\operatorname{Cov}(u-v). For state dimension jj, the corresponding correlation tolerance bound is τ0=Var⁡(uj−vj)/(2σmjσvj)\tau^0=\operatorname{Var}(u_j-v_j)/(2\sigma_{m_j}\sigma_{v_j}), which is a conservative bound valid for any covariance mapping satisfying Assumption 2. Additional slack comes from the conservativeness of the chosen covariance mapping, represented by the third term in Equation (47). For Algorithm 1, the bound incorporating this additional slack is $\tau^{\Psi}=[\operatorname{Var}(u_j-v_j)+\mathbb{E}[\Sigma^\xi_{jj}] -\operatorname{Var}(u_j)]/(2\sigma_{m_j}\sigma_{v_j})$. We average these bounds over all state dimensions and report the resulting τˉ0\bar{\tau}^{0} and τˉΨ\bar{\tau}^{\Psi} in Table 3 below. These results indicate that NICP can tolerate some correlation between model and observation errors, and that conservative covariance mappings can substantially increase this tolerance, with their values even exceeding 1 in some cases.

Table 3: Dimension-averaged correlation tolerance bounds across three trajectory world models and three Humanoid tasks. Here, τˉ0\bar{\tau}^0 is a conservative bound valid for any covariance mapping satisfying Assumption 2, while τˉΨ\bar{\tau}^{\Psi} is the bound computed for the specific covariance mapping used in Algorithm 1.

ModelHumanoid CrouchHumanoid Jog ArcHumanoid Walk
τˉ0\bar{\tau}^{0}τˉΨ\bar{\tau}^{\Psi}τˉ0\bar{\tau}^{0}τˉΨ\bar{\tau}^{\Psi}τˉ0\bar{\tau}^{0}τˉΨ\bar{\tau}^{\Psi}
TDM0.0883331.5311720.0740000.2859910.0877232.039277
TrajWorld0.0643002.9542980.0595670.3655570.0725092.504743
WestWorld0.3755162.3069940.0624070.3395360.3786462.611897

W3/Q3.2: Have you measured the correlation between clean-input model errors and future observation errors in your experiments, and examined whether it affects coverage?

A3.2: Table 4 below reports the mean absolute correlation coefficients between clean-input model errors and future observation errors, together with the average coverage achieved by Algorithm 1 across different tasks and models on the Humanoid dataset. These values are very close to zero and below the corresponding reference bounds reported in Table 3 (A3.1). Correlations of this magnitude do not prevent NICP from achieving the target average coverage in these experiments.

Table 4: Mean absolute correlation coefficients between clean-input model errors and future observation errors, together with the average empirical coverage of NICP, across three trajectory world models and three Humanoid tasks. The target coverage is 90%.

ModelHumanoid CrouchHumanoid Jog ArcHumanoid Walk
Avg. Corr.Coverage (%)Avg. Corr.Coverage (%)Avg. Corr.Coverage (%)
TDM0.01177491.690.05011391.930.02297690.59
TrajWorld0.01963791.940.04258192.070.01884190.13
WestWorld0.01060791.880.04231291.920.01894090.13

W4/Q4: About whether and how the constructed CP sets can improve safety in closed-loop control.

A4: We conduct a simple control experiment to demonstrate that improved CP intervals can translate into safer control. We employ MPPI to control a quadrotor as it tracks a reference trajectory while avoiding obstacles, using the environment provided by safe-control-gym. We first train a dynamics model on data collected without dynamics disturbances. We then introduce stochastic wind and use OpenVINS to provide noisy observations in the control environment. The quadrotor tracks the ellipse x(t)=3sin⁡(2πt/10)x(t)=3\sin(2\pi t/10), z(t)=3+2cos⁡(2πt/10)z(t)=3+2\cos(2\pi t/10), while avoiding two circular obstacles centered at (3,3)(3,3) and (−3,3)(-3,3), each with radius 0.250.25 m. MPPI performs this task by penalizing candidate trajectories for deviations from the reference trajectory and proximity to obstacles. CP expands predicted trajectories into tubes, whose boundaries are used to compute obstacle clearance. We collect 50 calibration trajectories by having the quadrotor track randomly deformed ellipses. Following Section 5.3, we construct 40-step joint CP sets$(\alpha=0.1)$ for the quadrotor’s 2D position at each control step and incorporate them into candidate-trajectory evaluation during control. We do not include Robust CP in this experiment, as its current construction does not readily extend to multi-step joint prediction sets.

We evaluate each controller over 100 trials, considering a trial successful if the quadrotor completes three laps without collision. The results are summarized in Table 5 below. NICP accounts for both model prediction errors and state observation errors, producing larger CP sets and consequently achieving a higher task success rate. This also comes with higher tracking RMSE: additional safety margins are needed to guard against collisions caused by potential trajectory deviations, and maintaining greater clearance from obstacles leads to temporary deviations from the reference trajectory. We will include the detailed experimental setup in our submission.

Table 5: Closed-loop control results for quadrotor trajectory tracking and obstacle avoidance over 100 trials per controller. A trial is considered successful if the quadrotor completes three laps without collision.

ControllerTask success rateMean tracking RMSE (m)Mean CP radius (m)
MPPI without CP52% (52/100)0.20150.0000
MPPI + Vanilla CP73% (73/100)0.28580.1624
MPPI + NICP99% (99/100)0.39310.2760

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.