- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks when reliability quantities in multimodal state estimation should be inferred from a sensor's own history and when they benefit from cross-modal evidence. It proposes a ``Local--Cross'' comparison for UAV optical-flow/IMU/GPS fusion. The Local variant feeds a task head only a modality's own causal TCN latent, while the Cross-Modal variant additionally uses the other two modalities' latents and learned directed residuals . Parameter counts are matched by adjusting hidden widths. Two reliability targets are studied: per-role covariance under stochastic covariance variation (SCV), and binary measurement consistency under sensing consistency disruption (SCD), including freezes, biases, drifts, and outliers; both are also evaluated under compound degradation. Experiments use PX4 SITL/Gazebo simulation with synthetic degradations injected in a SCV$\times$SCD design, and statistical comparisons use a family-stratified cluster bootstrap over 12 held-out trajectories. Across two matched-capacity tiers, Local achieves lower log-covariance MAE, whereas Cross-Modal achieves higher consistency AUROC/AUPRC. When integrated into a non-learned ESKF, combining Local covariance prediction with Cross-Modal consistency prediction (F3) reduces position and velocity error under SCD and compound degradation and improves several NEES-based calibration measures relative to Local consistency (F2), while leaving nominal performance essentially unchanged.
Critical strengths
-
The paper isolates a clear and meaningful question of whether reliability prediction benefits from modality-local or cross-modal information. The Local and Cross-Modal comparisons are tightly controlled, with the same prediction targets, data, optimization procedure, and closely matched parameter budgets. Evaluation is also statistically precise since it uses held-out trajectories, treats the clean trajectory as the unit of dependence, averages antithetic noise pairs, and reports family-stratified cluster-bootstrap confidence intervals. The inclusion of downstream NIS/NEES analysis further strengthens the evaluation beyond standalone prediction metrics.
-
The downstream evaluation is particularly useful. Continuous consistency modulation substantially improves state estimation under sensing consistency disruption relative to covariance-only adaptation and hard gating, while leaving nominal performance essentially unchanged. More specifically, replacing Local consistency prediction with Cross-Modal consistency prediction (F3 vs. F2) gives an additional improvement under both SCD and compound degradation, showing that the representation-level finding has practical consequences for the estimator.
Critical weaknesses
-
The central finding is closely tied to how the benchmark is constructed. SCV changes a sensor's own noise covariance, so the target is naturally reflected in that sensor's local statistics. SCD, by contrast, is defined through violations of the nominal relationship among sensors. The result that covariance favors Local while consistency favors Cross-Modal is therefore largely expected from the definitions, rather than establishing a broader principle about when cross-modal evidence is useful.
-
The matched-capacity comparison does not fully isolate representation choice. Parameter matching is achieved by reducing the Cross-Modal hidden width (e.g., covariance Small: Local vs. Cross ), so the comparison changes both access to cross-modal information and local representational capacity. A same-width control or shared/frozen local encoder would make the conclusion that cross-modal information hurts covariance prediction much stronger.
-
The statistical conclusions are based on a single trained model per task/configuration. The trajectory bootstrap captures evaluation-set variation, but not training or initialization variance, which matters given the relatively small consistency gains (Local AUROC is already around ).
-
The broader conclusion is supported only on one simulated UAV setup with synthetic degradations and one specific Cross-Modal construction. The downstream results are useful, but Cross-Modal covariance is never tested in the ESKF, and the filter remains far from statistically consistent under degradation despite the relative improvements.
Questions for authors
-
How much do the Local--Cross differences vary across independent training seeds? Can the authors retrain each configuration with several seeds and report uncertainty that incorporates both training and test-trajectory variation?
-
Parameter matching reduces the Cross-Modal hidden width relative to Local. Does Cross still underperform Local for covariance when the local encoder width is held fixed (e.g., Local vs. Cross ), even if total parameter counts are no longer matched?
-
Can the authors report consistency AUROC/AUPRC separately for each SCD type? In particular, is the Cross-Modal advantage concentrated in bias, drift, and coherent-incorrect measurements, while freezes/outliers are already detectable locally? If so, how should the claimed principle be interpreted beyond the observation that failures whose detection naturally depends on cross-modal disagreement benefit from relational evidence?
3 comments
I thank the authors for their thorough rebuttal and substantial additional experiments, which have addressed many of my original concerns. While I do think some limitations remain regarding the generalizability of the findings beyond the simulated setting (as in for real-world generalization), I believe the additional evidence and also the more carefully scoped conclusions have strengthened the paper. I will therefore raise my score.
Thank You for the Reassessment
We thank the reviewer for the thoughtful reassessment and positive recognition of our revisions. We appreciate your constructive feedback and will retain clear statements of the remaining limitations regarding real-world generalization.
Response to Reviewer PPwE: Training Variability, Degradation-Specific Findings, and Scope
We thank the reviewer for assessing our controls and downstream evaluation. Your questions prompted stronger tests of training variability, architecture, and degradation heterogeneity, and helped us refine the scope of the conclusions.
1. Does the finding follow from the benchmark definitions?
The SCV/SCD construction motivates the hypothesis, but does not determine the empirical ordering. Structured errors may leave detectable local temporal cues, and a cross-modal model can preserve a Local prediction pathway without using reference features.
The revised five-run results illustrate this distinction: Cross improves aggregate consistency AUPRC under CD, whereas SCD-only AUROC and AUPRC difference intervals include zero. Mechanism-specific results also include gains, uncertain differences, and reversals. Preserving the Local pathway reduces the covariance gap.
We frame the findings as an empirical reference for information selection, conditional on target, architecture, and degradation context, rather than a universal assignment of covariance to Local and consistency to Cross.
2. Training variability and capacity controls
Parameter-matched, width-matched, and fusion comparisons now use five runs. Their 95% CIs use 5,000 paired resamples crossing training runs with family-stratified trajectory clusters, accounting for both sources of variation (Appendix D).
Matching counts parameters connected to the task loss. Consistency Local/Cross counts are 120,307/121,103 (Small) and 198,343/197,123 (Large), differing by less than 1%; Local widths are 73/94 and Cross widths 50/64.
All four parameter-matched covariance contrasts favor Local. For Large CD, log-MAE is 0.1487 versus 0.1900: difference +0.0413, 95% CI [0.0300, 0.0512] (Table 3).
Consistency AUPRC differences are:
SCD-only AUROC intervals also include zero. We revised the abstract and conclusions to reflect this boundary (Table 9).
At fixed encoder/head hidden width h=64, covariance differences remain +0.0396 under SCV (95% CI [0.0304, 0.0485]) and +0.0410 under CD (95% CI [0.0288, 0.0524]). CD AUPRC improves by 0.0184 (95% CI [0.0116, 0.0262]); SCD-only AUPRC intervals include zero at both widths (Tables 4 and 10).
We compare Concat, Relational, and Preserved Local fusion. Preserved Local freezes the trained Local predictor and adds a zero-initialized residual branch; the initial checkpoint remains eligible for selection. Its CD covariance difference is +0.0017 (95% CI [−0.0017, 0.0055]). Thus, width alone does not explain the gap, but architecture materially affects it (Table 6).
3. Degradation-specific analyses
We extended your requested SCD breakdown to both targets: consistency AUROC/AUPRC for eight corruption mechanisms, and covariance error for five prediction roles under SCV/CD (Figure 7; Tables 23–26). The covariance analysis groups errors by predicted quantity, not by the role receiving the perturbation.
The consistency diagnostic set contains one SCD mechanism per case across 12 held-out trajectories. For Small CD, AUPRC differences are:
OF incorrect velocity benefits at both capacities under SCD/CD. GPS freeze benefits at both capacities under CD, but its SCD-only intervals include zero. For Large SCD-only GPS position drift, Local is better: difference −0.0109, 95% CI [−0.0139, −0.0079].
These results clarify where aggregate findings hold. Mechanism-wise intervals are exploratory, unadjusted for multiple comparisons, and conditional on one training run. They do not establish a causal explanation for each difference.
4. Downstream effects, calibration, and scope
We extended the covariance replacement to a full factorial evaluation (Tables 17–22). Under CD, changing Local to Cross consistency at fixed Local covariance changes position/velocity MAE by −0.0298 m (95% CI [−0.0438, −0.0157]) and −0.0271 m/s (95% CI [−0.0383, −0.0160]).
Changing Local to Cross covariance at fixed Cross consistency changes position/velocity MAE by +0.0015 m (95% CI [−0.0026, 0.0057]) and +0.0046 m/s (95% CI [0.0016, 0.0076]). Source swaps hold width fixed; these intervals capture trajectory variation conditional on one training realization.
We agree that relative calibration gains do not establish consistency. CD position NEES decreases from 1681.1 for F2 to 886.8 for F3 but remains far above one. We report absolute diagnostics and acknowledge persistent overconfidence.
The evidence remains simulation-specific despite added architectures and noise shifts. We appreciate your time and thoughtful scrutiny, which made the study more rigorous and its conclusions more precise.