- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper looks at the generalisation of learnt inertial odometry models and explores the sources of error when training a model using a source domain and then applying it to a target domain. It provides interesting insights into the sources of error, shows that the target errors are observed not only in high error source areas, but also in low error source areas with similar motion but different velocities. It also provides ideas on how different types of target domain adaptation data could be used to reduce different types of error.
Critical strengths
-
Critical review into cross domain failures in the context of data-driven inertial odometry estimation
-
Could inspire similar reviews in odometry systems using other modalities
-
A mixture of obvious (expected) and non-obvious (interesting) results e.g. ---- for example an expected result is that when you have a high error region in the source domain you expect high errors in the same region of the target domain ----- less obvious result that in low error motion regions in the source domain, sometimes one still sees high error motion regions in the target domain, and the explanation is the difference in velocities in that region
-
evaluation across multiple datasets spanning cars, drones, humans and animals
-
demonstration of recovery rates by calibration data
Critical weaknesses
In Table 20- the recovery gains from calibration data depend on the evaluated category, with some seeing benefits and others deterioration. The approach to recovery using calibration data is interesting but does not seem to guarantee performance improvements in a reliable way.
Questions for authors
In Table 20- the recovery gains from calibration data depend on the evaluated category, with some seeing benefits and others deterioration. If this is the case is there a good approach to add additional sessions from the target domain without risking deterioration in regions / categories that have low errors?
Is the analysis done for inertial odometry easy to expand and use in other types of odometry?
Can one use domain knowledge on the shift between source and target domains to improve failure detection?
5 comments
A2: The framework could extend to other type of odometry by adapting the interpretable features and failure criteria to their inputs and outputs.
An important aspect of our approach is to use physically meaningful and intuitively understandable motion features to relate model failures to observable motion conditions, informing failure-risk assessment in practical use.
For extensions to visual and LiDAR odometry, similarly observable and interpretable features, such as scene appearance and geometric structure, could be used to examine when models are prone to error and whether large errors can still occur under conditions common in the source domain. Failure criteria would also need to be adapted to outputs such as pose and displacement.
This could support a similar failure-analysis framework for identifying high-risk conditions and assessing a model’s range of applicability.
A3: Domain knowledge provides complementary cues for failure detection
Domain knowledge can be expressed through measurable motion descriptors to identify conditions requiring closer monitoring. We design five features based on IMU kinematics, describing rotational intensity, angular-velocity changes, and acceleration fluctuations (Section 2.1, Eqs. (1)–(2)). Sections 3.1 and 3.2 use these features to analyze source prediction difficulties and their cross-domain behavior. For Car and Human, some high-error regions involve rapid changes in angular velocity and acceleration; for Dog, the studied high-error region exhibits strong rotation. Drone shows that prediction error does not always increase with motion intensity. These findings connect prediction risk to specific motion conditions that warrant attention even within the source domain.
For target-domain failure detection, Section 4.2 compares motion features, internal model features, and their combination. Table 6 shows that combining them achieves higher AUROC than either feature set alone on KITTI, OxIOD, and QuadSLAM. For example, AUROC on OxIOD increases from 0.803 with internal features alone to 0.871 with the combined detector. This demonstrates that descriptors based on motion knowledge can complement internal model responses and improve failure detection.
A1: Non-failure samples mitigate calibration deterioration at a cost to failure recovery
We agree that readout calibration degrades predictions for some originally non-failed target windows. But including non-failure windows alleviated this deterioration.
Mean absolute velocity error of originally non-failed target windows increased from 0.1172m/s before calibration to 0.5730–0.7708m/s under failure-only calibration, as shown by the 0% NF-share rows below. Then we kept the total calibration budget fixed and replaced 50% of the calibration windows with original non-failure windows sampled to balance motion regions. We compared C1 (source-sparse failures), C2 (source-covered failures in source high-error regions), C3 (source-covered failures in source low-error regions), and a mixture of all three.
Effect of including non-failure calibration windows on OxIOD
\hline \text{Failure pool} & \text{NF share (\\%)} & \text{Overall EPE (m/s)} & \text{NF EPE (m/s)} & \text{New failures (\\%)} \\\\ \hline \mathrm{C1} & 0 & 0.5751 & 0.7708 & 88.11 \\\\ \mathrm{C1} & 50 & 0.5594 & 0.4142 & 69.92 \\\\ \hline \mathrm{C2} & 0 & 0.4543 & 0.6802 & 84.45 \\\\ \mathrm{C2} & 50 & 0.5938 & 0.3567 & 38.88 \\\\ \hline \mathrm{C3} & 0 & 0.5438 & 0.7185 & 85.02 \\\\ \mathrm{C3} & 50 & 0.5436 & 0.3054 & 41.67 \\\\ \hline \text{Mixed} & 0 & 0.4357 & 0.5730 & 65.09 \\\\ \text{Mixed} & 50 & 0.5379 & 0.3286 & 43.77 \\\\ \hline \end{array}$$ Footnote:NF denotes windows that were non-failures before calibration. NF share is their proportion in the calibration set; Mixed uses C1, C2, and C3. NF calibration windows are sampled with approximately equal motion-region quotas, capped by availability and without replacement. Each condition uses 1,600 calibration windows per fold and the same held-out sessions. Results are means over 10 sampling repetitions. Overall EPE is evaluated on all \$32,866 \$ test windows. NF EPE and new-failure percentages are evaluated on the \$1,754 \$ original NF test windows. Before calibration, overall EPE was 0.8136 m/s and NF EPE was \$0.1172 m/s \$. The result indicates that including non-failure windows reduced both the mean error and new-failure rate among original non-failures across all four groups. However, recovery of original failures declined, and overall error increased for C2 and mixed calibration. **These results show a trade-off between preserving low-error predictions and correcting failures. Including non-failure windows mitigates deterioration but does not restore their pre-calibration error levels.**