acceptodds
Under review as a conference paper at ICLR 2027

ZeroSync: Learning Motion Matching from Paired Relative Timestamp Shifts

Abstract

Estimating unknown time offsets between heterogeneous sensors requires matching motion observed at different rates and with varying reliability. Sparse or repeated motion can produce ambiguous alignment peaks, making fixed correlation scores unreliable. We introduce ZeroSync, which learns temporal matching through controlled timestamp shifts in a common motion representation. Sensor-specific front ends convert observations into angular-velocity sequences for a shared scoring network. Shifting only one stream in time preserves the recorded motion and supplies supervision without absolute clock labels. Shared branches refine component- and block-specific correlations, while the known shift supervises their relative offset distribution. A confidence head learns recovery success using observable and motion-erased examples. At evaluation, the difference between refined branch offsets estimates the correction that reverses the injected shift. Experiments on camera–IMU and LiDAR–IMU robot recordings show that learned residual scoring improves paired recovery without confidence selection. The improvement persists under sequence-equal averaging in each test dataset. These findings support learning a shared motion-matching rule across sensor pairs from known relative timing changes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
sensor synchronization, temporal alignment, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ps24OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a system to time synchronise heterogenous sensors (e.g. camera with IMU or LIDAR with IMU) matching motion that is being observed by different sensors at different rates and with different level of reliability.

The proposed system injects controlled timestamp shifts and uses that as supervision for learning how to time-synchronise the different sensor modalities.

Critical strengths
  • Interesting idea of using controlled time shifts as supervision
  • Significant performance improvements shown compared to baselines
Critical weaknesses

Related work is limited and needs to be expanded...

  • Closed form estimator approaches, e.g. Lightweight Standalone Temporal Calibration for Visual-Inertial Odometry on Resource-Constrained Embedded Systems: A Closed-Form Estimator with Statistical Hardening. Several other papers in this category
  • Implicit time synchronisation via selective sensor fusion, instead of treating calibration as a separate step: e.g. "Learning Selective Sensor Fusion for State Estimation". Many others in this category
  • Reinforcement based approaches e.g. L2Calib: SE(3)-Manifold Reinforcement Learning for Robust Extrinsic Calibration with Degenerate Motion Resilience

The paper is hard to read. The method in Section 3 and the various subsections are very hard to follow in the main architecture diagram of Figure 1. For example the loss function shown in the figure and its components are hard to track in the text, and the same applies for the modules in the architecture diagram and subsections of section 3.

Questions for authors

Is the idea of injecting controlled timestamp shifts applicable to many different architectures or specific to ZeroSync? Here you present it as a specific architecture, but could this be a high level approach that can be adapted to improve several state of the art methods?

What are the reasons that correlation does not work in some cases? It would be good to provide an illustrative example in the introduction or method sections to demonstrate the problem more clearly and then show how training with the proposed controlled timestamp shifts enables the network to learn a better time alignment.

Are there any specific challenging conditions that make the time alignment particularly difficult for existing methods ? If so, what are these conditions - and why does your approach manage to handle them better?

Reviewer_oiqjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ZeroSync learns temporal motion matching for camera–IMU and LiDAR–IMU pairs without absolute clock-offset labels. It converts observations into angular-velocity sequences and learns residual corrections to component- and block-specific correlations. Known timestamp shifts supervise differences between paired offset predictions. Experiments demonstrate improved relative-shift recovery over deterministic baselines and a residual-disabled prior, including under sparse observations.

Critical strengths
  1. The paired-shift formulation enables training without absolute clock labels and supports a shared scorer across camera–IMU and LiDAR–IMU inputs.
  2. The matched Prior-only control isolates the learned residual’s contribution. Without confidence selection, non-erased-window MAE improves from 1.341 to 1.208 ms.
Critical weaknesses
  1. Practical calibration benefits remain unvalidated. Evaluation measures differences between branch predictions, which can cancel shared alignment bias. Accurate relative recovery therefore does not establish accurate absolute calibration. No downstream VIO/LIO evaluation demonstrates improvements in trajectory accuracy or system reliability.
  2. The incremental learned contribution is modest. Compared with Prior-only, MAE improves by 0.133 ms and the 10 ms failure rate decreases from approximately 1.91% to 1.77%. Without multi-seed robustness evidence, the strength and stability of this gain remain uncertain.
Questions for authors
  1. Do you have single-branch absolute-offset results against an independent timing reference, or downstream VIO/LIO comparisons between uncorrected timestamps, Prior-only correction, and ZeroSync correction?
  2. How consistent is the full-versus-Prior-only improvement across training seeds and independent sequences? Sequence-level uncertainty estimates would help assess its robustness.
2 comments
Reviewer_oiqjOpenReview ↗

Official Comment by Reviewer oiqj

I appreciate the responses made by the authors, which solved most my concerns. However, I still have one concern about the seed selection, the results in Appendix J is different to Appendix F. What make this difference

Thank you very much for your extraordinarily careful review and for catching the discrepancy between Appendix F and Appendix J. We sincerely apologize for this inconsistency, which stemmed from an oversight in our version control and cross-checking workflow, and for the confusion and extra effort it caused you during the review process.

We thoroughly traced the calculation pipeline for both sets of numbers and confirmed that both discrepancies share the exact same root cause: an outdated evaluation script was inadvertently used when compiling the 10 ms10\text{ ms} row in Appendix F, and we failed to cross-check this entry against Table 1 and Appendix J prior to submission. When re-evaluated with the verified evaluation script across the 23,889 non-erased windows, both metrics match Table 1 and Appendix J identically:

Metric (23,889 Non-Erased Windows)Previous Appendix F (Outdated Script)Verified Result (Matches Table 1 & Appendix J)
Prior-only F10F_{10}1.9088%1.9088\% (456 failures)1.7414%1.7414\% (416 failures)
Full (seed 42) F10F_{10}1.7707%1.7707\% (423 failures)1.6326%1.6326\% (390 failures)

To ensure complete accuracy and consistency across the entire manuscript, we conducted a comprehensive, table-by-table audit of all experimental metrics, evaluation protocols, and code versions, and re-ran the relevant evaluations using the verified pipeline:

  1. Verified results: We confirmed that the 1 ms1\text{ ms}, 2 ms2\text{ ms}, and 5 ms5\text{ ms} metrics as well as the error quantiles in Appendix F were computed with the correct script and remain unaffected.
  2. Manuscript revisions: To avoid redundant reporting across sections, we have removed the erroneous 10 ms10\text{ ms} row from Appendix F so that the 10 ms10\text{ ms} comparison is reported consistently in Table 1 and Appendix J. We have also updated the full-window results in Appendix B.1 accordingly and clarified the exact calculation basis in the table captions.

Once again, we deeply apologize for this mistake and thank you for your meticulous scrutiny. Your feedback not only helped us correct this error promptly, but also prompted a rigorous end-to-end verification of all results in our paper.

Reviewer_kMUdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

ZeroSync estimates the time offset between an IMU and a camera or LiDAR from rotational motion. Both sensors are converted to angular-velocity sequences, and a matcher scores each candidate offset on a discrete grid (±300 ms at 10 ms spacing, 61 candidates) by quality-weighted correlation plus a small learned correction (Eqs. 3–6). Training needs no clock labels: a window is duplicated with its sensor timestamps shifted by a known Δ, and the two offset estimates are required to differ by Δ (Eqs. 7–8). Evaluation measures recovery of Δ, a relative change rather than absolute calibration (Section 3.6). On 21 sequences from three datasets at 2–20 Hz, the learned correction lowers non-erased MAE from 1.341 ms with the residual disabled ("Prior only") to 1.208 ms (Table 1), and the method has the lowest failure rate among nine matchers at five operating points (Table 2); larger margins appear under selection by its own confidence (Table 3; Table 7 versus Table 2).

Critical strengths
  1. A well-motivated, reusable formulation. The problem is practical: timing between an IMU and a camera or LiDAR when motion is sparse or repeated and fixed correlation peaks become ambiguous (p. 1). The underlying formulation is simple: a common angular-velocity representation, an explicit candidate-offset grid, and supervision from known timestamp shifts that needs no clock labels (Sections 3.1–3.5), with one network serving both sensor pairs.
  2. Improvement shown without confidence selection. On all 23,889 non-erased windows, MAE falls from 1.341 ms with the residual disabled to 1.208 ms, and the ordering holds in each dataset group (Table 1). This matched, unselected comparison is what makes the small gain credible.
  3. Candid diagnostics. The paper reports full-coverage results, alternative selectors, and shared-delay and anchor ablations, and discloses where it loses (Table 11; S13 in Table 5) and that windows overlap (Appendices A–B).
Critical weaknesses
  1. The main results cannot be reproduced from the paper, and the learned gain is not isolated where margins are largest.
    • Undefined inputs: the quality score that weights every correlation (Section 3.1) has no formula, and the IMU-side q_r is never specified (Eq. 3; Fig. 1a).
    • Unspecified comparator: weighted L1, the main internal baseline (Tables 1, 3, 4, 13), is the authors' own implementation with unstated channels, blocks, interpolation and refinement (Section 4.1, l. 287–289).
    • Under-specified populations: erasure is described for training (Section 3.5; Appendix A.2), not for test time (Appendix B.5); the two model-free selectors (Table 3) and the test-window sampling rule are not defined, yet these define the "non-erased" and selected sets.
    • Missing control: the deterministic "Prior only" is absent from the headline comparisons (Tables 2, 5, 7, 11; Fig. 2), so these do not show how much of their margins comes from learning.
    • Relative evidence only: every accuracy number is a difference of two branch estimates (Appendix B.6: e1 − e0), so shared errors cancel; the paper states this scope but reports no single-branch accuracy, leaving its application goal (p. 1, l. 33–34) untested.
    • Selection and sampling: Tables 4–10, Figs. 2 and 4, and the model-confidence row of Table 3 keep the 70% of windows ranked by ZeroSync's confidence, trained to predict its own success (Eq. 9). A repeat S15 draw moves its MAE from 2.976 to 4.014 ms (Tables 8, 10) with no Prior only column, so the residual gain's stability across draws is unknown.
    • Consequence: every same-window Full-versus-Prior-only comparison shows a single-run gain, of 0.10–0.64 ms in Tables 1, 3, 16 and 18; where a headline margin can be checked (S15: Table 16 versus Table 7), Prior only already holds 5.71 of the 7.55-point F10 margin over weighted L1, so the paper does not show that learning, not the fixed prior, drives those margins.
  2. Organization and readability.
    • Terms used before definition: branch (abstract, l. 52; explained l. 91, 193–194), erasure (abstract; Section 3.5), sequence-equal (abstract; Table 1 caption); q_r is never specified.
    • At least seven evaluation populations across main text and appendices, with no table relating them (Section 4.1; Appendix A.1).
    • Figure 1 uses notation absent from the text and labels the shifted input as a later time (t + Δt; panels a, f), though only timestamps change (Section 3.2); the text never refers to its panels.
    • Consequence: a careful reader must reconstruct the method and, for some tables (e.g., Tables 10 and 14), cannot tell which population they refer to.
Questions for authors

Questions whose answers could change the assessment:

  1. Single-branch stability. Every accuracy number is the difference of two branch estimates (Appendix B.6: e1 − e0), which cancels any error shared by both branches, yet the stated application goal is to estimate the offset itself (p. 1, l. 33–34). Under the paper's constant-offset, zero-drift assumption (l. 151, 158), f_θ(x) should not vary across windows of one sequence at one rate (precompensation differs by rate, Table 26), so no labels are needed. Could you report this spread over non-erased windows per configuration for Full, Prior only and weighted L1? It would test stability, not accuracy; if a recording with an independent timing reference is available, the single-branch error against it would be stronger still.
  2. Attribution of the gain. Same-window comparisons of Full with Prior only (Tables 1, 3, 13, 15, 16, 18) show gains of about 0.1–0.6 ms from a single training run, and Prior only is missing from the tables with the largest margins. Could you add Prior only to Tables 2, 5, 7 and 11 and Fig. 2 (no retraining needed) and report Full versus Prior only on the pooled non-erased set over at least three seeds? Since Table 21 removes only the anchors, a run with the same network, data and confidence objective but trained on the per-branch anchor targets alone (neither relative term of Eq. 8) would also show what the paired objective itself contributes.
  3. Selection. The confidence head is trained with erased windows as negatives, erased windows account for about 7.47 of the 8.58 ms all-window MAE (Appendix B.5), and the two model-free selectors are undefined, so Table 3 may mainly measure how well each selector detects erasure. Could you define the two model-free scores, report how many erased windows each keeps in Table 3, then rerun the comparison on non-erased windows only, re-ranked per configuration at 70% coverage?

Clarification requests (needed to verify the reported numbers):

  1. The quality score weights every correlation (Eq. 3) and enters precompensation, observability and the weighted L1 baseline, but Section 3.1 names only its ingredients. Please specify each camera and LiDAR component, its normalization, how the components are combined into q_s, and how the IMU-side q_r is set.
  2. Weighted L1 is the main internal comparator and your own implementation (Section 4.1, l. 287–289); please specify its channels, blocks, interpolation, sub-grid refinement, invalid-sample handling, and any tuning.
  3. Erasure defines the "non-erased" population and much of the selective gain; Section 3.5 and Appendix A.2 describe it for training only (motion removed, quality kept). Please state how erasure is applied at test time (one branch or both, sensor stream only, whole window), what it is meant to simulate, how the 350/700 windows per configuration are sampled, and why all three estimators give exactly 96.171 ms on erased windows.
  4. The scales 8 and 4 in Eq. 6 bound how far the residual can override the prior, and the loss weights in Eqs. 8–9 are given without a stated selection procedure; please state how they were chosen. Please also confirm the residual's receptive field: two padded 3×3 convolutions on a 16×1×J tensor appear to see about ±20 ms; does "using the full volume" (l. 202) mean all 16 curves rather than all offsets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.