- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies temporal offset estimation and 2D and 3D point tracking in unsynchronized multi-view videos. Given multi-view videos with stationary camera parameters, depth maps, and 2D/3D query points, SyncTAP jointly estimates each view's temporal offset relative to a reference view and predicts 2D and 3D trajectories and visibility. EpiInit estimates pairwise temporal offsets using initial 2D motion trajectories, epipolar constraints, and a second-order Taylor approximation, then derives per-view offsets through weighted global fitting. EpiRef constructs candidate offsets around the current estimate and predicts an offset correction using geometric matching information from Sampson errors. STCV aggregates 2D features from nearby frames in other views through spatio-temporal cross-view attention and updates trajectory-related predictions through unidirectional 2D to-3D attention. EpiRef and STCV alternates during refinement, using updated trajectories to guide subsequent temporal offset estimation. The authors construct the synthetic MV-Kubric2 dataset for training and testing and evaluate on real-world datasets (by introducing temporal offsets into originally synchronized videos). The evaluation shows the proposed method achieve sub-frame synchronization accuracy, significantly outperforming SOTA visual-based synchronization method.
Critical strengths
The motivation is well established. Temporal offsets cause cross-view trajectories and features to become misaligned, while errors in the trajectories themselves can weaken the geometric constraints used for synchronization.
The pipeline is generally technially sound. I think EpiInit has a clear advantage as it provides a geometrically grounded starting point for refinement. EpiRef and STCV's spatio-temporal cross-view attention captures the insights of the challenge of unsynchronized videos
The main experiments show substantial improvements over the SOTA visual-based synchronization methods.
Critical weaknesses
The gain from alternating refinement is not isolated. Table 3 adds components progressively. From the configuration with EpiInit to the full model, synchronization MAE changes from 31.6 to 30.2 ms, A@100 from 72.4 to 73.3, and 3D Average Jaccard from 65.1 to 67.9. Because the network architecture changes at the same time, these differences do not isolate the benefit of repeated feedback.
How robust is SyncTAP when camera extrinsics and depth are both inaccurate under large temporal offsets, especially with too few jointly visible moving points or slow motion? Although the authors evaluate some estimated inputs in the appendix, could you further assess robustness under these combined conditions?
The comparability of EpiRef's candidate matching scores may need clarification. Section 3.3.1 aggregates and normalizes Sampson errors over valid point pairs. If different candidate temporal offsets involve different sets of valid correspondences, the aggregated errors may depend on the number of available pairs.
The paper formulate pairwise temporal offset fitting as a weighted graph Laplacian system. This property requires a connected view graph. If insufficient jointly visible dynamic points make the graph disconnected, the offsets of some views relative to the reference camera may lack adequate constraints. The paper does not clearly explain how such cases are detected or handled.
The evaluation largely relies on synthetically introduced constant offsets. The appendix evaluates jointly estimated camera parameters and depth on DexYCB, but the combined effects of larger offsets and imperfect geometry remain unclear. CMU Panoptic results also vary across test configurations.
Questions for authors
I would like to understand how much of the improvement actually comes from alternating refinement. The refinement is performed over M = 4 iterations. Could the authors compare M = 1, 2, 3 and M = 4 under matched training and evaluation settings. It would also be good to include a variant that disables feedback from updated trajectories to temporal offset estimation
Sec 3.3.1 aggregates and normalizes Sampson errors over valid point pairs. Are the valid correspondence sets identical across temporal offset candidates?
I have a concern about the connectivity assumption underlying the one-dimensional null space. Does the method check whether the graph is connected? How are offsets estimated for a view that has no valid path, how about the reliability?
10 comments
Response to Reviewer Wd5S (2/5): Necessity of mutual alternating refinement (continued)
[W1 & Q1] Necessity of mutual alternating refinement (continued)
Experiment (b): The Number of Iterations M of alternating refinement SyncTAP is trained and evaluated for iterations in the paper, and we control the number of iterations at inference in experiment (b).
| Dataset | M (The number of alternating refinements) | 3D AJ | 3D PA | 3D OA | MTE | A@100 | A@500 | |||
|---|---|---|---|---|---|---|---|---|---|---|
| MV-Kubric2 | 1 | 55.4 | 66.8 | 88.6 | 22.7 | 29.7 | 24.1 | 18.4 | 74.6 | 93.9 |
| 2 | 62.3 | 72.4 | 91.4 | 17.2 | 26.9 | 20.7 | 17.3 | 76.5 | 94.4 | |
| 3 | 63.1 | 73.5 | 91.5 | 15.9 | 27.3 | 22.5 | 17.4 | 76.2 | 94.6 | |
| 4 | 62.0 | 73.6 | 90.2 | 15.8 | 30.2 | 25.4 | 18.2 | 73.3 | 94.1 | |
| Panoptic | 1 | 67.1 | 82.5 | 87.0 | 5.8 | 60.9 | 62.8 | 45.1 | 53.2 | 86.4 |
| 2 | 71.9 | 85.9 | 88.9 | 4.6 | 44.0 | 43.7 | 33.2 | 60.2 | 90.3 | |
| 3 | 70.6 | 86.3 | 87.7 | 4.6 | 37.1 | 34.5 | 23.4 | 67.7 | 92.4 | |
| 4 | 67.7 | 86.5 | 85.4 | 4.4 | 29.4 | 27.5 | 18.3 | 70.9 | 94.0 | |
| DexYCB | 1 | 71.1 | 81.7 | 89.6 | 3.48 | 45.2 | 31.0 | 33.4 | 69.3 | 91.1 |
| 2 | 74.2 | 84.3 | 90.8 | 3.07 | 44.2 | 31.2 | 32.2 | 70.2 | 91.3 | |
| 3 | 75.0 | 85.3 | 90.8 | 2.93 | 43.9 | 33.0 | 31.9 | 70.6 | 91.4 | |
| 4 | 74.4 | 85.3 | 89.6 | 2.95 | 44.7 | 32.4 | 33.3 | 70.4 | 91.4 |
- Synchronization performance: On Panoptic, where EpiInit is not sub-frame, every additional iteration improves synchronization accuracy. On MV-Kubric2 and DexYCB, where EpiInit is already sub-frame, synchronization performance is kept accurate by EpiRef over iterations. The reason for the difference is explained in b3u4 W2 & Q2.
- 3D tracking performance: On the datasets above, two iterations were enough to achieve good performance and later iterations have trade-off for Panoptic.
Even though it was an evaluation on limited datasets, we can figure out that mutual alternating refinement is essential but efficiently a few iterations may be enough.
Response to Reviewer Wd5S (3/5): Robustness to combined estimated inputs
[W2 & W5] - Robustness to combined estimated inputs
A: In Appendix D.4, D.5, and D.6, we demonstrated the robustness to estimated queries(ViTPose on Panoptic), estimated cameras and depths(DUSt3R and VGGT- on DexYCB), and varying time offset amounts, respectively. As the reviewer asked, to extend Table 14., we evaluated on combined conditions of estimated cameras, estimated depths, larger offsets, fewer valid points, and slow motion on DexYCB.
- Robustness to Estimated Cameras, Estimated Depths, and Larger Offset: As shown in the table below, SyncTAP is robust to the combined poor conditions of estimated cameras, estimated depths, and larger offsets. On DexYCB at 10 fps, one frame is 100 ms, so 1 5 frames 500 ms corresponds to large temporal offsets, which is 28% of the 18-frame length clip. Our SyncTAP is robust to estimated inputs because it is trained with Gaussian perturbations of camera poses and query positions as described in Appendix D.10.
- Effect of Fewer Valid Points: As described in Appendix D.7, the 3D tracking performance consistently degrades as the number of valid points decreases, whereas the synchronization performance remains relatively stable as long as sufficient valid points are available. However, the synchronization performance is also degraded when the number of valid points becomes too few (approximately 10 or fewer).
- Effect of Motion Threshold: We vary the motion threshold only at : at smaller , the threshold leaves too few valid point pairs. If we use only trajectories of which motion is slower than 8 px/frame for time offset estimation in EpiInit and EpiRef on 10fps 384x512 DexYCB, the performance is degraded. Since we estimate point-pairwise time offset by for , each sample gives a noisy offset if the motion of a point pair is so small that the change due to 2D velocity is small compared with the residual caused by tracking noises or geometry errors. In our evaluation below, SyncTAP remained robust for motion > 8 px/frame, which assumes co-visible dynamic point pair between views for motion-based (visual-based) synchronization.
Overall Performance for combined poor conditions
| Camera and Depth Source | Motion threshold (px/frame) | A@100 | A@500 | 3D AJ | 3D PA | 3D OA | MTE | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GT cam. + DUSt3R depth | 5 | 8 | 4 | 161.8 | 185.7 | 49.6 | 39.0 | 71.0 | 57.4 | 73.6 | 83.5 | 5.21 | |
| 5 | 16 | 6 | 114.6 | 111.2 | 45.4 | 55.6 | 82.0 | 64.3 | 77.3 | 85.9 | 4.72 | ||
| 5 | 64 | 23 | 43.2 | 45.5 | 24.4 | 69.6 | 90.5 | 70.6 | 82.4 | 88.6 | 3.44 | ||
| 5 | 300 | 104 | 53.1 | 59.2 | 26.4 | 63.8 | 89.2 | 73.0 | 84.5 | 89.0 | 3.19 | ||
| 5 | 300 | < 16 | 63 | 54.2 | 58.4 | 23.0 | 64.2 | 89.2 | 72.5 | 83.9 | 89.1 | 3.33 | |
| 5 | 300 | < 8 | 27 | 98.4 | 105.8 | 37.6 | 55.9 | 84.0 | 70.4 | 82.7 | 87.7 | 3.75 | |
| 5 | 300 | < 2 | 1 | 221.7 | 222.4 | 52.8 | 16.8 | 56.3 | 65.5 | 81.0 | 82.6 | 3.71 | |
| VGGT- cam. + VGGT- depth | 5 | 8 | 4 | 134.9 | 146.2 | 29.0 | 41.4 | 75.7 | 59.4 | 74.1 | 84.5 | 5.16 | |
| 5 | 16 | 6 | 126.3 | 122.1 | 62.7 | 47.5 | 78.7 | 63.7 | 76.9 | 86.3 | 4.25 | ||
| 5 | 64 | 23 | 47.0 | 54.3 | 20.0 | 67.1 | 90.0 | 69.6 | 81.2 | 88.2 | 3.20 | ||
| 5 | 300 | 104 | 51.0 | 56.5 | 23.8 | 62.6 | 89.0 | 73.0 | 84.6 | 89.0 | 2.94 | ||
| 5 | 300 | < 16 | 63 | 63.2 | 69.1 | 29.5 | 58.3 | 87.4 | 72.2 | 84.2 | 88.2 | 2.93 | |
| 5 | 300 | < 8 | 27 | 92.6 | 109.8 | 42.6 | 53.9 | 83.8 | 71.7 | 83.4 | 88.8 | 3.04 | |
| 5 | 300 | < 2 | 2 | 224.9 | 233.8 | 60.9 | 16.2 | 55.3 | 65.9 | 81.5 | 82.6 | 3.47 |
Introduced constant offsets and Panoptic Variance: Our experiments model constant per-camera offsets to simulate unsynchronized multi-view videos from publicly avaliable synchronized multi-view benchmarks; clock drift and rolling shutter remain outside the evaluated scope. The random seed configuration is described in Appendix D.9.
Response to Reviewer Wd5S (1/5): Necessity of mutual alternating refinement
We thank the reviewer Wd5S for recognizing our strong motivation of synchronization task in tracking and highlighting the technical soundness of our method with its substantial performance gain over SOTA visual-based synchronization methods. We also sincerely appreciate reviewer's constructive comments, which motivated us to conduct and present several valuable additional experiments to reflect in our paper.
[W1 & Q1] Necessity of mutual alternating refinement
A: As the reviewer asked, we isolate the gain from alternating refinement via three variants of SyncTAP in experiment (a). We also compare the performance across the number of iterations between EpiRef and STCV tracking network in experiment (b). As analyzed in b3u4 W2 & Q2, EpiInit already achieves sub-frame synchronization on MV-Kubric2 dataset. We therefore show the gain from alternating refinement on Panoptic dataset where EpiInit is not that accurate.
Experiment (a): Necessity of Mutual Alternating Refinement between Offset and Trajectory
As a result, each refinement benefits from the other, showing the necessity of mutual alternating refinement beyond a simpler cascaded pipeline.