acceptodds
Under review as a conference paper at ICLR 2027

SyncTAP: Tracking Any Point from Unsynchronized Multi-View Videos

Abstract

Multi-view 3D point tracking is an important prior to understand dynamic 3D scenes, but typically assumes temporally aligned input videos. Even small time offsets collapse the underlying epipolar geometry, disrupting cross-view feature aggregation and severely degrading tracking performance. We propose SyncTAP, the first unified tracking framework that jointly estimates sub-frame per-view time offsets and high-fidelity one-to-one correspondent 2D/3D trajectories from unsynchronized multi-view videos. First, our Epipolar-based Time Offset Initialization (EpiInit) analytically finds initial time offsets using an epipolar constraint and a 2nd-order Taylor approximation. Then, a transformer-based refinement stage alternates two operations: (i) an Epipolar-based Time Offset Refinement (EpiRef) module updates time offsets via Sampson error landscapes; (ii) a Spatio-Temporal-Cross-View (STCV) attention module refines 2D/3D trajectories by introducing novel attention layers for robust feature aggregation under temporal misalignment. This joint optimization yields a mutually beneficial cycle between synchronization accuracy and tracking accuracy. Extensive experiments demonstrate that our SyncTAP achieves sub-frame synchronization accuracy, significantly outperforming SOTA visual-based synchronization method and establishing a robust new baseline for multi-view tracking under unsynchronized conditions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Synchronization, Tracking, Multiview, Time Offset, Epipolar Geometry, Taylor Series

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Wd5SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal offset estimation and 2D and 3D point tracking in unsynchronized multi-view videos. Given multi-view videos with stationary camera parameters, depth maps, and 2D/3D query points, SyncTAP jointly estimates each view's temporal offset relative to a reference view and predicts 2D and 3D trajectories and visibility. EpiInit estimates pairwise temporal offsets using initial 2D motion trajectories, epipolar constraints, and a second-order Taylor approximation, then derives per-view offsets through weighted global fitting. EpiRef constructs candidate offsets around the current estimate and predicts an offset correction using geometric matching information from Sampson errors. STCV aggregates 2D features from nearby frames in other views through spatio-temporal cross-view attention and updates trajectory-related predictions through unidirectional 2D to-3D attention. EpiRef and STCV alternates during refinement, using updated trajectories to guide subsequent temporal offset estimation. The authors construct the synthetic MV-Kubric2 dataset for training and testing and evaluate on real-world datasets (by introducing temporal offsets into originally synchronized videos). The evaluation shows the proposed method achieve sub-frame synchronization accuracy, significantly outperforming SOTA visual-based synchronization method.

Critical strengths

The motivation is well established. Temporal offsets cause cross-view trajectories and features to become misaligned, while errors in the trajectories themselves can weaken the geometric constraints used for synchronization.

The pipeline is generally technially sound. I think EpiInit has a clear advantage as it provides a geometrically grounded starting point for refinement. EpiRef and STCV's spatio-temporal cross-view attention captures the insights of the challenge of unsynchronized videos

The main experiments show substantial improvements over the SOTA visual-based synchronization methods.

Critical weaknesses

The gain from alternating refinement is not isolated. Table 3 adds components progressively. From the configuration with EpiInit to the full model, synchronization MAE changes from 31.6 to 30.2 ms, A@100 from 72.4 to 73.3, and 3D Average Jaccard from 65.1 to 67.9. Because the network architecture changes at the same time, these differences do not isolate the benefit of repeated feedback.

How robust is SyncTAP when camera extrinsics and depth are both inaccurate under large temporal offsets, especially with too few jointly visible moving points or slow motion? Although the authors evaluate some estimated inputs in the appendix, could you further assess robustness under these combined conditions?

The comparability of EpiRef's candidate matching scores may need clarification. Section 3.3.1 aggregates and normalizes Sampson errors over valid point pairs. If different candidate temporal offsets involve different sets of valid correspondences, the aggregated errors may depend on the number of available pairs.

The paper formulate pairwise temporal offset fitting as a weighted graph Laplacian system. This property requires a connected view graph. If insufficient jointly visible dynamic points make the graph disconnected, the offsets of some views relative to the reference camera may lack adequate constraints. The paper does not clearly explain how such cases are detected or handled.

The evaluation largely relies on synthetically introduced constant offsets. The appendix evaluates jointly estimated camera parameters and depth on DexYCB, but the combined effects of larger offsets and imperfect geometry remain unclear. CMU Panoptic results also vary across test configurations.

Questions for authors

I would like to understand how much of the improvement actually comes from alternating refinement. The refinement is performed over M = 4 iterations. Could the authors compare M = 1, 2, 3 and M = 4 under matched training and evaluation settings. It would also be good to include a variant that disables feedback from updated trajectories to temporal offset estimation

Sec 3.3.1 aggregates and normalizes Sampson errors over valid point pairs. Are the valid correspondence sets identical across temporal offset candidates?

I have a concern about the connectivity assumption underlying the one-dimensional null space. Does the method check whether the graph is connected? How are offsets estimated for a view that has no valid path, how about the reliability?

10 comments
AuthorsOpenReview ↗

Response to Reviewer Wd5S (1/5): Necessity of mutual alternating refinement

We thank the reviewer Wd5S for recognizing our strong motivation of synchronization task in tracking and highlighting the technical soundness of our method with its substantial performance gain over SOTA visual-based synchronization methods. We also sincerely appreciate reviewer's constructive comments, which motivated us to conduct and present several valuable additional experiments to reflect in our paper.

[W1 & Q1] Necessity of mutual alternating refinement

A: As the reviewer asked, we isolate the gain from alternating refinement via three variants of SyncTAP in experiment (a). We also compare the performance across the number of iterations between EpiRef and STCV tracking network in experiment (b). As analyzed in b3u4 W2 & Q2, EpiInit already achieves sub-frame synchronization on MV-Kubric2 dataset. We therefore show the gain from alternating refinement on Panoptic dataset where EpiInit is not that accurate.

Experiment (a): Necessity of Mutual Alternating Refinement between Offset and Trajectory

  • (1) No EpiRef + Trajectory refinement only: Time offset is fixed as EpiInit initialization, and only trajectories are updated with CVTAL and U3DAL.
  • (2) SyncTAP: Time offsets are updated by EpiRef, and trajectories are updated with CVTAL and U3DAL, by a mutually beneficial alternating manner.
  • (3) No feedback from updated trajectories to offset estimation Time offsets are updated by EpiRef, and trajectories are updated with CVTAL and U3DAL, but EpiRef only reads not-updated trajectories.
  • (4) Without the tracking-network modifications: Time offsets are updated by EpiRef, and trajectories are updated, by a mutually beneficial alternating manner, but no use of CVTAL and U3DAL at inference.
Setting3D AJ ↑\uparrow3D PA ↑\uparrow3D OA ↑\uparrowMTE ↓\downarrowδmean\delta_{mean} ↓\downarrowδmedian\delta_{median} ↓\downarrowδstd\delta_{std} ↓\downarrowA@100 ↑\uparrowA@500 ↑\uparrow
(1)66.986.783.64.460.962.845.153.286.4
(2)67.786.585.44.429.427.518.370.994.0
(3)65.986.183.24.769.270.045.847.185.6
(4)58.182.276.35.438.930.928.770.192.0
  • Offset refinement is beneficial for trajectory estimation in alternating refinement, (1) →\rightarrow (2): Refining time offsets together with tracking improves both the synchronization performance and the 3D tracking performance since more accurate offsets better align temporal features for each step. It shows that offset refinement by EpiRef in alternating refinement is beneficial for trajectory estimation.
  • Trajectory refinement is beneficial for offset estimation in alternating refinement, (3) →\rightarrow (2): Feeding the refined trajectories to EpiRef module improves both the synchronization performance and the 3D tracking performance since refined trajectories provide clear Sampson landscape with less noise. It shows that trajectory refinement by STCV network in alternating refinement is beneficial for time offset estimation.
  • The modified tracking network, (4) →\rightarrow (2): CVTAL and U3DAL give large 3D tracking performance gain, where CVTAL tolerates residual time offset error by attending to a temporal neighborhood and U3DAL tolerates by utilizing rich 2D evidence.

As a result, each refinement benefits from the other, showing the necessity of mutual alternating refinement beyond a simpler cascaded pipeline.

AuthorsOpenReview ↗

Response to Reviewer Wd5S (2/5): Necessity of mutual alternating refinement (continued)

[W1 & Q1] Necessity of mutual alternating refinement (continued)

Experiment (b): The Number of Iterations M of alternating refinement SyncTAP is trained and evaluated for M=4M=4 iterations in the paper, and we control the number of iterations at inference in experiment (b).

DatasetM (The number of alternating refinements)3D AJ ↑\uparrow3D PA ↑\uparrow3D OA ↑\uparrowMTE ↓\downarrowδmean\delta_{mean} ↓\downarrowδmedian\delta_{median} ↓\downarrowδstd\delta_{std} ↓\downarrowA@100 ↑\uparrowA@500 ↑\uparrow
MV-Kubric2155.466.888.622.729.724.118.474.693.9
262.372.491.417.226.920.717.376.594.4
363.173.591.515.927.322.517.476.294.6
462.073.690.215.830.225.418.273.394.1
Panoptic167.182.587.05.860.962.845.153.286.4
271.985.988.94.644.043.733.260.290.3
370.686.387.74.637.134.523.467.792.4
467.786.585.44.429.427.518.370.994.0
DexYCB171.181.789.63.4845.231.033.469.391.1
274.284.390.83.0744.231.232.270.291.3
375.085.390.82.9343.933.031.970.691.4
474.485.389.62.9544.732.433.370.491.4
  • Synchronization performance: On Panoptic, where EpiInit is not sub-frame, every additional iteration improves synchronization accuracy. On MV-Kubric2 and DexYCB, where EpiInit is already sub-frame, synchronization performance is kept accurate by EpiRef over iterations. The reason for the difference is explained in b3u4 W2 & Q2.
  • 3D tracking performance: On the datasets above, two iterations were enough to achieve good performance and later iterations have trade-off for Panoptic.

Even though it was an evaluation on limited datasets, we can figure out that mutual alternating refinement is essential but efficiently a few iterations may be enough.

AuthorsOpenReview ↗

Response to Reviewer Wd5S (3/5): Robustness to combined estimated inputs

[W2 & W5] - Robustness to combined estimated inputs

A: In Appendix D.4, D.5, and D.6, we demonstrated the robustness to estimated queries(ViTPose on Panoptic), estimated cameras and depths(DUSt3R and VGGT-Ω\Omega on DexYCB), and varying time offset amounts, respectively. As the reviewer asked, to extend Table 14., we evaluated on combined conditions of estimated cameras, estimated depths, larger offsets, fewer valid points, and slow motion on DexYCB.

  • Robustness to Estimated Cameras, Estimated Depths, and Larger Offset: As shown in the table below, SyncTAP is robust to the combined poor conditions of estimated cameras, estimated depths, and larger offsets. On DexYCB at 10 fps, one frame is 100 ms, so 1 στ\sigma_{\tau} == 5 frames == 500 ms corresponds to large temporal offsets, which is 28% of the 18-frame length clip. Our SyncTAP is robust to estimated inputs because it is trained with Gaussian perturbations of camera poses and query positions as described in Appendix D.10.
  • Effect of Fewer Valid Points: As described in Appendix D.7, the 3D tracking performance consistently degrades as the number of valid points decreases, whereas the synchronization performance remains relatively stable as long as sufficient valid points are available. However, the synchronization performance is also degraded when the number of valid points becomes too few (approximately 10 or fewer).
  • Effect of Motion Threshold: We vary the motion threshold only at N=300N = 300: at smaller NN, the threshold leaves too few valid point pairs. If we use only trajectories of which motion is slower than 8 px/frame for time offset estimation in EpiInit and EpiRef on 10fps 384x512 DexYCB, the performance is degraded. Since we estimate point-pairwise time offset by Tijn,t=−2CB+sign⁡(B)B2−4ACT_{ij}^{n,t} = \frac{-2C}{B + \operatorname{sign}(B)\sqrt{B^{2}-4AC}} for A=12aj⊤Fijxi,B=vj⊤Fijxi,C=xj⊤FijxiA = \frac{1}{2}a_j^{\top} F_{ij} x_i, B = v_j^{\top} F_{ij} x_i, C = x_j^{\top} F_{ij} x_i, each sample gives a noisy offset if the motion of a point pair is so small that the change due to 2D velocity is small compared with the residual caused by tracking noises or geometry errors. In our evaluation below, SyncTAP remained robust for motion > 8 px/frame, which assumes co-visible dynamic point pair between views for motion-based (visual-based) synchronization.

Overall Performance for combined poor conditions

Camera and Depth Sourceστ\sigma_{\tau}NNMotion threshold (px/frame)NvalidN_{valid}δmean\delta_{mean} ↓\downarrowδmedian\delta_{median} ↓\downarrowδstd\delta_{std} ↓\downarrowA@100 ↑\uparrowA@500 ↑\uparrow3D AJ ↑\uparrow3D PA ↑\uparrow3D OA ↑\uparrowMTE ↓\downarrow
GT cam. + DUSt3R depth58∞\infty4161.8185.749.639.071.057.473.683.55.21
516∞\infty6114.6111.245.455.682.064.377.385.94.72
564∞\infty2343.245.524.469.690.570.682.488.63.44
5300∞\infty10453.159.226.463.889.273.084.589.03.19
5300< 166354.258.423.064.289.272.583.989.13.33
5300< 82798.4105.837.655.984.070.482.787.73.75
5300< 21221.7222.452.816.856.365.581.082.63.71
VGGT-Ω\Omega cam. + VGGT-Ω\Omega depth58∞\infty4134.9146.229.041.475.759.474.184.55.16
516∞\infty6126.3122.162.747.578.763.776.986.34.25
564∞\infty2347.054.320.067.190.069.681.288.23.20
5300∞\infty10451.056.523.862.689.073.084.689.02.94
5300< 166363.269.129.558.387.472.284.288.22.93
5300< 82792.6109.842.653.983.871.783.488.83.04
5300< 22224.9233.860.916.255.365.981.582.63.47

Introduced constant offsets and Panoptic Variance: Our experiments model constant per-camera offsets to simulate unsynchronized multi-view videos from publicly avaliable synchronized multi-view benchmarks; clock drift and rolling shutter remain outside the evaluated scope. The random seed configuration is described in Appendix D.9.

Reviewer_JW43OpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles multi-view 2D/3D point tracking when the input videos are not synchronized. SyncTAP jointly estimates sub-frame per-view time offsets and point trajectories. It first initializes the offsets in closed form using the epipolar constraint and a second-order Taylor approximation of the tracks (EpiInit). It then alternates a learned offset refinement based on Sampson-error profiles (EpiRef) with a tracker extended to handle temporal misalignment (STCV attention). The authors train the method on a new synthetic dataset and evaluate it there and on four real datasets with simulated offsets, reporting large gains in synchronization accuracy and runtime over VisualSync alongside improved tracking.

Critical strengths

S1. SyncTAP cuts synchronization error across all five datasets (e.g., 30 vs. 94 ms on MV-Kubric2 and 87 vs. 269 ms on Panoptic Studio against VisualSync). Even its worst seed beats VisualSync's best on every dataset in Table 24, and it runs about 12x faster (4.7 vs. 58.2 s) without external foundation models. This is the main reason I lean toward acceptance.

S2. The paper tackles a well-motivated problem with thorough experiments. Table 8 shows that unsynchronized multi-view 2D tracking can fall below single-view tracking, which clearly justifies the setup, and the appendix carefully tests offset sizes, sub-frame offsets, estimated cameras and queries, view and query counts, and failure modes.

Critical weaknesses

W1. The experiments do not cleanly separate synchronization's contribution from the backbone and training setup. In Table 8, ground-truth offsets lift MVTracker from 51.6 to 53.8 in 3D AJ while SyncTAP reaches 62.0 in the same setting. Yet that baseline is not a clean synchronization ceiling, since Table 15 shows plain MVTracker at 72.0 AJ on synchronized video (sigma = 0) while SyncTAP drops only from 69.8 to 62.0 over the same offset range in Table 4. The authors should explain how they apply sub-frame offsets to a frame-indexed tracker and include a proper control, such as MVTracker retrained on MV-Kubric2 with ground-truth offsets. Table 3 also uses a different evaluation interval from the main results, and the baselines were apparently not retrained: Sec. 4 trains the backbones for 30k iterations on MV-Kubric2, and nothing indicates the baselines were.

W2. The paper does not clearly distinguish EpiInit from prior work or isolate EpiRef's added value. It cites Albl et al. (2017) but never discusses the overlap: they plug a first-order Taylor motion model into the epipolar constraint, iterate for larger offsets, and jointly estimate the fundamental matrix, so assuming a known fundamental matrix makes EpiInit an easier special case that extends the motion model to second order. Since EpiInit already includes a first-order fallback (Appendix A.1), a first-order-only comparison is a one-line ablation that would show whether the second-order term helps. For EpiRef, Table 3 lacks an EpiInit-only row and does not compare against continuous minimization of the Sampson error from the same initialization.

Questions for authors

Q1. SyncTAP assumes known camera parameters (Sec. 3.1), whereas VisualSync estimates relative poses from off-the-shelf reconstruction. Because Appendix D.4 says the modified VisualSync skips cross-view matching with MASt3R, it is unclear where its fundamental matrices come from. Please confirm what camera inputs VisualSync received in Table 1 and Table 13, and report VisualSync with ground-truth cameras if it did not receive them.

18 comments
AuthorsOpenReview ↗

Response to Reviewer JW43 (1/7): Comparison with baseline retrained on MV-Kubric2, and Evaluation protocols

We thank the reviewer JW43 for recognizing our strong motivation of synchronization task in tracking and highlighting our substantial performance gain over SOTA VisualSync with our extensive experiments. We also sincerely appreciate reviewer's constructive comments, which motivated us to conduct and present several valuable additional experiments to reflect in our paper.

[W1] Comparison with baseline retrained on MV-Kubric2, and Evaluation protocols

A: We sincerely appreciate the reviewer's constructive comments, which motivated us to conduct and present valuable additional experiment to reflect in our paper. During the rebuttal, for a fair comparison, we re-trained MVTracker baseline on MV-Kubric2 using the same training setup of SyncTAP since SyncTAP also started with pretrained weights of MVTracker backbone, in order to separate the contribution of our modules from the backbone and training setup. If we use re-trained MVTracker for cascaded baselines, the synchronization performance remains the same with Table 1 in the main paper, and only 3D tracking performance differs. We will add re-trained MVTracker baseline in the revised manuscript for more fair comparison.

  • Evaluation intervals as (i) whole-interval and (ii) all-view-overlap interval: Our tracking metric covers the whole frame of each video because our objective is to recover point trajectories of all frames throughout the recorded videos. Only Table 3 uses the all-view-overlap interval to examine component contributions with all four views available. Correct temporal alignment pushes boundary intervals (the end frames of each video) to observe only other 1~3 views, whereas zero-offset matches incorrect frames but all frames observe 4 views. Consequently, limited boundary observations can partly mask the benefit of synchronization. Therefore, on MV-Kubric2, GT offsets improve the original MVTracker's AJ from 55.7 to 63.8 within all-view-overlap interval, compared with from 51.6 to 53.8 over the whole interval. This also explains why unsynchronized, but offset-corrected inputs in Table 8 are not equivalent to originally synchronized inputs in Table 15. Since both protocols are meaningful to evaluate, we will report both evaluation protocols in the revised manuscript.

  • Comparison: Retraining baseline improves its tracking performance, which makes us separate the contribution of backbone training from the proposed modules. The baseline still does not have specific architecture that makes the model robust to unsynchronized inputs. On all-view-overlap interval protocol, it is more visible that more accurate synchronization leads to higher 3D tracking performance, where we can see the synchronization's contribution to 3D tracking performance more clearly. And our SyncTAP still achieves better performance than any other cascaded baselines even with the re-trained MVTracker and even with the ground-truth offsets (except the 3D tracking performance against GT offset + MVTracker(re-trained) on the all-view-overlap protocol of Panoptic) thanks to the contribution of both synchronization modules(EpiInit, EpiRef) and modified tracking modules(CVTAL, U3DAL) that are robust to unsynchronized videos of different time interval coverage for each video.

On Whole-interval/All-view-overlap protocol, Synchronization performance (Table 1. in the main paper), with re-trained MVTracker

OffsetsMV-Kubric2PanopticDexYCB
δmean\delta_{\text{mean}} ↓\downarrowδmedian\delta_{\text{median}} ↓\downarrowδstd\delta_{\text{std}} ↓\downarrowA@100 ↑\uparrowA@500 ↑\uparrowδmean\delta_{\text{mean}} ↓\downarrowδmedian\delta_{\text{median}} ↓\downarrowδstd\delta_{\text{std}} ↓\downarrowA@100 ↑\uparrowA@500 ↑\uparrowδmean\delta_{\text{mean}} ↓\downarrowδmedian\delta_{\text{median}} ↓\downarrowδstd\delta_{\text{std}} ↓\downarrowA@100 ↑\uparrowA@500 ↑\uparrow
No Sync.351.1325.3188.79.939.3361.1355.6163.68.333.4193.3190.057.120.061.5
MASt3R + MVTracker (re-trained)157.0147.679.240.074.7479.6511.1209.714.442.0233.3230.0112.518.362.0
VisualSync + MVTracker (re-trained)94.473.371.756.985.0168.5166.789.231.567.8133.3100.0100.845.073.7
SyncTAP (Ours)30.225.418.273.394.129.427.518.370.994.044.732.433.370.491.4
AuthorsOpenReview ↗

Response to Reviewer JW43 (2/7): Comparison with baseline retrained on MV-Kubric2, and Evaluation protocols (continued)

[W1] Comparison with baseline retrained on MV-Kubric2, and Evaluation protocols (continued)

(i) On Whole-interval protocol, 3D tracking performance (Table 2. in the main paper), with re-trained MVTracker and GT offset added

MethodSync.MV-Kubric2PanopticDexYCB
AJ ↑\uparrowPA ↑\uparrowOA ↑\uparrowMTE ↓\downarrowAJ ↑\uparrowPA ↑\uparrowOA ↑\uparrowMTE ↓\downarrowAJ ↑\uparrowPA ↑\uparrowOA ↑\uparrowMTE ↓\downarrow
MVTracker51.670.676.820.456.984.167.67.867.482.780.63.22
MASt3R + MVTracker✓\checkmark50.869.677.320.353.081.666.27.666.482.279.03.37
VisualSync + MVTracker✓\checkmark52.871.178.419.851.780.665.07.067.283.479.72.98
GT offset + MVTrackerGT53.871.779.018.750.878.167.07.169.886.281.32.34
MVTracker (re-trained)50.370.676.020.859.383.670.87.965.581.482.13.53
MASt3R + MVTracker (re-trained)✓\checkmark54.872.381.116.557.583.171.47.364.481.180.73.64
VisualSync + MVTracker (re-trained)✓\checkmark58.274.383.415.859.683.476.66.066.882.282.43.33
GT offset + MVTracker (re-trained)GT60.175.584.914.161.084.879.54.769.385.282.82.85
SyncTAP (Ours)✓\checkmark62.073.690.215.867.786.585.44.474.485.389.62.95

(ii) On All-view-overlap protocol, 3D tracking performance, with re-trained MVTracker and GT offset added

MethodSync.MV-Kubric2PanopticDexYCB
AJ ↑\uparrowPA ↑\uparrowOA ↑\uparrowMTE ↓\downarrowAJ ↑\uparrowPA ↑\uparrowOA ↑\uparrowMTE ↓\downarrowAJ ↑\uparrowPA ↑\uparrowOA ↑\uparrowMTE ↓\downarrow
MVTracker55.767.881.420.767.491.674.84.374.382.988.32.89
MASt3R + MVTracker✓\checkmark59.269.985.519.966.790.974.94.573.782.785.83.05
VisualSync + MVTracker✓\checkmark62.071.987.319.268.088.377.65.175.183.988.32.79
GT offset + MVTrackerGT63.872.988.818.769.785.579.76.479.287.389.72.16
MVTracker (re-trained)57.469.483.619.673.490.382.94.571.281.688.03.27
MASt3R + MVTracker (re-trained)✓\checkmark61.071.986.716.472.790.582.14.470.981.187.93.59
VisualSync + MVTracker (re-trained)✓\checkmark64.674.488.815.876.490.686.04.372.582.488.33.14
GT offset + MVTracker (re-trained)GT66.976.290.213.780.592.688.13.976.485.989.52.65
SyncTAP (Ours)✓\checkmark67.976.991.912.179.291.888.33.979.385.893.72.72

Sub-frame offsets. The tracker's sliding window runs on the reference view's discrete frame axis: global time tt at the reference view's frame axis corresponds to local time t−τvt - \tau_v of view vv with predicted time offset τv\tau_v. For fractional τv\tau_v, we linearly interpolate 2D features, trajectories, and camera parameters between the two nearest frames (rotation by SLERP), while the 3D branch takes the nearest frame's point cloud since blending two depth maps of two frames would create points on neither surface. Frames outside each view's video are masked not to perform attention on them. Predicted 2D tracks are mapped back to each view's frame grid by the same interpolation, so that both predicted 3D trajectories and 2D trajectories can be compared with their frame-indexed ground-truth.

AuthorsOpenReview ↗

Response to Reviewer JW43 (3/7): Comparison with first-order taylor model

[W2-1] Comparison with first-order taylor model

A: As the reviewer asked, we ran a first-order-only EpiInit and discussed the relation to Albl et al. As shown in the table below, the first-order variant is as accurate as ours, so both methods provide good estimates in EpiInit for datasets in the main paper. However, for another dataset having strongly accelerating motion, we guess the second-order taylor model may perform better.

Synchronization Performance of EpiInit-only Estimate: 1st vs. 2nd order taylor approximation

DatasetMethodδmean\delta_{\text{mean}} (ms) ↓\downarrowδmean\delta_{\text{mean}} (frames) ↓\downarrowδmedian\delta_{\text{median}} ↓\downarrowδstd\delta_{\text{std}} ↓\downarrowA@100 ↑\uparrowA@500 ↑\uparrow
MV-Kubric2 (12 fps)1st order EpiInit28.90.3522.918.974.094.1
2nd order EpiInit + 1st-order fallback29.70.3624.118.474.693.9
Panoptic (30 fps)1st order EpiInit58.81.7750.043.550.186.6
2nd order EpiInit + 1st-order fallback60.91.8362.845.153.286.4

Relation to Albl et al. (2017). Albl et al. jointly estimates the fundamental matrix and temporal shift using a first-order taylor approximation within RANSAC, and iterates to handle larger offsets. Both EpiInit and Albl et al. utilize Taylor-based epipolar constraints to estimate time offsets, but EpiInit adapts the initialization to an unified network of joint synchronization and multi-view tracking. We assume given cameras which simplify geometric estimation so that it enables closed-form time offset estimates for individual point pair. Beyond the second-order Taylor approximation, EpiInit robustly aggregates point-pairwise estimates into view-pairwise estimates by orthogonality-weighted histogram, and into per-view estimates by confidence-weighted Laplacian solver, as it iterates between the pairwise-fitting and the global-fitting.

Reviewer_b3u4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SyncTAP, a unified framework for joint temporal synchronization and 2D/3D point tracking from unsynchronized multi-view videos. The method first estimates sub-frame time offsets through epipolar constraints and a second-order Taylor approximation. It then iteratively refines time offsets and point trajectories using a learning-based synchronization module and STCV attention. Built upon existing 2D and 3D tracking backbones, the framework combines geometric estimation with learned iterative refinement. Experiments on synthetic and real-world datasets demonstrate significant improvements in synchronization accuracy and multi-view tracking performance over existing baselines.

Critical strengths
  1. This paper presents a comprehensive framework for joint temporal synchronization and multi-view point tracking. The overall method is reasonably designed, with clearly defined roles for each component. The technical details and implementation are also well explained.

  2. Compared with existing baselines, the proposed method achieves substantial improvements in both temporal synchronization accuracy and 2D/3D point tracking performance, demonstrating the effectiveness of the overall framework.

  3. The paper provides extensive quantitative and qualitative evaluations across multiple datasets, along with robustness analyses under various experimental settings. Additionally, the ablation studies sufficiently demonstrate the effectiveness of individual components and provide empirical support for the proposed design choices.

Critical weaknesses
  1. The overall framework appears to rely heavily on combining existing techniques and introducing multiple task-specific modules to improve performance. Although each component is reasonably motivated, the method lacks a unified and deeper methodological insight, making the overall contribution relatively limited despite the seemingly substantial amount of work involved.

  2. The introduction of EpiInit already brings significant improvements in synchronization accuracy, but the authors do not sufficiently explain why a complex learning-based refinement stage is still necessary. Without a direct comparison against simpler geometry-based optimization approaches, it is difficult to assess the unique advantages of this hybrid design. Alternatively, could the traditional optimization-based initialization be removed entirely in favor of a fully end-to-end learning-based approach? Given the capabilities of recent powerful feed-forward 3D/4D models, such an approach seems potentially feasible and might lead to a more distinctive methodological contribution.

  3. Although the authors emphasize the mutual benefits between synchronization and tracking, the current experiments do not sufficiently disentangle the improvements brought by more accurate synchronization, modifications to the tracking network, and bidirectional iterative refinement. Therefore, the results do not convincingly demonstrate that the proposed joint optimization strategy is superior to a simpler cascaded pipeline.

  4. The proposed method assumes known camera parameters, depth information, and initial cross-view corresponding query points. However, these conditions may be difficult to satisfy in real-world unsynchronized multi-view scenarios, potentially limiting the practical applicability of the method.

Questions for authors

My main questions concern the interdependence and necessity of the different stages and modules, particularly EpiInit, EpiRef, and the subsequent refinement process. Are all these components truly necessary?

  1. Table 3 reports the results of EpiRef-only and EpiInit + EpiRef, but does not provide the performance of EpiInit alone. Could the authors report the synchronization accuracy achieved by EpiInit alone, as well as the tracking performance obtained by directly feeding its results into the existing MV-TAP/MVTracker? This would help clarify the actual contribution of the geometry-based initialization to the overall performance improvements.

  2. Given that EpiInit already provides a reasonably accurate initialization based on epipolar geometry, why is an additional Transformer-based EpiRef necessary instead of directly minimizing the Sampson error through further geometric optimization? Could the authors compare their approach against a simpler geometry-based refinement strategy to better justify the unique advantages and necessity of the learning-based refinement?

  3. The current ablation studies do not evaluate the full refinement network without EpiInit. Considering that recent powerful feed-forward 3D/4D models have demonstrated strong geometric and motion reasoning capabilities, could the authors discuss or investigate the feasibility of a fully learning-based temporal synchronization approach without traditional geometric initialization? This would help clarify whether EpiInit is truly indispensable to the proposed framework.

9 comments
AuthorsOpenReview ↗

Response to Reviewer b3u4 (1/6): Concerns about combining existing backbones and lack of an unified insight

We thank the reviewer b3u4 for recognizing our clear framework design and highlighting our significant performance gains with extensive experiments. We also sincerely appreciate reviewer's constructive comments, which motivated us to conduct and present several valuable additional experiments to reflect in our paper. As the reviewer asked, we focused on additional experiments to show the necessity of each component in our SyncTAP.

[W1] Concerns about combining existing backbones and lack of an unified insight

A: Our key insight is that synchronization and multi-view tracking are inherently coupled tasks under unsynchronized videos; the time offset is derived from the epipolar consistency of point tracking across views, while multi-view tracking requires temporal alignment for feature aggregation. Therefore, our goal is to design an unified framework to bridge the gap between synchronization and multi-view tracking in order to give existing multi-view trackers the ability to handle unsynchronized videos, rather than redesigning the tracking backbone itself. Specifically, we utilize 2D trajectories and epipolar constraints as geometric cues for synchronization. Rather than just combining task-specific modules, each module below shares the common goal for synchronization in multi-view tracking and serves mutually complementary motivations that we use the epipolar consistency as a common signal to connect synchronization with tracking.

  • EpiInit - a good starting point: It estimates time offsets as pre-processing by a closed-form root of the epipolar constraint with taylor approximation. Its purpose is to provide a good starting point for EpiRef, which only searches ±4\pm4 frames around the current time offset estimate per update, or to provide a good starting point for other downstream models.
  • EpiRef - recovering from initialization errors: As in b3u4 W2 & Q2, EpiInit predicts time offsets accurately for certain cases, but may not for other cases, e.g. those caused by noisy trajectory estimates. Since EpiInit is a pre-processing step, the residual error cannot be corrected in cascaded models. Therefore, EpiRef maintains the time offsets as variables which a learned module can refine by a local search over the Sampson-error landscape around the current estimate, in order to predict time offsets more precisely while tracking also gets more precise.
  • CVTAL / U3DAL - robust tracking under unsynchronization: Until the time offsets are estimated accurately, tracking has to tolerate the residual temporal misalignment. CVTAL tolerates by attending to a temporal neighborhood in the other views, and U3DAL tolerates by passing rich 2D latent evidence to the 3D stream.
  • Alternating refinement - letting each estimate use the current best version of the other: Better offsets more precisely aligns 2D features from which tracking module benefits, and better trajectories under better temporal alignment gives a cleaner signal to the time offset estimation.
AuthorsOpenReview ↗

Response to Reviewer b3u4 (2/6): Necessity of learning-based EpiRef module

[W2 & Q2] Necessity of learning-based EpiRef module

A: As the reviewer asked the necessity of learning-based EpiRef module, on two representative datasets(synthetic MV-Kubric2 and real Panoptic), we (i) measured the effect of EpiInit and EpiRef, and (ii) compared geometry-based refinement with learning-based refinement(our EpiRef) where both start from the same time offset initialization of EpiInit using SyncTAP tracker. EpiInit alone is accurate in sub-frame level for MV-Kubric2 dataset, but not for Panoptic dataset. Geometry-based refinement(directly minimizing the Sampson error) did not improve EpiInit's time offset estimation on evaluated datasets and drifted away from the ground-truth on MV-Kubric2. However, learning-based refinement(EpiRef) corrected the residual error of EpiInit when EpiInit's time offset estimation is not yet accurate in sub-frame level, and also kept accurate when EpiInit's time offset estimation is already accurate.

Table Setup.

  • Method a) VisualSync: no iterative refinement (cascaded).
  • Method b) EpiInit only: no iterative refinement (cascaded).
  • Method c) EpiInit + geometry-based (per-view argmin) refinement: Δτj=arg⁡min⁡δ∈CEsamj(δ)\Delta \tau_j = \arg\min_{\delta \in \mathcal C} E_{sam}^{j}(\delta) for each view jj at each refinement step.
  • Method d) EpiInit + geometry-based (per-pair argmin, and global fit) refinement: δ^ij=arg⁡min⁡δ∈CEsamij(δ)\hat \delta_{ij} = \arg\min_{\delta \in \mathcal C} E_{sam}^{ij}(\delta) for each view pair i,ji, j at each refinement step, and then Δτj=arg⁡min⁡Δτj∑i≠jqij ρHuber(Δτj−Δτi−δ^ij)\Delta \tau_j = \arg\min_{\Delta \tau_j} \sum_{i\neq j}q_{ij}\,\rho_{\text{Huber}}(\Delta\tau_j - \Delta\tau_i - \hat \delta_{ij}), where ρHuber\rho_{\text{Huber}} is the Huber loss and the equation is solved with an iteratively reweighted least squares(IRLS) as in VisualSync's global offset estimation.
  • Method e) EpiInit + learning-based (EpiRef) refinement (SyncTAP): Δτj=fθ(Esamij(δ),qij,Njvp,τi)\Delta \tau_j = f_{\theta}(E_{sam}^{ij}(\delta), q_{ij}, N_j^{vp}, \tau_{i}), learned by network.
  • Note that Esamij(δ)E_{sam}^{ij}(\delta) is mean Sampson error between views i, j with view j shifted by δ∈C={0,±1,±2,±4}\delta \in \mathcal C = \{0, \pm1, \pm2, \pm4\}, and qijq_{ij} is view pair quality defined by the concentration of epipolar roots, and Esamj(δ)E_{sam}^{j}(\delta) is qijq_{ij}-weighted mean of Esamij(δ)E_{sam}^{ij}(\delta) over ii.
  • Methods a), b) are a cascaded version where time offset is estimated as pre-processing, and methods c), d), e) iteratively refine time offsets along with tracker.
  • Methods b), c), d), e) start with the same time offset initialization from EpiInit, and differ only in the way of time offset update at each refinement iteration.
  • Methods a) and d) utilize the same strategy for minimizing the Sampson error, but method a) searches over the whole range of discrete time offsets as pre-processing, and method d) searches over the local time offset candidates at each refinement step.

Synchronization Performance for Different Offset Refinement Strategies

DatasetMethodδmean\delta_{\text{mean}} (ms) ↓\downarrowδmean\delta_{\text{mean}} (frames) ↓\downarrowδmedian\delta_{\text{median}} ↓\downarrowδstd\delta_{\text{std}} ↓\downarrowA@100 ↑\uparrowA@500 ↑\uparrow
MV-Kubric2 (12 fps)a)94.41.1373.371.756.985.0
b)29.70.3624.118.474.693.9
c)199.72.40177.9126.724.161.2
d)166.92.00135.496.725.368.8
e)30.20.3625.418.273.394.1
Panoptic (30 fps)a)168.55.05166.789.231.567.8
b)60.91.8362.845.153.286.4
c)63.71.9129.962.255.885.3
d)79.52.3962.947.955.185.7
e)29.40.8827.518.370.994.0

(analysis is continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer b3u4 (3/6): Necessity of learning-based EpiRef module

[W2 & Q2] Necessity of learning-based EpiRef module (continued)

Table Results.

  • MV-Kubric2: EpiInit is already accurate, and only learning-based EpiRef keeps its sub-frame accuracy. EpiInit alone already achieves sub-frame synchronization (δmean=0.36\delta_{\text{mean}}=0.36 frames), and learning-based EpiRef keeps it accurate (δmean=0.36\delta_{\text{mean}}=0.36 frames). However, geometry-based refinement moves away from it to 2.4 or 2.0 frames.
  • Panoptic: EpiInit is inaccurate, and only learning-based EpiRef fixes it. EpiInit leaves time offsets residual error of δmean=1.83\delta_{\text{mean}}=1.83 frames. Starting from the residual error, learning-based refinement(EpiRef) reduces to δmean=0.88\delta_{\text{mean}}=0.88 frames, but geometry-based refinement does not as 1.91 or 2.39 frames.

Analysis.

  1. Why EpiInit is sub-frame on MV-Kubric2, but not on Panoptic: EpiInit relies on derivatives of estimated 2D trajectories and can become less reliable when inter-frame displacement is small relative to tracking noise. A higher frame rate of Panoptic reduces this displacement and may increase the sensitivity. The sub-frame criterion is also stricter on Panoptic: one frame is 33.3 ms for 30 fps Panoptic and 83.3 ms for 12 fps MV-Kubric2.

  2. Why directly minimizing the Sampson error fails.: With estimated trajectories, the Sampson error curve is often flat or multi-modal: slow or epipolar-parallel motion almost does not change the Sampson error, and occlusion or tracking noise may make the fake minima. Blindly trusting the minimum value causes the time offset estimation to drift away from the ground-truth time offset, even if it started from alreay accurate time offset.

  3. How learning-based refinement deals with.: It does not blindly trust a single minimum. EpiRef not only considers each view's Sampson error landscape, but also jointly considers view pair quality, valid view pair coverage, and all views' current offsets to predict the offset refinement. As recent 2D point trackers predict a pixel position residual from a local feature correlation volume together with other tracking features, EpiRef also predicts a time offset residual from a local Sampson landscape together with other inputs. Rather than directly jumping to Sampson error minimum, EpiRef predicts a supervised offset residual from the local landscape where the network is trained to robustly predict time offsets from noisy inputs, allowing it to learn corrections when the minimum of the estimated geometric error does not coincide with the true temporal alignment.

For A fully end-to-end learning-based approach without optimization-based initialization(EpiInit), please see b3u4 Q3 for more details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.