acceptodds
Under review as a conference paper at ICLR 2027

Track2Recon: Persistent 3D Object Detection and Tracking for Dynamic Scene Reconstruction

Abstract

Instance-level dynamic 3D Gaussian Splatting (3DGS) for autonomous-driving scene reconstruction typically relies on per-frame 3D bounding boxes and persistent track IDs, whereas unlabeled driving logs contain only multi-camera images, LiDAR point clouds, calibration, and ego poses. Recovering persistent 3D objects from such data poses two main challenges: sparse and incomplete LiDAR observations make reliable association between 2D instances and 3D components difficult, while limited viewpoints and occlusion lead to underestimated object sizes and shifted centers. To address these issues, we present Track2Recon. Its hull-supported 2D–3D object association measures object-level spatial consistency using the projected envelope of an entire 3D component and combines it with support from actual LiDAR points to determine object assignment. Its complete-geometry recovery for partial observations estimates the full object extent from complementary observations across frames and uses visible-surface anchoring to complete missing geometry primarily toward unobserved regions, thereby jointly recovering object size and center. Track2Recon requires neither manual 3D annotations in the target domain nor additional supervised training. Experiments on PandaSet and Waymo demonstrate strong performance in 3D detection and persistent tracking, and the recovered 3D bounding boxes and track IDs can be directly used for instance-level dynamic scene reconstruction and editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dynamic scene reconstruction, autonomous driving, 3D object detection, multi-object tracking, 3D Gaussian Splatting, LiDAR

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_38UROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Track2Recon, an offline method that recovers 3D object boxes and persistent identities from driving images and LiDAR without manual target-domain 3D annotations. It associates image instances with projected point-cloud components, then combines observations across frames to recover object dimensions and correct box placement. Detection experiments on PandaSet and Waymo, together with PandaSet tracking and OmniRe reconstruction results, evaluate whether these outputs can replace annotated object inputs for dynamic scene reconstruction.

Critical strengths
  1. Recovering object boxes and tracks without manual target-domain 3D labels is useful for scene reconstruction. Detection improves substantially on both datasets, and the online variant also outperforms the listed baselines in mAP. Using the recovered objects in OmniRe gives a dynamic-region PSNR of 25.73 dB, compared with 26.56 dB using GT objects. The reconstruction results are a useful check beyond detection accuracy.

  2. The ablations help explain which parts of the method work. Tables A1–A2 compare association rules within the same pipeline, while Table 3 shows the effects of multi-frame geometry recovery and visible-face anchoring. These comparisons give more support for the design than the full-system comparison alone.

  3. The authors provide an anonymous code link to support reproducibility.

Critical weaknesses
  1. I could not fully follow the alignment step in Section 3.3. Each frame uses its own estimated center ct0c_t^0 before the boundaries are combined. If that center shifts as different parts of an object become visible, the boundaries from different frames may not line up correctly. The paper mentions earlier visible-face compensation; please explain how this handles the alignment for moving objects before applying Eq. 3.

  2. The reconstruction evaluation needs more detail. Please confirm that all methods in Table 4 are evaluated on the same dynamic pixels, since OmniRe's masks depend partly on projected object boxes and those boxes are being replaced here. Also, which PandaSet sequences were used, and are the scores measured on images used to fit the reconstruction or on held-out frames? Appendix D.6 lists detection sequences but does not clarify the reconstruction setup.

Questions for authors

Please see the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 38UR: Cross-Frame Alignment and Reconstruction Evaluation

Reviewer 38UR

We thank the reviewer for the positive assessment of Track2Recon and for identifying two points that require further clarification: cross-frame coordinate alignment before Equation (3), and the reconstruction evaluation protocol.

1. Cross-frame alignment before Equation (3)

Our implementation uses a two-stage alignment procedure.

First, we transform the raw LiDAR point cloud in each frame into the corresponding object-local coordinate system using the initial object center and a stabilized horizontal principal axis:

qt=Rt⊤(pt,xy−ct,xy0).\mathbf{q}_t = \mathbf{R}_t^\top (\mathbf{p}_{t,xy}-\mathbf{c}^0_{t,xy}).

This operation compensates for the object's dominant translation and rotation and keeps the physical axis directions consistent across observations.

Second, we select a reliable reference observation frame rr and use ICP to further estimate residual rigid misalignment after the coarse object-local transformation:

qt→r=Rt→rqt+tt→r.\mathbf{q}_{t\rightarrow r} = \mathbf{R}_{t\rightarrow r}\mathbf{q}_t + \mathbf{t}_{t\rightarrow r}.

The transformed point clouds are then represented in the reference coordinate system. Only after registration do we extract the lower and upper object boundaries and perform reliability-weighted quantile fusion in the unified reference coordinate system.

For moving rigid vehicles, the object-local coordinate transformation first compensates for the dominant motion in each frame, after which ICP estimates the residual alignment error. Therefore, this procedure does not assume that a vehicle remains at a fixed world-coordinate position throughout the sequence.

The three operations serve different purposes:

  • Initial-center and coordinate-axis transformation: Compensates for the dominant translation and rotation.
  • Coordinate-axis stabilization: Ensures consistency in the physical axis directions.
  • Residual ICP registration: Aligns the remaining point-cloud differences in a unified reference coordinate system.

Therefore, the boundaries fused in Equation (3) are not measured in independent, unrelated coordinate systems on a frame-by-frame basis.

2. Reconstruction sequences, shared dynamic masks, and evaluation images

The reconstruction experiments use the following 30 PandaSet sequences:

001, 004, 006, 008, 011, 012, 015, 016, 021, 024, 028, 035, 041, 042, 044, 053, 063, 065, 070, 084, 088, 089, 101, 106, 109, 120, 122, 123, 149, 158.

All methods use exactly the same selected images, camera configurations, and OmniRe optimization settings. The only varying inputs are the recovered 3D bounding boxes and persistent track IDs.

The dynamic evaluation masks are generated in advance by projecting the dataset-provided ground-truth 3D bounding boxes of dynamic objects onto the image plane. All methods use the same masks, which are independent of each method's predicted bounding boxes and track information.

This ensures that even if an object is missed by a method, its corresponding region is not excluded from evaluation.

Except for the explicitly labeled GT-object-input control experiment, ground-truth bounding boxes are used only to define these shared evaluation regions.

We further clarify that all selected reconstruction images are used during OmniRe optimization and are also used to compute the final evaluation metrics. Table 4 does not contain an independent held-out-frame or held-out-view split.

Therefore, Table 4 measures reconstruction quality on the training views, rather than held-out novel-view synthesis performance.

The dynamic-region results are as follows:

Object inputPSNR ↑SSIM ↑LPIPS ↓
GT 3D Boxes + Track IDs26.55690.84880.0715
IS-Fusion + PubTracker23.62800.75510.1450
VESPA + MCTrack22.53880.70020.1570
Track2Recon25.72790.81540.0930

We will add the explicit sequence list, the construction procedure for the shared dynamic masks, and the training-view evaluation protocol to the revised manuscript.

We again thank the reviewer for these comments. They have helped us clarify the geometric validity of cross-frame boundary fusion and ensure that the downstream reconstruction results are interpreted accurately within the scope of the actual evaluation protocol.

AuthorsOpenReview ↗

Further Clarification of Methodological Novelty and Additional Evidence

Dear Reviewer,

We sincerely thank you again for your positive assessment and constructive feedback. We are pleased that you recognized the practical value of Track2Recon and the effectiveness of its core components.

We would like to briefly highlight that we have carefully addressed both of your concerns in our detailed rebuttal and revised manuscript.

Regarding cross-frame alignment, we clarified that our method first transforms observations into stabilized object-local coordinates to compensate for the dominant translation and rotation, and then applies residual ICP registration to align observations in a common reference frame before boundary fusion. This also applies to moving rigid objects and avoids directly combining boundaries measured in inconsistent coordinate systems.

Regarding reconstruction evaluation, we have explicitly clarified the PandaSet sequence IDs, the use of identical images and shared ground-truth-based dynamic evaluation masks across all methods, and the fact that the reported results measure training-view reconstruction quality rather than held-out novel-view synthesis.

We have also revised the manuscript to make these procedures clearer and provided anonymous code to facilitate reproducibility. In addition, Figure 2 illustrates our core geometry-recovery mechanisms, and the supplementary materials provide further qualitative comparisons.

We hope these clarifications have adequately addressed your concerns. We would greatly appreciate it if you could revisit our responses and revised manuscript when convenient.

Thank you again for your time and support.

Thank you for clarifying the alignment procedure and reconstruction evaluation. This addresses my main concerns, and I will keep my score unchanged.

Reviewer_W9HMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Track2Recon recovers 3D boxes, headings, and persistent identities from calibrated camera images, LiDAR, and ego poses. It combines frozen SAM 3 instances with association using projected point cloud hulls, extent estimation across frames, anchoring at visible faces, and identity repair. Its outputs replace manual object inputs to an unchanged OmniRe reconstructor. The method requires no manual 3D annotations in the target domain or additional supervised training. Experiments report detection improvements on PandaSet and Waymo, and tracking and reconstruction improvements on PandaSet.

Critical strengths
  1. The pipeline addresses a consequential practical bottleneck: obtaining explicit object geometry and persistent identities for reconstruction without manual 3D labels in the target domain. Table 4 keeps OmniRe fixed and varies its object inputs. PSNR in dynamic regions improves from 23.6280 dB with IS-Fusion + PubTracker to 25.7279 dB with Track2Recon. This directly supports the usefulness of the proposed interface to reconstruction.
  2. Detection improvements are substantial across two datasets. Offline [email protected] reaches 42.41% on PandaSet and 47.04% on Waymo, versus 15.94% and 22.36% for the strongest listed baseline. The online variant also exceeds the listed baselines. This makes access to future frames an insufficient explanation for the entire improvement and strengthens the empirical case for the pipeline.
Critical weaknesses
  1. The evidence does not sufficiently distinguish methodological novelty from useful engineering integration. The main contribution combines existing models with geometric rules and temporal refinement under known calibration. Section 2 already acknowledges corner alignment in Auto4D; VESPA also expands incomplete boxes and shifts them away from the ego vehicle. Track2Recon adds observations across frames and adjusts anchoring with viewing direction, but Table 3 tests anchoring against removal rather than a simpler alternative that preserves visible surfaces under the same extent estimate. Likewise, hull association changes mAP from 23.76% to 23.41% in Table 3. Appendix Table A1 better supports the complete association rule, but the accessible implementation appears to change support and coverage gates as well as ranking. Unless these factors are controlled, the gain cannot be attributed specifically to hull scoring. This leaves the central algorithmic advance insufficiently established and is the main reason for my rejection recommendation.

  2. The experiments leave important uncertainty about the source and scope of the reported benefits. Section 4.1 retains the baselines’ original inference modes, while segmentation and subsequent processing differ; the online variant only partly controls this comparison. Table 2 changes detections and tracking together, so it does not isolate identity repair. For Table 4, Sections 4.1 and 4.5 do not clearly specify reconstruction sequences, the training/test image split, or how common dynamic masks are defined. Appendix D.6 lists detection sequences but does not resolve these details. Consequently, it is difficult to assess the specific identity contribution or determine whether reconstruction gains hold on identical regions in images excluded from reconstruction training. These gaps weaken the evidence needed to justify acceptance of a primarily empirical pipeline contribution.

Questions for authors
  1. Clarification of existing ablations: For Table 3 and Appendix Table A1, which candidate sets, support/coverage gates, and subsequent modules remain fixed? Does the released association code match these experiments, and do the existing results separate hull scoring from candidate filtering?
  2. Clarification of the contribution: What specific operation distinguishes the proposed anchoring from the cited methods that preserve corners or visible surfaces? Which existing result supports its benefit beyond extent estimation across frames? Also, is any existing tracking result based on identical detection boxes, allowing the contribution of identity repair to be separated?
  3. Clarification of reconstruction evaluation: Please provide sequence IDs, the training/test image split, and the source of the dynamic masks for Table 4. Are all methods evaluated on the same images and masks, and are the evaluated images excluded from reconstruction training? These questions concern the reported experiments and do not request a new benchmark study.
6 comments
AuthorsOpenReview ↗

Response to Reviewer W9HM (1/4): Methodological Contributions and Association

Reviewer W9HM

We thank the reviewer for recognizing the practical significance of Track2Recon, particularly the goal of recovering explicit object geometry and persistent identity information without manual target-domain 3D annotations.

Your main concern is whether the proposed method offers genuine methodological innovations beyond an effective integration of existing models and geometric operations. To address this concern, we further clarify the role of each component and report controlled experiments that isolate their effects.

1. Methodological contributions beyond engineering integration

Track2Recon addresses a unified problem: recovering complete, temporally consistent object representations from unlabeled driving logs, with the explicit goal of providing reliable 3D bounding boxes and persistent identity information for downstream dynamic-scene reconstruction.

We propose two core mechanisms to address the main technical challenges.

1.1 Convex-hull-supported association: object-level projection consistency combined with actual LiDAR support

Existing methods commonly rely on the number of points inside a 2D mask, point-hit ratios, or local clustering results. However, under sparse observations, occlusion, and ambiguous multi-camera projections, spatially incorrect point-cloud components may still obtain high local hit scores.

Track2Recon combines two complementary types of evidence:

  1. Component-level projection consistency: We compare the projected convex hull of an entire 3D component with the corresponding 2D instance to evaluate whether the component explains the 2D instance at the object level.
  2. Actual LiDAR point support: We use the number and ratio of point hits to verify that an association is genuinely supported by physical observations, rather than relying solely on overlap in the projection space.

The first component provides a global spatial-consistency constraint, while the second prevents geometrically plausible but unsupported associations.

In response to the reviewer's concern that the original Table 3 did not independently validate the effect of convex-hull scoring, we conducted a controlled comparison of association rules.

The controlled results are as follows:

Association rulemAP ↑HOTA ↑IDF1 ↑Dynamic PSNR ↑SSIM ↑LPIPS ↓
Largest cluster29.7636.8840.6624.12640.78050.1193
Max hit-count30.9738.9338.0124.31860.79870.1278
Max hit-ratio25.2734.6336.8723.94620.75610.1114
Hull-only*40.3753.0862.4725.48370.80760.1018
Full42.4154.7165.3425.72790.81540.0930

All five rules use the same:

  • input instance masks;
  • generated 3D candidate components;
  • candidate-generation and component-splitting procedures;
  • convex-hull validity conditions and J>0J>0 condition;
  • point-cloud support and coverage gating conditions;
  • bidirectional best-match strategy and fixed-ID tie-breaking rule; and
  • cross-camera merging and subsequent processing configuration.

Specifically, the shared candidate-validity conditions include n≥10n\geq10, ρ≥0.50\rho\geq0.50, and r≥0.30r\geq0.30.

The only component that changes is the final ranking criterion.

Hull-only* ranks eligible candidate matches solely by their envelope IoU JJ, using fixed candidate IDs to break ties.

Full Association lexicographically ranks the same eligible candidate matches according to (J,r,n,ρ)(J,r,n,\rho).

An important detail is that both Hull-only* and Full prioritize Hull IoU when ranking candidate components. However, Hull IoU is computed on a discretized image-pixel grid. As a result of pixel-level quantization, different 3D components may receive the same projected-overlap score.

When multiple candidates have the same Hull IoU, Full further resolves the ambiguity using the coverage ratio and actual LiDAR point-support information. In contrast, Hull-only* uses fixed candidate IDs to determine the final selection.

This distinction is important because even a small number of changed object associations can affect subsequent trajectory construction and the aggregation of cross-frame observations. Through temporal association and multi-frame geometry fusion, these local association differences may propagate to multiple subsequent observations of the same object.

AuthorsOpenReview ↗

Response to Reviewer W9HM (2/4): Association Evidence and Visible-Face Anchoring

Therefore, the difference between Hull-only* and Full is not merely a different numerical ranking of isolated candidates. The additional evidence helps resolve projection-level ambiguities and prevents them from further affecting the consistency of reconstructed object trajectories and their recovered geometry.

Under the controlled conditions above, Hull-only* achieves mAP, HOTA, and IDF1 scores of 40.37%, 53.08%, and 62.47%, respectively. Full Association further improves these metrics to 42.41%, 54.71%, and 65.34%.

Dynamic-region PSNR also increases from 25.4837 dB to 25.7279 dB.

The corresponding tracking metrics are:

Association ruleDetA ↑AssA ↑MOTA ↑IDS ↓Frag ↓
Largest cluster26.6553.6615.08148310
Max hit-count28.3051.659.34124406
Max hit-ratio26.4048.27-6.66171577
Hull-only*41.9370.8438.264681
Full43.1172.9944.013968

These results support the importance of component-level projection consistency and validate the additional gains obtained by combining it with complementary support evidence.

Regarding implementation consistency, the experiments above use the same controlled association implementation as the anonymous code repository. The corresponding configuration files and audit entry points are:

  • configs/association_ablation.json
  • run_association_ablation.py
  • postprocess/method_config.json

The audit procedure compares fingerprints of candidate components and candidate matching pairs before ranking, and verifies that the subsequent processing configuration remains unchanged. The corresponding records are saved in association_audit.json and run_manifest.json.

Therefore, the reported controlled-experiment results are not obtained by changing candidate-filtering conditions or subsequent processing modules across different association rules.

1.2 Visible-face anchoring: determining the direction in which missing geometry should be completed

Auto4D uses a corner-alignment-based size-adjustment strategy, whereas VESPA compensates for incomplete bounding boxes using category-size information and orientation relative to the ego vehicle.

Our method addresses a more specific coupled problem: after estimating an object's complete dimensions from complementary observations across frames, how should the complete bounding box be positioned so that it remains geometrically consistent with the LiDAR surfaces actually observed in the current frame?

We divide geometry recovery into three operations:

  1. Multi-frame extent estimation: Determines how much geometric extent is missing from the object.
  2. Hard visible-face anchoring: Determines the direction in which the missing extent should be completed.
  3. Viewpoint-aware correction: Determines how much of the resulting center compensation should ultimately be retained.

This distinction is important. Symmetric expansion around the original center may move both the already-observed surface and the unobserved surface. Hard anchoring, in contrast, preserves the LiDAR-supported visible boundary and completes the missing extent primarily toward the unobserved side.

However, under grazing-angle observations, the reliability of inferring the visible side decreases. We therefore introduce a continuous viewpoint-confidence weight:

γ(a)=min⁡(∣v^Ta∣τview,1),\gamma(\mathbf{a}) = \min\left( \frac{|\hat{\mathbf{v}}^{T}\mathbf{a}|} {\tau_{\mathrm{view}}}, 1 \right),

where v^\hat{\mathbf{v}} is the normalized observation direction from the sensor to the object, and a\mathbf{a} is the direction of the object's axis.

The final center correction is:

Δcnet=γ(a)Δchard.\Delta\mathbf{c}_{\mathrm{net}} = \gamma(\mathbf{a}) \Delta\mathbf{c}_{\mathrm{hard}}.

When the observation direction provides sufficient axial support, γ=1\gamma=1, and the correction produced by hard anchoring is fully retained. Under grazing-view conditions, γ\gamma gradually approaches zero, thereby attenuating the correction.

This mechanism prevents unreliable estimates of the visible side from causing excessive shifts in the object center.

AuthorsOpenReview ↗

Response to Reviewer W9HM (3/4): Anchoring Ablations and Identity Repair

We provide the following comparison:

StrategymAP ↑3D IoU ↑Center error ↓Size error ↓HOTA ↑IDF1 ↑
Without Anchoring35.780.68480.26137.573451.1356.97
Hard Anchoring39.830.70120.24917.648353.4262.87
Viewpoint-aware42.410.70750.24477.621654.7165.34

Hard anchoring increases mAP from 35.78% to 39.83% and 3D IoU from 0.6848 to 0.7012, while reducing center error from 0.2613 m to 0.2491 m.

Adding viewpoint-aware correction further increases mAP to 42.41% and 3D IoU to 0.7075, and reduces center error to 0.2447 m.

We note that size error does not decrease monotonically with the addition of the anchoring mechanism: the full viewpoint-aware configuration has a size error of 7.6216%, slightly higher than the 7.5734% obtained without anchoring. The main additional benefit of anchoring is more accurate joint positioning and localization of the recovered complete bounding box, rather than a guaranteed improvement in every individual metric.

These controlled experiments demonstrate that visible-surface preservation and viewpoint-dependent center correction provide additional contributions beyond multi-frame extent estimation alone.

1.3 Clarification of the convex-hull association results in the original Table 3

We do not interpret the result at this intermediate stage as an independent improvement in mAP.

At this stage, however, 3D IoU increases from 0.5584 to 0.5749, center error decreases from 0.2911 m to 0.2796 m, and size error decreases from 10.9748% to 10.4931%. Meanwhile, HOTA increases from 41.88% to 42.73%, and IDF1 increases from 44.46% to 45.66%.

Thus, although mAP decreases slightly, the intermediate association stage still improves geometric localization accuracy and identity consistency.

The controlled association experiments in Appendix Table A2 provide more direct evidence of the independent contribution of the association module. The subsequent multi-frame geometry-recovery and visible-surface-anchoring stages further increase the full system's mAP to 42.41%.

We will clearly distinguish the performance changes at the intermediate stage from the results of the controlled end-to-end association comparison in the revised manuscript.

2. Separating detection improvements from the contribution of identity repair

We conducted the fixed-detection-input experiment described above.

Specifically, we use the frame-by-frame detections obtained after the first geometry-recovery stage and before Identity Repair and Geometry Feedback, and freeze them into a shared detection file before comparing the trackers.

PubTracker, MCTrack, and our method receive exactly the same object centers, dimensions, orientations, categories, confidence scores, and timestamps. The input file does not contain identity information repaired by Track2Recon.

We disable Geometry Feedback and observation recovery. During the comparison, the trackers update only identity assignments; they cannot modify bounding boxes or add or remove detections.

This protocol eliminates the effects of differences in detection inputs and subsequent geometric updates, allowing us to compare different tracking pipelines directly under identical detection conditions.

The results are as follows:

TrackerHOTA ↑AssA ↑IDF1 ↑MOTA ↑IDS ↓Frag ↓
PubTracker46.1554.1950.8737.36102198
MCTrack42.9258.3153.2635.7264142
Ours52.8369.2263.6442.814389

With identical detection inputs, our method improves HOTA and IDF1 by 6.68 and 12.77 percentage points, respectively, compared with PubTracker. Compared with MCTrack, the improvements are 9.91 and 10.38 percentage points, respectively.

This experiment directly isolates the contribution of the tracking pipeline itself from the effect of detection-box quality.

The improvement is consistent with the design of our identity-maintenance mechanism. After the initial geometry recovery, Track2Recon constructs a time-ordered identity graph and connects different trajectory fragments using 3D motion information, size consistency, undirected principal-axis consistency, and auxiliary 2D-source evidence.

Importantly, 2D identity information is not directly used as an unconditional identity label. It provides only auxiliary evidence, while 3D motion information and geometric consistency remain the primary constraints.

Reviewer_HESpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Track2Recon estimates 3D object boxes and persistent track IDs from multi-camera images, LiDAR, calibration, and ego poses without manual target-domain 3D annotations. It combines hull-supported 2D–3D association, multi-frame size estimation, visible-face anchoring, and track repair. The resulting boxes and tracks can replace manual object annotations in OmniRe. Experiments evaluate detection on PandaSet and Waymo, and tracking and reconstruction on PandaSet.

Critical strengths
  • The geometry recovery has a clear motivation and supporting experiments. Partial LiDAR observations can underestimate object size and shift box centers toward visible surfaces. Estimating size across frames and preserving the observed face directly addresses these errors. Table 3 shows that multi-frame recovery and anchoring increase 3D IoU from 0.5749 to 0.7075. Table A1 also supports combining projected hulls with actual point support for association.

  • The recovered objects improve the intended reconstruction task. Under fixed OmniRe settings, Track2Recon improves dynamic-region PSNR from 23.63 to 25.73 dB over IS-Fusion + PubTracker, compared with 26.56 dB using ground-truth objects (Table 4). Together with the detection and tracking improvements in Tables 1–2, this provides useful evidence that the output boxes and identities support scene reconstruction.

Critical weaknesses
  1. Differences between the compared systems need clearer documentation. Track2Recon uses SAM 3 and, in its offline version, complete sequences; the baselines retain their original inference modes (Section 4.1). The online results are helpful, but it remains unclear whether the baselines use comparable 2D inputs. Table 2 also changes both detections and tracking methods, so it does not show how much improvement comes specifically from the proposed track repair.

  2. The geometry ablations do not separate several important design choices. Table 3 groups multiple operations into multi-frame extent recovery. Separate results for span estimation, boundary fusion, and the size priors in Appendix B.3 would clarify which operations account for the gain. Equation (3) also needs a clearer explanation of how boundaries measured relative to different per-frame centers are aligned before fusion.

  3. Results for the difficult cases motivating the method are missing. The paper emphasizes sparse observations and occlusion, but reports aggregate results over Vehicle and Pedestrian on selected sequences (Section 4.1). Results by class, distance, and occlusion would help establish where recovery succeeds or fails. At the reported operating point, Waymo recall is also lower than OV-SCAN’s—75.68% versus 90.40% (Table 1)—which matters when the goal is to recover objects for subsequent reconstruction and editing.

  4. The reconstruction evaluation needs more detail. Sections 4.1 and 4.5 do not specify the number of reconstruction sequences, the training/test frame or camera split, or how dynamic-region masks are defined. These omissions make Table 4 difficult to assess fully, particularly whether missed objects are included consistently in the dynamic-region evaluation.

  5. Several implementation settings and processing costs are not reported. The manuscript does not give the complete observation-weight formula or values for several fusion, anchoring, stitching, and smoothing parameters (Section 3; Appendices B and D.4). The sequence split used for heading calibration is also unclear. Runtime and memory measurements, including SAM 3 processing, would help assess the cost of applying the method to large driving-log collections.

Questions for authors
  1. Which 2D models and temporal inputs are used by each baseline? With Track2Recon detections held fixed, how do PubTracker or MCTrack compare with the proposed tracking pipeline?
  2. How are local coordinate origins and axis directions aligned before Equation (3)? What are the separate contributions of boundary fusion and fixed size priors?
  3. Can the authors report detection and association results by class, distance, and occlusion, and explain the recall difference from OV-SCAN?
  4. Which sequences and held-out frames or views are used in Table 4? Are dynamic-region masks shared across methods and defined independently of their predicted detections?
  5. Can the authors provide the full parameter configuration, the development and held-out sequences for heading calibration, and end-to-end runtime and memory measurements?
7 comments
AuthorsOpenReview ↗

Response to Reviewer HESp (1/5): Baselines and Fixed-Detection Tracking

Reviewer HESp

We thank the reviewer for recognizing the motivation and practical value of our geometry-recovery module. We address the five questions and related concerns raised by the reviewer below.

1. Baseline comparability and attribution of tracking improvements

Our goal in the main comparison is to evaluate the end-to-end performance of each method according to its original inference pipeline.

Specifically, we use the following configurations for the compared methods:

  • CM3D: Uses Detic and SAM to generate 2D instances and aggregates LiDAR observations from the preceding three frames during pseudo-label generation.
  • VESPA: Uses Grounding DINO and SAM, fuses DINOv2 appearance features, and uses five temporally aligned LiDAR scans. It also employs adjacent-frame matching and ICP-based motion estimation.
  • OV-SCAN: Uses Grounding DINO and SAM to discover candidate objects and CLIP for semantic alignment.
  • SAM3D: Applies SAM ViT-H to a BEV representation generated from LiDAR.
  • IS-Fusion + PubTracker: IS-Fusion uses a Swin-T image backbone and multi-frame LiDAR inputs; PubTracker then performs temporal association on the resulting 3D detections.
  • VESPA + MCTrack: First transforms VESPA detections into world coordinates, then feeds them into MCTrack with frame-rate adaptation.

Track2Recon uses a frozen SAM 3 model. Its online version accesses only the current frame and observations available up to that point, whereas its offline version uses the complete sequence for cross-frame geometry recovery and global identity repair.

In addition, we conducted a controlled experiment in which the detection inputs are strictly fixed.

Tracking evaluation with fixed detection inputs

We use the frame-by-frame detections obtained after the first Geometry Recovery stage and before Identity Repair and Geometry Feedback. Before comparing the trackers, we freeze these detections into a single shared detection file that is used by all methods.

The three trackers—PubTracker, MCTrack, and our method—receive exactly the same detection information, including:

  • 3D bounding-box centers;
  • 3D bounding-box dimensions;
  • 3D bounding-box orientations;
  • object categories;
  • confidence scores; and
  • timestamps.

The frozen detection file does not contain identity information repaired by Track2Recon. Therefore, none of the trackers can use track IDs generated by Track2Recon's post-processing stages.

For this experiment, we disable Geometry Feedback and observation recovery. Each tracker updates only the object-identity assignments; it cannot modify the input bounding boxes, add new detections, or remove existing detections.

This protocol ensures that all three methods operate on exactly the same geometric detections. It therefore isolates the tracking process itself, without confounding effects from differences in detection inputs or subsequent geometric updates.

The results are as follows:

Detection inputTrackerHOTA ↑AssA ↑IDF1 ↑MOTA ↑IDS ↓Frag ↓
Fixed Track2Recon detectionsPubTracker46.1554.1950.8737.36102198
Fixed Track2Recon detectionsMCTrack42.9258.3153.2635.7264142
Fixed Track2Recon detectionsOurs52.8369.2263.6442.814389

With identical detection inputs, our tracking pipeline improves HOTA by 6.68 percentage points and IDF1 by 12.77 percentage points over PubTracker. It also reduces the number of identity switches from 102 to 43 and the number of trajectory fragments from 198 to 89.

Compared with MCTrack, our method improves HOTA by 9.91 percentage points and IDF1 by 10.38 percentage points.

This experiment directly demonstrates that the tracking improvement does not arise solely from more accurate detection boxes. Our identity-association and trajectory-repair mechanisms make an independent contribution.

AuthorsOpenReview ↗

Response to Reviewer HESp (2/5): Geometry Ablations and Cross-Frame Alignment

2. Fine-grained geometry ablations, size priors, and coordinate alignment

We thank the reviewer for pointing out that the original geometry ablation combined multiple operations. In response, we conducted a cumulative ablation that separately evaluates multi-frame span estimation and cross-frame boundary fusion.

The results are as follows:

ConfigurationSize error ↓3D IoU ↑Center error ↓mAP ↑
Baseline10.49310.57490.279623.41
+ Span Estimation9.18460.62870.272430.16
+ Boundary Fusion7.91830.66920.264133.47
Full Recovery7.57340.68480.261335.78

Here, the Baseline corresponds to the configuration after convex-hull-supported association, consistent with the starting point of the supplementary geometry ablation.

The results demonstrate two distinct contributions.

Span Estimation.

Adding reliability-weighted multi-frame span estimation increases mAP from 23.41% to 30.16% and 3D IoU from 0.5749 to 0.6287. This indicates that information from different viewpoints provides complementary evidence and helps estimate the true physical dimensions of partially observed objects.

Boundary Fusion.

Further adding cross-frame boundary fusion increases mAP to 33.47% and 3D IoU to 0.6692, while reducing size error from 9.1846% to 7.9183%. This demonstrates that aligning and fusing complementary boundary observations provides additional benefits beyond trajectory-level span statistics alone.

Complete-Size Recovery.

In addition to span estimation and boundary fusion, the full configuration introduces a non-shrinking size constraint, vehicle-size bounds, and pedestrian body-size constraints. It also performs an additional boundary-fusion operation for stationary vehicles that satisfy the geometric-consistency conditions. These operations are designed to reduce persistent size underestimation caused by incomplete observations.

When these components are enabled together, mAP increases from 33.47% to 35.78%, 3D IoU increases from 0.6692 to 0.6848, and size error decreases from 7.9183% to 7.5734%.

These results show that the complete geometry-recovery configuration provides additional gains over span estimation and boundary fusion alone.

Regarding the size priors, Appendix B.3 describes the category-specific constraints used in our implementation in detail. These priors are lower-bound constraints, not fixed object dimensions, and are intended for objects that are incompletely observed over time. When instance observations support larger dimensions, the final estimated dimensions can exceed these lower bounds.

In the revised manuscript, we will clearly distinguish the contributions of span estimation and boundary fusion, which are independently validated by the ablation, from the combined contribution of the additional complete-size recovery operations.

Cross-frame coordinate alignment before Equation (3)

For the same object observed at different timestamps, we first transform the raw LiDAR point cloud into the corresponding object-local coordinate system using the initial object center and a stabilized horizontal principal axis:

qt=Rt⊤(pt,xy−ct,xy0).\mathbf{q}_{t} = \mathbf{R}_{t}^{\top} \left(\mathbf{p}_{t,xy} - \mathbf{c}^{0}_{t,xy}\right).

This operation compensates for the object's dominant translation and rotation and establishes a consistent convention for the physical axis directions.

However, because the initial centers and coordinate axes are estimated independently in different frames, this coarse transformation may not completely eliminate all residual alignment errors.

We therefore select a reliable observation frame as the reference frame rr, and initialize ICP using the coarsely aligned object-local coordinates. ICP estimates a residual rigid transformation for each observation:

qt→r=Rt→rqt+tt→r.\mathbf{q}_{t\rightarrow r} = \mathbf{R}_{t\rightarrow r}\mathbf{q}_{t} + \mathbf{t}_{t\rightarrow r}.

We then apply this transformation to the corresponding point cloud and extract the lower and upper object boundaries in the unified reference coordinate system. Only observations with sufficient overlap and valid registration are used for boundary fusion. The final boundaries are fused using reliability-weighted quantiles:

ℓz,u=Qβw(ℓt,u).\ell_{z,u} = Q_{\beta}^{w}\left(\\{\ell_{t,u}\\}\right).

rz,u=Q1−βw(rt,u).r_{z,u} = Q_{1-\beta}^{w}\left(\\{r_{t,u}\\}\right).

Both boundaries are measured in the same reference coordinate system. Thus, the complete procedure consists of a coarse object-local coordinate transformation, residual ICP registration, boundary extraction, and weighted fusion, in that order.

AuthorsOpenReview ↗

Response to Reviewer HESp (3/5): Challenging Scenarios and Waymo Recall

3. Analysis of challenging scenarios and the recall gap on Waymo

We added subset analyses along three dimensions: object category, observation distance, and the degree of LiDAR point support.

The results are as follows:

GroupSubsetP@2m ↑R@2m ↑mAP ↑3D IoU ↑AssA ↑IDF1 ↑
CategoryVehicle94.7273.1650.630.749176.8369.82
CategoryPedestrian89.8462.9134.190.665969.1560.73
DistanceNear94.7375.6250.860.751377.2470.38
DistanceMiddle92.3166.8441.720.698772.8664.73
DistanceFar88.1653.4727.830.628463.7154.82
LiDAR supportHigh94.1873.2648.370.739175.8268.74
LiDAR supportMedium92.0767.8241.930.704272.6464.91
LiDAR supportLow89.6260.9335.460.668168.7361.27

The results reveal three important trends.

Category-dependent performance.

Track2Recon performs well on both Vehicle and Pedestrian objects. For Vehicles, mAP is 50.63% and 3D IoU is 0.7491; for Pedestrians, mAP is 34.19% and 3D IoU is 0.6659. Their corresponding AssA scores are 76.83% and 69.15%, respectively.

Distance-dependent performance.

Performance gradually decreases as the observation distance increases. Specifically, 3D IoU decreases from 0.7513 in the Near subset to 0.6284 in the Far subset, while AssA decreases from 77.24% to 63.71%.

In the Far subset, precision remains at 88.16%, whereas recall decreases to 53.47%. This indicates that the method can still localize detected distant objects relatively accurately, but discovering and recovering objects with limited observations is more challenging.

LiDAR-support-dependent performance.

As the number of LiDAR points inside the ground-truth object boxes decreases, the detection and geometry metrics also decline. Nevertheless, in the Low-support subset, the method still achieves a 3D IoU of 0.6681, an AssA of 68.73%, and an IDF1 of 61.27%.

This suggests that complementary observations across frames can partially compensate for sparse single-frame point-cloud support.

We use the number of LiDAR points inside the ground-truth object boxes to characterize observation sparsity. Low LiDAR support may result from occlusion, long observation distance, or limited sensor coverage.

Why is recall on Waymo lower than that of OV-SCAN?

At a confidence threshold of 0.3, Track2Recon achieves a recall of 75.68%, whereas OV-SCAN achieves 90.40%.

This gap reflects the relatively conservative object-acceptance and recovery criteria adopted by Track2Recon.

The association between components and masks must satisfy projection consistency, actual LiDAR point support, and bidirectional best-match constraints simultaneously. Specifically, our association mechanism requires a non-degenerate convex hull, J>0J>0, n≥10n\geq10, ρ≥0.50\rho\geq0.50, and a coverage ratio r≥0.30r\geq0.30. In addition, observation recovery requires sufficient 2D instance evidence and LiDAR point-cloud support in the current frame.

These conditions filter out some weakly supported candidates that may be retained by strategies that prioritize recall more strongly.

Although recall is lower, this trade-off yields substantially higher mAP and 3D IoU, as well as more accurate center and size estimates.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Track2Recon — Author Rebuttal

ICLR 2027 | Submission ID: 55886

We sincerely thank all three reviewers for their careful evaluation of our work and constructive feedback. We are pleased that the reviewers recognized the practical importance of recovering temporally consistent 3D object geometry and identity information from unlabeled autonomous-driving logs, as well as the potential of the recovered object representations for downstream dynamic-scene reconstruction.

The reviewers' comments primarily focus on three aspects:

(1) Whether Track2Recon makes genuine methodological contributions beyond integrating existing models and geometric operations;

(2) Whether the experiments adequately isolate and validate the individual contributions of its components;

(3) Whether the evaluation protocols and implementation details are described transparently enough to ensure reproducibility.

To address these concerns, we have added controlled experiments, more fine-grained ablation studies, and implementation details. More importantly, we further clarify the interplay among Object Association, Complete-Geometry Recovery, and Persistent Identity Maintenance. These three mechanisms jointly form the core design of Track2Recon.

AuthorsOpenReview ↗

Final Summary of Track2Recon: Key Contributions, Practical Value, and Supporting Evidence

We sincerely thank the Area Chair and all reviewers for their time, careful evaluation, and constructive feedback. As the discussion period concludes, we would like to summarize the central contribution and significance of Track2Recon.

The fundamental value of our work is to bridge the gap between raw autonomous-driving logs and object-level dynamic scene reconstruction. Existing reconstruction frameworks typically require accurate per-frame 3D bounding boxes and persistent object identities, which are unavailable in ordinary unlabeled driving logs and expensive to annotate manually.

Track2Recon addresses this practical bottleneck. Given calibrated multi-camera images, LiDAR point clouds, and ego poses, our method automatically recovers the 3D bounding boxes and persistent track IDs needed by existing reconstruction frameworks, without manual target-domain 3D annotations or additional supervised detector training. In other words, our goal is not merely to improve detection or tracking metrics, but to make unlabeled driving logs directly usable for dynamic scene reconstruction.

To achieve this, we introduce two key methodological contributions:

1. Hull-supported 2D–3D object association. Our method combines the projected convex hull of complete 3D components with actual LiDAR point support to establish reliable 2D–3D correspondences. This provides a consistent geometric foundation for cross-camera object association and persistent identity recovery. Figure 2 illustrates the proposed mechanism, Figure A1 presents qualitative comparisons among association strategies, and Table A2 provides controlled quantitative evidence isolating the contribution of hull-based ranking.

2. Complete-geometry recovery from partial observations. Our method combines complementary cross-frame observations with visible-face anchoring to jointly recover incomplete object dimensions and correct center displacement. Rather than simply enlarging partial bounding boxes, it estimates the missing physical extent, determines the appropriate completion direction from observed surfaces, and accounts for viewpoint uncertainty. The ablations in Tables A1 and 3 validate these complementary mechanisms.

An important practical advantage is adaptability across sensor configurations. Unlike approaches that depend on supervised 3D detector training for a target domain, Track2Recon relies on calibrated camera–LiDAR geometry without requiring target-domain 3D annotations. Experiments on PandaSet and Waymo demonstrate its applicability across different camera and LiDAR configurations, achieving 42.41% and 47.04% [email protected], respectively.

Most importantly, the recovered geometry and identities directly benefit downstream reconstruction. With unchanged OmniRe settings, Track2Recon achieves 25.73 dB dynamic-region PSNR, compared with 23.63 dB using IS-Fusion + PubTracker and 26.56 dB using ground-truth object inputs (Table 4). These training-view reconstruction results demonstrate that our automatically recovered objects can substantially reduce reliance on manual 3D annotations.

We have also publicly released our anonymized code and included visualization videos in the supplementary material, enabling reproducibility and providing direct visual evidence of the recovered objects and their temporal consistency.

We believe Track2Recon contributes not only effective geometric association and recovery mechanisms, but also a practical, reusable interface that enables existing dynamic scene reconstruction methods to operate on unlabeled driving logs. We hope the additional clarifications, controlled experiments, and released resources help establish both its methodological novelty and broader practical significance.

We sincerely thank the Area Chair and reviewers again for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.