- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Track2Recon, an offline method that recovers 3D object boxes and persistent identities from driving images and LiDAR without manual target-domain 3D annotations. It associates image instances with projected point-cloud components, then combines observations across frames to recover object dimensions and correct box placement. Detection experiments on PandaSet and Waymo, together with PandaSet tracking and OmniRe reconstruction results, evaluate whether these outputs can replace annotated object inputs for dynamic scene reconstruction.
Critical strengths
-
Recovering object boxes and tracks without manual target-domain 3D labels is useful for scene reconstruction. Detection improves substantially on both datasets, and the online variant also outperforms the listed baselines in mAP. Using the recovered objects in OmniRe gives a dynamic-region PSNR of 25.73 dB, compared with 26.56 dB using GT objects. The reconstruction results are a useful check beyond detection accuracy.
-
The ablations help explain which parts of the method work. Tables A1–A2 compare association rules within the same pipeline, while Table 3 shows the effects of multi-frame geometry recovery and visible-face anchoring. These comparisons give more support for the design than the full-system comparison alone.
-
The authors provide an anonymous code link to support reproducibility.
Critical weaknesses
-
I could not fully follow the alignment step in Section 3.3. Each frame uses its own estimated center before the boundaries are combined. If that center shifts as different parts of an object become visible, the boundaries from different frames may not line up correctly. The paper mentions earlier visible-face compensation; please explain how this handles the alignment for moving objects before applying Eq. 3.
-
The reconstruction evaluation needs more detail. Please confirm that all methods in Table 4 are evaluated on the same dynamic pixels, since OmniRe's masks depend partly on projected object boxes and those boxes are being replaced here. Also, which PandaSet sequences were used, and are the scores measured on images used to fit the reconstruction or on held-out frames? Appendix D.6 lists detection sequences but does not clarify the reconstruction setup.
Questions for authors
Please see the weaknesses above.
3 comments
Further Clarification of Methodological Novelty and Additional Evidence
Dear Reviewer,
We sincerely thank you again for your positive assessment and constructive feedback. We are pleased that you recognized the practical value of Track2Recon and the effectiveness of its core components.
We would like to briefly highlight that we have carefully addressed both of your concerns in our detailed rebuttal and revised manuscript.
Regarding cross-frame alignment, we clarified that our method first transforms observations into stabilized object-local coordinates to compensate for the dominant translation and rotation, and then applies residual ICP registration to align observations in a common reference frame before boundary fusion. This also applies to moving rigid objects and avoids directly combining boundaries measured in inconsistent coordinate systems.
Regarding reconstruction evaluation, we have explicitly clarified the PandaSet sequence IDs, the use of identical images and shared ground-truth-based dynamic evaluation masks across all methods, and the fact that the reported results measure training-view reconstruction quality rather than held-out novel-view synthesis.
We have also revised the manuscript to make these procedures clearer and provided anonymous code to facilitate reproducibility. In addition, Figure 2 illustrates our core geometry-recovery mechanisms, and the supplementary materials provide further qualitative comparisons.
We hope these clarifications have adequately addressed your concerns. We would greatly appreciate it if you could revisit our responses and revised manuscript when convenient.
Thank you again for your time and support.
Thank you for clarifying the alignment procedure and reconstruction evaluation. This addresses my main concerns, and I will keep my score unchanged.
Response to Reviewer 38UR: Cross-Frame Alignment and Reconstruction Evaluation
Reviewer 38UR
We thank the reviewer for the positive assessment of Track2Recon and for identifying two points that require further clarification: cross-frame coordinate alignment before Equation (3), and the reconstruction evaluation protocol.
1. Cross-frame alignment before Equation (3)
Our implementation uses a two-stage alignment procedure.
First, we transform the raw LiDAR point cloud in each frame into the corresponding object-local coordinate system using the initial object center and a stabilized horizontal principal axis:
qt=Rt⊤(pt,xy−ct,xy0).
This operation compensates for the object's dominant translation and rotation and keeps the physical axis directions consistent across observations.
Second, we select a reliable reference observation frame r and use ICP to further estimate residual rigid misalignment after the coarse object-local transformation:
qt→r=Rt→rqt+tt→r.
The transformed point clouds are then represented in the reference coordinate system. Only after registration do we extract the lower and upper object boundaries and perform reliability-weighted quantile fusion in the unified reference coordinate system.
For moving rigid vehicles, the object-local coordinate transformation first compensates for the dominant motion in each frame, after which ICP estimates the residual alignment error. Therefore, this procedure does not assume that a vehicle remains at a fixed world-coordinate position throughout the sequence.
The three operations serve different purposes:
Therefore, the boundaries fused in Equation (3) are not measured in independent, unrelated coordinate systems on a frame-by-frame basis.
2. Reconstruction sequences, shared dynamic masks, and evaluation images
The reconstruction experiments use the following 30 PandaSet sequences:
001, 004, 006, 008, 011, 012, 015, 016, 021, 024, 028, 035, 041, 042, 044, 053, 063, 065, 070, 084, 088, 089, 101, 106, 109, 120, 122, 123, 149, 158.
All methods use exactly the same selected images, camera configurations, and OmniRe optimization settings. The only varying inputs are the recovered 3D bounding boxes and persistent track IDs.
The dynamic evaluation masks are generated in advance by projecting the dataset-provided ground-truth 3D bounding boxes of dynamic objects onto the image plane. All methods use the same masks, which are independent of each method's predicted bounding boxes and track information.
This ensures that even if an object is missed by a method, its corresponding region is not excluded from evaluation.
Except for the explicitly labeled GT-object-input control experiment, ground-truth bounding boxes are used only to define these shared evaluation regions.
We further clarify that all selected reconstruction images are used during OmniRe optimization and are also used to compute the final evaluation metrics. Table 4 does not contain an independent held-out-frame or held-out-view split.
Therefore, Table 4 measures reconstruction quality on the training views, rather than held-out novel-view synthesis performance.
The dynamic-region results are as follows:
We will add the explicit sequence list, the construction procedure for the shared dynamic masks, and the training-view evaluation protocol to the revised manuscript.
We again thank the reviewer for these comments. They have helped us clarify the geometric validity of cross-frame boundary fusion and ensure that the downstream reconstruction results are interpreted accurately within the scope of the actual evaluation protocol.