- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents LucidLyra, a framework for generating 3D Gaussian scenes with jointly predicted geometry, appearance, language-aligned semantic features, and instance embeddings. Building on Lyra, the method distills semantic and instance supervision from 2D foundation models. Track-based target aggregation, input-image feature conditioning, and joint reconstruction–understanding training produce queryable 3D representations without per-scene optimization. The framework also extends to time-conditioned dynamic reconstruction. Experiments demonstrate substantial improvements over the evaluated external segmentation baselines while preserving static reconstruction quality, with performance generally competitive with direct teacher lifting at substantially lower reported per-scene processing cost.
Critical strengths
- The motivation is natural and practically relevant. Integrating semantic and instance prediction into the generative 3D reconstruction pipeline yields a unified, jointly trained Gaussian decoder, reducing the need for expensive per-scene teacher processing and optimization.
- The distillation pipeline is carefully developed. Cross-view track aggregation, rendered and direct per-Gaussian supervision, and hierarchical instance features provide a coherent approach to learning from imperfect 2D teachers without manual 3D training annotations.
Critical weaknesses
- The benefit of joint learning is not adequately isolated. Table 2 identifies C-RADIO conditioning as the largest contributor among the evaluated components, but does not establish the additional benefit of jointly optimizing reconstruction and understanding.
- Instance supervision has limited coverage of newly revealed objects. The supervision excludes objects that appear in later generated views without a first-frame seed, and the paper does not separately evaluate performance on these objects.
Questions for authors
- Could the authors include a sequential-training baseline that first fine-tunes Lyra for reconstruction, then freezes its geometry and appearance predictions and trains only the understanding branch? This would help isolate the benefit of joint optimization.
- Could the authors provide a runtime breakdown for teacher lifting, including RGB decoding, semantic feature extraction, SAM/SAM2 processing, and feature assignment?
- How would a simple baseline—lifting RADSeg+ features onto frozen Lyra Gaussians, followed by geometry-aware 3D clustering—perform?
- Does the lifted-teacher reference share the same restriction to instances seeded in the first frame? New objects can be initialized with SAM in later generated frames and tracked with SAM2, with appropriate cross-view identity association. If so, it would be better to report the results for both initially visible and all objects under the same evaluation protocol.
6 comments
Official Comment by the Authors
Table: Observed/unobserved scene regions and objects.
| Full | Observed Scene | Unobserved Scene | Observed Objects | Unobserved Objects | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | PSNR | mIoU | mTrackPR | PSNR | mIoU | mTrackPR | PSNR | mIoU | mTrackPR | mTrackPR | mTrackPR25 | mTrackPR | mTrackPR25 |
| LucidLyra-SDG | |||||||||||||
| Lyra + Lifted Teachers | 24.5 | 58.4 | 19.8 | 25.6 | 60.5 | 25.0 | 22.6 | 50.1 | 12.1 | 22.9 | 32.5 | 6.6 | 14.0 |
| Lyra + LangSplatV2 | 24.5 | 31.9 | -- | 25.6 | 32.5 | -- | 22.6 | 28.4 | -- | -- | -- | -- | -- |
| Lyra + SFS | 24.5 | 31.6 | -- | 25.6 | 32.7 | -- | 22.6 | 25.7 | -- | -- | -- | -- | -- |
| Lyra + Unified-Lift | 24.5 | -- | 9.9 | 25.6 | -- | 12.6 | 22.6 | -- | 7.7 | 11.5 | 16.3 | 3.2 | 6.9 |
| Lyra + Chorus | 24.5 | 12.0 | -- | 25.6 | 12.1 | -- | 22.6 | 11.5 | -- | -- | -- | -- | -- |
| Lyra + IGGT | -- | 30.1 | 5.9 | -- | 31.2 | 5.9 | -- | 27.1 | 4.3 | 5.4 | 9.2 | 4.9 | 9.2 |
| LucidLyra (Ours) | 25.0 | 63.1 | 19.0 | 26.3 | 64.9 | 23.1 | 23.0 | 55.2 | 12.8 | 21.9 | 30.4 | 10.9 | 21.7 |
| LucidLyra+ (Ours) | 25.0 | 67.4 | 20.6 | 26.3 | 69.6 | 25.0 | 23.0 | 57.6 | 13.9 | 23.7 | 33.3 | 9.1 | 17.4 |
| LucidLyra-ADE20K | |||||||||||||
| Lyra + Lifted Teachers | 24.6 | 49.3 | 16.5 | 25.4 | 50.3 | 19.1 | 22.1 | 42.5 | 11.9 | 17.5 | 26.5 | 5.3 | 9.6 |
| Lyra + LangSplatV2 | 24.6 | 36.3 | -- | 25.4 | 37.4 | -- | 22.1 | 25.2 | -- | -- | -- | -- | -- |
| Lyra + SFS | 24.6 | 34.4 | -- | 25.4 | 35.4 | -- | 22.1 | 24.6 | -- | -- | -- | -- | -- |
| Lyra + Unified-Lift | 24.6 | -- | 7.2 | 25.4 | -- | 8.1 | 22.1 | -- | 6.6 | 7.5 | 11.2 | 3.4 | 7.1 |
| Lyra + Chorus | 24.6 | 9.4 | -- | 25.4 | 9.7 | -- | 22.1 | 7.7 | -- | -- | -- | -- | -- |
| Lyra + IGGT | -- | 32.7 | 9.2 | -- | 33.7 | 9.7 | -- | 26.0 | 7.1 | 9.2 | 15.6 | 6.9 | 12.9 |
| LucidLyra (Ours) | 24.7 | 49.5 | 17.8 | 25.5 | 50.2 | 19.9 | 22.3 | 43.3 | 14.8 | 18.4 | 29.4 | 12.0 | 19.8 |
| LucidLyra+ (Ours) | 24.7 | 53.6 | 18.2 | 25.5 | 54.3 | 19.6 | 22.3 | 46.2 | 15.6 | 18.1 | 27.6 | 14.9 | 22.4 |
Q2: Runtime breakdown for teacher lifting.
Below, we break down the per-scene runtime of the Lifted Teachers reference measured on a single NVIDIA A100 (80 GB) GPU. As in other SAM-based pipelines, SAM automatic mask generation and mask propagation dominates runtime (51.7 min, or 81% of the total). Notably, even if SAM/SAM2 preprocessing were parallelized across multiple GPUs, the remaining stages alone (6.4 min for RGB decoding, SigLIP2 extraction, and 3D lifting) still require over 3x the total runtime of feed-forward LucidLyra (2.1 min).
Official Comment by the Authors
| Runtime in min/scene | |
|---|---|
| Video latent-to-RGB decoding | 1.0 |
| SAM automatic mask generation | 1.3 |
| SAM2 mask propagation, including tracking I/O and setup | 51.7 |
| Coverage-based track consolidation | 4.6 |
| Extract projected SigLIP2 semantic features | 3.6 |
| Lift semantic features and instance IDs | 1.6 |
| Post-lifting classification, rendering, and ID decoding | 0.2 |
| Total estimated runtime | 64.0 |
Q3: ``...lift RADSeg+ features onto frozen Lyra ... geometry-aware 3D clustering''
We implemented the suggested baseline by lifting RADSeg+ features directly onto frozen Lyra Gaussians and sweeping clustering configurations over 3D positions alone as well as joint RADSeg+ features and 3D positions. The strongest configuration runs HDBSCAN within each predicted semantic class over normalized RADSeg+ features and scene-normalized, weighted Gaussian positions.
While lifting RADSeg+ yields competitive semantic segmentation, approaching Lifted Teachers on SDG (57.1 vs. 60.5 mIoU) and improving over it on ADE20K (53.2 vs. 50.1 mIoU), it remains below LucidLyra+ across both datasets (67.9 and 53.7 mIoU). More critically, heuristic feature–geometry clustering fails at 3D instance segmentation (4.8 vs. 22.0 mTrackPR on SDG; 7.8 vs. 18.5 on ADE20K). Adjacent same-class objects with similar features and positions frequently merge into a single cluster, whereas intra-object appearance or geometric variation splits single objects into fragments, confirming the need for learned instance embeddings.
Table: RADSeg+ lifting and geometric clustering
| Method | PSNR ↑ | LPIPS ↓ | mIoU ↑ | mAcc ↑ | mTrackPR ↑ | TrackPR25 ↑ |
|---|---|---|---|---|---|---|
| LucidLyra-SDG | ||||||
| Lifted Teachers | 24.49 | 0.211 | 60.5 | 75.3 | 22.8 | 33.2 |
| Suggested Method | 24.49 | 0.211 | 57.1 | 75.1 | 4.8 | 10.1 |
| LucidLyra (Ours) | 25.02 | 0.186 | 64.1 | 79.2 | 21.5 | 29.3 |
| LucidLyra+ (Ours) | 25.02 | 0.186 | 67.9 | 81.7 | 22.0 | 31.1 |
| LucidLyra-ADE20K | ||||||
| Lifted Teachers | 24.59 | 0.217 | 50.1 | 64.5 | 17.3 | 26.8 |
| Suggested Method | 24.59 | 0.217 | 53.2 | 70.6 | 7.8 | 14.2 |
| LucidLyra (Ours) | 24.72 | 0.199 | 49.8 | 65.6 | 18.5 | 29.7 |
| LucidLyra+ (Ours) | 24.72 | 0.199 | 53.7 | 71.6 | 18.5 | 28.5 |
Official Comment by the Authors
Thank you for your time, effort, and the positive review of our work. Below, we address the questions and requested experiments. We appreciate your feedback and will incorporate it into the revised version of the paper.
W1/Q1: "...sequential-training baseline that first fine-tunes Lyra for reconstruction, then ... trains only the understanding branch?"
We conducted the suggested sequential-training experiment: we first fine-tuned with C-RADIO conditioning for reconstruction (30k iterations), then froze the backbone and trained the attached understanding head of the 3DGS decoder for another 30k iterations. Because reconstruction-only training underutilizes C-RADIO features and locks in a latent space tailored solely to RGB appearance, semantic and instance segmentation drop substantially (-13.7 mIoU, -6.8 mTrackPR) compared to joint training, where both objectives shape the shared representation.
Table: Sequential versus joint training evaluated on LucidLyra-SDG (following Table 2 main paper).
W2 Separate evaluation of observed and unobserved regions/objects in generated scenes.
We thank the reviewer for this suggestion. To directly evaluate scene understanding beyond the input view, we break down the Table 1 evaluation into regions observed versus unobserved in the input image, obtaining visibility masks by lifting the input view into 3D. An object is unobserved when the majority of its tracklet pixels are unseen. While our primary benchmark already captures newly revealed surfaces of partially occluded objects and background ("stuff") regions via propagation from the input view, we also extended the benchmark ground truth to cover entirely unseeded objects appearing later along the trajectories. These extended annotations are used to evaluate all methods, with no changes to LucidLyra's training or inference. Specifically, we prompted SAM in unlabeled benchmark regions, propagated masks bidirectionally with SAM2, and used a GPT-6 Astra pipeline to resolve overlaps, assign ADE20K-150 labels, and verify depth-consistent 3D cross-trajectory correspondences, all followed by human verification.
Evaluated on this extended benchmark, LucidLyra consistently outperforms all baselines across both observed and unobserved regions, matching or surpassing the 3D-lifted teachers even in newly synthesized areas. While absolute metrics are naturally higher in observed regions (a trend shared across all baselines and already present in underlying RGB reconstruction) LucidLyra maintains strong semantic and instance coherence well beyond the input view, confirming that our 3D representation generalize effectively to newly generated content.