acceptodds
Under review as a conference paper at ICLR 2027

LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation

Abstract

We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into 3D Gaussians that jointly encode geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization, while training requires neither manual 3D annotations nor captured multi-view data. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, while preserving reconstruction quality. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, asset extraction, and robotic simulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Generation, 3D Scene Understanding, 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dciLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents LucidLyra, a framework for generating 3D Gaussian scenes with jointly predicted geometry, appearance, language-aligned semantic features, and instance embeddings. Building on Lyra, the method distills semantic and instance supervision from 2D foundation models. Track-based target aggregation, input-image feature conditioning, and joint reconstruction–understanding training produce queryable 3D representations without per-scene optimization. The framework also extends to time-conditioned dynamic reconstruction. Experiments demonstrate substantial improvements over the evaluated external segmentation baselines while preserving static reconstruction quality, with performance generally competitive with direct teacher lifting at substantially lower reported per-scene processing cost.

Critical strengths
  1. The motivation is natural and practically relevant. Integrating semantic and instance prediction into the generative 3D reconstruction pipeline yields a unified, jointly trained Gaussian decoder, reducing the need for expensive per-scene teacher processing and optimization.
  2. The distillation pipeline is carefully developed. Cross-view track aggregation, rendered and direct per-Gaussian supervision, and hierarchical instance features provide a coherent approach to learning from imperfect 2D teachers without manual 3D training annotations.
Critical weaknesses
  1. The benefit of joint learning is not adequately isolated. Table 2 identifies C-RADIO conditioning as the largest contributor among the evaluated components, but does not establish the additional benefit of jointly optimizing reconstruction and understanding.
  2. Instance supervision has limited coverage of newly revealed objects. The supervision excludes objects that appear in later generated views without a first-frame seed, and the paper does not separately evaluate performance on these objects.
Questions for authors
  1. Could the authors include a sequential-training baseline that first fine-tunes Lyra for reconstruction, then freezes its geometry and appearance predictions and trains only the understanding branch? This would help isolate the benefit of joint optimization.
  2. Could the authors provide a runtime breakdown for teacher lifting, including RGB decoding, semantic feature extraction, SAM/SAM2 processing, and feature assignment?
  3. How would a simple baseline—lifting RADSeg+ features onto frozen Lyra Gaussians, followed by geometry-aware 3D clustering—perform?
  4. Does the lifted-teacher reference share the same restriction to instances seeded in the first frame? New objects can be initialized with SAM in later generated frames and tracked with SAM2, with appropriate cross-view identity association. If so, it would be better to report the results for both initially visible and all objects under the same evaluation protocol.
6 comments
AuthorsOpenReview ↗

Official Comment by the Authors

Thank you for your time, effort, and the positive review of our work. Below, we address the questions and requested experiments. We appreciate your feedback and will incorporate it into the revised version of the paper.

W1/Q1: "...sequential-training baseline that first fine-tunes Lyra for reconstruction, then ... trains only the understanding branch?"

We conducted the suggested sequential-training experiment: we first fine-tuned with C-RADIO conditioning for reconstruction (30k iterations), then froze the backbone and trained the attached understanding head of the 3DGS decoder for another 30k iterations. Because reconstruction-only training underutilizes C-RADIO features and locks in a latent space tailored solely to RGB appearance, semantic and instance segmentation drop substantially (-13.7 mIoU, -6.8 mTrackPR) compared to joint training, where both objectives shape the shared representation.

Table: Sequential versus joint training evaluated on LucidLyra-SDG (following Table 2 main paper).

PSNR ↑LPIPS ↓mIoU ↑mAcc ↑mTrackPR ↑TrackPR25 ↑
Sequential training24.860.18950.464.714.725.3
LucidLyra (Ours)25.020.18664.179.221.529.3

W2 Separate evaluation of observed and unobserved regions/objects in generated scenes.

We thank the reviewer for this suggestion. To directly evaluate scene understanding beyond the input view, we break down the Table 1 evaluation into regions observed versus unobserved in the input image, obtaining visibility masks by lifting the input view into 3D. An object is unobserved when the majority of its tracklet pixels are unseen. While our primary benchmark already captures newly revealed surfaces of partially occluded objects and background ("stuff") regions via propagation from the input view, we also extended the benchmark ground truth to cover entirely unseeded objects appearing later along the trajectories. These extended annotations are used to evaluate all methods, with no changes to LucidLyra's training or inference. Specifically, we prompted SAM in unlabeled benchmark regions, propagated masks bidirectionally with SAM2, and used a GPT-6 Astra pipeline to resolve overlaps, assign ADE20K-150 labels, and verify depth-consistent 3D cross-trajectory correspondences, all followed by human verification.

Evaluated on this extended benchmark, LucidLyra consistently outperforms all baselines across both observed and unobserved regions, matching or surpassing the 3D-lifted teachers even in newly synthesized areas. While absolute metrics are naturally higher in observed regions (a trend shared across all baselines and already present in underlying RGB reconstruction) LucidLyra maintains strong semantic and instance coherence well beyond the input view, confirming that our 3D representation generalize effectively to newly generated content.

AuthorsOpenReview ↗

Official Comment by the Authors

Table: Observed/unobserved scene regions and objects.

FullObserved SceneUnobserved SceneObserved ObjectsUnobserved Objects
MethodPSNRmIoUmTrackPRPSNRmIoUmTrackPRPSNRmIoUmTrackPRmTrackPRmTrackPR25mTrackPRmTrackPR25
LucidLyra-SDG
Lyra + Lifted Teachers24.558.419.825.660.525.022.650.112.122.932.56.614.0
Lyra + LangSplatV224.531.9--25.632.5--22.628.4----------
Lyra + SFS24.531.6--25.632.7--22.625.7----------
Lyra + Unified-Lift24.5--9.925.6--12.622.6--7.711.516.33.26.9
Lyra + Chorus24.512.0--25.612.1--22.611.5----------
Lyra + IGGT--30.15.9--31.25.9--27.14.35.49.24.99.2
LucidLyra (Ours)25.063.119.026.364.923.123.055.212.821.930.410.921.7
LucidLyra+ (Ours)25.067.420.626.369.625.023.057.613.923.733.39.117.4
LucidLyra-ADE20K
Lyra + Lifted Teachers24.649.316.525.450.319.122.142.511.917.526.55.39.6
Lyra + LangSplatV224.636.3--25.437.4--22.125.2----------
Lyra + SFS24.634.4--25.435.4--22.124.6----------
Lyra + Unified-Lift24.6--7.225.4--8.122.1--6.67.511.23.47.1
Lyra + Chorus24.69.4--25.49.7--22.17.7----------
Lyra + IGGT--32.79.2--33.79.7--26.07.19.215.66.912.9
LucidLyra (Ours)24.749.517.825.550.219.922.343.314.818.429.412.019.8
LucidLyra+ (Ours)24.753.618.225.554.319.622.346.215.618.127.614.922.4

Q2: Runtime breakdown for teacher lifting.

Below, we break down the per-scene runtime of the Lifted Teachers reference measured on a single NVIDIA A100 (80 GB) GPU. As in other SAM-based pipelines, SAM automatic mask generation and mask propagation dominates runtime (51.7 min, or 81% of the total). Notably, even if SAM/SAM2 preprocessing were parallelized across multiple GPUs, the remaining stages alone (6.4 min for RGB decoding, SigLIP2 extraction, and 3D lifting) still require over 3x the total runtime of feed-forward LucidLyra (2.1 min).

AuthorsOpenReview ↗

Official Comment by the Authors

Runtime in min/scene
Video latent-to-RGB decoding1.0
SAM automatic mask generation1.3
SAM2 mask propagation, including tracking I/O and setup51.7
Coverage-based track consolidation4.6
Extract projected SigLIP2 semantic features3.6
Lift semantic features and instance IDs1.6
Post-lifting classification, rendering, and ID decoding0.2
Total estimated runtime64.0

Q3: ``...lift RADSeg+ features onto frozen Lyra ... geometry-aware 3D clustering''

We implemented the suggested baseline by lifting RADSeg+ features directly onto frozen Lyra Gaussians and sweeping clustering configurations over 3D positions alone as well as joint RADSeg+ features and 3D positions. The strongest configuration runs HDBSCAN within each predicted semantic class over normalized RADSeg+ features and scene-normalized, weighted Gaussian positions.

While lifting RADSeg+ yields competitive semantic segmentation, approaching Lifted Teachers on SDG (57.1 vs. 60.5 mIoU) and improving over it on ADE20K (53.2 vs. 50.1 mIoU), it remains below LucidLyra+ across both datasets (67.9 and 53.7 mIoU). More critically, heuristic feature–geometry clustering fails at 3D instance segmentation (4.8 vs. 22.0 mTrackPR on SDG; 7.8 vs. 18.5 on ADE20K). Adjacent same-class objects with similar features and positions frequently merge into a single cluster, whereas intra-object appearance or geometric variation splits single objects into fragments, confirming the need for learned instance embeddings.

Table: RADSeg+ lifting and geometric clustering

MethodPSNR ↑LPIPS ↓mIoU ↑mAcc ↑mTrackPR ↑TrackPR25 ↑
LucidLyra-SDG
Lifted Teachers24.490.21160.575.322.833.2
Suggested Method24.490.21157.175.14.810.1
LucidLyra (Ours)25.020.18664.179.221.529.3
LucidLyra+ (Ours)25.020.18667.981.722.031.1
LucidLyra-ADE20K
Lifted Teachers24.590.21750.164.517.326.8
Suggested Method24.590.21753.270.67.814.2
LucidLyra (Ours)24.720.19949.865.618.529.7
LucidLyra+ (Ours)24.720.19953.771.618.528.5
Reviewer_VnUsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LucidLyra, which extends single-image generative 3D reconstruction from producing renderable scenes to generating semantically queryable, instance-aware 3D representations. Building upon Lyra, the method jointly predicts Gaussian geometry, appearance, semantic features, and instance embeddings through multi-task distillation from generated multi-view observations and frozen 2D foundation models. This enables feed-forward 3D scene understanding without requiring manual 3D annotations or per-scene optimization. Experiments on synthetic, real-image-conditioned, and dynamic generative benchmarks demonstrate effective improvements over several existing 3DGS understanding baselines while largely preserving reconstruction quality.

Critical strengths
  1. The paper introduces a useful method of unifying generative 3D scene reconstruction with semantic and instance understanding. Unlike existing approaches that primarily focus on visual reconstruction or understanding observed multi-view scenes, this formulation enables structured 3D scene generation from a single image.

  2. LucidLyra achieves improvements over several existing 3DGS understanding baselines and demonstrates advantages over direct teacher lifting in certain settings, while preserving reconstruction quality. These results support the effectiveness of incorporating semantic and instance learning into feed-forward generative 3D reconstruction.

Critical weaknesses

1. Incremental extension of Lyra and standard 2D-to-3D distillation.

LucidLyra inherits Lyra's generative reconstruction framework and primarily extends it with semantic/instance prediction heads, C-RADIO conditioning, and 2D foundation model distillation. The key components, such as SAM/SAM2 tracking, semantic feature distillation, track-level aggregation, and contrastive instance learning, are well-established techniques.

  • The paper does not introduce a new mechanism for learning 3D-consistent semantic or instance representations. Cross-view consistency largely relies on tracked 2D supervision and the shared Gaussian representation inherited from Lyra.
  • The distinction from existing approaches, particularly IGGT's instance-grounded supervision and prior 3DGS feature distillation methods, is not sufficiently established.

2. The claimed benefits of joint learning.

The paper claims that joint learning allows geometry and semantics to reinforce each other. However, the reported results are equally consistent with a simpler explanation: strong 2D teacher knowledge is effectively amortized into an existing reconstruction model.

  • Marginal geometric gains. The PSNR gain over the reconstruction-only control is only 0.14 dB (Table 2), and rendering quality alone does not establish improved geometry. Meanwhile, removing C-RADIO reduces mIoU by 7.2 points, indicating that stronger visual conditioning is a major contributor.
  • Strong performance of direct teacher lifting. The Lyra + Lifted Teachers baseline achieves higher instance mTrackPR on SDG Static (22.8 vs. 21.5), slightly higher semantic mIoU on ADE20K (50.1 vs. 49.8), and identical semantic mIoU on the dynamic benchmark (53.2).

These results make a reasonable case for distillation efficiency, but fall short of showing that joint training actually improves the 3D representation. Controlled comparisons with a frozen backbone would be needed to support this point.

3. Evaluation setup relative to the open-world claims.

  • Open-vocabulary understanding is evaluated only on ADE20K categories present in each reference scene. This does not establish generalization to unseen concepts or compositional language queries.
  • Instance understanding is evaluated within ground-truth thing regions, bypassing full-scene instance discovery and penalization of false positives outside those regions.

Each benchmark contains only 25 scenes, is the number of scenes sufficient?

Questions for authors

Q1. What does the proposed semantic/instance learning mechanism offer beyond existing 2D-to-3D distillation? In particular, does track-level semantic pooling outperform direct pixel-level feature distillation under identical supervision and decoder settings? How does the proposed instance supervision differ from IGGT's instance-grounded learning, beyond the use of generated trajectories?

Q2. Are the improvements robust under matched input and teacher supervision? IGGT uses 11 generated RGB views at 518 × 280 resolution, while LucidLyra processes video latents from six generated camera trajectories. The baselines also rely on different semantic teachers. How do the results change with comparable input budgets and identical teacher targets?

Q3. How does the method perform under genuinely open-world evaluation? Could the authors evaluate unseen semantic categories and full-scene instance segmentation without reference foreground cropping? These experiments are essential to distinguish open-world scene understanding from classification and grouping within known semantic and spatial support.

Q4. How are newly revealed objects represented and supervised? SAM/SAM2 tracks are initialized from the first frame, leaving objects visible only in later generated views without instance supervision. What proportion of generated content is affected? More importantly, can the model reliably recover the semantic identity of such objects, and how is consistency enforced across independently tracked camera trajectories?

12 comments
AuthorsOpenReview ↗

Official Comment by the Authors

Thank you for your time, effort, and the constructive feedback on our work. Below, we address the questions and requested experiments. We appreciate your feedback and will incorporate it into the revised version of the paper.

W1/Q1: What does the proposed semantic/instance learning mechanism offer beyond existing 2D-to-3D distillation?

Our core contribution is unifying single-image generative 3D reconstruction, open-vocabulary 3D semantic segmentation, and 3D instance segmentation within a single feed-forward model. While we build on established feature distillation (Hinton et al., 2015; Kobayashi et al., 2022) and supervised contrastive learning (Khosla et al., 2020; Bhalgat et al., 2023), prior methods predominantly focus either on open-vocabulary semantic/instance understanding or generative 3D reconstruction. In contrast, LucidLyra must predict 3D geometry, appearance, semantics, and instances. Our technical contribution lies in designing an architecture and supervision signals, while balancing them so all tasks train jointly without degrading, as validated by our ablations in Table 2 (main paper).

How does the proposed instance supervision differ from IGGT's instance-grounded learning, beyond the use of generated trajectories?

First, unlike IGGT (Li et al., 2026), which takes explicit multi-view RGB images as input, LucidLyra decodes directly from generative video latents and supervises semantic and instance attributes on a generated 3D Gaussian scene via differentiable rendering. IGGT uses a pairwise pull/push margin loss with separate weights and a margin. We instead apply SupCon (Khosla et al., 2020) to rendered embeddings across views. This avoids tuning a separate attraction–repulsion margin while its softmax formulation automatically emphasizes hard, confusable negatives.

Semantic labels: track-level pooling vs. pixel-level feature distillation

We thank the reviewer for this suggestion. We repeated the Table 1 experiment under identical settings, replacing track-pooled semantic targets with unpooled pixel-level features. Track pooling improves semantic segmentation (+7.0 mIoU on SDG, +2.2 mIoU on ADE20K) and simultaneously boosts instance segmentation (+2.4 and +1.6 mTrackPR), showing that cleaner semantic supervision is beneficial. As discussed in Sec. 3.3, α\alpha-blending mixes multiple Gaussian descriptors along each ray, making rendered supervision ambiguous, while direct per-Gaussian supervision remains sensitive to local and view-dependent fluctuations in 2D teacher features. Track pooling denoises these fluctuations prior to distillation, providing a much more consistent cross-view supervision signal. Practically, offline track pooling also reduces storage from millions of dense pixel embeddings per scene to a few hundred track descriptors, accelerating data loading without the iteration time and GPU memory overhead of online feature extraction.

Table: Track-level pooling vs. pixel-level LucidLyra feature distillation.

MethodPSNR ↑mIoU ↑mTrackPR ↑
LucidLyra-SDG
LucidLyra pixel-level24.9157.119.1
LucidLyra (pooled, Ours)25.0264.121.5
LucidLyra-ADE20K
LucidLyra pixel-level24.6647.616.9
LucidLyra (pooled, Ours)24.7249.818.5
AuthorsOpenReview ↗

Official Comment by the Authors

W2/Q2: Benefits of joint learning, teacher-matched alternatives, comparisons against frozen-backbone.

``...paper argues that joint learning allows geometry and semantics to reinforce each other.''

Our goal is to distill complementary generative and perceptual teachers into a unified model without sacrificing single-task performance. We acknowledge that phrasing in the introduction suggests mutual reinforcement of geometry and understanding and will revise it to align better with the rest of the paper (e.g., Section 4.2 "LucidLyra preserves reconstruction fidelity while adding understanding fields"). Crucially, however, joint training is essential for learning an expressive shared representation (as verified by our frozen-backbone ablation below). Our method is on par width or even improves upon Lifted Teachers while preserving reconstruction quality and cutting per-scene runtime from 64 minutes to 2.1 minutes (2.3 minutes for LucidLyra+).

``IGGT uses 11 generated RGB views at 518 × 280 resolution, while LucidLyra processes video latents from six generated camera trajectories ... baselines also rely on different semantic teachers''

Because related works rely on different 2D teachers, we report Lifted Teachers as a direct reference. For all baselines, we used official implementations and tuned inputs and hyperparameters to maximize competitiveness without altering the underlying methods. For IGGT, we sample 11 views maximizing viewpoint diversity and scene coverage. Increasing resolution or adding more views improves none of the metrics while substantially increasing runtime due to instance-feature clustering.

Table: IGGT accuracy and runtime across different input configurations.

Method /configurationmIoU ↑mAcc ↑mTrackPR ↑TrackPR25 ↑Runtime min/scene ↓
LucidLyra-SDG
Lyra + IGGT29.238.35.29.33.6
w/ higher input resolution 784×42023.933.51.83.813.7
w/ 21 input images27.336.34.88.810.4
w/ 31 input images (batched)27.537.03.46.69.8
LucidLyra-ADE20K
Lyra + IGGT32.741.29.516.63.6
w/ higher input resolution 784×42027.235.04.89.513.7
w/ 21 input images30.839.29.116.010.4
w/ 31 input images (batched)32.641.07.213.29.8
AuthorsOpenReview ↗

Official Comment by the Authors

To further compare under identical semantic teachers, we reran LangSplatV2, SFS, and IGGT using our semantic preprocessing and teacher (Chorus already shares our underlying teacher). While stronger teacher features improve all three baselines, none exceeds Lifted Teachers or our method. This demonstrates that the underlying vision-language representation in open-vocabulary understanding methods is a major design choice. In addition, LangSplatV2 and SFS require hours of preprocessing and per-scene optimization (697 and 168 minutes vs. 2.1 minutes feed-forward for LucidLyra).

Table: Integrating our semantic teacher into the baseline methods.

MethodPSNR ↑LPIPS ↓mIoU ↑mAcc ↑mTrackPR ↑TrackPR25 ↑runtime minutes per scene
LucidLyra-SDG
Lyra + Lifted Teachers24.490.21160.575.322.833.264
Lyra + LangSplatV224.490.21157.974.9----697
Lyra + SFS24.490.21158.271.9----168
Lyra + Chorus24.490.21111.519.3----1.0
Lyra + IGGT----39.957.65.29.33.6
LucidLyra (Ours)25.020.18664.179.221.529.32.1
LucidLyra+ (Ours)25.020.18667.981.722.031.12.3
LucidLyra-ADE20K
Lyra + Lifted Teachers24.590.21750.164.517.326.864
Lyra + LangSplatV224.590.21749.864.4----697
Lyra + SFS24.590.21749.064.5----168
Lyra + Chorus24.590.2179.316.1----1.0
Lyra + IGGT----42.460.39.516.63.6
LucidLyra (Ours)24.720.19949.865.618.529.72.1
LucidLyra+ (Ours)24.720.19953.771.618.528.52.3

Comparisons against frozen-backbone fine-tuned for understanding

We conducted the suggested sequential-training experiment: we first fine-tuned with C-RADIO conditioning for reconstruction (30k iterations), then froze the backbone and trained the attached understanding head of the 3DGS decoder for another 30k iterations. Because reconstruction-only training underutilizes C-RADIO features and locks in a latent space tailored solely to RGB appearance, semantic and instance segmentation drop substantially (-13.7 mIoU, -6.8 mTrackPR) compared to joint training, where both objectives shape the shared representation.

Table: Sequential versus joint training evaluated on LucidLyra-SDG (following Table 2 main paper).

PSNR ↑LPIPS ↓mIoU ↑mAcc ↑mTrackPR ↑TrackPR25 ↑
Sequential training24.860.18950.464.714.725.3
LucidLyra (Ours)25.020.18664.179.221.529.3
Reviewer_PknQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LucidLyra, a framework for jointly generating and understanding 3D scenes from a single image. Building on Lyra, it distills semantic and instance-level knowledge from frozen 2D foundation models into a feed-forward 3D Gaussian decoder using generated multi-view videos. The resulting representation supports novel-view rendering, open-vocabulary segmentation, and instance grouping without per-scene optimization. Experiments on static and dynamic scenes demonstrate improved scene understanding while largely preserving reconstruction quality.

Critical strengths
  • The paper presents a practical framework that jointly predicts geometry, appearance, semantic, and instance features in a feed-forward 3DGS representation. This enables direct querying and manipulation of generated scenes without expensive per-scene optimization.

  • LucidLyra achieves competitive semantic and instance segmentation performance while preserving reconstruction quality. In particular, it achieves comparable accuracy to direct teacher lifting while substantially reducing per-scene processing time, demonstrating the practical value of the proposed framework.

Critical weaknesses
  • A key motivation of generative 3D reconstruction is to model regions that are not visible in the input image. However, LucidLyra extracts C-RADIO features only from the input image, while Section 3.3 excludes newly revealed objects without first-frame masks from instance supervision. Although the overall segmentation results are promising, the paper does not separately evaluate semantic and instance understanding in these newly revealed regions. This leaves uncertainty about whether the proposed framework can provide reliable scene understanding beyond the initially observed content, which is central to its generative 3D setting.
Questions for authors
  • While LucidLyra improves reconstruction quality on static scenes, its dynamic extension slightly underperforms the original Lyra in both PSNR and LPIPS (Table 4). Could the authors clarify what causes this performance degradation?

  • Given that C-RADIO features are extracted only from the input image and broadcast across generated views, could the authors analyze whether semantic and instance understanding performance degrades under large viewpoint changes, particularly in regions not visible in the input image?

5 comments
AuthorsOpenReview ↗

Official Comment by the Authors

Thank you for your time, effort, and the positive review of our work. Below, we address the questions and requested experiments. We appreciate your feedback and will incorporate it into the revised version of the paper.

W1/Q1 Separate evaluation of observed and unobserved regions in generated scenes.

We thank the reviewer for this constructive suggestion. To directly evaluate scene understanding beyond the input view, we break down the Table 1 evaluation into regions observed versus unobserved in the input image, obtaining visibility masks by lifting the input view into 3D. An object is unobserved when the majority of its tracklet pixels are unseen.

While our primary benchmark already captures newly revealed surfaces of partially occluded objects and background ("stuff") regions via propagation from the input view, we also extended the benchmark ground truth to cover entirely unseeded objects appearing later along the trajectories. These extended annotations are used to evaluate all methods, with no changes to LucidLyra's training or inference. Specifically, we prompted SAM in unlabeled benchmark regions, propagated masks bidirectionally with SAM2, and used a GPT-6 Astra pipeline to resolve overlaps, assign ADE20K-150 labels, and verify depth-consistent 3D cross-trajectory correspondences, all followed by human verification.

AuthorsOpenReview ↗

Official Comment by the Authors

Evaluated on this extended benchmark, LucidLyra consistently outperforms all baselines across both observed and unobserved regions, matching or surpassing the 3D-lifted teachers even in newly synthesized areas. While absolute metrics are naturally higher in observed regions (a trend shared across all baselines and already present in underlying RGB reconstruction) LucidLyra maintains strong semantic and instance coherence well beyond the input view, confirming that our 3D representation generalize effectively to newly generated content.

Table: Observed/unobserved scene regions and objects.

FullObserved SceneUnobserved SceneObserved ObjectsUnobserved Objects
MethodPSNRmIoUmTrackPRPSNRmIoUmTrackPRPSNRmIoUmTrackPRmTrackPRmTrackPR25mTrackPRmTrackPR25
LucidLyra-SDG
Lyra + Lifted Teachers24.558.419.825.660.525.022.650.112.122.932.56.614.0
Lyra + LangSplatV224.531.9--25.632.5--22.628.4----------
Lyra + SFS24.531.6--25.632.7--22.625.7----------
Lyra + Unified-Lift24.5--9.925.6--12.622.6--7.711.516.33.26.9
Lyra + Chorus24.512.0--25.612.1--22.611.5----------
Lyra + IGGT--30.15.9--31.25.9--27.14.35.49.24.99.2
LucidLyra (Ours)25.063.119.026.364.923.123.055.212.821.930.410.921.7
LucidLyra+ (Ours)25.067.420.626.369.625.023.057.613.923.733.39.117.4
LucidLyra-ADE20K
Lyra + Lifted Teachers24.649.316.525.450.319.122.142.511.917.526.55.39.6
Lyra + LangSplatV224.636.3--25.437.4--22.125.2----------
Lyra + SFS24.634.4--25.435.4--22.124.6----------
Lyra + Unified-Lift24.6--7.225.4--8.122.1--6.67.511.23.47.1
Lyra + Chorus24.69.4--25.49.7--22.17.7----------
Lyra + IGGT--32.79.2--33.79.7--26.07.19.215.66.912.9
LucidLyra (Ours)24.749.517.825.550.219.922.343.314.818.429.412.019.8
LucidLyra+ (Ours)24.753.618.225.554.319.622.346.215.618.127.614.922.4
AuthorsOpenReview ↗

Official Comment by the Authors

Q2: LucidLyra reconstruction quality in dynamic scenes

The minor gap in Table 4 (0.28 dB PSNR, 0.004 LPIPS) stems from optimization schedule trade-off. In the dynamic setting, LucidLyra jointly optimizes scene dynamics, RGB reconstruction, and 3D semantic/instance representations. To stabilize these multi-task gradients, we apply a reduced learning rate to the shared network parameters, which slightly slows RGB convergence under the default iteration budget. To verify this, we trained both Vanilla Lyra and our model for an additional 5k iterations under the exact Table 4 setup (see Table below). With +5k iterations, LucidLyra and LucidLyra+ gain +0.50 dB PSNR (21.88 →\rightarrow 22.38 dB) and close the LPIPS gap entirely (0.252 →\rightarrow 0.248), slightly surpassing compute-matched Vanilla Lyra (22.32 dB PSNR, 0.248 LPIPS), whose LPIPS remains unchanged and PSNR improves by only +0.16 dB. Meanwhile, longer training further boosts our scene understanding metrics, raising mIoU to 56.3 (+3.1) for LucidLyra and 63.7 (+1.1) for LucidLyra+, alongside consistent gains in mAcc and mTrackPR.

Table: Dynamic reconstruction and understanding on LucidLyra-SDG Dynamic.

MethodPSNR ↑LPIPS ↓mIoU ↑mAcc ↑mTrackPR ↑TrackPR25 ↑
Vanilla Lyra22.160.248--------
Vanilla Lyra w/ longer training22.320.248--------
LucidLyra (Ours)21.880.25253.269.311.719.7
LucidLyra (Ours) w/ longer training22.380.24856.371.911.919.8
LucidLyra+ (Ours)21.880.25262.679.312.320.4
LucidLyra+ (Ours) w/ longer training22.380.24863.779.812.420.4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.