acceptodds
Under review as a conference paper at ICLR 2027

Lens3D: Target-Conditioned Visual Foveation for Fine-Grained 3D Understanding

Abstract

Existing 3D large language models often overlook fine-grained attributes and less visually salient objects and parts, even when relevant evidence is present in scene videos. We introduce Lens3D to improve fine-grained object understanding through external visual assistance and knowledge transfer. Its LensUnd pipeline adopts 3D localization to select informative, complementary views for an external 2D vision-language model, supporting fine-grained object captioning, small-object grounding, and fine-grained object question answering. LensDistill transfers the resulting fine-grained knowledge to 3D LLMs through detailed caption supervision, enabling captioning from native inputs without external VLM calls. We also construct LensBench, a held-out evaluation set of 2,068 objects with three silver-standard reference descriptions per object. Experiments with Video-3D LLM and 3DRS demonstrate that LensDistill substantially improves fine-grained object captioning while preserving existing grounding and scene-level QA performance. These results establish the feasibility of transferring externally acquired fine-grained knowledge into native 3D LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
3D LLMs, Fine-grained Perception, Distillation, 3D Grounding, VLMs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HzPyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Lens3D for fine-grained object captioning in 3D scenes. LensUnd selects complementary target views for an external VLM, while LensDistill uses the generated descriptions to train 3D LLMs without external VLM calls at inference. Experiments on LensBench show improved captioning on two backbones while largely preserving their existing grounding and QA performance.

Critical strengths
  1. The paper addresses a useful problem: existing 3D models often locate objects correctly but miss their detailed appearance and parts. Captioning improves substantially on both tested backbones, with human ratings and target-masking experiments providing evidence beyond text-overlap scores.

  2. The evaluation includes controlled comparisons of view selection, target marking, and predicted versus ground-truth boxes. Training and evaluation scenes are explicitly separated, and tests on standard grounding and QA benchmarks show that detailed-caption training does not substantially weaken existing capabilities.

Critical weaknesses
  1. The student gains are not clearly separated from the benefit of richer caption supervision. The proposed selector improves Summary F1 from 0.436 to 0.445 over ExCap3D’s selection strategy, but the paper does not compare students trained on equally detailed captions produced by the two methods. It is therefore unclear whether the new selector contributes meaningfully to the large student gains or whether similar gains could be achieved using an existing caption-generation pipeline.

  2. The practical benefit of distillation is not fully evaluated. GPT and Gemini are used to generate supervision and references, but are not included as direct captioning baselines. Removing external VLM calls does not by itself establish lower deployment cost, particularly since the student processes 32 views while the teacher uses four selected views. Without a quality–cost comparison, it remains unclear when the distilled student is preferable to directly querying an external VLM.

Questions for authors
  1. With the teacher, prompts, caption count, and student training budget fixed, how does student performance differ when training captions are generated using the proposed selector versus ExCap3D or quality-pool sampling?

  2. How do the students compare with directly querying GPT/Gemini in caption quality, end-to-end latency, and computational or API cost? Please include view-selection overhead and assess quality without matching a teacher against its own reference captions.

  3. Are teacher-generated details checked for visibility in the student’s 32 input views? If some details are absent, how is the risk of training the student to produce visually unsupported descriptions assessed?

3 comments

We sincerely thank Reviewer HzPy for the time and efforts spent on our manuscript.

Q1: Controlled student ablation

A1: We fixed the teacher, prompts, objects, caption count, and student training budget, and varied only the view-selection strategy. The teacher is Qwen3.8 27B, which did not generate any LensBench reference. Each student was trained for two epochs on 10% of the training data.

Table Q1:

View selectionSummary F1Summary Sem
Quality-pool sampling0.37230.6057
Uniform sampling0.36180.5861
ExCap3D visibility ranking0.37040.6059
bbox-voxel0.37720.6106

The student results follow the LensUnd results in Table 6 of the main paper. Table Q1 shows that bbox-voxel is best and Uniform sampling is lowest at both metrics. This shows that the quality of the data-construction scheme is positively correlated with the performance of the distilled student.

Q2: Distilled students versus external VLMs

A2: The relevant comparison is the complete localization-and-captioning pipeline. LensUnd uses 3DRS for localization, then selects and marks four views for an external VLM. LensDistill performs both tasks within one 3DRS model using its native 32-view input.

Local LensUnd with Qwen3-VL-8B requires 16.9B total parameters and approximately 36 GB peak GPU memory, versus 8.1B and approximately 19 GB for LensDistill student. Both pipelines can reuse the scene-level inputs across queries by caching the scene's visual tokens in the KV cache, so the scene is encoded only once. Given the cached scene, the student then needs only one model inference per query, whereas LensUnd still runs its localization inference, view selection and wireframe rendering, and a separate VLM inference on the additional target-dependent views.

Table Q2:

Queries per sceneLensUnd, local VLM (s/query)LensUnd, GPT (s/query)LensDistill (s/query)
19.3120.356.38
56.9918.034.06
156.6117.653.68

Table Q2 shows that LensDistill is 1.46–1.80× faster than local LensUnd and 3.19–4.80× faster than the GPT pipeline, while Table 6 in A.5 shows that LensUnd with the 8B VLM and the LensDistill student perform comparably, so the speedup comes at a comparable performance level.

Q3: Visibility of teacher supervision

A3: To check whether the teacher's supervision is grounded in what the student can see, we use Qwen3.8 27B to judge 4,260 attributes (parts, materials, and states) from the teacher's captions against the student's 32 views. Only 5.7% were visible in the teacher's views but not verifiable in the student's, so nearly all of the fine-grained details the student is trained on are supported by its own native input.

Coverage gaps and context prior. Although bbox-voxel covers more of the target surface, some regions are still not observed, and for these the model may rely on context priors. Indoor scenes are regular, and objects that belong together co-occur and tend to share a style, so the surrounding scene and the objects that typically accompany the target can supplement what the target views leave out. To check whether this reliance on context priors leads to hallucination, we evaluated hallucination directly through a user study. Raters scored 30 objects (114 responses, about 4 per object) on detail level and hallucination separately (higher hallucination-free scores mean fewer hallucinations). Table Q3 shows that our distillation does not come with more hallucination. More systematic hallucination suppression, such as uncertainty-aware supervision and hallucination-negative preference optimization, is left for future work.

Table Q3:

ModelDetailHallucination-free
3DRS (before distillation)2.163.16
Lens3D (3DRS)3.843.82
Video-3D LLM (before distillation)2.143.19
Lens3D (Video-3D LLM)3.843.81

Dear Reviewer HzPy,

Thank you for taking the time to review our paper and for your valuable feedback. With one day left in the author–reviewer discussion period, we would be glad to address any remaining questions or concerns you may have.

Best regards, Authors of Submission 11626

Dear Reviewer HzPy,

Thank you for your review and for the constructive feedback. With less than 12 hours left in the author–reviewer discussion period, we would appreciate it if you could let us know whether our response has addressed your concerns. We are happy to clarify or address any remaining points.

Best regards, Authors of Submission 11626

Reviewer_hhFfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Lens3D to improve fine-grained object understanding in 3D scenes. LensUnd selects complementary views of a target object, marks the target in these images, and uses an external VLM to generate detailed descriptions. LensDistill uses these descriptions to fine-tune existing 3D LLMs, so that external VLM calls are no longer needed at inference. The paper also introduces LensBench for evaluating detailed object descriptions. Results on two backbones show clear improvements in captioning while largely preserving their original grounding and QA performance.

Critical strengths
  1. The problem is useful and well motivated. Existing 3D models often miss object details such as materials, parts, and states. The examples clearly illustrate why these details matter and where current models fall short.

  2. The approach is practical. It works with existing 3D models and VLMs, and the distilled model needs no additional view selection or external model calls at inference. Results on two backbones suggest that the approach is not specific to one model.

  3. The paper provides several useful checks. These include view-selection ablations, experiments with predicted boxes, human evaluation, and target masking. The results support improved description quality, and the standard benchmark results suggest that this improvement does not substantially hurt existing capabilities.

Critical weaknesses
  1. The novelty over ExCap3D appears limited. ExCap3D already addresses the lack of fine-grained object and part descriptions, uses multiview images and VLMs to generate detailed annotations, and trains a 3D captioning model with them. Lens3D mainly changes the view-selection rule and applies the resulting supervision to general-purpose 3D LLMs. These are useful changes, but I am not convinced that LensDistill constitutes a new distillation method: Eq. 6 is standard next-token training on generated captions. X-Trans2Cap also studies transferring training-time 2D information into a 3D captioner. The paper would be stronger if it showed what its particular design enables beyond these existing approaches, or provided a clear finding about when and why complementary views improve the student.

  2. The main improvements are not clearly attributed to the proposed method. Table 6 shows that bbox-voxel selection improves Summary F1 from 0.436 to 0.445 over ExCap3D visibility selection, and attribute mean F1 from 0.394 to 0.397. However, this comparison is only made for the external VLM. It does not show whether the difference leads to a better distilled student. The much larger student gains also include richer captions, a new response format, and additional fine-tuning. The most useful comparison would be to train the same student using captions generated with ExCap3D-style view selection, with the same teachers, prompts, objects, and training budget. A continuation baseline using the original caption data would also help separate the benefit of additional training from that of the new supervision.

  3. The evaluation may partly reward matching the teachers’ descriptions. The same three VLMs generate both the training captions and the test references. Although the scenes are separate, the annotations can still share wording, preferred details, and systematic errors. Token F1 and sentence similarity do not directly measure whether individual claims are visually correct. In addition, attribute scores are not reported for the base models because their outputs lack the required fields, making the attribute-level improvement difficult to assess directly. The human studies help, but the reference audit uses only one rating per reference and shows the same selected views. I would like to see a comparison using the same output format for all models, along with a human-checked subset that separately measures correct details, missing details, and hallucinations.

  4. Some conclusions go beyond what the experiments establish. More covered voxels do not necessarily mean better visibility of a small handle, a material cue, or an object’s state. The paper does not examine this link directly. It is also unclear how often details visible in the teacher’s selected views remain visible in the student’s uniformly sampled views. Masking the target shows that the student uses target pixels, but without a corresponding base-model or ordinary fine-tuning comparison, it does not show that LensDistill teaches a new ability to inspect fine details. Finally, fine-grained QA and small-object grounding are shown mainly through qualitative examples using the external VLM. These examples do not establish that the distilled model acquires those abilities. A controlled evaluation on attribute questions, including whether the relevant evidence is visible to the student, would make the claims more convincing.

Questions for authors
  1. Have you compared final student performance when the training captions are generated using ExCap3D-style visibility selection instead of bbox-voxel, with the other settings fixed? If so, how large is the difference?

  2. How often are the details described by the teacher actually visible in the student’s 32 input views after preprocessing? Have you checked cases where the teacher can see a detail but the student cannot?

  3. Were the base models asked to produce the same detailed, structured output as the distilled models? In the human comparison, was factual correctness assessed separately from the amount of detail, and how many raters evaluated each output?

  4. For the predicted-box results in Table 6, what queries were used to obtain the boxes? Were failed grounding cases included, with predictions still evaluated against references for the originally requested object?

5 comments

We sincerely thank Reviewer hhFf for the time and efforts spent on our manuscript.

Q1: Contribution beyond existing methods

A1: Lens3D studies how target-focused visual evidence can improve generalist 3D LLMs while preserving their existing capabilities. ExCap3D and X-Trans2Cap focus on dedicated captioners. LensUnd instead provides a shared evidence pipeline for captioning, object QA, and small-object grounding. LensDistill uses its captions to improve fine-grained object captioning within existing 3D LLMs.

Eq. 6 uses a standard autoregressive objective. The contribution lies in the teacher–student observation design: target-specific inspection produces supervision during training, while the student retains its native inputs at deployment. Results on two backbones show that this transfer improves captioning without substantially weakening existing grounding and scene-level QA.

Q2: Controlled student ablation and reference bias

A2: We fixed the teacher, prompts, objects, caption count, and student training budget, and varied only the view-selection strategy. The teacher is Qwen3.8 27B, which did not generate any LensBench reference. Each student was trained for two epochs on 10% of the training data.

Table Q2:

View selectionSummary F1Summary Sem
Quality-pool sampling0.37230.6057
Uniform sampling0.36180.5861
ExCap3D visibility ranking0.37040.6059
bbox-voxel0.37720.6106

The student results follow the LensUnd results in Table 6. Our bbox-voxel is best and Uniform sampling is lowest at both metrics. This shows that the quality of the data-construction scheme is positively correlated with the performance of the distilled student. Our controls address different explanations for the gains. Qwen3.8 27B was not involved in reference generation, avoiding direct teacher–reference overlap in the new ablation. The VLMs evaluated within LensUnd are also distinct from the reference generators in Table 6.

Q3: Output-format control, and factual correctness

A3: For output-format control, the base models received the same detailed, structured-output instructions but did not reliably provide the requested attribute fields. We therefore added the following experiments to decompose the score gains and show that they come from improved description quality.

Separating format, context prior, and target observation. To test whether the scores reflect matching the output format and the teachers' style rather than observing the target, we evaluate Lens3D with a controlled progression of inputs, always scoring against the references of the requested target (Supp. A.4, Table 5, plus a new format control). Level 1 keeps only the output format. Level 2 sees 32 views, but the target region is replaced with a gray mask. It can still infer the target from the surrounding objects and the scene context, but cannot observe the target's appearance. Level 3 is the full model. Table Q3.1:

LevelPredictionSummary F1Overall F1Summary Sem.
ReferenceBase 3DRS (original captions)0.2300.1520.549
1(a) Random-object Lens3D caption0.2860.2540.391
1(b) Base answer + random-object Lens3D caption0.3170.2850.582
2Lens3D (3DRS), target region masked0.3690.3370.560
3Lens3D (3DRS), full0.4890.4700.752

Table Q3.1 shows that the score rises at every level, and the largest step is observing the target. Format and context priors do contribute, but only a limited part of the gain. Giving the base model the Lens3D format does not lift it substantially, and the model's real improvement depends on complete observation of the target and on the knowledge transferred from the teacher.

Human evaluation of detail and hallucination. To identify the source of this gain in Table 1, we ran a follow-up study with 30 objects and 114 responses (about 4 per object), in which raters scored detail level and hallucination separately (higher hallucination scores mean fewer hallucinations). Raters were shown eight views of each target selected by quality-pool uniform sampling, a strategy different from the bbox-voxel selection used for supervision, and judged the outputs against these views.

Table Q3.2:

ModelDetailHallucination-free
3DRS (before distillation)2.163.16
Lens3D (3DRS)3.843.82
Video-3D LLM (before distillation)2.143.19
Lens3D (Video-3D LLM)3.843.81

Table Q3.2 shows that the perceived improvement comes from substantially richer detail, and from fewer hallucinations. Humans viewing the images judge the distilled models both more detailed and more accurate, which format matching alone would not produce.

Q4: Surface coverage and teacher–student visibility

A4: Independent visibility ranking can repeatedly select views of the same surface. Bbox-voxel instead rewards additional coverage, increasing the evidence available within a fixed view budget.

Table Q4:

Teacher view selectionTeacher-visible voxelsVoxels shared with student
Quality-pool sampling1,172.8893.2
Uniform sampling491.9425.4
ExCap3D visibility ranking1,564.61,041.7
bbox-voxel2,057.71,346.7

Table 4 in Supp. A.3 shows that under the same four-view budget, redundancy decreases from 47.8% to 38.0% relative to ExCap3D. Table Q4 shows that shared coverage also increases: the broader teacher observations include more surface evidence accessible to the student. This supports organizing available evidence as a source of teacher advantage.

To check whether the teacher's supervision is grounded in what the student can see, we use Qwen3.8 27B to judge 4,260 attributes (parts, materials, and states) from the teacher's captions against the student's 32 views. Only 5.7% were visible in the teacher's views but not verifiable in the student's, so nearly all of the fine-grained details the student is trained on are supported by its own native input. For the remaining cases, the student can often still infer the missing detail from context. As the context-prior analysis in Response 3 shows (Level 2), even when the target is not fully visible, the surrounding scene and the objects that typically accompany it give useful information about it, so a partially visible object can still be described well.

Q5: Quantitative support for additional tasks and the scope of experiments established in the paper

A5: We agree that qualitative results alone are not sufficient to show LensUnd's small-object grounding and object QA capabilities, so we added a set of quantitative results for both.

Small-object grounding. We built a test set of 109 small objects from ScanNet scenes. We compare four evidence-selection strategies inside LensUnd with the 3D LLM baseline 3DRS, which matches text against Mask3D proposals.

Table Q5.1:

MethodEvidence selection[email protected][email protected]
LensUndQuality-pool uniform25.7%11.9%
LensUndUniform11.9%6.4%
LensUndExCap3D26.6%15.6%
LensUndbbox-voxel39.4%19.3%
3D LLM3DRS28.4%23.9%

Table Q5.1 shows the LensUnd variants, bbox-voxel is best at both thresholds. Compared with 3DRS, LensUnd (bbox-voxel) improves [email protected] from 28.4% to 39.4%, but does not improve [email protected] (19.3% vs. 23.9%). A likely explanation is that 3DRS matches text against Mask3D proposals, whose boundaries are relatively precise but which miss some small objects, while LensUnd can find more of them but obtains the 3D box by back-projecting per-view 2D predictions, so the box may not fit the object tightly. Grounding depends on spatial agreement with the ground-truth box, so these results provide evidence beyond caption-format matching.

Fine-grained object QA. We constructed a QA set with the LensBench protocol and evaluated LensUnd with the same four evidence-selection strategies, table Q5.2 shows the LensUnd variants, bbox-voxel gives the best fine-grained QA performance among all evidence-selection strategies.

Table Q5.2:

MethodEvidence selectionQA exact matchQA token F1
LensUndQuality-pool uniform0.3770.416
LensUndUniform0.2650.294
LensUndExCap3D0.4160.461
LensUndbbox-voxel0.4620.510
3D LLM3DRS0.3670.421

Q6: Predicted-box source and failed cases

A6: We will add these clarifications on the box source to the revised paper.

Source. For the predicted-box results in Table 6, the boxes were produced by the grounders listed in the Box source column (3DRS and Video-3D LLM). Each grounder received the target's ScanRefer referring expression and predicted a 3D box, which LensUnd then used to select and mark the views.

Failed cases. Failed grounding cases are included. All 2,068 LensBench targets are kept for every box source, with no filtering by IoU or by grounding success. When a predicted box misses the target, the generated description is still scored against the references of the originally requested object, so a wrong localization lowers the score and is not excluded.

Dear Reviewer hhFf,

Thank you for taking the time to review our paper and for your valuable feedback. With one day left in the author–reviewer discussion period, we would be glad to address any remaining questions or concerns you may have.

Best regards,

Authors of Submission 11626

Reviewer_3e1COpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Lens3D, a framework for improving fine-grained 3D scene understanding. The proposed LensUnd pipeline first grounds a target object using an existing 3D model, then selects four complementary views based on 3D voxel coverage, and uses an external VLM to generate detailed object captions. LensDistill further uses these generated captions to fine-tune existing 3D LLMs. The authors also introduce LensBench to evaluate fine-grained object descriptions.

Critical strengths

The paper addresses an important and practical problem. Existing 3D LLMs are relatively good at scene-level understanding but often struggle with fine-grained object attributes.

I like the overall design, especially the use of 3D geometry to select complementary views. The bbox-voxel strategy is intuitive and makes good use of the available 3D information.

The approach is simple and can be integrated with existing 3D LLMs. The experimental results show substantial improvements in fine-grained captioning across two backbones, while largely preserving their original grounding and QA performance.

Critical weaknesses

The contribution of the proposed view selection to the final student is unclear. While bbox-voxel improves geometric coverage, its captioning performance is only slightly better than existing methods such as ExCap3D (0.445 vs. 0.436 Summary F1). It is unclear whether this difference leads to meaningful improvements after distillation. I would like to see an ablation comparing students trained using captions generated from different view selection strategies.

It is unclear how much of the captioning improvement reflects better visual understanding. The reported improvements are substantial, but both the training captions and evaluation references are generated by VLMs. Since the student is trained to produce more detailed and structured descriptions, some of the F1 gains may come from better matching the reference format rather than recognizing additional visual details. Side-by-side qualitative comparisons with the original models and a more independent evaluation of factual correctness would help clarify this.

The main captioning evaluation relies on ground-truth target centers. This assumes accurate target localization, which is not available in a realistic end-to-end setting. Although the authors evaluate LensUnd with predicted boxes, the final distilled student is primarily evaluated with ground-truth centers. I consider this an important limitation and would like to see student results using predicted target centers, including an analysis of localization errors.

Limited evaluation of the other claimed capabilities. The paper also presents small-object grounding and fine-grained object QA as applications of LensUnd, but these are only supported by qualitative examples. Quantitative evaluations on relevant benchmarks would help establish whether the method provides consistent improvements on these tasks.

Questions for authors

Please see the questions on Weaknesses.

5 comments

We sincerely thank Reviewer 3e1C for the time and efforts spent on our manuscript.

Q1: Contribution of view selection to the distilled student

A1: To better verify our contribution, we conduct a controlled training ablation by directly measuring what each selection strategy contributes to the distilled student.

Setup. We fixed the teacher, prompt, training objects, number of captions per object, and training budget, and varied only the view-selection strategy used to produce the four teacher views. To avoid overlap with the LensBench references (GPT 5.4, Gemini 3, Qwen 3.5), we used Qwen3.8-27B, as the teacher, which did not generate any reference. Students were trained on 10% of the training objects for two epochs. We evaluate on LensBench.

Table Q1:

View selectionSummary F1Summary Sem
Uniform0.36180.5861
ExCap3D0.37040.6059
Quality-pool uniform0.37230.6057
Ours (bbox-voxel)0.37720.6106

The student results follow the LensUnd results in Table 6 of the paper. Our bbox-voxel is best and Uniform sampling is lowest at both metrics. Table Q1 shows that the quality of the data-construction scheme is positively correlated with the performance of the distilled student.

Q2: Visual correctness versus format matching

A2: Output format and priors account for a rather limited part of the scores on LensBench; the high scores require the observation of the target. We separate the contributions with two quantitative analyses: (1) a controlled progression of settings and (2) a blind user study. In the revision, we will add side-by-side qualitative comparisons of the base and distilled models as supporting evidence.

(1) Progressive decomposition: format, context prior, target observation (Supp. A.4, Table 5). We evaluate Lens3D (3DRS) at three levels that progressively add information, always scoring against the references of the requested target.

  • Level 1, format only. (a) A Lens3D caption of an object drawn uniformly at random from the evaluation set, which has the format but describes the wrong object. (b) The original 3DRS answer with such a random-object caption appended, which gives the base model's own answer the Lens3D format.
  • Level 2, format + context prior. The model sees 32 views, but the target region is replaced with a gray mask. It can still infer the target from the surrounding objects and the scene context, but cannot observe the target's appearance.
  • Level 3, full model. Native settings.

Table Q2.1:

LevelPredictionSummary F1Overall F1Summary Sem.
ReferenceBase 3DRS (original captions)0.2300.1520.549
1(a) Random-object Lens3D caption0.2860.2540.391
1(b) Base answer + random-object Lens3D caption0.3170.2850.582
2Lens3D (3DRS), target region masked0.3690.3370.560
3Lens3D (3DRS), full0.4890.4700.752

Table Q2.1 shows that format and context priors do contribute, but only a limited part of the gain. Regularizing the output format does not lift the base model substantially, and the model's real improvement depends on complete observation of the target and on the knowledge transferred from the teacher.

(2) Human evaluation of detail and hallucination. Our blind user study (Table 1) shows that raters clearly prefer the distilled models. To identify the source of this gain, we ran a follow-up study with 30 objects and 114 responses (about 4 per object), in which raters scored detail level and hallucination separately (higher hallucination scores mean fewer hallucinations).

Table Q2.2:

ModelDetailHallucination-free
3DRS (before distillation)2.163.16
Lens3D (3DRS)3.843.82
Video-3D LLM (before distillation)2.143.19
Lens3D (Video-3D LLM)3.843.81

Table Q2.2 shows that the perceived improvement comes from substantially richer detail and fewer hallucinations, rather than from matching the style or format of the reference captions.

Q3: End-to-end evaluation

A3: Using the ground-truth center follows the standard object-captioning setting and matches a user clicking the object's center. To measure the loss under an imperfect grounder, we added an end-to-end experiment. For each of the 2,068 LensBench targets, the Lens3D grounder predicts a box from the ScanRefer referring expression, and the box center replaces the ground-truth center as the <coord> input.

Table Q3:

SubsetGT-center F1Predicted-center F1GT-center SemPredicted-center Sem
All objects0.490.480.750.74
Predicted-box IoU ≥ 0.50.500.500.770.77
Predicted-box IoU < 0.50.480.460.730.72

Table Q3 shows that predicted centers reduce Summary F1 and Sem by 0.01 each. Comparing GT and predicted centers within each subset separates the effect of localization from differences in object difficulty. The scores are unchanged when the box is well localized (IoU ≥ 0.5) and drop by 0.02 F1 and 0.01 Sem when it is not.

Localization error analysis. The distance between the predicted and ground-truth centers has a median of 0.094 m but a mean of 0.831 m over all 2,068 objects. Most targets are therefore localized within about 10 cm, while a minority of grounding failures produce large errors and dominate the mean.

Q4: Small-object grounding and fine-grained object QA

A4: We agree that qualitative results alone are not sufficient to show LensUnd's small-object grounding and object QA capabilities, so we added a set of quantitative results for both.

Small-object grounding. We built a test set of 109 small objects from ScanNet scenes. We compare four evidence-selection strategies inside LensUnd with the 3D LLM baseline 3DRS, which matches text against Mask3D proposals.

Table Q4.1:

MethodEvidence selection[email protected][email protected]
LensUndQuality-pool uniform25.7%11.9%
LensUndUniform11.9%6.4%
LensUndExCap3D26.6%15.6%
LensUndbbox-voxel39.4%19.3%
3D LLM3DRS28.4%23.9%

Table Q4.1 shows that among the LensUnd variants, bbox-voxel is best at both thresholds. Compared with 3DRS, LensUnd (bbox-voxel) improves [email protected] from 28.4% to 39.4%, but does not improve [email protected]. A likely explanation is that 3DRS matches text against Mask3D proposals, whose boundaries are relatively precise but which miss some small objects, while LensUnd can find more of them but obtains the 3D box by back-projecting per-view 2D predictions, so the box may not fit the object tightly.

Fine-grained object QA. We constructed a QA set with the LensBench protocol and evaluated LensUnd with the same four evidence-selection strategies, Table Q4.2 shows that bbox-voxel gives the best fine-grained QA performance among all evidence-selection strategies.

Table Q4.2:

MethodEvidence selectionQA exact matchQA token F1
LensUndQuality-pool uniform0.3770.416
LensUndUniform0.2650.294
LensUndExCap3D0.4160.461
LensUndbbox-voxel0.4620.510
3D LLM3DRS0.3670.421

Thanks for the detailed responses and additional experiments. They address most of my concerns. My main remaining concern is the relatively small improvement from the proposed view selection strategy. Although bbox-voxel performs best in the new student ablation, the gain over ExCap3D is still quite small (0.3772 vs. 0.3704). This suggests that much of the overall student improvement may come from richer caption supervision rather than the view selection itself.

Overall, I appreciate the authors' efforts in addressing my concerns, and I will maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the area chair and the reviewers for their valuable feedback. We have clarified our core scientific questions and added roughly 10 new experiments and analyses, which together address six shared concerns.


Our scientific question:

Lens3D studies how to extend an existing generalist 3D LLM, which tends to favor spatial perception and overlook details, so that it perceives fine-grained attributes without substantially weakening its original capabilities. LensUnd is our solution to this question. Furthermore, LensDistill examines whether the capability LensUnd provides can be internalized into the 3D LLM itself.


Concern 1: The contribution of view selection to the distilled student is unclear (Reviewer.3e1C, Reviewer.hhFf, Reviewer.HzPy)

Our response: We ran a controlled distillation ablation. The teacher (Qwen3.8-27B) generated none of the LensBench references, and only the view-selection strategy varied. bbox-voxel is best (F1 0.377 vs. 0.362 for Uniform), so data-construction quality translates into student performance.


Concern 2: Gains may come from format or reference matching rather than visual correctness (Reviewer.3e1C, Reviewer.hhFf)

Our response: We decomposed the score into format, context prior and target observation (F1 0.317 vs. 0.369 vs. 0.489). Format and context priors explain only a limited part of the gain. A blind user study (30 objects, 114 responses) shows the distilled models score much higher on detail (3.84 vs. 2.16) and on hallucination-free (3.82 vs. 3.16).


Concern 3: Teacher supervision may not be visible to the student, and hallucination risk (Reviewer.hhFf, Reviewer.HzPy)

Our response: bbox-voxel raises teacher-visible voxels to 2,057.7 (ExCap3D: 1,564.6) and shared voxels to 1,346.7 (ExCap3D: 1,041.7), and cuts redundancy from 47.8% to 38.0%. Only 5.7% of the 4,260 teacher attributes are unverifiable in the student's views. The user study shows distillation does not increase hallucination.


Concern 4: Small-object grounding and object QA lack quantitative support (Reviewer.3e1C, Reviewer.hhFf)

Our response: We added new quantitative benchmarks. bbox-voxel is best among the LensUnd variants. On small-object grounding (109 objects), [email protected] is 39.4% vs. 28.4% for 3DRS, though [email protected] is lower (19.3% vs. 23.9%), and we explain why. It also gives the best QA scores (EM 0.462, F1 0.510).


Concern 5: Reliance on GT centers/boxes, and handling of failed grounding (Reviewer.3e1C, Reviewer.hhFf)

Our response: In an end-to-end test with predicted centers, F1 and Sem each drop by only 0.01. The median center error is 0.094 m. We clarified that all 2,068 targets are kept, with no filtering by IoU or grounding success, and that failed cases lower the score.


Concern 6: Contribution beyond existing methods, and cost versus external VLMs (Reviewer.hhFf, Reviewer.HzPy)

Our response: We clarified the contribution as a shared evidence pipeline (LensUnd) plus a teacher–student observation design (LensDistill). LensDistill is 1.46–1.80× faster than local LensUnd and 3.19–4.80× faster than the GPT pipeline, at about half the memory (19 GB vs. 36 GB), with comparable performance (Table 6, A.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.