acceptodds
Under review as a conference paper at ICLR 2027

Look Where You Can: Active View Selection For CAD Reconstruction Under Scene Occlusion

Abstract

CAD reconstruction methods assume a luxury reality rarely grants: unrestricted visual access to the object, photographed from any desired angle. Real objects, however, are scene-embedded, bolted against walls, wedged into corners, resting on floors, where the scene renders much of the view sphere unreachable and the remaining views unequally informative. We introduce **SightCAD**, a framework for parametric CAD reconstruction that treats view feasibility as a first-class constraint. In this work we consider objects from standard CAD benchmarks embedded in realistic indoor scenes with physically derived visibility constraints over a discrete view sphere. A learned view selector must choose **K** feasible views for a vision–language model (VLM) that generates executable CadQuery code, scored by geometric fidelity of the executed solid. Because reward arrives only after discrete view selection, autoregressive generation, and CAD-kernel execution, we propose a joint training paradigm in which the view selector and the CAD-generation VLM are trained together against this reward. The learned selection policy departs sharply from random, uniform, and coverage-greedy alternatives, outperforming surface-area maximization (SA-max) by up to Intersection-over-Union (IoU) points across budgets **K** . The full system surpasses strong external baselines on scene-embedded, occluded multi-view renders of DeepCAD and Fusion360 objects ( and effective-mIoU points over the best baseline, respectively), as well as on test-time domain-canonicalized real images from the industrial T-LESS benchmark and on both synthetic and real images from the MP6D industrial metal-parts benchmark, while producing the highest rate of executable programs of any method compared (invalid-code rate ).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Models, Reinforcement Learning, CAD Reconstruction, 3D

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xpteOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new problem of parametric CAD reconstruction of objects, when these objects are inserted into a real environment with occlusions through walls, floors and corners. This means, that the standard paradigm of reconstructing these objects, which assumes that they are floating freely in space and that views can be selected freely does not apply anymore. The paper introduces a new, large dataset of such scenes, where CAD models from CAD-Recode and Text2CAD are inserted into SceneNet office rooms. 36 views of each object are rendered across the viewing hemisphere. The paper then proposes SlichtCAD which has a lightweight view selector, which is tasked to pick K out of 36 views, and a Qwen2-VL generator, which generates CadQuerry code. The paper has three training stages. 1. distilling a greedy coverage policy into the view selector, 2. joint training of selector and generator and 3. a generator only training. On the test split, the proposed selector beats greedy selection at every budget. The full system outperforms external CAD baselines on DeepCAD, Fusion360, T-LESS and MP6D with very low invalid-code rates.

Overall, I believe the problem is relevant, the dataset is useful, and the internal policy comparisons are carefully controlled. My main questions/concerns are: The large margins over external baselines do not isolate the contribution of view selection well enough. The “active” and “under occlusion” framing is stronger than the evidence: the gains are as large on mildly occluded floor placements as in corners. Real-capture performance depends on an author-designed canonicalization protocol and trails baselines on raw photographs.

I currently lean on borderline to weak accept, but am willing to raise my score if my concers (see Weaknesses section) are addressed

Critical strengths
  1. The problem is interesting, relevant and well introduced. Reconstruction of CAD objects from images alone is a challenging problem and being able to handle real world occlusions is an interesting question, which prior work largely ignores.
  2. I like that the paper creates a large dataset and I hope this will be released as soon as possible. Given that the dataset contains 567k objects under varying degrees of occlusion I believe this could be a valuable resource for the community. I would also encourage the authors to consider releasing the data generation protocol/code.
  3. The method is technically sound and the experiments are well controlled. All methods are evaluated on the identical data and results on n=4000 include paired Wilcoxon tests, bootstrap confidence intervals and a per-regime breakdown (Table 8). I also like that the policy neutral generator (Table 1b) isolates view quality from co-adaptation.
  4. The budget-dependent strategy (Table 5, Fig. 5) and the controlled same-scene study (Fig. 6) give a clear, interpretable picture of what the selector learns.
  5. The evaluation section is broad and fair. It covers synthetic CAD benchmarks and two real industrial datasets. The paper openly reports that the method trails on raw T-LESS photographs. Invalid-code rates are consistently the lowest of all methods.
  6. The paper is well written, the figures are informative and reproducibility is good. Architecture, warm-start targets, hyperparameters (Table 6), the alignment network and the real-capture selection protocol are described in detail, and code is provided.
Critical weaknesses
  1. I believe the comparisons do not fully isolate the contribution of view selection. Tables 2-4 and 9 compare full systems that differ in multiple ways (training data, generator data and number/type of input views). The +21 and +17 mIoU_eff therefore cannot be attributed to selection only. The paper’s own Table 1 suggests selection accounts for a few points. A direct way to show how much of the gap selection explains is to report the SA-max co-trained generator (with +GRPO) under the cross-method protocol on DeepCAD, Fusion360 and the 4k split.
  2. The active view selection and under occlusion framings are too strong given the evidence. The selector takes renders of all 36 candidate views as input, so the method assumes every feasible view has already been captured. The selector essentially does not save aquisition effort. The paper should state this explicitly, since the title and introduction suggest deciding where to look. Similarly, Table 8 shows the advantage over SA-max is at least as large for mildly occluded floor placements as for corners (e.g., +6.0 vs. +5.3 at K=4). This suggests the gain comes mostly from choosing views that make program parameters legible, not from handling occlusion. That is a valuable finding in itself, but it shifts the paper’s central message. The paper also deals with occlusion arising mainly from walls removing whole half-spaces of views rather than from partial occluders.
  3. The cross-method metric protocol is ambiguous. Sec. 5 says predicted and ground-truth meshes are normalized independently to the unit cube. However, the captions of Fig. 11 and later figures say the printed IoU is computed after a per-object best-fit rotation. Please clarify which protocol the tables use. A best-fit rotation would hide orientation errors and could favor methods that output arbitrary frames.
  4. I’m less convinced by the real-capture results, as they rely on several additional assumptions. On raw T-LESS photographs, SightCAD trails the baselines substantially (8.26 vs. 25.66 mIoU_eff for CAD-Recode←TripoSR). The favorable “canon” condition is a protocol designed by the authors themselves. The real-capture pipeline also depends on an alignment network with an exact top-1 accuracy of only 28.2%, and on camera poses taken from ground-truth object-pose annotations. I appreciate that the paper reports the raw T-LESS result openly, but I would ask the authors to reflect it in the wording of Contribution 4, which currently groups T-LESS with the benchmarks where the method leads.
  5. I don’t think the mechanistic conclusion is quantitatively supported. App. A.1.2 states that the supply of in-plane and face-on views “correlates strongly” with extrude-depth and sketch accuracy, but this correlation is never computed, and Figs. 7 and 9 show clear exceptions. fw-SA supplies the most face-on views (13% vs. 8% for the learned policy) but does not achieve the best sketch accuracy. Similarly, the uniform policy supplies more feasible in-plane views than SA-max (19% vs. 13%) yet has much worse extrude-depth accuracy. The paper attributes these exceptions to occluded (contaminated) views, but this explanation is not tested.
Questions for authors
  1. Could you report the SA-max co-trained generator (with +GRPO) under the cross-method protocol on DeepCAD, Fusion360 and the 4k split? This would show how much of the gap to the external baselines comes from view selection and how much from training on the scene-embedded data.
  2. Why are DeepCAD and Fusion360 evaluated on only 100 objects each? Could you report results on the full test splits, or at least confidence intervals for Tables 2–4?
  3. The selector takes renders of all 36 candidate views as input. Could you clarify what the selection saves in practice? As I understand it, 4. it reduces the generator’s input rather than the number of views that need to be captured.
  4. Table 8 shows the gain over SA-max is at least as large on floor placements as in corners. Do you see this as evidence that the benefit comes mainly from parameter legibility rather than from handling occlusion, and how does this fit the paper’s framing?
4 comments

Response to Reviewer xpte

We thank the reviewer for the constructive review. We ran the requested control, evaluated every method on the complete DeepCAD and Fusion360 splits (Table R, general response), and computed the missing correlation. The updated supplementary adds the code for the new experiments, and a revised PDF of the main paper incorporating all changes described below will be uploaded shortly.

W1/Q1: How much of the external gap comes from view selection?

We scored the SA-max co-trained generator (+GRPO) with its own views under the cross-method protocol (paired 95% CIs):

SA-max pipelineSightCADΔ
4k (n=4,000)61.1666.50+5.34 [4.68, 6.01]
DeepCAD (n=8,046)62.0565.72+3.67 [3.25, 4.08]
Fusion360 (n=1,725)52.5255.37+2.84 [2.00, 3.72]

As suspected, in-domain training explains most of the external gap: the SA-max pipeline alone leads cadrille by +16 to +23. Learned selection adds a significant +2.8 to +5.3 on top, on the scale of Table 1, and +5.7 / +2.9 on canonicalized T-LESS and MP6D real. The paper will report this split explicitly.

Q2: Can you report the complete test splits, with confidence intervals?

The 100 + 100 objects were a fixed random draw, sized by the slowest baselines. Table R now evaluates every method on the complete splits with paired CIs, except CAD-Fit (~5 min per object). The ranking is unchanged: our margin over the strongest baseline is +26.4 / +18.9 with native cameras, and +21.3 / +16.2 when blocked cameras are replaced by feasible views. Tables 2–4 will report both.

W2/Q3: What does view selection save in practice?

We agree that K limits what the generator reads, not what is captured, and will rename the setting budgeted view selection. Selection pays off at inference (Table K): all ~24 feasible views score 22.5 against 60.5 at K=4, which uses 5.9× fewer prompt tokens. The forward pass takes 43 ms at K=4 against 198 ms with all feasible views (batch size 1, one H200); the selector adds 7 ms.

W2/Q4: Does the gain come from legibility rather than from handling occlusion?

Yes, and we will adopt this as the central message. Since SA-max is itself occlusion-aware, the gain comes from the legibility of the chosen views, and it is significant in every regime (Table 8). Occlusion instead defines the candidate set and breaks fixed camera rigs. We will note that our occlusion is dominated by walls and floors, and list partial occluders as a limitation.

W3: Which protocol do the cross-method tables use?

We do not rotate per object when computing metrics. Each mesh is normalized to the unit cube, and one axis rotation per method is fixed for all its objects to absorb conventions such as y-up versus z-up, which penalizes rather than favors arbitrary per-object frames. Table 1 compares our own programs with exact mesh IoU and no rotation; the cross-method tables use 32³ voxel IoU because baseline meshes are often not watertight, and re-scoring them with exact IoU only widens our margins. The best-fit rotation only aligns the solids in the qualitative figures. We will state this; the scorer is in the supplementary.

W4: How reliable are the real-capture results?

The reviewer is right to question it and we will reword Contribution 4: SightCAD leads on DeepCAD, Fusion360, synthetic T-LESS and MP6D and canonicalized captures, but trails on raw T-LESS photographs. Canonicalization will be presented as our diagnostic of the image-domain gap, and we will list every ground-truth input used; the selector and generator never see the GT mesh. Indeed our Align-net's low exact top-1 (28.2%) reflects part symmetry: it is correct up to a 90° rotation 90.3% of the time, and residual errors count against us. We would like to mention though that canonicalization is still useful to our method in case of actual deployment.

W5: Is the mechanistic conclusion quantitatively supported?

Over 30 cells (six policies × K=1–5, same policy-neutral generator), face-on supply predicts sketch accuracy (r = 0.67, p < 1e-4) but in-plane supply does not predict depth (r = 0.19, n.s.); each occluded view costs 3.7 pp depth and 2.9 pp sketch. Because policy-level correlations confound view type with coverage, we added an intervention on all 8,443 held-out objects. In the learned K=4 picks, one view is replaced, with the other three fixed, either by a feasible view of the same type (control) or by a feasible oblique view (15–75° to the nearest sketch normal), each the closest match in visible coverage.

(continued) Same generator; Δ = oblique − control, paired boot-strap 95% CI:

Replaced viewnCoverage (removed / control / oblique)Δ extrude depth (pp)Δ sketch dims (pp)Δ IoU (pp)
In-plane (≥ 75°)2,2120.22 / 0.24 / 0.36−4.5 [−6.6, −2.4]+4.1 [+2.1, +6.1]−0.4 [−1.5, +0.7]
Face-on (≤ 15°)623 (IoU: 719)0.32 / 0.31 / 0.37+1.6 [−1.9, +5.1]−3.0 [−7.1, +1.0]+0.4 [−1.1, +1.9]

Although the oblique view covers more of the object, losing an in-plane view costs 4.5 pp of depth accuracy, so in-plane views do carry depth, which the policy-level correlation missed. Sketch information, by contrast, is not specific to face-on views: replacing one changes sketch accuracy by −3.0 pp (n.s.), and an oblique view in place of an in-plane one raises it by 4.1 pp. This explains the fw-SA exception. We will add this table to App. A.1.2, replace "correlates strongly" with these findings, and release the code (supplementary, analysis/viewswap/).

We hope this addresses the reviewer's major concerns and would be grateful if they would consider raising their score further.

Edit: The revised PDF now addresses your points: the SA-max pipeline control and complete splits with CIs (Table 2), views vs. tokens (Table 3), the scoring protocol (Sec. 5), the reworded Contribution 4 and limitations, and the measured correlations with an extended view-swap intervention (App. A.1.2, Table 7).

Update (Nov 15): the contamination explanation, tested directly.

As the reviewer noted, our explanation of the uniform exception by occluded views was untested. On the 4k test split (policy-neutral generator), we remove one feasible view from each policy's K=4 picks and fill the slot in two ways, keeping the other three views fixed:

  • Feasible (control): an unused feasible view of the same view type (angle to the sketch normal: face-on ≤ 15°, oblique, in-plane ≥ 75°), with the closest visible coverage.
  • Occluded: an unused occluded view (visible coverage ≤ 1%) of the same view type.

The two inputs therefore differ only in whether the object is visible.

PolicyObjectsExtrude depth acc. (%) feasible → occludedSketch dims acc. (%) feasible → occludedIoU (%) feasible → occluded
Learned1,01846.2 → 41.2 (−5.0 [−8.0, −2.0])40.2 → 37.9 (−2.3 [−5.3, +1.0])54.1 → 51.0 (−3.1 [−4.6, −1.6])
SA-max1,03941.9 → 35.5 (−6.4 [−9.0, −3.4])40.1 → 34.6 (−5.5 [−8.4, −2.6])52.5 → 48.4 (−4.1 [−5.8, −2.6])
fw-SA1,04040.9 → 36.2 (−4.7 [−7.7, −1.8])40.2 → 36.2 (−3.9 [−6.8, −1.0])52.4 → 50.1 (−2.3 [−3.9, −0.8])

Paired bootstrap 95% CIs; bold: CI excludes zero. Objects: sketch planes recoverable from the GT program (1,710 of 4,000), a single extrusion axis, and both replacement types available; counts differ because the removed view comes from each policy's own picks.

An occluded view is also worse than leaving the slot empty (pooled: depth −2.5 pp [−4.0, −0.8], sketch −3.1 pp [−4.8, −1.6]), so it actively misleads the generator.

Conclusion for W5. Both exceptions are now explained by interventions rather than policy-level correlations. Uniform: an in-plane view carries extrude depth only if the object is visible in it, so uniform's frequent occluded picks (Fig. 9) cancel its larger supply of in-plane views. fw-SA: its extra face-on views add little because oblique views already carry much of the sketch (Table 7). App. A.1.2 replaces "correlates strongly" with these interventional results, and we will add this experiment to it.

Reviewer_xpteOpenReview ↗

Response

I would like to thank the authors for the unusually thorough and candid rebuttal. The SA-max cotrained pipeline controll of the full splits addresses my main concern. The clarifications on the scoring protocolm the view swap and the revised contribution statement mainly address my leftover concerns. Overall I think it is good that the work is getting reframed around budgeted view selection.

A few remaining points:

  • Since the gain does not come mainly from handling occlusion, please also revise the title and abstract (“Active View Selection … Under Scene Occlusion”, “view feasibility as a first-class constraint”) so they match the new framing
  • On the industrial sets, the rebuttal reports the selection gain against the SA-max pipeline (+5.7 on canonicalized T-LESS, +2.9 on MP6D real). With the generator held fixed, Table R shows only +1.8 and +0.4, and on synthetic T-LESS, SA-max views do better (56.5 vs. 54.0). Please report the fixed-generator numbers and state that selection gives no clear benefit on the real-image sets.

One more question:

  • Feasibility is only learned, and occluded views actively mislead the generator. How often does the learned selector pick an occluded view at test time?

The rebuttal did a great job overall at clarifying the scope of the paper and I am happy to raise my score to reflect this.

We sincerely thank the reviewer for the score increase and for the kind words. The reviewer's questions led to the controls, the view-swap interventions and the reframing that made the paper substantially stronger, and we are grateful for the careful engagement throughout.

Title and abstract. We agree. The title will become "Look Where You Can: Budgeted View Selection for CAD Reconstruction of Scene-Embedded Objects", and the abstract will replace "treats view feasibility as a first-class constraint" with a description of the budgeted setting: the scene determines which views are available, and the selector chooses the K that make the program's parameters legible to the generator.

Fixed-generator results on the industrial sets. Tables 4–5 will now report the same-generator comparison (learned vs. SA-max views, Table R) alongside the pipeline comparison, and Sec. 5 and the limitations will state that, with the generator held fixed, learned selection shows little benefit on the industrial sets.

How often does the learned selector pick an occluded view? Rarely. On the 4k test split at K=4, 0.6% of learned selections contain an occluded view (0.2% of all picked views): 0.8% for floor, 0.2% for wall and 0.6% for corner placements. We will report this in App. A.1.2.

Reviewer_6un6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies parametric CAD reconstruction when the object is embedded in an indoor scene, so that walls and floors make part of the view sphere infeasible. The contributions are as follows: (i) A large synthetic dataset that places CAD-Recode and Text2CAD objects in SceneNet rooms under three occlusion regimes (floor/wall/corner). Each object is rendered from a fixed 36-view grid with ray-cast visibility and feasibility masks. (ii) SightCAD, which pairs a learned view selector with a Qwen2-VL-2B code generator initialized from cadrille. Training has multiple stages. Both modules are first warm-started, with the selector distilled from SA-max. The selector is then trained with GRPO on the executed-IoU reward while the generator is fine-tuned on the best-of-G view subset. Finally, the generator is RL-tuned with the selector frozen. (iii) Analysis arguing that the learned policy trades surface coverage for parameter legibility, supported by view statistics and token-ablation experiments. For validation, on a 4k held-out split the learned selector outperforms SA-max at every budget K=1–5. Against external baselines under matched scene occlusion, the full system reports large gains on DeepCAD and Fusion360, synthetic T-LESS/MP6D, canonicalized T-LESS, and real MP6D, with low invalid-code rates. It underperforms on raw real T-LESS photographs.

Critical strengths
  1. A well-motivated problem setting with a meaningful empirical finding. The paper formalizes CAD reconstruction under scene-imposed view feasibility, a realistic setting that prior CAD-as-code methods ignore. Within it, the authors show that the learned selector covers markedly less surface than greedy coverage yet reconstructs better. It suggests that, for structured outputs like CAD programs, views should be chosen for how legible they make the downstream parameters. The insight is likely to transfer beyond this specific system.
  2. A controlled evaluation that isolates view quality from generator co-adaptation. The policy-neutral comparison decodes all policies with a single shared generator. Under this design, the learned selector beats SA-max at every budget and within every regime. Without this separation, the gains could be explained by the generator adapting to its own inputs. The controlled design makes the claim that learned selection outperforms coverage heuristics credible.
Critical weaknesses
  1. The gains over external baselines are not attributed to the paper’s core contribution. SightCAD generator is cadrille further trained on scene-embedded objects through SFT and RL stages. The baselines are evaluated zero-shot on out-of-distribution inputs, and often with fewer views. In wall/corner placements cadrille frequently receives only one of its four views (Fig. 10). CAD-Coder and GenCAD receive blank images that count as invalid outputs. The key control is missing: the same co-adapted generator paired with SA-max or random feasible views, evaluated under the same protocol. The huge point gains are the paper’s main evidence of practical significance. Yet the isolated selection gain is modest at K=4. Most of the reported gap are not certain whether from in-domain fine-tuning with feasible views or from learned view selection.
  2. The “active view selection under occlusion” framing is not matched by what is tested. The selector takes all 36 rendered views as input. The budget therefore limits the VLM’s input, not image acquisition. Yet no upper bound that feeds the generator all feasible views is reported. The learned advantage is not larger under heavier occlusion. At K=5, corners show the smallest gain. The real-world evidence is somewhat weak. On raw T-LESS photographs the method ranks second to last . The canonicalized condition re-composites objects onto training-like backgrounds, which also removes real occlusion. MP6D real has no embedding scene and covers few distinct objects.
Questions for authors
  1. How were the 100 DeepCAD and 100 Fusion360 test objects selected?
  2. Text2CAD is derived from DeepCAD, and the paper uses a 44k subset of it. Can you confirm that DeepCAD-test objects and their near-duplicates are excluded from training?
  3. How many distinct objects do the 138 T-LESS real and 200 MP6D real multi-view sets cover?
  4. The CAD-Fit mIoU_eff values shown in Figs. 10, 12 and 14 (38.8, 34.7, 47.9) do not match Tables 2–3 (44.26, 33.47, 50.03), whereas the other baselines’ values agree. Which are correct?
  5. CAD-Fit is a mesh-to-CAD method, yet Appendix A.2 groups it with the single-view lift methods. How does it obtain a mesh from the single input image?
  6. Please evaluate the same co-adapted generator paired with SA-max and with random feasible views on DeepCAD and Fusion360, under the cross-method protocol.
  7. The selector already sees all 36 rendered views. What is the performance when the generator receives all feasible views (or K=8/12)? If the motivation for small K is inference cost, how large is that cost? My rating could be changed when all the concerned issues are resolved.
3 comments

Response to Reviewer 6un6

We thank the reviewer for asking for the control that separates training from selection. We ran it on the complete DeepCAD and Fusion360 test splits and on all industrial sets, and re-evaluated every baseline under two input protocols (Table R, general response). The updated supplementary adds the code for the new experiments.

W1/Q6: Where does the gain over external baselines come from?

The reviewer is right that most of the gap comes from in-domain training. The SA-max co-trained pipeline (our generator and training, coverage-selected views) already reaches 62.1 / 52.5 against cadrille's 39.4 / 36.5. Learned selection adds a significant part on top: +3.7 / +2.8 over that pipeline, and +5.7 / +2.9 on canonicalized T-LESS and MP6D real. Swapping only the views of a fixed generator gains more (+6.9 / +5.5), partly from co-adaptation. We will report this split instead of attributing the external gains to selection.

No baseline receives our in-domain scene renders: each gets its native input, and the scene only determines which native cameras are available. When blocked cameras are replaced by feasible views (SA1), SightCAD still leads on all benchmarks except raw T-LESS.

W2: Is the "active view selection under occlusion" framing supported?

We agree that the selector ranks already-rendered views, so K limits what the generator reads rather than what is captured. We will drop "active" and call the setting budgeted view selection; its practical benefit is the inference cost quantified in Q7.

As the paper argues, learned views help mainly because they make CAD parameters legible, as supported by our mechanistic analysis, rather than by handling occlusion which instead defines the candidate set and disqualifies fixed camera rigs. We will revise the occlusion framing accordingly.

On real data, we accept the limitation. On raw T-LESS photographs we rank second to last (8.3 vs. 25.7 / 32.6, Table R) because of the image-domain gap. Canonicalization removes much of the real occlusion but is a practical step towards deployment; on canonicalized T-LESS and MP6D real learned selection still outperforms SA-max (Table R).

Q7: What happens with all feasible views, and what does a small K save?

Table K reports accuracy and prompt tokens up to all feasible views (~24 of 36) for both pipelines; each view adds ~85 visual tokens. Within the trained budget every added view helps significantly, but beyond five views accuracy declines, since neither model was trained on more; all ~24 feasible views cost 5.9× the prompt tokens of K=4.

Learned selection leads the SA-max pipeline at every trained budget (+2.7 to +6.4), and a single learned view (116 tokens) already matches SA-max at K=4 (398 tokens; +0.7). We will present this accuracy-per-token trade-off as a contribution in the main text.

Q1: How were the 100 DeepCAD and 100 Fusion360 objects selected?

They were drawn at random from the first ~1.1k files of each test release, before any evaluation. Table R now uses the complete test splits for every method except CAD-Fit, whose ~5 min of fitting per object keeps it at 100 + 100.

Q2: Are DeepCAD-test objects and their near-duplicates excluded from training?

Yes: Text2CAD inherits DeepCAD's split, our loader excludes its test portion, and none of the 8,046 test IDs is among the 39,260 Text2CAD training objects. Comparing every test object against all training objects (maximum volumetric IoU over 24 axis rotations), a near-duplicate (IoU ≥ 0.95) exists for 62% of DeepCAD and 42% of Fusion360 objects. With all matched objects removed (the novel columns of Table R), SightCAD still leads the strongest baseline by +17.9 on DeepCAD (n=3,055) and +12.4 on Fusion360 (n=1,002), and learned selection still adds +3.1 / +2.3 over the SA-max co-trained pipeline.

Q3: How many distinct objects do the real sets cover?

T-LESS real covers all 30 objects across 20 scenes, and MP6D real covers all 20 objects across 74 videos.

Q4: Which CAD-Fit values are correct?

We thank the reviewer for noticing this. The table values are correct. The figures were produced from an earlier run on scene renders rather than CAD-Fit's native inputs, which the tables use, and we will regenerate Figs. 10–15 in the modified pdf.

Q5: How does CAD-Fit obtain a mesh from a single image?

As in its authors' pipeline, Hunyuan3D-2 lifts the image to a mesh, to which the program is then fitted (to be stated in A.2).

We hope this rebuttal resolves all major reviewer's concerns and helps motivate a positive assesment of our work. We would be grateful if they consider improving their score.

Edit: The revised PDF now covers your questions: complete DeepCAD/Fusion360 splits with both protocols and the SA-max pipeline control (Table 2), the budgeted framing with views vs. tokens and latency (Table 3), near-duplicates (Table 16), real-set coverage (Sec. 5), and regenerated figures (App. A.2).

Thank you for the thorough rebuttal. Running the co-adapted controls on the complete DeepCAD and Fusion360 test splits, with deduplicated novel subsets and paired confidence intervals, directly addresses my main concern (W1). I likewise accept the reframing as budgeted view selection, in which occlusion defines the candidate set rather than being the source of the gain. Conditional on these revisions, I am raising my rating to weak acceptance.

We appreciate the questions asked by the reviewer during the rebuttal about cause of gain and problem framing. All changes would be reflected in the future versions of the manuscript. We thank the reviewer for going thoroughly over the rebuttal and the score increase.

Reviewer_oimKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper does parametric CAD reconstruction for objects sitting in a scene, where only part of a fixed 36-view grid is usable. SightCAD pairs a learned view selector with a Qwen2-VL generator initialized from cadrille that writes CadQuery code. The selector is trained with GRPO on the IoU of the executed solid, then the generator gets SFT on each group's best views followed by GRPO with the selector frozen. Learned selection beats coverage-greedy SA-max at every budget from 1 to 5 views, by 1.4-3.2 effective mIoU with one policy-neutral generator and 2.7-6.4 after co-adaptation. Against five baselines the full system wins on DeepCAD-test, Fusion360, MP6D and synthetic T-LESS, but loses to four of them on raw real T-LESS photos. An appendix analysis links in-plane views to extrude-depth tokens and face-on views to sketch-coordinate tokens.

Critical strengths
  1. The main result survives a strong decoder control. With one policy-neutral generator, learned views win by 1.4-3.2 effective mIoU at every budget with paired tests and bootstrap intervals, and at K = 3 the ordering holds even when the heuristics get adapted generators. This is my main reason to lean accept.
  2. The token interventions give a mechanism, not just a benchmark gap. Dropping in-plane-view tokens raises extrude-depth NLL, dropping face-on tokens raises sketch-coordinate NLL, and learned views cover less surface yet score better.
Critical weaknesses
  1. The scoring protocol is ambiguous between paper and code. The text says internal comparisons penalize absolute size and placement, and a caption mentions a per-object best-fit rotation, but the released internal evaluation centers and scales both GT and prediction. I cannot tell which path made which table. Until that is clear, the large external margins should not be read as a decomposition of the internal selection effect.
  2. The motivating physical setting is not demonstrated. On raw real T-LESS the system scores 8.26 effective mIoU against 25.66 for the best baseline, and the good real-image results come from canonicalization or plain backgrounds. The paper does acknowledge GT camera poses, and the released scripts use visibility and mask annotations for cropping. That is not the same as the selector needing a GT mesh at deployment, but the stages need separating.
Questions for authors
  • The released loader excludes Text2CAD test IDs and the README points at standard test resources. Could you give the post-filter counts and the exact cadrille checkpoint, SFT or RL-tuned?
  • Could you map each table to its scoring script and flags: normalization, rotation search, and whether the same transforms precede IoU and Chamfer for every method? And confirm the SA1 rule is the one used for the reported numbers.
  • Do you have results from independently trained selectors at, say, K = 3? If not, saying that the reported uncertainty is over test objects would be enough.
  • Which preprocessing path produced the T-LESS real and canonicalized inputs, and the MP6D real inputs? The scripts use visible and label masks plus visibility annotations, while the paper describes SAM canonicalization without ground truth. Separating annotation use for cropping and poses from the selector's own input would clarify the deployment assumption.
2 comments

Response to Reviewer oimK

We thank the reviewer for checking the paper against the code; inconsistencies found are resolved below. Tables R and K are in the general response. The updated supplementary adds the code for the new experiments

W1/Q2: Which scoring path produced each table, and which SA1 rule was used?

ResultScriptIoUNormalizationRotation
Tables 1, 7, 8 (4k)train_joint_grpo.py --eval-onlyexact (boolean intersection of the solids)GT and prediction each centered, longest side 1none (shared frame)
Tables 2–4, 9, Table Rscoring/rescore_axisframe.py32³ voxelsameone axis rotation per method, all objects

The same transforms precede IoU and Chamfer distance for every method, and each table applies one scorer to all of its methods. Exact boolean IoU needs watertight solids in a shared frame, which baseline meshes often lack, so the cross-method tables voxelize; Sec. 4.1 wrongly calls the reward voxel-based and will be corrected. Re-scoring Table 2 with exact IoU (scoring/exact_iou_tables.py) widens our margin over the strongest baseline (+21.4 / +17.2 → +28.2 / +24.7). The sentence in Sec. 5 claiming that absolute size and placement are penalized is incorrect and will be removed: the GRPO reward and every table normalize each mesh, so orientation and proportions are scored, but not absolute size or position. No number changes. Tables 1, 7 and 8 compare our own models, which share the ground truth's frame, so no rotation is applied; baselines follow other axis conventions, so one rotation is fixed per method.

The SA1 rule was indeed applied inconsistently: only CAD-Fit and CAD-Recode←TripoSR received the SA1 view, while blocked views of the other baselines were blank, and the MP6D-synthetic inputs ignored the room (cadrille: 12.8 instead of 38.7 once corrected; Table 4 will be fixed). Table R now evaluates every baseline under both rules on the complete splits and all industrial sets; under either rule SightCAD leads on all benchmarks except raw T-LESS.

We agree that the external margins alone do not isolate selection. In Table R, the SA-max co-trained pipeline, with our data and training, accounts for most of the gap, and learned selection adds +5.3 [4.7, 6.0] on 4k and +3.7 / +2.8 on DeepCAD / Fusion360.

W2/Q4: How were the real inputs produced, and which annotations do they use?

Dataset annotations are used only to crop the object, choose candidate frames and label view poses; they never enter the selector or the generator:

StageT-LESS rawT-LESS canonMP6D realMP6D canon
Cropdataset visible mask (mask_visib)dataset visible mask (mask_visib)dataset label mask (label.png)dataset label-mask box
Segmentation / backgroundnone (raw photo crop)SAM, center-point prompt, no annotation; pasted on a training-style platecut out with the dataset label mask, pasted on whiteSAM prompted from the dataset label mask
Candidate framesobject visible in dataset maskobject visible in dataset mask≥200 dataset-mask pixels≥200 dataset-mask pixels
View az/el labelsdataset camera posesdataset camera posesdataset object posesdataset object poses
Object orientationpredicted (align-net)predicted (align-net)predicted (align-net)predicted (align-net)

At deployment, masks would come from a segmenter and poses from a rig, robot or SfM. The GT mesh is used only for scoring and, on MP6D, for the visibility labels of the SA-max and random arms. We will replace "without ground truth" with this description and state raw T-LESS as the main limitation.

Q1: What are the post-filter counts, and which cadrille checkpoint is used?

CAD-RecodeText2CADTotal
Training502,96639,260542,226
Val2,6521,7914,443
Test (4k)2,3581,6424,000

We use Text2CAD's official split, whose test split is DeepCAD's, so no DeepCAD-test object is in training, validation or the 4k set. Both the cadrille baseline and our initialization use the authors' SFT release (maksimko123/cadrille); Table R now also reports their RL model.

Q3: Are there results from independently trained selectors?

No; we trained one selector per budget, and the captions will state that CIs are over test objects (paired bootstrap), conditional on the checkpoints. At K=3 (as in Tables 1 and 8):

GeneratorLearnedSA-maxΔ [95% CI]
policy-neutral47.4646.02+1.44 [0.69, 2.20]
SFT51.6947.75+3.94 [3.17, 4.72]
+GRPO58.5154.72+3.80 [3.20, 4.40]

Learned selection leads at every budget K=1–5 (+2.7 to +6.4, Table K).

We hope these clarifications strengthen the case for acceptance and would be grateful if the reviewer would consider raising their score.

Edit: The revised PDF now addresses your points in the paper itself: both scorers and input protocols are defined in Sec. 4.1 and 5, post-filter counts and the cadrille checkpoint in Sec. 3–4, the annotations used on real captures in Table 15, and the per-generator K=3 results in Table 13.

Dear Reviewer, as the discussion period draws to a close, we wanted to briefly check whether our response addresses your concerns. Every table as of now is mapped to its scorer (Sec. 4.1 and 5), the SA1 inconsistency and the MP6D-synthetic room error are corrected under both input protocols (Table R / Tables 2, 4, 5), and the annotations used on the real captures are listed in Table 15, separately from the selector's and generator's inputs. We would be glad to clarify anything that remains open.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General response

General response

We thank all reviewers. Results requested by several reviewers are reported once here; each reply quotes the numbers relevant to it. The supplementary code has been uploaded.

Table R. mIoU_eff (%, invalid = 0; cross-method scorer of Tables 2–4 and 9: unit-cube normalization, 32³ voxel IoU, one axis rotation per method). 4k: scene-embedded test split, as in Table 9. DeepCAD / F360: complete test splits; novel: objects without a near-duplicate in our training data. Baselines: SA1 / available views; CIs: 95%, paired bootstrap over objects.

mIoU_eff4k (4,000)DeepCAD (8,046)F360 (1,725)DeepCAD novel (3,055)F360 novel (1,002)T-LESS syn (30)raw (138)canon (138)MP6D syn (20)real (200)
SightCAD +GRPO66.565.755.447.842.754.08.332.548.325.6
SightCAD SFT59.360.450.641.637.345.29.317.536.718.6
Co-adapted controls, K=4
SightCAD gen. + SA-max views57.358.849.842.839.256.59.830.745.525.2
SightCAD gen. + random feasible views55.257.250.642.539.652.09.329.042.024.4
SA-max pipeline61.262.152.544.740.452.510.726.847.022.7
SA-max gen. + learned views61.962.453.245.841.050.710.927.344.424.5
Sequential (+GRPO)65.364.054.146.842.251.610.426.443.825.3
Baselines
cadrille (4 SA / rig)11.5 / 45.016.8 / 39.413.7 / 36.512.8 / 29.811.7 / 30.321.2 / 31.410.6 / 10.19.2 / 10.917.5 / 12.8‡13.1 / 12.8
cadrille-rl (4 SA / rig)13.5 / 40.920.2 / 31.016.7 / 28.215.8 / 25.114.9 / 25.128.0 / 33.41.8 / 1.816.2 / 5.415.1 / 9.3‡17.2 / 19.4
CAD-Fit†31.4 / 36.644.3 / 33.833.5 / 23.128.9 / 29.926.4 / 19.150.0 / 41.114.4 / 19.418.0 / 20.933.9 / 25.417.2 / 20.2
CAD-Coder34.2 / 25.144.5 / 32.739.1 / 29.129.0 / 21.425.7 / 19.427.7 / 16.716.5 / 11.926.3 / 26.323.4 / 10.219.8 / 10.9
CAD-Recode←TripoSR16.6 / 10.517.0 / 17.315.5 / 13.914.9 / 14.413.7 / 12.632.5 / 27.925.7 / 32.620.0 / 15.626.3 / 17.815.9 / 23.2
GenCAD5.7 / 4.39.3 / 7.45.2 / 4.04.4 / 3.72.6 / 2.00.5 / 0.50.4 / 0.58.6 / 8.61.8 / 0.42.6 / 1.3
SightCAD − best baseline+27.0 / +21.8+21.3 / +26.4+16.2 / +18.9+18.7 / +17.9+13.5 / +12.4+4.0 / +12.9−17.4 / −24.3+6.2 / +6.2+14.4 / +22.9+5.7 / +2.4
SightCAD − SA-max pipeline+5.3 [4.7, 6.0]+3.7 [3.3, 4.1]+2.8 [2.0, 3.7]+3.1 [2.4, 3.7]+2.3 [1.2, 3.3]+1.5 [−2.8, 6.4]−2.5 [−4.5, −0.4]+5.7 [3.6, 7.7]+1.3 [−3.8, 6.5]+2.9 [1.0, 4.8]
SightCAD − Sequential+1.2 [0.7, 1.8]+1.7 [1.4, 2.1]+1.3 [0.5, 2.0]+1.0 [0.4, 1.5]+0.5 [−0.4, 1.5]+2.4 [−2.4, 7.7]−2.2 [−4.4, 0.1]+6.1 [4.0, 8.2]+4.5 [−1.0, 11.2]+0.3 [−1.3, 1.8]
  • SA1: a scene-blocked native camera is replaced by the first view of the occlusion-aware SA-greedy chain, in the baseline's input style (cadrille: the first four SA views as its 2×2 grid; on 4k, 1,313 objects). Available views: native cameras only, blank where blocked (real data: nearest real frame within 20°).
  • Every method ran on every object except † CAD-Fit (~5 min per object): the paper's 100 + 100 DeepCAD / F360 objects (39 / 60 novel) and 100 random 4k objects; its margins are paired on these.
  • Novel: the closest Text2CAD training object has volumetric IoU < 0.95 under all 24 axis rotations.
  • On the objects evaluated in the paper, every cell under each baseline's paper protocol reproduces Tables 2–4 and 9, except two errors we will correct: ‡ Table 4's MP6D-synth inputs ignored the room (cadrille 38.7 → 12.8 with blocked cameras blank; the other available-view cells are likewise room-corrected), and Table 9 placed the iso camera and cadrille's rig in the mesh frame (correcting it lowers cadrille from 45.0 to 34.2, so the 4k margins are conservative).

Table K. Views vs. accuracy and tokens, 4k split (mIoU_eff as in Table 1, exact mesh IoU). K≤5: each K's own jointly trained selector and generator; K>5 (*): the K=5 pair, beyond its training budget. SA-max coverage saturates after a median of three views, so its chain is extended with the remaining feasible views by visible area.

K123458*12*16*24*all feasible* (≈24.3)
SightCAD, learned views55.8656.9558.5160.5261.7956.0743.3840.0624.8822.54
SA-max pipeline, SA-max views53.0554.2854.7255.1655.4050.9142.9439.3025.87—
learned − SA-max+2.8+2.7+3.8+5.4+6.4+5.2+0.4+0.8−1.0—
prompt tokens (visual)116 (81)216 (171)307 (252)398 (333)499 (423)784 (675)1,164 (1,008)1,553 (1,350)2,332 (2,025)2,359 (2,049)

24.3 of the 36 views are feasible on average; each view adds ~85 visual tokens.

Edit: We have uploaded the revised PDF, with all changes. It incorporates the analyses and corrections described in our replies (new Tables 2–3 in Sec. 5, Appendix A.6, and figures under both input protocols). Table numbers in our replies refer to the original submission.

Response to Reviewer U8fi

We thank the reviewer for the precise questions and ran new experiments for each (Tables R and K, general response). The updated supplementary adds the code for the new experiments, and a revised PDF of the main paper incorporating all changes described below will be uploaded shortly.

W1/Q1: What does K measure?

K is the number of views the generator reads: the selector ranks views that are already available, and only the K selected ones reach the generator. We agree that the acquisition framing overstated this, and will rename the setting budgeted view selection.

The budget still reduces cost (Table K). Each view adds ~85 visual tokens, and accuracy improves with every view up to the trained budget of five. Learned selection leads SA-max at every trained budget (+2.7 to +6.4), and a single learned view beats SA-max at K=4 (+0.7 [0.1, 1.3]) with 29% of its tokens. At K=4 the forward pass takes 43 ms, against 198 ms with all ~24 feasible views (batch size 1, one H200); the selector adds 7 ms.

W2/Q2: How much of the external advantage comes from replacing blocked viewpoints?

We agree that blank inputs mix reconstruction quality with access to views, so Table R evaluates every baseline both with its native cameras and with SA1, where each blocked camera is replaced by an available view. CAD-Coder benefits and becomes the strongest baseline (32.7 → 44.5 / 29.1 → 39.1), whereas cadrille degrades with four SA-max views (39.4 → 16.8 / 36.5 → 13.7), as it reads only its training rig. Under either protocol SightCAD leads on all benchmarks except raw T-LESS, by +21.3 / +16.2 over the strongest baseline on the complete splits. Most of this gap comes from in-domain training; learned selection adds +3.7 [3.3, 4.1] / +2.8 [2.0, 3.7] over the SA-max co-trained pipeline.

The lead also holds on shapes without a near-duplicate in our training data (at least +17.9 / +12.4). A full deployment study is beyond this work for lack of real-world training data, but the framework needs only captured images and their poses (from a rig, robot or SfM).

Q3: Is co-adaptation necessary?

We trained the suggested sequential alternative with the same initialization, data and checkpoint rule: selector GRPO against the frozen generator, then generator SFT and GRPO on its views, in three one-epoch stages instead of our two (4k columns as in Table 1, DeepCAD / Fusion360 as in Table R).

mIoU_eff4k, SFT4k, +GRPODeepCADFusion360
Joint (ours)52.9060.5265.7255.37
Sequential50.4159.0564.0154.11
Joint − seq.+2.49 [1.79, 3.19]+1.47 [0.95, 1.99]+1.71 [1.36, 2.08]+1.26 [0.50, 2.02]

Co-adaptation is not strictly necessary, since the sequential pipeline also outperforms the SA-max pipeline. Joint training is, however, more efficient: with one epoch fewer it is significantly better everywhere above, also at a matched GRPO step. Each generator performs best with its own selector, and given identical views the two generators perform alike, so the gain stems from the views the generator learned to read.

Q4: What drives the real-image gain?

On T-LESS real (n=138), with generator, selector and align-net fixed:

rawcanonicalized
learned views8.332.5
SA-max (GT-mesh oracle)9.830.7
random feasible9.329.0

Canonicalization raises every policy by 20–24 points, so the main gain comes from the image domain. On canonicalized images, learned views outperform random feasible views (+3.5 [1.0, 6.0]) and match the SA-max oracle.

Q5: How is feasibility enforced at inference?

By learned image scores, without a hard mask or ground-truth geometry. Eq. 2 masks only already selected views, and inference takes the top-K logits; infeasible candidates are renders of walls or occluders, which the selector learns to score low. Ground-truth feasibility (ray-casting the mesh) is used only for training targets and baseline inputs, so an unseen object requires only its captured images.

We hope this rebuttal motivates a positive score by the reviewer and we would be grateful if they would consider revising their score.

Edit: The revised PDF now addresses your questions: budgeted framing, tokens and latency (Sec. 5, Table 3), every baseline under both input protocols (Tables 2, 4, 5, 11), joint vs. sequential training (Table 12), the T-LESS policy × domain analysis (Table 14), and learned feasibility (Sec. 4.1).

Reviewer_U8fiOpenReview ↗

Response after rebuttal

Thank you for the detailed response and additional experiments. The clarification of what K measures, together with the token and latency results, addresses my concern about the budget definition. I agree that selecting from already available images is a useful setting.

The comparisons with replacement views and the sequential-training baseline also strengthen the evidence. I appreciate the distinction between gains from in-domain training and the smaller, consistent gains from learned view selection. The feasibility clarification addresses my concern about requiring ground-truth geometry at inference, although learned scores do not guarantee that every selected view is feasible.

My main remaining reservation concerns real images. On T-LESS, most of the improvement comes from canonicalization, and the reported results do not establish a clear advantage over SA-max on canonicalized images. The practical claims should reflect these limits.

Overall, the additional evidence substantially reduces my original concerns. Together with the fixed-generator experiments and the analysis connecting selected views to CAD parameter errors, it supports a useful contribution within the clarified scope. I am raising my rating from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.