acceptodds
Under review as a conference paper at ICLR 2027

Retrieval-Augmented Transparent Object Perception for Autonomous Laboratories

Abstract

Transparent objects are ubiquitous in autonomous laboratories, yet their geometry remains difficult to perceive due to refraction, reflection, and unreliable visual cues. Existing methods largely infer transparent geometry from observations alone, overlooking a distinctive property of laboratory environments: they are semi-open worlds, where many instruments are standardized products with known or retrievable 3D models. We introduce Retrieval-Augmented Transparent Depth Estimation (RATDE), a framework that grounds visual depth recovery with external object geometry. Given a transparent laboratory object, our approach retrieves its corresponding CAD model, encodes the CAD geometry as point-cloud features, and establishes image-to-CAD correspondence to obtain pose-aware local geometric tokens. These geometric priors condition a transparent depth model, enabling it to jointly reason over ambiguous visual evidence and known 3D structure. Importantly, the framework uses pose supervision during training while requiring no ground-truth pose at inference time. To study this problem at scale, we construct a new laboratory transparent-object benchmark, LabSense-Glass, containing 231 laboratory instruments across 222 laboratory scenes, explicitly associating visual observations with retrievable 3D assets. We evaluate depth recovery together with CAD retrieval and 6D geometric alignment and further study generalization under unseen objects, missing CAD models, and imperfect retrieval. Our results demonstrate that exploiting retrievable geometry provides a strong prior for transparent perception, suggesting a broader principle for robotic laboratories: when geometry already exists in the world, perception should retrieve and ground it rather than infer it entirely from scratch.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Transparent Object Perception, Retrieval-Augmented Vision, Autonomous Laboratories

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h7zgOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into depth estimation for transparent objects in autonomous laboratories, and puts forward Retrieval‑Augmented Transparent Depth Estimation (RATDE). The core thinking here is that lab environments are kind of a semi‑open world. Lots of lab glassware and equipment are standard items, so their corresponding CAD models might already live in some external object memory bank. That means we don’t have to rely entirely on the tricky visual signals from transparent objects themselves — those signals get messed up easily by refraction and reflection — to figure out object geometry. The authors also built the dataset LabSense‑Glass. For their actual experiments, they used nine simulated lab setups and 1,336 stereo samples. They also provide annotations like paired renderings for transparent / opaque versions, depth maps, normal maps, instance masks, and camera calibration data.

Critical strengths

Treating lab environments as a semi‑open world is a very reasonable angle. Existing CAD libraries really can supply more trustworthy shape priors than noisy RGB signals from transparent objects. Importantly, the method does not get told the ground‑truth CAD identity or object pose during inference; it has to retrieve and align CAD models from memory on its own. The connections between silhouette retrieval, Utonia point features, DINOv2 / MoGe‑2 descriptors, PnP pose fitting, local geometry tokens, SeeClear and LingBot‑Depth are easy enough to follow. I especially like how they separate global retrieved shape priors from pose‑aware local geometry; this is sensible design.

Critical weaknesses
  1. The paper needs to spell out exactly how they split training / validation / test data. Are splits done by picking random individual frames? Or do they keep whole scenes separate across splits? Do evaluation scenes never show up in training at all? Have target objects used in evaluation already appeared in data used to train the conditioning modules? Could the same object instance or nearby camera views leak across train and test sets? This matters a lot. The core setup assumes the exact matching CAD exists inside memory — that premise is fine. But if test images share scenes, assets or nearby camera trajectories with training data, then the measured results mainly show reconstruction performance within familiar scenes instead of true generalization.
  2. One super important baseline is missing: an image‑only baseline adapted for lab data. The authors first LoRA‑fine‑tune the detector and the image‑only SeeClear model, and only then train the point‑feature conditioning and local residual modules. However, the SeeClear baseline they compare against in Table 2 is the publicly released checkpoint without this lab‑specific LoRA tuning. The authors themselves admit the gap between plain SeeClear and full RATDE mixes two factors: domain adaptation gains plus improvements from CAD‑based conditioning. Therefore we really need results for lab‑adapted image‑only SeeClear + LingBot + SGBM. That means applying exactly the same LabSense‑Glass LoRA domain adaptation, but completely removing both CAD‑based conditioning branches. Running this baseline would clearly tell us how much improvement comes purely from lab‑domain fine‑tuning and how much actual benefit comes from CAD conditioning. Their existing Random‑CAD control experiment is useful, but it cannot fully substitute for this baseline. The full model was trained expecting normal retrieval‑conditioned inputs. Swapping in random CADs only during testing shifts the input distribution for conditioning.
  3. All main experiments run on just nine Isaac Sim lab environments. Even though the appendix lists ClearDepth as an external benchmark resource, the main text shows no results on outside real‑world or external benchmark datasets. We know simulation‑to‑real gaps can be huge for transparent‑object perception. Even some small‑scale sanity‑check experiments on real‑world data, or tests on external benchmarks, would make the paper’s conclusions much stronger. I won’t reject the paper purely for lacking large‑scale real‑world tests, since this CAD‑conditioned task requires CAD model associations that most existing datasets don’t provide.
  4. There are clear inconsistencies when describing LabSense‑Glass dataset size. Early manuscript sections make it sound like “231 assets / 222 scene entries” are part of LabSense‑Glass itself. But the appendix clarifies that the actual experimental LabSense‑Glass dataset only has 9 environments and 1,336 stereo samples. Moreover, only a subset out of those 231 assets act as transparent target objects in experiments.
  5. Equation (6) relies on cjc_j and LjL_j to convert normalized CAD coordinates back into real‑world metric geometry. Yet the appendix says nominal object dimensions are used “when available”. The authors should clarify: do every transparent‑target CAD model in their experiments have reliable real‑world size information? If some CAD assets lack nominal dimension metadata, how do they set values for LjL_j, handle Equation (6), and fix the reference depth zref=1 mz_\mathrm{ref}=1\,\mathrm{m} for those objects?
Questions for authors
  1. Can you give us the exact protocol you used for train / validation / test splitting, as brought up in Weakness point 1?
  2. Can you report results for that lab‑adapted image‑only SeeClear + LingBot + SGBM baseline mentioned in Weakness point 2?
  3. Would it be possible for you to run those small‑scale real‑world experiments discussed in Weakness point 3?
  4. Can you clear up confusion about LabSense‑Glass dataset scope from Weakness point 4?
  5. Please address the metric‑dimension issue raised in Weakness point 5.
Reviewer_udxtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides a method for depth estimation of transparent objects using a given set of CAD models, with CAD retrieval and pose estimation used as intermediate steps. While the approach is based on a combination of known methods, the overall pipeline itself is novel. The paper also provides a rendered dataset based on 231 CAD assets and 9 environments. It demonstrates that the proposed approach can improve depth-estimation accuracy on the synthetic benchmark.

Critical strengths
  1. While the pipeline is mostly based on existing methods, it combines them in a novel way.

  2. The potential ability to incorporate new CAD models without retraining for each object could be very useful.

  3. The method reports higher depth-estimation accuracy than the compared methods and also performs CAD pose estimation as part of the pipeline, which could be useful for robotic applications.

  4. The dataset and collection of 231 laboratory CAD assets could be useful for future work.

  5. While the target application robotic laboratories (is relatively narrow) but it is an important and relevant research area.

Critical weaknesses
  1. No real-world benchmark is provided, and no evaluation is performed on real-world images, making it unclear how well the reported results would transfer to real world settings.

  2. While the method appears designed to work with CAD models that were unseen during training, it is unclear whether the CAD identities used for testing are disjoint from those used during training. Therefore, the claimed ability to generalize to new CAD models without retraining is not demonstrated.

  3. The number of rendered environments is relatively small (9), making it unclear how well the method generalizes to new laboratory settings. In addition, the paper does not describe the use of HDRI-based illumination or extensive lighting domain randomization. It also does not discuss realistic surface variations such as stains, printed labels, scratches, or other appearance changes that are common in real laboratory objects. This makes it unclear how well the synthetic benchmark captures real-world setting.

  4. It is unclear how much of the full CAD collection will be publicly released, and I could not find a link to the dataset or code in the submission.

Overall, adding a comprehensive real-world benchmark, including objects whose CAD identities were unseen during training, would address concerns 1-3, which are the main weaknesses of the paper. In addition, evaluating pose-estimation accuracy would strengthen the paper’s relevance to downstream robotic applications.

Questions for authors

Open Questions

  1. Does the method output pose/position, or is it only used internally?
  2. How are liquids inside the vessels simulated?
  3. How are the materials for the objects and liquids generated?
  4. What is the Top-1 CAD retrieval accuracy?
  5. Since the method seems useful for transparent objects in general why limit it to the laboratory setting?
Reviewer_z1GpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work presents Retrieval-Augmented Transparent Depth Estimation (RATDE), a framework that incorporates object-specific CAD geometry into depth estimation for transparent objects. Given a image with transparent objects, RATDE segments transparent objects and retrieves candidate CAD models from a predefined library. Their shape features and estimated alignment guide a generative model to produce an image in which the transparent objects appear opaque. The new images without transparent surface could then be fed to other depth models for depth estimation. The authors also introduce LabSense-Glass, a synthetic laboratory dataset, on which RATDE outperforms the selected baselines.

Critical strengths

The synthetic dataset, labsense-glass, could be an useful contribution to the community.

Critical weaknesses
  1. The method targets laboratory transparent-object perception and assumes that the exact CAD asset of every target object is available in the retrieval library. This severely limits this work's scope. Also, its robustness when the exact asset is missing or only an approximate model is available is not evaluated. All results are only evaluated on the proposed synthetic data. Without real-world laboratory experiments or independent benchmark evaluation, the practical applicability and generalization of the method remain unclear.
  2. The added pipeline complexity is insufficiently justified. The method retrieves and aligns CAD models, uses their geometry to generate opaque-looking images, and then estimates depth from those images together with stereo depth. Since the pipeline already renders depth from the aligned CAD models, a natural baseline would use that rendered depth directly, potentially combined with the same depth-completion backend.
  3. Baseline comparisons are incomplete. Some recent general depth estimation model (e.g. Depth Anything 3 [1], Moge 3 [2]), or depth estimation model for transparent objects (e.g. SeeGroup [3]) is not compared against with.

[1] Depth Anything 3: Recovering the Visual Space from Any Views [2] MoGe-3: Fine-Detail Monocular Geometry Estimation with Self-Guided Sparse Volumetric Refinement [3] SeeGroup: Multi-Layer Depth Estimation of Transparent Surfaces via Self-Determined Grouping

Questions for authors
  1. Can RATDE generalize to real-world laboratory environments and broader scenarios beyond the laboratory setting?
  2. How does RATDE perform when the exact CAD model of a target object is absent from the retrieval library?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.