acceptodds
Withdrawn from ICLR 2027

AuTO-Pose: Autoregressive Transparent Object 6D Pose Estimation with Ray-Background Context

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present AuTO-Pose, a novel framework for transparent object 6D pose estimation from RGB videos that causally decodes each target frame window conditioned on previously processed frames. Existing state-of-the-art methods for transparent object pose estimation either adapt opaque object pose estimators or learn cues from isolated transparent observations, largely overlooking temporal context and remaining vulnerable to background-dependent appearance variations. In contrast, AuTO-Pose predicts poses causally from streaming visual observations while accumulating object-centric evidence from previously processed frames. By constructing context from ray-background interaction cues across views, AuTO-Pose exploits how transparent objects refract, reflect, attenuate, and rearrange background evidence over time. This allows the model to resolve ambiguous single-frame cues and generalize better to challenging scenarios with changing backgrounds, weak boundaries, occlusion, and long sequences. We further adopt symmetry-aware supervision to avoid inconsistent training signals for transparent objects with indistinguishable orientations. Extensive experiments on TRansPose, ClearPose, and our self-collected TOS-6D benchmarks show that AuTO-Pose consistently improves transparent object pose accuracy and achieves state-of-the-art performance across benchmark accuracy, sequence generalization, and scenario-level robustness protocols. More details can be found on our [project page](https://anonymous.auto-pose.pages.dev).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
6D Pose Estimation, Transparent Object Pose Estimation, Autoregressive Modeling, Streaming Pose Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KxPeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the ambiguity of 6D pose estimation for transparent objects, whose lack of stable surface texture makes single-frame RGB evidence unreliable due to refraction and background appearance. AuTO-Pose uses a short memory of previously decoded views to provide context for the current target window. It stores object-centered RoI features from decoded frames for retrieval before dense coordinate prediction and pose regression, and introduces symmetry-aware supervision for visually indistinguishable orientations. Overall, the method is clearly described and improves several non-BOP metrics on TRansPose, TOS-6D, and ClearPose.

Critical strengths

(1) The context flow is constrained. Current-frame queries use only RoI features, RGB, 2D coordinates, mask confidence, and temporal embeddings, without geometry derived from the current pose estimate. Geometric anchors come only from previously decoded views and are fused into the current RoI features through a residual connection. (2) The symmetry supervision accounts for unobservable rotations. For axially symmetric objects such as cups, tubes, and bottles, rotation supervision preserves only axis alignment in camera coordinates. Surface points are represented by axial height and radial distance, with ADD-S used for point matching. Removing symmetry handling reduces ADD(-S) on New BG from 87.32 to 58.74 and 5∘2cm5^\circ 2\mathrm{cm} from 89.29 to 58.95. (3) Improvements appear across multiple metrics against strong baselines. On TRansPose, ARADD\mathrm{AR}_{\mathrm{ADD}} increases from 68.01 to 73.34, ARMSSD\mathrm{AR}_{\mathrm{MSSD}} from 25.05 to 31.71, and IoU@50 from 38.55 to 45.03. On TOS-6D New BG, ADD(-S) improves from 80.31 with GDRNPP to 87.32. (4) The ablations help isolate the contribution of temporal context. With the same DINOv3 backbone, the single-frame variant achieves an ADD(-S) of 80.54 on New BG, close to GDRNPP's 80.31. Retaining cross-frame attention while removing the context memory raises the score to 82.58, and the full model reaches 87.32. Under the stricter 5∘2cm5^\circ 2\mathrm{cm} metric, the score also increases from 87.04 for the single-frame variant to 89.29 for the full model.

Critical weaknesses

(1) Potential inconsistencies in metric calculation. In Tables 1 and 2, ARBOP\mathrm{AR}_{\mathrm{BOP}} consistently equals the average of ARMSSD\mathrm{AR}_{\mathrm{MSSD}} and ARMSPD\mathrm{AR}_{\mathrm{MSPD}}. However, the official BOP average recall also includes ARVSD\mathrm{AR}_{\mathrm{VSD}}, which is not reported. The symmetry ablation in Table 4 also deviates from this calculation. The average of 42.96 and 74.47 is 58.72, whereas the reported value is 53.93. In addition, GDR-Net reports identical values of 75.51 for 5∘2cm5^\circ 2\mathrm{cm} and ADD(-S) on New BG, while Reflow6D reports 69.58 for both metrics on Long Seq. These values should be verified. (2) Confounding factors in the ablations and inconsistencies with the reported conclusions. The non-autoregressive variant uses a six-frame window, whereas the no-context variant uses a three-frame window. This prevents the isolated effect of the context memory from being assessed. On New BG, removing the context memory improves both ARMSPD\mathrm{AR}_{\mathrm{MSPD}} and 5∘5cm5^\circ 5\mathrm{cm}, while the single-frame variant achieves even higher scores. These results are not adequately discussed in the paper. Furthermore, the symmetry ablation removes both nearest-pose supervision and continuous-axis handling, making it difficult to attribute the performance drop solely to the proposed symmetry rule.

Questions for authors

(1) Insufficient clarity in context construction. In Eq. (4), xc,ux_{c,u} is obtained from the predicted object-coordinate map, yet the paper states that the object coordinate system is defined by the cached pose. The relationship between these two components needs clarification. The method also does not explicitly explain how refraction, reflection, or attenuation is handled. Moreover, training anchors are constructed from perturbed ground-truth poses, whereas inference relies on the model's own previous predictions. The authors should analyze how this discrepancy affects error accumulation. (2) Insufficient evidence for some central claims. The novelty of introducing autoregression to 6D pose estimation is not fully established, as existing pose-tracking methods also use historical estimates. The real-time inference claim lacks supporting frame-rate, memory, and latency measurements. The metric difference between New BG and Long Seq cannot directly demonstrate reduced error accumulation because the two results come from different test distributions. Furthermore, Table 3 shows that AuTO-Pose underperforms IRPosef_f on several metrics under semi-transparent occlusion and in liquid-filled scenes. The abstract's claim of consistently achieving state-of-the-art performance should therefore be revised. (3) Incomplete experimental details and dataset description. The method section does not clearly specify the pose, coordinate, mask, and symmetry losses, or whether test crops are obtained from ground-truth boxes or detector predictions. As a newly introduced dataset, TOS-6D also lacks key details, including the number of objects, sequence scale, annotation procedure, and data splits. In addition, the single-frame variant using the same DINOv3 backbone achieves an ADD(-S) of 80.54, close to GDRNPP's 80.31. The authors should provide single-frame baselines and closed-loop training comparisons under consistent training settings and fixed window lengths to distinguish the contributions of the backbone, context memory, and symmetry-aware supervision.

Reviewer_5WFrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses 6D pose estimation for transparent objects from streaming RGB video. Because transparent objects refract and reflect their surroundings, their appearance depends strongly on the background, making pose estimation from a single frame ambiguous. To address this challenge, it introduces AuTO-Pose, a windowed autoregressive framework that retains object-centric evidence from previously processed views to guide pose prediction in subsequent frames. At its core is a ray–background interaction context bank that links visual features to object-coordinate anchors while preserving information about their foreground confidence and spatial and temporal locations. This representation allows the model to relate current observations to historical evidence through cross-view attention. Building on DINOv3 features, the framework dynamically zooms in at the feature level to predict dense object coordinates, with symmetry-aware supervision accounting for equivalent poses under discrete and continuous symmetries. Evaluation on TRansPose, TOS-6D, and ClearPose assesses benchmark accuracy, generalization to unseen backgrounds, and long-sequence inference. Further tests examine performance when objects are occluded, covered, or filled with liquid, as well as in scenes containing distractors or non-planar surfaces.

Critical strengths
  1. The framework reduces single-frame ambiguities by retrieving historical observations anchored in object coordinates, allowing it to combine evidence across viewpoints while keeping the context bank bounded for streaming inference.

  2. The method accounts for both discrete and continuous symmetries during supervision to avoid conflicting training targets for visually indistinguishable orientations. Ablation results show that this symmetry-aware supervision improves pose estimation accuracy.

  3. Experiments on three datasets evaluate pose estimation accuracy and generalization to unseen backgrounds, alongside performance on long video sequences and in challenging scenes. The method outperforms the evaluated baselines on TRansPose and TOS-6D and achieves further gains in several ClearPose scenarios.

Critical weaknesses
  1. Although the paper reports results on sequences of 120–180 frames, it does not systematically examine how performance changes with stream length or the sizes of the context and target windows. Aggregate accuracy alone offers limited insight into whether errors accumulate during extended rollouts or how sensitive the framework is to these temporal configurations. Evaluations on longer streams and across different window settings would help substantiate the claim of sustained streaming inference.

  2. The proposed framework exploits historical observations and cached predictions, yet the reported comparisons primarily involve methods evaluated through single-frame inference. Without comparisons against established temporal pose tracking approaches under comparable input, initialization, and temporal-access settings, it remains unclear how much of the improvement follows from the proposed autoregressive context design and how much can be obtained through existing temporal estimation techniques. This omission limits the assessment of the method’s contribution relative to prior approaches that also exploit sequential information.

Questions for authors
  1. How does performance vary with stream length and context and target window sizes?

  2. Can the manuscript compare the framework with established temporal pose-tracking methods under matched input, initialization, and temporal-access settings to clarify whether the gains stem from the proposed autoregressive context design rather than the use of temporal information alone?

Reviewer_TMg7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper targets 6D pose estimation of transparent objects from streaming RGB video. Its central premise is that the visual difficulty of transparent objects (refraction, reflection, weak texture, and unstable boundaries) is better resolved from the temporal evolution of ray-background interactions than from any single frame. The proposed system, AuTO-Pose, uses a windowed architecture. A DINOv3 ViT-S/16 encoder produces frame-level tokens, and a lightweight cross-view decoder exchanges information within a window. The main proposed component is a ray-background interaction context bank that stores object-anchored records (RGB and DINO region-of-interest features, normalized 2D coordinates, foreground masks, and a learnable temporal embedding) from previously decoded target views, and retrieves them through cross-attention before dense coordinate and pose regression in a GDR-Net-style head. The design is causal across windows but jointly decodes all target views inside a window, and it uses symmetry-aware supervision for continuous axial and discrete symmetries of objects such as tubes and bottles. Experiments are reported on TRansPose, a self-collected TOS-6D benchmark with New-BG and Long-Seq subsets, and ClearPose, together with ablations for autoregression, symmetry handling, and context.

Critical strengths
  1. The premise is well motivated and practically relevant. Reframing transparent-object pose from independent per-frame appearance to the accumulation of ray-background interaction over time is a sensible response to a genuine failure mode, because the background distortion caused by a transparent object is only partially observable in any single frame. The streaming setting is realistic for robotics and AR applications, and Figure 1 communicates the idea clearly.
  2. The system is reasonably complete and the evaluation is broad. The context bank, cross-view decoder, and symmetry-aware supervision are integrated into a single pipeline, and the paper evaluates cross-background generalization, long-sequence inference, and difficult ClearPose scenarios (heavy occlusion, non-planar surfaces, distractors). The ablations cover autoregression, symmetry handling, context, and a single-frame variant, and the qualitative boxes in Figures 3 and 4 align tightly with the ground truth.
Critical weaknesses

1. The core claim that the ray-background autoregressive context drives the gains is not established. The model is not strictly causal: Equation (1) jointly decodes all targets within a window (“bidirectional within windows, unidirectional across windows”), making it closer to a sliding-window memory than to frame-by-frame autoregression. Every reported baseline is single-frame, with no non-autoregressive multi-frame control, and the gains are at the same time confounded with a much stronger DINOv3 ViT-S/16 encoder over the weaker backbones used by GDRNPP and others. Without a single-frame GDR-Net using the identical DINOv3 encoder and a non-autoregressive temporal model over the same frames, the improvements cannot be attributed to the proposed context rather than to a stronger backbone or ordinary multi-frame aggregation. 2. Internal inconsistencies and overstated claims undermine the reported results. The tables do not support the claims. On TRansPose, AuTO-Pose IoU@25 (76.57) is slightly below GDRNPP (76.73), yet the text claims “consistent gains across pose and 3D-overlap metrics,” and the identical TOS-6D New-BG full model gives conflicting numbers in Tables 2 and 4 (AR_MSPD 92.55 versus 81.05; IoU@75 85.17 versus 95.06) with no protocol difference stated. The tables must be reconciled and the blanket “state-of-the-art scenario-level robustness” wording narrowed to the metrics actually won, since the method is also not best on the most diagnostic ClearPose scenes such as translucent covers and filled liquids.

Questions for authors
  1. Please add non-autoregressive temporal controls under the identical backbone and window: (a) single frame, (b) six-frame joint multi-view decoding without cross-window recurrence, and (c) simple pooling or concatenation of recent-frame features without a recurrent bank. How much of the Table 4 gain survives without autoregression?
  2. Please provide a single-frame GDR-Net baseline with the exact DINOv3 ViT-S/16 encoder and training recipe, so the backbone contribution and the context-bank contribution can be separated.
  3. Please report per-frame latency, frames per second, peak GPU memory, and bank memory with and without the context bank, as a function of sequence length. What is the evidence for “real-time throughput”?
  4. Please reconcile Tables 2 and 4 for the TOS-6D New-BG split (AR_MSPD 92.55 versus 81.05, 5°2 cm 89.99 versus 89.29, and IoU@75 85.17 versus 95.06), and explain the TRansPose IoU@25 result (76.57 versus GDRNPP 76.73) relative to the “consistent gains” claim.
  5. Please report accuracy versus rollout length on Long-Seq and a robustness test in which the cached history pose is perturbed by increasing rotation or translation, or replaced by a wrong frame. What training-noise magnitude is used, and how does it match the measured test-time pose-error distribution?
  6. Please report per-category results (tubes, bottles, cubes, and so on) and state how symmetry-aware ADD-S is applied to each object. Please also provide mean and standard deviation over at least three seeds for the main tables and key ablations.
  7. Does the context bank use any explicit refraction or reflection geometry, or is it purely learned appearance-coordinate memory? Please clarify the positioning relative to Reflow6D and other refraction-guided or NeRF-based transparent-pose methods.
  8. Please provide in the main paper the TOS-6D dataset construction, ground-truth annotation procedure, object, scene, and sequence statistics, and overlap with existing benchmarks, and confirm whether all baselines were retrained under the same split and resolution. For ClearPose, how is comparability with cross-paper, downsampled numbers ensured?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.