acceptodds
Under review as a conference paper at ICLR 2027

EgoTrack: Robust 6D Object Tracking via Reliability-Aware Pose Hypothesis Reasoning in Extreme Egocentric Vision

Abstract

Robust 6D object pose estimation is critical for egocentric and embodied systems operating under challenging conditions such as low light, motion blur, smoke and occlusion. Under such degradation, frame-wise pose estimates can become unreliable: selecting a single estimate may discard useful alternatives, while even the entire candidate set may be uninformative. We introduce EgoTrack, which retains multiple pose hypotheses and keeps their influence revisable during trajectory reconstruction. For each frame, a learned policy selects the object template appearance most useful for pose recovery, and and a frozen estimator retains its top-K ranked pose hypotheses. Trajectory diffusion then repeatedly retrieves candidate evidence conditioned on the evolving trajectory state and reliability cues; temporal denoising updates the trajectory, which changes subsequent retrieval. A learned NULL token provides an alternative to real-candidate evidence when the local set is uninformative. Experiments show substantial improvements in 6D pose accuracy and trajectory consistency over frame-wise and temporal baselines under severe visual degradation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
6D Object Pose Tracking, Egocentric Vision, Extreme Conditions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a78POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a 6D pose tracking framework named EgoTrack in extreme visual degradation scenarios. This framework retains multiple pose hypotheses and passes them to the diffusion model for time trajectory reconstruction, thereby addressing the issue of unreliable single-frame estimation in extreme visual degradation scenarios. In the EgoXtreme benchmark test, this method outperforms existing methods significantly.

Critical strengths
  1. The motivation of this paper is very clear, and it addresses the issue that the performance of traditional 6D object pose tracking significantly deteriorates under extreme degraded conditions.

  2. The quantitative results of the paper have been significantly improved, and the components have been thoroughly validated through ablation studies.

Critical weaknesses
  1. The 6D object pose estimation has high requirements for real-time performance in practical applications, but this paper does not evaluate the computational efficiency of the framework at all. It is suggested to conduct experiments on the inference time, memory usage, and the trade-off between accuracy and efficiency.

  2. The paper claims that the NULL marker enables the model to learn to reject unreliable candidates, but this mechanism lacks direct verification. The ablation results in Table 4 also show that the AUC improves after adding NULL, but 5 out of 6 RPE metrics decrease, and its actual help for temporal consistency remains unclear.

  3. When EgoTrack is compared with the baseline, both the candidate generation method and the model architecture change simultaneously. It is impossible to determine whether the advantage shown in Table 1 is due to the proposed reasoning method. It is suggested to conduct additional control variable experiments, comparing different models under the condition of fixed candidate inputs.

  4. This paper only used the EgoXtreme dataset in the quantitative experiments and did not verify its generalization ability on different datasets; moreover, all the experiments relied on real object masks, but in the case of severe visual degradation, the segmentation itself might fail. It is suggested to supplement the experiments by introducing predicted masks or adding segmentation noise, and to conduct validations on more datasets.

Questions for authors

Please see the weaknesses.

8 comments

Weaknesses

The 6D object pose estimation has high requirements for real-time performance in practical applications, but this paper does not evaluate the computational efficiency of the framework at all. It is suggested to conduct experiments on the inference time, memory usage, and the trade-off between accuracy and efficiency.

Response:

We thank the reviewer for highlighting the importance of computational efficiency. We conducted additional inference-time and GPU-memory profiling of EgoTrack and the compared baselines.

Inference time and memory. GigaPose requires 63.37 ms/frame (5.777 GiB). Our policy runs before GigaPose and requires 10.20 ms/frame (0.235 GiB), while DDIM-10 diffusion takes 10.08 ms/window (0.056 GiB). Conservatively counting one complete diffusion-window evaluation per frame without amortization, the estimated EgoTrack pipeline cost is 83.65 ms/frame for inference computation, or 143.27 ms/frame including component-level online overhead. These are component-based estimates, not directly measured end-to-end latencies.

For comparison, FreePose refinement requires 532.83 ms/unique object-frame (4.746 GiB), and GoTrack tracking requires 3149.19 ms/frame (1.513 GiB), excluding GigaPose initialization. The profiling scopes differ across methods.

Runtime and memory comparison.

MethodInference time ↓Online runtime ↓GPU memory (GiB) ↓
Measured components
GigaPose (1-bank, Top-5)63.37 ms116.48 ms5.777
Policy (B=1)10.20 ms14.85 ms0.235
Diffusion (B=1, DDIM-10)10.08 ms11.94 ms0.056
Estimated pipeline costs
GigaPose + FreePose596.20 ms876.31 ms10.523
GigaPose + GoTrack3212.56 ms3374.36 ms7.290
GigaPose + EgoTrack83.65 ms143.27 ms6.068

Pipeline times are estimated by summing separately profiled components, with different timing protocols across methods. Pipeline memory values are sums of component peak GPU allocations, not directly measured end-to-end peaks.

These results demonstrate that EgoTrack introduces relatively low computational and memory overhead beyond its GigaPose backbone, although we do not claim real-time operation. We will include the runtime and memory analysis in the revised manuscript.

The paper claims that the NULL marker enables the model to learn to reject unreliable candidates, but this mechanism lacks direct verification. The ablation results in Table 4 also show that the AUC improves after adding NULL, but 5 out of 6 RPE metrics decrease, and its actual help for temporal consistency remains unclear.

Response:

To examine NULL's role more directly, we analyzed its attention weights and its effects on pose accuracy and temporal consistency under different candidate-quality conditions.

NULL behavior under candidate uncertainty. We divide frames into two groups based on the best Top-K candidate's ADD(-S) error: reliable (≤0.3d) and unreliable (>0.3d). NULL attention is higher in the unreliable group across all three EgoXtreme scenarios. For example, its average weight increases from 0.0815 to 0.1375 on Sports and from 0.1592 to 0.2511 on Maintenance.

Contribution to pose accuracy. Adding NULL improves ADD(-S) AUC in the unreliable group by 2.20, 5.88, and 1.48 percentage points on Sports, Maintenance, and Emergency, respectively. The improvement is larger than in the reliable group for Sports and Maintenance, suggesting that NULL helps reduce reliance on inaccurate candidate evidence.

Temporal consistency. NULL participates in state-conditioned, per-frame candidate aggregation at each denoising step, before temporal modeling. It provides a learned alternative to unreliable candidate evidence but does not explicitly regularize inter-frame motion. While NULL improves AUC, RPE-T and RPE-R do not consistently improve. We therefore clarify that NULL primarily supports pose reconstruction, while temporal dependencies are modeled by the trajectory denoiser, without claiming that NULL directly improves temporal consistency.

NULL analysis on EgoXtreme-Sports.

Top-K qualityNULL weightAUC w/o ↑AUC w/ ↑RPE-T w/o ↓RPE-T w/ ↓RPE-R w/o ↓RPE-R w/ ↓
Oracle ≤0.3d0.081549.4850.75 (+1.27)0.07280.07745.245.29
Oracle >0.3d0.137541.5843.78 (+2.20)0.05260.05425.535.98

NULL analysis on EgoXtreme-Maintenance.

Top-K qualityNULL weightAUC w/o ↑AUC w/ ↑RPE-T w/o ↓RPE-T w/ ↓RPE-R w/o ↓RPE-R w/ ↓
Oracle ≤0.3d0.159228.5630.70 (+2.14)0.00770.00814.274.44
Oracle >0.3d0.251116.7022.58 (+5.88)0.00900.00935.095.70

NULL analysis on EgoXtreme-Emergency.

Top-K qualityNULL weightAUC w/o ↑AUC w/ ↑RPE-T w/o ↓RPE-T w/ ↓RPE-R w/o ↓RPE-R w/ ↓
Oracle ≤0.3d0.195437.7039.54 (+1.84)0.01540.01542.853.00
Oracle >0.3d0.215214.3315.81 (+1.48)0.02310.02264.654.31

Candidate quality is determined using GT poses for post-hoc analysis only. GT information is not used to compute NULL attention during inference. "w/o" and "w/" denote configurations without and with NULL, respectively. Values in parentheses indicate AUC improvements in percentage points.

We will include this analysis in the appendix and clarify NULL's role in the main paper.

When EgoTrack is compared with the baseline, both the candidate generation method and the model architecture change simultaneously. It is impossible to determine whether the advantage shown in Table 1 is due to the proposed reasoning method. It is suggested to conduct additional control variable experiments, comparing different models under the condition of fixed candidate inputs.

Response:

Isolating Stage 2. We evaluate EgoTrack using Clean Aug-1 Rank-5, which disables learned augmentation selection and uses GigaPose's Top-5 hypotheses from the unaugmented template bank. This configuration achieves ADD(-S) AUCs of 43.87, 27.07, and 20.70 on Sports, Maintenance, and Emergency, respectively. Adding learned augmentation selection improves these results to 47.07, 27.24, and 23.29, demonstrating that substantial gains persist without the proposed selection policy.

Comparison under a fixed candidate pool. We additionally compare against FreePose (Top-5 Oracle) using the same Selected Top-5 candidate pool. FreePose selects the GT-best hypothesis for initialization, providing a privileged-initialization reference. Despite this advantage, FreePose achieves AUCs of 12.51, 17.38, and 14.08, compared with 47.07, 27.24, and 23.29 for EgoTrack.

Controlled comparisons on EgoXtreme.

ScenarioMethodAUC ↑RPE-T ↓RPE-R ↓
SportsGigaPose Top-18.333.6018114.28
SportsFreePose (Top-5 Oracle)12.511.939980.30
SportsEgoTrack (Clean Aug-1 Rank-5)43.870.272929.49
SportsEgoTrack (Selected Aug-1 Rank-5)47.070.272126.45
MaintenanceGigaPose Top-15.520.970591.47
MaintenanceFreePose (Top-5 Oracle)17.380.095838.55
MaintenanceEgoTrack (Clean Aug-1 Rank-5)27.070.031119.97
MaintenanceEgoTrack (Selected Aug-1 Rank-5)27.240.032820.75
EmergencyGigaPose Top-14.023.090692.28
EmergencyFreePose (Top-5 Oracle)14.080.553134.00
EmergencyEgoTrack (Clean Aug-1 Rank-5)20.700.067512.90
EmergencyEgoTrack (Selected Aug-1 Rank-5)23.290.066814.42

Clean Aug-1 Rank-5 disables learned augmentation selection. FreePose (Top-5 Oracle) and full EgoTrack share the same Selected Top-5 candidate pool, but FreePose uses a single GT-selected initialization, whereas EgoTrack jointly processes the full candidate set.

Together, these results show that the improvements cannot be attributed solely to candidate generation. Although the FreePose comparison does not provide identical model inputs or training supervision, it controls the underlying candidate pool while preserving each method's refinement architecture. We will clarify these comparisons in the revised manuscript.

Reviewer_DhesOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a method for refining trajectories of objects 6 DoF tracks from an off-the-shelf model based object pose estimator (GigaPose). For template augmentation they train a policy that selects an augmentation to the template rendering for more effective render and compare within GigaPose, while retaining the top K candidates produced by the frozen estimator. The final trajectory refinement step uses a diffusion model conditioned on the top K hypotheses per frame along with a null hypothesis to denoise a the object pose trajectory conditioned on the temporal candidate hypothesis.

Critical strengths
  • Refining the trajectory via a diffusion model conditioned on multi-hypothesis is an interesting idea akin to a modern day particle filter.
  • Training a policy for template augmentation selection is well motivated as it can be cheaper and could be adapted for different front-ends.
Critical weaknesses
  • The set of augmentation used is limited and biases can be introduced when selecting these augmentation, as opposed to optimising for them on the fly, which challenges the scalability of this approach.
  • There is no clear ablation isolating the effects of stage 2 without the effects of stage 1.
  • Evaluation on a single dataset is weak, further evaluation on datasets like YCB-V and HO-3D can strengthen the paper, since some of the sequences have challenging lighting. More importantly, EgoTrack have been trained on EgoXtreme while the baselines, including GigaPose, have not, which makes the evaluation setup unfair.
  • The paper claims to be doing tracking however, if I understood it correctly it seems to be more of an offline trajectory refinement, especially in the bidirectional setup. If that is correct then this should be communicated more clearly.
Questions for authors

In addition to my concerns mentioned up in the weaknesses section:

  • What is the distribution of template selection?
  • Can you please provide a run time analysis?
  • How long are the evaluated sequences?
  • Can you provide visualisations for the trajectories estimated compared to GT?
  • The reliability cue is not defined in the main paper. Can you please define it?
6 comments

Weaknesses

The set of augmentation used is limited and biases can be introduced when selecting these augmentation, as opposed to optimising for them on the fly, which challenges the scalability of this approach.

Response:

We thank the reviewer for raising concerns about augmentation-bank coverage, selection bias, and scalability.

Augmentation-bank scalability. We expanded the template augmentation bank from 20 to 30 transformations on EgoXtreme-Sports. At 0.3d, the best-of-bank Rank-1 candidate coverage increases from 51.91% to 53.33%, while our selected Aug-1 Rank-5 coverage changes only from 44.69% to 44.82%. This suggests diminishing returns from simply expanding the bank, rather than establishing 20 transformations as optimal. Rather than optimizing augmentations on the fly, our learned policy selects from a precomputed template bank, avoiding iterative optimization at inference time while restricting the search to predefined transformations. We will provide the bank composition and transformation ranges in the appendix.

Applicability across datasets. We also applied the same augmentation-bank design and selection protocol to HOT3D-Aria. Our selection improves Top-5 candidate coverage at 0.3d from 42.91% with the unaugmented template bank to 45.83%. This demonstrates applicability beyond EgoXtreme, without assuming zero-shot transfer of the learned policy.

Candidate coverage comparison (%)

MethodSports(20) @0.1dSports(20) @0.2dSports(20) @0.3dSports(30) @0.1dSports(30) @0.2dSports(30) @0.3dHOT3D @0.1dHOT3D @0.2dHOT3D @0.3d
Clean Aug-11.4410.5723.341.4410.5723.341.6810.5720.50
Clean Aug-1 Rank-54.5522.4843.514.5522.4843.513.5524.3442.91
Retrieval Global Aug Rank-52.7018.9537.572.6218.4836.765.1619.9834.71
RANSAC Global Aug Rank-53.4320.3038.973.3320.0438.545.2420.6934.19
Ours Selected Aug-5 Rank-13.1920.6140.242.9119.4139.125.0421.9939.87
Ours Selected Aug-1 Rank-54.3924.1444.694.5924.0044.824.7626.5345.83
Global Aug Rank-16.0728.8751.916.4729.8453.339.5031.0952.46

Sports(20) and Sports(30) denote EgoXtreme-Sports with 20- and 30-transformation banks, respectively. Results report candidate oracle success rates at the specified ADD(-S) thresholds, not final trajectory AUC. Global-bank configurations use all 20 or 30 augmentations, respectively.

There is no clear ablation isolating the effects of stage 2 without the effects of stage 1.

Response:

Isolating Stage 2. To separate the contributions of the two stages, we compare GigaPose Top-1, EgoTrack with Clean Aug-1 Rank-5, and the full EgoTrack configuration. Clean Aug-1 Rank-5 disables learned augmentation selection and uses GigaPose's Top-5 hypotheses from the unaugmented template bank.

Without learned augmentation selection, EgoTrack achieves ADD(-S) AUCs of 43.87, 27.07, and 20.70 on Sports, Maintenance, and Emergency, respectively. Adding Stage 1 further improves AUC to 47.07, 27.24, and 23.29, showing that substantial gains are retained independently of the selection policy.

The comparison against GigaPose Top-1 reflects the combined effects of multiple hypotheses and trajectory refinement, rather than temporal denoising alone.

Stage-wise ablation on EgoXtreme.

ScenarioMethodAUC ↑RPE-T ↓RPE-R ↓
SportsGigaPose Top-18.333.6018114.28
SportsEgoTrack (Clean Aug-1 Rank-5)43.870.272929.49
SportsEgoTrack (Selected Aug-1 Rank-5)47.070.272126.45
MaintenanceGigaPose Top-15.520.970591.47
MaintenanceEgoTrack (Clean Aug-1 Rank-5)27.070.031119.97
MaintenanceEgoTrack (Selected Aug-1 Rank-5)27.240.032820.75
EmergencyGigaPose Top-14.023.090692.28
EmergencyEgoTrack (Clean Aug-1 Rank-5)20.700.067512.90
EmergencyEgoTrack (Selected Aug-1 Rank-5)23.290.066814.42

Clean Aug-1 Rank-5 uses the unaugmented GigaPose template bank without learned augmentation selection. Selected Aug-1 Rank-5 uses the full EgoTrack framework.

Evaluation on a single dataset is weak, further evaluation on datasets like YCB-V and HO-3D can strengthen the paper, since some of the sequences have challenging lighting. More importantly, EgoTrack have been trained on EgoXtreme while the baselines, including GigaPose, have not, which makes the evaluation setup unfair.

Response:

Additional benchmark. We extend our evaluation to HOT3D-Aria, where EgoTrack achieves 19.87 ADD(-S) AUC, outperforming GigaPose (13.56) and FreePose (13.74). Compared with GigaPose, it reduces rotation error from 52.69° to 22.10° and RPE-T from 1.2432 to 0.0606 m. We also report FreePose with privileged Top-5 oracle initialization (23.69 AUC).

HOT3D-Aria benchmark.

MethodAUC ↑Trans. ↓Rot. ↓RPE-T ↓RPE-R ↓
GigaPose13.560.076152.691.243283.94
FreePose13.740.072854.496.572730.68
FreePose (Top-5 Oracle)23.690.035647.950.405829.55
EgoTrack (Bidirectional)19.870.061722.100.060624.62

Training fairness. We acknowledge that EgoTrack uses target-domain GT supervision, whereas the baselines were not originally trained under identical conditions. To investigate this difference, we fine-tuned FreePose's CoTracker3 [2] on EgoXtreme-Sports, following the official real-video fine-tuning procedure while excluding potentially occluded handle regions. All FreePose variants were evaluated under identical settings.

The AUCs with frozen CoTracker2 [1], frozen CoTracker3 [2], and fine-tuned CoTracker3 [2] are 6.55, 5.99, and 6.40, respectively. Fine-tuning yields only a marginal AUC improvement over frozen CoTracker3, without improving RPE. This suggests that adapting the point-tracking component alone provides limited benefits under our evaluation setting.

FreePose target-domain adaptation on EgoXtreme-Sports.

MethodAUC ↑Trans. ↓Rot. ↓RPE-T ↓RPE-R ↓
FreePose (CoTracker2, Frozen)6.550.2240122.322.391079.94
FreePose (CoTracker3, Frozen)5.990.2238120.662.127878.88
FreePose (CoTracker3, Fine-tuned)6.400.2235121.672.656481.24

We will clarify the differences in training supervision and include these additional results in the appendix.

[1] Karaev et al., "CoTracker: It is Better to Track Together," ECCV, 2024. [2] Karaev et al., "CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos," arXiv:2410.11831, 2024.

The paper claims to be doing tracking however, if I understood it correctly it seems to be more of an offline trajectory refinement, especially in the bidirectional setup. If that is correct then this should be communicated more clearly.

Response:

We agree that EgoTrack is more precisely characterized as hypothesis-based 6D object pose trajectory refinement, rather than a standalone tracking system.

EgoTrack refines pose sequences generated by a frozen frame-level pose estimator under two temporal settings:

  • Bidirectional: Uses past and future observations within a temporal window for offline trajectory refinement.
  • Causal: Uses only current and past observations, without accessing future frames.

The term "tracking" refers to temporally refining object poses, rather than performing object detection or identity association. We will clarify this distinction in the abstract, introduction, and experimental setup, explicitly distinguishing offline bidirectional refinement from causal inference.

Questions

What is the distribution of template selection?

Response:

Augmentation selection distribution. Stage 1 selects template-bank augmentations, not individual GigaPose templates. Our 20-transformation bank consists of blur (60%), contrast (15%), gamma (10%), and clean, brightness, and color shift (5% each).

We report the selection enrichment ratio (observed selection frequency divided by bank proportion), where 1.0 indicates proportional selection. HOT3D favors clean (1.55) and contrast (1.24), while EgoXtreme favors blur (1.19–1.31), demonstrating dataset-dependent preferences rather than proportional selection.

Top-1 augmentation selection enrichment ratios.

DatasetBlurContrastCleanGammaBrightnessColor shift
HOT3D-Aria1.101.241.550.290.530.40
EgoXtreme-Maintenance1.190.790.610.820.610.43
EgoXtreme-Sports1.270.590.380.510.970.58
EgoXtreme-Emergency1.310.650.310.380.740.46

We will provide the full augmentation-bank specifications in the appendix.

Can you please provide a run time analysis?

Response:

Runtime analysis. We profiled GigaPose, FreePose, GoTrack, and EgoTrack's components. GigaPose requires 63.37 ms/frame, while our policy runs before GigaPose and requires 10.20 ms/frame. DDIM-10 diffusion takes 10.08 ms/window after GigaPose generates the pose hypotheses.

Conservatively counting one complete diffusion-window evaluation per frame without amortization, the estimated EgoTrack pipeline cost is 83.65 ms/frame for inference computation, or 143.27 ms/frame including component-level online overhead. These are component-based estimates rather than directly measured end-to-end latencies.

For comparison, FreePose refinement requires 532.83 ms/unique object-frame and GoTrack tracking requires 3149.19 ms/frame, excluding GigaPose initialization. Peak PyTorch GPU allocations are 5.777 GiB for GigaPose, 0.235 GiB for our policy, and 0.056 GiB for diffusion.

Runtime and memory comparison.

MethodInference time ↓Online runtime ↓GPU memory (GiB) ↓
Measured components
GigaPose (1-bank, Top-5)63.37 ms116.48 ms5.777
Policy (B=1)10.20 ms14.85 ms0.235
Diffusion (B=1, DDIM-10)10.08 ms11.94 ms0.056
Estimated pipeline costs
GigaPose + FreePose596.20 ms876.31 ms10.523
GigaPose + GoTrack3212.56 ms3374.36 ms7.290
GigaPose + EgoTrack83.65 ms143.27 ms6.068

Pipeline times are estimated by summing separately profiled components, with different timing protocols across methods. Pipeline memory values are conservative sums of component peak GPU allocations, not directly measured end-to-end peaks.

We will include the runtime and memory analysis in the revised manuscript.

How long are the evaluated sequences?

Response:

Each EgoXtreme scene contains 1,650 consecutive frames. EgoTrack processes object trajectories using temporal windows of up to 64 frames, with a stride of 16. Trajectories with at least 8 valid frames are retained through padding and validity masking, while overlapping predictions are aggregated at the frame level.

In our evaluation, full-length 64-frame windows account for 98.68%, 59.85%, and 95.08% of the windows in Maintenance, Sports, and Emergency, respectively. The remaining windows contain 8–63 valid frames, with shorter windows occurring more frequently in Sports.

Distribution of evaluated window lengths.

Scenario8–1516–3132–4748–6364
Maintenance33 (0.32%)47 (0.45%)34 (0.33%)23 (0.22%)10,258 (98.68%)
Sports534 (16.37%)501 (15.35%)202 (6.19%)73 (2.24%)1,953 (59.85%)
Emergency24 (1.26%)38 (1.99%)14 (0.73%)18 (0.94%)1,818 (95.08%)

Values indicate the number and percentage of evaluated windows, grouped by valid frame count.

Thus, trajectory refinement operates within 64-frame windows rather than across entire scenes simultaneously. We will clarify this distinction in the revised manuscript and provide the detailed window-length distribution in the appendix.

Can you provide visualisations for the trajectories estimated compared to GT?

Response:

We have added qualitative trajectory visualizations to the appendix, comparing GT, GigaPose Top-1, and EgoTrack across Emergency, Maintenance, and Sports scenarios. These visualizations illustrate how EgoTrack recovers trajectories closer to GT and reduces temporal fluctuations compared with frame-wise GigaPose estimates.

The reliability cue is not defined in the main paper. Can you please define it?

Response:

The reliability cue is described in the appendix, but we agree that a concise definition in the main paper would improve clarity.

The reliability cue rt,k\mathbf{r}_{t,k} is an eight-dimensional feature vector associated with each pose hypothesis. It consists of six policy/selection features (selector probability, selector score, selector rank, selection rank, selection score, and diversity penalty) and two GigaPose confidence features (template retrieval score and RANSAC score).

These features are normalized over valid candidates, embedded, and fused with the corresponding pose representations to inform trajectory refinement. All cues are available at inference time without GT poses or oracle labels.

We will add this definition to the main paper, with further details provided in the appendix.

Reviewer_gAXfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EgoTrack addresses robust 6D object pose tracking in egocentric video under severe visual degradation. Stage 1 uses a learned policy to select one of 20 appearance-transformed template banks for each frame, runs a frozen GigaPose estimator, and retains K=5 ranked pose hypotheses with reliability cues. Stage 2 uses a diffusion trajectory denoiser whose state-conditioned query re-attends to the candidates at every denoising step and can select a learned NULL token when local evidence is unreliable. On the three EgoXtreme scenarios (Sports, Maintenance, and Emergency), EgoTrack substantially improves ADD(-S) AUC and relative pose error over GigaPose, FreePose, and GoTrack. And ablations support multi-hypothesis reasoning, state-conditioned retrieval, and NULL-based abstention. The evidence is strongest on this benchmark and the real-world examples are qualitative and have no ground-truth annotations.

Critical strengths
  1. The central design is well motivated and technically coherent: retaining multiple GigaPose hypotheses while allowing the evolving trajectory state to reweight them addresses a concrete failure mode of frame-wise top-1 selection. The NULL token is a useful explicit mechanism for abstaining when all local candidates are unreliable.
  2. The empirical gains are substantial and consistent across difficult scenarios. The candidate-set and mechanism ablations support the claimed roles of multi-hypothesis evidence and state-conditioned association, and the causal variant shows that the approach is not limited to bidirectional inference.
Critical weaknesses
  1. The main evaluation is confined to EgoXtreme and uses ground-truth object masks for every method. This establishes robustness after segmentation, but does not establish end-to-end robustness or transfer to other egocentric datasets. The real-world examples are qualitative and have no quantitative ground truth.
  2. Practical and statistical evidence is incomplete. The paper does not report inference time, memory, or the number of template/GigaPose evaluations, despite the 10-step diffusion sampler and an offline exhaustive transformation sweep used to train the policy. It also gives no confidence intervals or seed variation, and the policy's thresholded 0.3d training label is only partially connected to the final AUC objective.
Questions for authors
  1. The paper is primarily motivated by the unreliability of frame-level candidate ranking under severe degradation. However, the candidate generation stage still relies on a frozen GigaPose ranking to retain only the top-5 hypotheses. While current results show that top-5 outperforms top-1, they do not sufficiently justify the choice of this specific truncation threshold.
  2. Please specify exactly which ground-truth signals are used during policy and trajectory training and when. In particular, confirm how the scene-level cross-fitting prevents leakage and whether all baselines were retrained or evaluated under the same data and mask protocol.
  3. What are the end-to-end inference time, memory footprint, and number of GigaPose/template evaluations per frame for each variant, compared with GigaPose, FreePose, and GoTrack? This affects the claimed egocentric and embodied use case.
  4. If existing baseline data is available, or if reproduction is feasible, could the authors include one or two additional benchmarks under similar extreme conditions, along with metrics for more comparative methods on EgoXtreme, as well as real-world demonstrations under other extreme scenarios such as occlusion, to more comprehensively demonstrate the method's effectiveness?
4 comments

Questions

The paper is primarily motivated by the unreliability of frame-level candidate ranking under severe degradation. However, the candidate generation stage still relies on a frozen GigaPose ranking to retain only the top-5 hypotheses. While current results show that top-5 outperforms top-1, they do not sufficiently justify the choice of this specific truncation threshold.

Response:

We thank the reviewer for raising this concern. To better justify our choice of K=5K=5, we extended our analysis to K=1,3,5,7,10K=1,3,5,7,10 on EgoXtreme-Sports without additional training.

Candidate coverage. Top-KK oracle accuracy at 0.3d increases from 24.14% (K=1K=1) to 46.25% (K=5K=5) and 55.94% (K=10K=10), indicating that larger candidate sets provide greater coverage.

K@0.1d ↑@0.2d ↑@0.3d ↑
11.4311.2624.14
33.2119.6437.55
54.4624.2846.25
75.4327.6350.89
106.6431.2855.94
*Results on EgoXtreme-Sports.

Trajectory refinement. In contrast, EgoTrack's ADD(-S) AUC improves from 40.84 (K=1K=1) to 47.07 (K=5K=5), but shows negligible additional benefit at K=7K=7 (46.74) and K=10K=10 (46.99).

KAUC ↑Trans. ↓Rot. ↓RPE-T ↓RPE-R ↓
140.840.090616.390.267927.01
344.550.082716.150.281228.51
547.070.077714.710.272126.45
746.740.078314.280.276127.17
1046.990.077214.220.278727.49
*Results on EgoXtreme-Sports.

Thus, we adopt K=5K=5 as a practical choice that maintains refinement performance with fewer hypotheses. Although larger candidate sets improve oracle coverage, the additional hypotheses do not translate into meaningful improvements in final refinement performance.

Please specify exactly which ground-truth signals are used during policy and trajectory training and when. In particular, confirm how the scene-level cross-fitting prevents leakage and whether all baselines were retrained or evaluated under the same data and mask protocol.

Response:

We clarify the ground-truth supervision and evaluation protocols as follows.

Ground-truth supervision. Policy training uses GT 6D poses to label each template-bank augmentation based on whether GigaPose's rank-0 prediction satisfies the 0.3d ADD(-S) threshold. Trajectory training uses GT poses to supervise diffusion, symmetry-aware targets, and translation/rotation uncertainty calibration, while model inputs consist of actual policy-selected GigaPose hypotheses rather than oracle candidates. GT poses are used for validation loss computation but not for trajectory inference.

Scene-level cross-fitting. Training scenes are divided into two disjoint folds. Each policy is trained on one fold and generates out-of-fold candidates for the other, preventing the policy from generating training candidates for scenes whose GT labels it has seen. Validation and test inference use an ensemble of both policies without GT-based candidate selection.

Baseline protocol. All methods use the same evaluation splits and GT object masks for localization in the main experiments. However, baselines are not retrained with identical supervision; existing models are evaluated with method-specific initialization procedures. We acknowledge that the original description did not make this distinction sufficiently clear and will explicitly document the retraining status and initialization procedure for each baseline in the revision.

Weaknesses & Questions

Practical and statistical evidence is incomplete. The paper does not report inference time, memory, or the number of template/GigaPose evaluations, despite the 10-step diffusion sampler and an offline exhaustive transformation sweep used to train the policy. It also gives no confidence intervals or seed variation, and the policy's thresholded 0.3d training label is only partially connected to the final AUC objective.

What are the end-to-end inference time, memory footprint, and number of GigaPose/template evaluations per frame for each variant, compared with GigaPose, FreePose, and GoTrack? This affects the claimed egocentric and embodied use case.

Response:

We provide additional profiling results and analyses of computational efficiency and statistical reliability below, and will incorporate them into the revision.

Computational efficiency and memory. GigaPose requires 63.37 ms/frame (5.777 GiB). Our policy runs before GigaPose to select a template-bank augmentation, requiring 10.20 ms/frame (0.235 GiB). After GigaPose generates the Top-5 hypotheses, DDIM-10 diffusion takes 10.08 ms/window (0.056 GiB). For comparison, FreePose refinement requires 532.83 ms/unique object-frame (4.746 GiB), and GoTrack tracking requires 3149.19 ms/frame (1.513 GiB), excluding GigaPose initialization.

Estimated pipeline cost. Conservatively counting one complete diffusion-window evaluation per object-frame without amortization, the estimated EgoTrack pipeline cost is 83.65 ms/frame for inference computation, or 143.27 ms/frame including component-level online overhead. These estimates sum separately measured components rather than directly measuring end-to-end latency. The accompanying table also reports estimated pipeline costs for the compared methods.

MethodInference time ↓Online runtime ↓GPU memory (GiB) ↓
Measured components
GigaPose (1-bank, Top-5)63.37 ms116.48 ms5.777
Policy (B=1)10.20 ms14.85 ms0.235
Diffusion (B=1, DDIM-10)10.08 ms11.94 ms0.056
Estimated pipeline costs
GigaPose + FreePose596.20 ms876.31 ms10.523
GigaPose + GoTrack3212.56 ms3374.36 ms7.290
GigaPose + EgoTrack83.65 ms143.27 ms6.068

Component memory values are measured peak PyTorch GPU allocations; pipeline memory values are conservative sums of component peaks, not measured end-to-end peaks. Pipeline times are component-based estimates, and baseline timing and initialization protocols differ.

GigaPose evaluations. Our final configuration selects one template-bank augmentation and runs GigaPose only once per frame, retaining its Top-5 hypotheses. The Aug-5 and full-bank configurations, evaluated only for ablation, require 5 and 20 GigaPose runs, respectively. The exhaustive augmentation sweep is performed offline for policy supervision and is not repeated during inference.

Sampling efficiency. We evaluate DDIM sampling with 10, 20, and 30 steps. Increasing the steps from 10 to 30 increases sampling time from 10.08 to 26.82 ms/window (2.66×) without improving ADD(-S) AUC. We therefore retain 10 steps as a practical setting balancing accuracy and computational cost.

DDIM stepsSampling time ↓Sports AUC ↑Maintenance AUC ↑Emergency AUC ↑
1010.08 ms47.0727.2423.29
2017.91 ms47.0027.0822.66
3026.82 ms46.8126.9622.55

Statistical reliability and supervision. We computed 95% confidence intervals for all our experimental results reported in the main paper using 2,000 scene-level bootstrap resamples. These quantify evaluation-set variability rather than training-seed variability. The policy's 0.3d success label corresponds to the upper limit of the ADD(-S) AUC range and serves as a surrogate for selecting usable hypotheses, rather than directly optimizing AUC.

We will include the runtime and memory analysis in the revised manuscript, with detailed profiling protocols, DDIM ablations, and confidence intervals provided in the appendix.

The main evaluation is confined to EgoXtreme and uses ground-truth object masks for every method. This establishes robustness after segmentation, but does not establish end-to-end robustness or transfer to other egocentric datasets. The real-world examples are qualitative and have no quantitative ground truth.

If existing baseline data is available, or if reproduction is feasible, could the authors include one or two additional benchmarks under similar extreme conditions, along with metrics for more comparative methods on EgoXtreme, as well as real-world demonstrations under other extreme scenarios such as occlusion, to more comprehensively demonstrate the method's effectiveness?

Response:

To address these concerns, we conducted additional experiments on HOT3D-Aria, analyzed robustness to partial visibility, and evaluated EgoTrack using predicted CNOS masks.

Additional benchmark. On HOT3D-Aria, EgoTrack achieves 19.87 ADD(-S) AUC, outperforming GigaPose (13.56) and FreePose (13.74). It also reduces rotation error (52.69° → 22.10°) and RPE-T (1.2432 → 0.0606 m) compared with GigaPose. We additionally report FreePose with privileged Top-5 oracle initialization (23.69 AUC).

HOT3D-Aria benchmark.

MethodAUC ↑Trans. ↓Rot. ↓RPE-T ↓RPE-R ↓
GigaPose13.560.076152.691.243283.94
FreePose13.740.072854.496.572730.68
FreePose (Top-5 Oracle)23.690.035647.950.405829.55
EgoTrack (Bidirectional)19.870.061722.100.060624.62

Occlusion and truncation. On HOT3D, EgoTrack improves AUC from 3.64 to 18.16 under low visibility (≤0.7), compared with 16.09 to 18.50 at higher visibility. On EgoXtreme-Sports, AUC improves from 0.02 to 18.20 under severe truncation (visibility below 0.2). HOT3D visibility reflects both occlusion and truncation, whereas the EgoXtreme analysis measures truncation. Additional occlusion examples are provided in the appendix.

HOT3D-Aria: Visibility analysis.

MethodVisibilityAUC ↑Trans. ↓Rot. ↓
GigaPose0.00–0.703.640.1438115.15
GigaPose0.70–1.0016.090.063940.57
EgoTrack0.00–0.7018.160.082331.67
EgoTrack0.70–1.0018.500.061720.99

Visibility analysis uses a subset of windowed frames.

Predicted-mask evaluation. We evaluate two settings using CNOS predictions: all original evaluation frames, including detection failures, and confidence-filtered frames (CNOS score > 0.5). In the all-frame setting, EgoTrack improves GigaPose AUC from 2.89 to 7.05 on Sports, 0.64 to 1.62 on Maintenance, and 2.05 to 3.73 on Emergency. For confidence-filtered frames, covering 9.4% of the original evaluation frames, AUC improves from 28.03 to 54.55, 3.87 to 9.15, and 8.88 to 15.76, respectively. Both methods follow the same evaluation protocol without GT-based detection matching.

EgoXtreme: CNOS predicted masks (Score > 0.5).

ScenarioMethodAUC ↑Trans. ↓Rot. ↓RPE-T ↓RPE-R ↓
SportsGigaPose28.030.1431109.260.7504107.35
SportsEgoTrack54.550.083648.970.398741.81
MaintenanceGigaPose3.870.096782.490.298674.50
MaintenanceEgoTrack9.150.066118.320.024718.56
EmergencyGigaPose8.880.078248.750.588665.29
EmergencyEgoTrack15.760.076624.530.05948.30

EgoXtreme: CNOS predicted masks (All frames).

ScenarioMethodAUC ↑Trans. ↓Rot. ↓RPE-T ↓RPE-R ↓
SportsGigaPose2.891.1202122.28229.3163108.17
SportsEgoTrack7.050.498188.070.529367.17
MaintenanceGigaPose0.640.3256121.9828.8943111.87
MaintenanceEgoTrack1.620.149068.670.098938.34
EmergencyGigaPose2.050.9379149.6092.012594.85
EmergencyEgoTrack3.730.071653.490.440028.62

The all-frame evaluation includes detection failures, whereas the confidence-filtered evaluation reports performance on the retained frame subset.

Scope and limitations. EgoTrack refines existing pose hypotheses rather than replacing object detection or initial pose estimation. Temporal evidence can compensate for unreliable local observations, but recovery remains challenging when informative evidence is unavailable across most of the temporal window. The all-frame CNOS results incorporate upstream detection failures, which limit absolute performance despite consistent improvements from EgoTrack. Improving detection and segmentation remains a direction for future work.

Dear Reviewer gAXf,

Thank you for your constructive feedback regarding our work. We hope our responses and additional experiments have addressed your concerns. Please feel free to let us know if you have any further questions or comments.

Best regards, the Authors

Comments on the paper

2 comments

We thank all three reviewers (R1, R2, and R3) for their thoughtful and constructive feedback. We appreciate their recognition of the importance of robust 6D object pose estimation under challenging visual conditions and the potential of hypothesis-based trajectory refinement.

Our main goal is to recover reliable object pose trajectories from ambiguous or inaccurate frame-level hypotheses by combining task-oriented candidate selection with temporal reasoning.

The reviewers raised several important concerns regarding generalization, contribution isolation, training fairness, computational efficiency, and methodological clarity. We conducted additional experiments and analyses to address these concerns, as summarized below. Detailed results are provided in our individual responses and the revised manuscript.

Evaluation diversity and robustness (R1, R2, R3)

  • We extended quantitative evaluation to HOT3D-Aria, where EgoTrack achieves 19.87 ADD(-S) AUC, compared with 13.56 for GigaPose and 13.74 for FreePose. We also analyzed performance under partial visibility and evaluated CNOS predicted masks, including an all-frame evaluation that retains frames with missed detections. These results demonstrate improvements beyond the original evaluation setting while highlighting the remaining dependence on upstream detection quality.

Contribution isolation and training fairness (R1, R2, R3)

  • We evaluated Stage 2 using Clean Aug-1 Rank-5 without learned augmentation selection, confirming that substantial gains persist without the proposed selection policy. We also compared EgoTrack against FreePose using the same Stage 1 Top-5 candidate pool, from which FreePose receives the GT-best pose as privileged oracle initialization. To investigate domain adaptation, we evaluated frozen and fine-tuned CoTracker3 under both standard and privileged Top-5 oracle initialization, observing modest AUC gains from fine-tuning in both settings. We acknowledge that these comparisons do not fully equalize training supervision and explicitly state this limitation in the revised manuscript.

Candidate selection and refinement mechanisms (R1, R2, R3)

  • We analyzed the effect of candidate-set size (K=1,3,5,7,10K=1,3,5,7,10), expanded the augmentation bank from 20 to 30 transformations, and examined dataset-dependent augmentation selection distributions. We also analyzed NULL attention under different candidate-quality conditions, finding greater reliance on NULL when candidate evidence is unreliable. While NULL improves pose accuracy, we clarify that it does not consistently improve temporal RPE.

Computational efficiency and statistical reliability (R1, R2, R3)

  • We profiled inference time and GPU memory for EgoTrack and the compared baselines, including the number of GigaPose evaluations. We additionally evaluated DDIM sampling with 10, 20, and 30 steps, assessed sensitivity to DDIM sampling seeds, and computed 95% confidence intervals using 2,000 scene-level bootstrap resamples. We report measured component-level runtimes and distinguish pipeline cost estimates from directly measured end-to-end latency.

Methodological clarity (R1, R2)

  • We explicitly characterize EgoTrack as hypothesis-based 6D pose trajectory refinement, distinguishing offline bidirectional refinement from causal inference. We also clarify the temporal window length, reliability cues, GT supervision, scene-level cross-fitting, and baseline evaluation protocols.

These additional experiments and clarifications strengthen the empirical evidence for EgoTrack while making its scope and limitations more explicit. The corresponding results and methodological details have been incorporated into the revised manuscript and appendix.

Area_Chair_RvNxOpenReview ↗

Please check the authors' rebuttal before the discussion deadline again

Dear Reviewers,

The authors have posted their responses to the issues you raised. As the discussion period will end soon, please take a look or a last look to see if their latest responses have addressed your concerns or further concerns well at your convenience. Thank you very much!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.