acceptodds
Under review as a conference paper at ICLR 2027

Learning Not to Discard Prematurely: Trajectory Learning for Interactive 3D Visual Grounding

Abstract

3D visual grounding aims to localize the query-matched target object in a 3D scene. In real-world scenarios such as human–robot interaction, human queries are often ambiguous and may be compatible with multiple candidate objects, necessitating iterative interaction to identify the intended target. However, as the interaction progresses, candidate filtering based solely on current query–object matching often fails to explicitly account for its consequences for subsequent interaction. We observe that interaction trajectories reveal which early filtering decisions successfully retain the target until disambiguating evidence arrives, and which make later clarification ineffective. Building on this observation, we formulate interactive 3D visual grounding as sequential candidate refinement, and propose a trajectory learning framework that uses ordered candidate transitions, subsequent feedback, and final outcomes to optimize early candidate filtering decisions. Through trajectory-level preference optimization, the framework learns to retain plausible candidates until sufficient clarification evidence arrives. To effectively propagate trajectory-level supervisory signals to the early candidate generation, we introduce a trajectory-aware adapter that integrates interaction history and candidate belief to produce residual updates for candidate representations and grounding scores. We evaluate our framework on AmbiRefer3D, an interactive 3D visual grounding dataset, and construct a more challenging subset requiring multiple rounds of clarification to uniquely identify the target. Experimental results demonstrate that our method improves candidate retention and downstream interaction performance, outperforming the direct-training baseline by an average of 20.56% in TS and 3.66% in Acc across grounding backbones. These results highlight that learning how to avoid prematurely discarding candidates is crucial for achieving reliable interactive 3D visual grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Visual Grounding, Interactive Visual Grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bm3xOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies premature candidate filtering in interactive 3D visual grounding, where the target may be incorrectly discarded before sufficient clarification is available. The authors propose a trajectory learning framework that uses interaction history, candidate transitions, and trajectory-level preference optimization to improve candidate retention and subsequent disambiguation. A trajectory-aware adapter is introduced to refine candidate features and grounding scores. Experiments on AmbiRefer3D and a newly constructed challenging subset show consistent improvements across four grounding backbones.

Critical strengths
  1. Well-motivated problem. The paper identifies an important issue in interactive grounding: early candidate filtering can affect all subsequent interactions. I find the motivation convincing, and the idea of optimizing candidate selection based on future interaction outcomes is reasonable.

  2. Solid experimental validation. The method is evaluated on four different grounding backbones, with consistent improvements in target survival and final accuracy. The comparison between ordered trajectories and shuffled pairs is particularly useful, as it demonstrates the benefit of modeling sequential dependencies rather than simply adding more supervision. The ablations also provide reasonable evidence for the proposed components.

Critical weaknesses
  1. The method combines several components and training objectives, including the trajectory-aware adapter, retention supervision, gating, and preference optimization. While the ablations demonstrate their effectiveness, the overall framework is somewhat complicated, and it is unclear how much additional training and inference overhead is introduced.

  2. Limited improvements in final accuracy. Although the gains in target survival are substantial, the improvements in final grounding accuracy are relatively modest for some backbones. This suggests that better candidate retention does not always translate into successful

Questions for authors

What is the additional training and inference overhead of the proposed framework compared to direct training? In particular, how much overhead comes from the trajectory-aware adapter?

1 comment

Q1: Additional training and inference overheads, particularly from the trajectory-aware adapter.

Thank you for this important question and for recognizing our motivation and multi-backbone evaluation. To evaluate the training and inference overhead, we analyze adapter parameters, training cost, peak GPU memory usage, and module-wise inference latency across Direct, Shuffled pairs, and Ordered trajectory on an NVIDIA A40 GPU.

SchemeAdapter Params (M)GPU-hPeak GBGrounding (ms/query)Adapter (ms/query)Disamb. (ms/query)Total (ms/query)
Direct0.0006.124.87100.640.0019.96120.60
Shuffled pairs0.3978.6511.1397.431.9822.16121.58
Ordered trajectory0.3978.3013.6395.951.8820.37118.19

As shown in Table, compared with Direct, our framework introduces only 0.397M additional parameters, while training cost and peak GPU memory increase by 2.18 GPU-h and 8.76 GB, respectively. Compared with Shuffled pairs, Ordered trajectory requires comparable training time, with an additional 2.50 GB of peak GPU memory usage.

During inference, the trajectory-aware adapter introduces only 1.88 ms per query, accounting for approximately 1.6% of the total inference latency. Despite this additional overhead, the overall inference latency decreases from 120.60 to 118.19 ms per query, demonstrating comparable inference efficiency.

W2: Limited improvements in final accuracy

Thank you for bringing up this phenomenon. For some backbones, the limited accuracy gains may be attributed to more severe ambiguity in the newly retained instances, making subsequent disambiguation more challenging. Another factor is that the extracted features are not sufficiently exploited by the downstream disambiguation module to distinguish the intended target from similar candidates.

Reviewer_22uyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors aim to address premature candidate removal in interactive 3D visual grounding. To solve the task, the authors propose a trajectory learning framework to retain plausible candidates until there is enough evidence. The authors also propose a trajectory-aware adapter that refines candidates. The proposed method is evaluated on AmbiRefer3D and a newly constructed multi-round subset.

Critical strengths
  1. The authors address an important challenge in interactive 3D grounding, where the prematurely discarded objects will prevent disambiguation. The proposed method consistently improves target survival across different grounding backbones, which show the potential of applying the trajectory-aware training for interactive 3D grounding.
Critical weaknesses
  1. The claimed task novelty and method novelty are limited. To the reviewer's understanding, the proposed formulation of interactive 3D visual grounding as a sequential candidate refinement is closely related to AmbiRefer3D. In their pipeline, there already exists multi-round interactions and progressive filtering. The authors should better clarify the key differences from the prior work. Moreover, the proposed method introduces a trajectory-aware adapter and trajectory-level supervision. It is not clear what new insights are introduced beyond adapting the existing recurrent state modeling and preference learning techniques to this interactive 3D grounding task.
  2. The experimental evidence are not sufficient. For different training schemes in Tab.1, the Direct baseline freezes the grounding backbone, while the proposed method jointly optimizes both the grounding backbone and additional modules (L352-358). Based on appendix B, The training modules are not well controlled over different training schemes. Furthermore, it seems the improvements in target survival (TS) does not consistently means better grounding accuracy. For example, BUTD-DETR in Tab. 1, target survival improves from 64.69 to 83.93, while accuracy barely changes from 26.92 to 26.99. It's hard to determine whether trajectory learning itself effectively improves the interactive grounding.
Questions for authors
  1. Based on weakness 1, could the authors further clarify the key differences in the task formulation and the methods between the proposed framework and prior work (AmbiRefer3D)?
  2. Based on weakness 2, though the authors compare against shuffled pairs scheme, the ordered trajectory scheme introduces several changes simultaneously. Could the authors provide a more controlled comparison to isolate the contributions of these components? For example, it would be helpful to identify whether the improvements primarily come from modeling sequential dependencies, or from the additional supervision losses?
  3. Based on weakness 2, the improvement in TS is larger than the final grounding accuracy. Could the authors analyze why better candidate retention does not consistently lead to better grounding performance?
  4. Could the authors include a computation analysis of the overall proposed scheme against the baseline schemes Direct and Shuffled pairs, including training cost, memory usage and inference latency? This better clarifies the tradeoff between accuracy improvement and computation cost.
1 comment

Q1:The key differences between our framework and AmbiRefer3D in task formulation and method.

Thank you for this insightful question. Our work follows the same task formulation as AmbiRefer3D, focusing on interactive 3D visual grounding through multi-round clarification and progressive candidate filtering. The key difference is that AmbiRefer3D does not explicitly consider how early candidate filtering affects subsequent disambiguation. We observe that interaction trajectories provide valuable supervision for early candidate retention decisions and accordingly propose a trajectory learning framework.

Q2: Can controlled experiments isolate the contributions of those components?

To better isolate the effects of sequential modeling from those of model capacity and backbone fine-tuning, we introduce an additional Direct variant with the same trainable modules as the Ordered trajectory scheme, while matching the total number of training rounds. As shown in the table below, these results isolate the effects of increased trainable capacity and provide further evidence for the importance of modeling sequential dependencies through trajectory-level learning.

Training schemeTSAccAvgR
Direct variant71.6829.431.24
Ordered trajectory77.1634.063.20

We have reported loss and component ablations in Table 3 of the main text. The first row corresponds to a jointly fine-tuned component baseline with ordered interaction training, where the adapter, retention supervision, gate, and preference optimization are disabled, while the original grounding and local interaction losses remain active. These experiments validate the contributions of different supervision losses within Ordered trajectory.

Q3: Why do improvements in TS not consistently translate into higher grounding accuracy?

The reason is that the newly retained instances may involve more severe ambiguity, making the intended targets difficult to distinguish from similar candidates. Additionally, the extracted features may not be sufficiently exploited by the downstream disambiguation module to resolve the remaining ambiguity.

Q4: Computation analysis of the overall proposed scheme against the baseline schemes Direct and Shuffled pairs, including training cost, memory usage and inference latency.

Thank you for requesting this practical comparison. We provide a computation analysis of Direct, Shuffled pairs, and Ordered trajectory on an NVIDIA A40 GPU, covering additional parameters, training cost, peak GPU memory usage, and module-wise inference latency.

SchemeAdapter Params (M)Training Cost (GPU-h)Peak GPU Memory (GB)Inference Latency (ms/query)
Direct0.0006.124.87120.60
Shuffled pairs0.3978.6511.13121.58
Ordered trajectory0.3978.3013.63118.19

As shown in Table, our framework introduces only 0.397M additional parameters. Ordered trajectory increases training cost from 6.12 to 8.30 GPU-h compared with Direct, while remaining comparable to Shuffled pairs at 8.65 GPU-h. It requires an additional 2.50 GB of peak GPU memory compared with Shuffled pairs. Notably, its overall inference latency decreases to 118.19 ms/query, compared with 120.60 and 121.58 ms/query for Direct and Shuffled pairs, respectively. Despite the additional memory overhead, our framework maintains comparable training and inference efficiency, achieving substantial improvements in grounding accuracy with limited additional computational cost.

Reviewer_Ww6wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies interactive 3D visual grounding, where an ambiguous query matches several objects and the target is identified over several rounds of clarification. The authors argue that a grounding model which prunes candidates only by current query–object matching can discard the target before disambiguating feedback arrives. They propose to train candidate generation from interaction trajectories. A trajectory-aware adapter (GRU history plus geometric context) applies residual corrections to the backbone's candidate features and scores. It is trained jointly with an interactive disambiguation module, using a retention loss and a DPO-inspired preference loss over logged alternative actions. Experiments on the Mixture split of AmbiRefer3D with four backbones (M3DRefCLIP, BUTD-DETR, 3D-VisTA, TSP3D) compare three training schemes (Direct, Shuffled pairs, Ordered trajectory). The authors also build a 220-example challenging subset with edited ScanNet scenes. Ordered trajectory training improves target survival (TS) by 20.56 points and accuracy by 3.66 points on average over Direct.

Critical strengths
  1. The problem is well motivated. Fig. 1(b) shows that enlarging the candidate budget increases target coverage but dilutes the candidate set. The point that early pruning should be judged by its consequences for later clarification is a sensible way to frame interactive grounding.
  2. The trajectory-structure controls are informative. Shuffling round order or feedback signs (Table 4) clearly hurts, and the gains over Direct are consistent in TS across four different backbones. Together these suggest that the ordered training signal, and not only the extra supervision, matters for retention.
Critical weaknesses
  1. There is no comparison with any external method, and the main baseline is confounded. All results compare the authors' own three training schemes. The method of AmbiRefer3D itself and other interactive or ambiguity-aware grounding approaches are not reported. "Direct" freezes the backbone and disables the adapter, while "Ordered trajectory" fine-tunes the backbone and adds the adapter and roughly a dozen additional loss terms. Part of the gap may therefore come from extra trainable capacity and fine-tuning rather than from trajectory learning. Shuffled pairs partly controls for this, but the improvement over it is modest in accuracy (2.34 points on the test set). For BUTD-DETR, Ordered trajectory is essentially equal to Direct in accuracy (26.99 vs. 26.92).
  2. The improvement is mainly in TS, which can be raised by keeping more candidates, and the downstream effect is small and noisy. TS measures whether the target survives the initial filter (threshold 0.01). The paper never reports the size of the initial candidate set C0 for each method, so a TS gain could partly reflect larger candidate sets. Fig. 3 addresses this only for M3DRefCLIP. Accuracy gains are much smaller than TS gains (3.66 vs. 20.56 on average). On the 220-example challenging subset, one example equals 0.45 points, so differences such as 19.09 vs. 17.73 (3D-VisTA) amount to about three examples. No seeds or confidence intervals are given. The claim that AvgR decreases "across all four backbones" also does not hold on the subset (3D-VisTA 3.44 → 3.72). Finally, the "interaction" uses annotated constraints with oracle yes/no feedback, and the "rejected trajectories" reuse logged states rather than real rollouts. The claims about human–robot interaction and trajectory-level preference optimization are therefore stronger than what is tested.
Questions for authors

Clarifications that could change my assessment:

  1. How does your framework compare with the interaction method proposed in AmbiRefer3D and with other published methods on the same split, under the same evaluation protocol?
  2. What is the average size of C0 for Direct, Shuffled pairs and Ordered trajectory on each backbone? What are the TS and Acc of Direct when its threshold is tuned to give the same average |C0|?
  3. Can you add a Direct variant that fine-tunes the backbone and includes the adapter with only per-round supervision, to separate capacity and fine-tuning from trajectory learning?
  4. Please report mean and standard deviation over several seeds, at least for M3DRefCLIP on the test set and on the challenging subset.
  5. What configuration does the first row of Table 3 (no components; TS 65.43 / Acc 40.18) correspond to? It differs from both Direct and Shuffled pairs in Table 1.

Suggestion (not required for my decision): evaluate with noisy or free-form feedback (e.g., an LLM-simulated user) to support the human–robot interaction framing. Since S@6 always equals Acc with a 6-round budget, one of the two columns could be dropped.

3 comments

Q1. Comparison with external methods.

We additionally evaluate AmbiRefer3D and two adapted ambiguity-aware baselines, AmbiVer and CandVG. For the adapted AmbiVer baseline, GroundingDINO is used for multi-view object detection, followed by cross-view instance unification and Qwen3-VL-8B-Instruct to generate initial 3D candidates. For CandVG, we use Qwen3.5-9B to predict candidate boxes from multiple views and map them to 3D scene instances. We integrate their generated 3D candidates into the same trajectory-aware adapter and interactive disambiguation pipeline to enable multi-round grounding. The corresponding results are shown as follows:

MethodTSAccAvgR
AmbiRefer3D65.0040.033.51
AmbiVer-adapter59.7729.552.91
CandVG-adapter71.6836.073.15
Our framework87.0244.372.88

Q2. Initial candidate-set size and count-matched Direct baseline.

We report the average initial candidate-set size, ∣C0∣|\mathcal{C}0|, for all four backbones with the fixed threshold of 0.01:

BackboneDirectShuffled pairsOrdered trajectory
M3DRefCLIP5.565.365.88
BUTD-DETR9.219.749.90
3D-VisTA2.633.503.84
TSP3D12.076.548.33

Since exactly matching average candidate counts across all backbones on the full test set by threshold tuning is computationally prohibitive, we use 256 sampled test queries for a controlled comparison. We adjust the thresholds to obtain the same average ∣C0∣≈5.996|\mathcal{C}_0| \approx 5.996 and report the Direct results:

BackboneCount-matched Direct TS/AccOriginal Direct TS/Acc
M3DRefCLIP66.14 / 37.8964.71 / 37.54
BUTD-DETR57.42 / 21.9760.55 / 22.48
3D-VisTA94.92 / 36.1076.09 / 31.77
TSP3D67.58 / 23.0379.60 / 25.61

These results suggest that improved retention and subsequent disambiguation cannot be explained solely by increasing the initial candidate count, which may also increase the burden of later interaction.

Q3. Capacity- and fine-tuning-controlled Direct variant.

We compare a Direct variant with per-round supervision against Ordered trajectory while matching the training data and trainable modules. The experiment uses M3DRefCLIP with matched training rounds:

Training schemeTSAccAvgR
Direct variant71.6829.431.24
Ordered trajectory77.1634.063.20

These results provide further evidence that modeling cross-round dependencies through trajectory-level supervision improves both TS and Acc beyond the benefits of additional trainable capacity and backbone fine-tuning.

Q4. Multi-seed reliability.

We conduct three independent runs (seeds 101, 123, and 555) with M3DRefCLIP and report mean ±\pm standard deviation for TS and Acc on the Mixture test set and challenging subset:

SplitDirect TS / AccShuffled pairs TS / AccOrdered trajectory TS / Acc
Test set57.84±0.65 / 38.04±0.9478.31±0.70 / 38.56±0.6587.60±0.35 / 44.16±0.60
Challenging54.29±0.52 / 20.30±0.4568.25±0.52 / 20.75±0.4575.75±0.78 / 25.25±0.91

Q5. Table 3 first-row configuration.

The first row of Table 3 (TS 65.43 / Acc 40.18) is a jointly fine-tuned component baseline with ordered interaction training. The adapter, retention supervision, gate, and preference optimization are disabled, while the original grounding and local interaction losses remain active. It therefore differs from both Direct and Shuffled pairs. We have clarified this configuration in the revised manuscript.

Corrections concerning W2.

Regarding your note in W2 on AvgR in Table 2, we apologize for the imprecise phrasing in Section 4.3. On the challenging subset (Table 2), AvgR decreases on three backbones but increases on 3D-VisTA. We also narrowed the human–robot interaction framing and clarified that preference optimization is conducted offline over recorded interaction states. These changes are highlighted in blue in the revised manuscript.

Suggestion: noisy feedback and redundant S@6.

We inject feedback corruption of 10%, 20%, and 30% at inference time on the AmbiRefer3D Mixture test set with M3DRefCLIP:

SchemeMetric0%10%20%30%
DirectAcc38.7436.1334.3831.55
DirectAvgR3.433.693.914.12
Shuffled pairsAcc39.1136.5134.2831.86
Shuffled pairsAvgR3.163.373.593.80
Ordered trajectoryAcc44.3742.2839.7037.45
Ordered trajectoryAvgR2.882.973.153.37

We have also removed the redundant S@6 column from the revised manuscript.

Thank you for the response and the additional experiments. The AvgR correction, the narrower HRI framing, and the clarification of Table 3 are appreciated. Some numbers still need clarification:

  1. Raw counts. TS, Acc and S@K are per-example binary outcomes. On the 220-example challenging subset, every value must therefore be a multiple of 1/220, yet none of the S@1/S@3 values in Table 2 are (e.g., 7.41, 16.58), and neither are the mean ± std values in Q4. The same problem appears in Q2 with 256 queries (e.g., 66.14, 21.97). Please report the raw correct counts.
  2. Q3 vs. Table 1. Ordered trajectory on M3DRefCLIP is 77.16 / 34.06 in Q3 but 87.02 / 44.37 in Table 1. Why do these differ?
  3. AmbiRefer3D row in Q1. Its numbers (65.00 / 40.03 / 3.51) nearly match your own baseline in the first row of Table 3 (65.43 / 40.18 / 3.48). Was this row produced with the original AmbiRefer3D pipeline?
  4. Q2 matching. Please match |C0| to each backbone's own Ordered-trajectory value, and report Ordered trajectory on the same queries.

I keep my current rating until these are clarified.

We are sorry for the delayed response, as we needed additional time to rerun the evaluations and verify the corrected results. We sincerely appreciate your patience and constructive feedback.

Q1. Raw counts and metric consistency.

We sincerely appreciate the your careful identification of these inconsistencies. We confirm that the challenging subset contains exactly 220 queries, while the 256 queries were sampled from the AmbiRefer3D Mixture test set for the backbone-wise |C0| matching experiment in Q2, as previously reported. After tracing the evaluation logs, we found that the discrepancies originated from an aggregation error in our auxiliary evaluation script.

To accelerate the experiments, our auxiliary evaluation launcher divided all evaluation sets into fixed size query chunks and assigned the next pending chunk to whichever GPU became available. Because workers pulled chunks dynamically from a shared queue, faster GPUs processed more chunks, resulting in unequal worker-level sample counts. The previous collation script incorrectly averaged the normalized metrics reported by individual workers. Consequently, some reported binary rates did not correspond to integer multiples of 1/N1/N over the complete evaluation set.

We have corrected this issue by collecting deduplicated per-query outcomes and computing the metrics as ∑gcg/∑gng\sum_g c_g / \sum_g n_g. We are rerunning the complete evaluation for all trained models and will replace all affected entries in the revised manuscript. The current updated results for the Backbone-wise |C0| matching experiment are reported in our response to Q4 below.

Q2. Discrepancy between Q3 and Table 1.

The performance difference arises from the different amounts of training data used in the two experiments. As described in our previous Q3 response, to isolate the effects of sequential modeling while controlling for training data size and trainable capacity, we use 8,366 independent per-round training samples for the Direct variant, while Ordered trajectory uses 3,146 ordered interaction trajectories containing the same total of 8,366 rounds. Both schemes share the same trainable modules. Since the Direct variant operates on independent per-round samples, cross-round supervision objectives requiring sequential trajectories are not applicable.

Importantly, this controlled experiment uses fewer training trajectories than the full training setting in Table 1, which explains the relatively lower performance of Ordered trajectory. ** By controlling training data size, trainable modules, and the applicable supervision losses, these results provide strong evidence for the effectiveness and importance of modeling sequential dependencies.

Q3. Clarification of the AmbiRefer3D baseline.

When the additional supervision losses and trajectory-aware adapter are removed,our framework becomes largely consistent with the original AmbiRefer3D framework. The first row of Table 3 follows this configuration, retaining the original grounding and interaction losses, with only minor modifications to the disambiguation module for candidate retention. Therefore, its performance is naturally close to that of the original AmbiRefer3D.

Q4. Backbone-wise |C0| matching.

Thank you for the clarification. We mistakenly interpreted the requirement as enforcing the same |C0| across all Direct backbones, instead of matching Direct to its corresponding Ordered trajectory within each backbone.

We have now conducted this comparison on the same AmbiRefer3D Mixture test set used in Table 1. For each backbone, we computed the |C0| of Ordered trajectory and then adjusted the filtering threshold of the corresponding Direct model to match this value. The original |C0| for both Ordered trajectory and Direct are consistent with those reported in Q2 of our previous response. The updated results, computed using the corrected aggregation procedure described in Q1, are shown below:

BackboneOrigional C0Matched C0Updated(Q1) Origional Direct TS / AccUpdated(Q1) Count-matched Direct TS / AccUpdated(Q1) Ordered Trajectory TS / Acc
M3DRefCLIP5.565.8858.35/ 38.1562.90 / 38.5885.40/43.57
BUTD-DETR9.219.9062.03 / 26.3765.61 /23.3083.35/26.73
3D-VisTA2.633.8475.29 / 30.5685.51 / 36.7895.77/38.37
TSP3D12.078.3381.97/28.5070.12 / 22.4391.04/29.34

After fully correcting the aggregation bug, our core conclusions remain unchanged. We sincerely appreciate the valuable suggestions raised during the discussion, and we will incorporate the additional experiments conducted in response to these suggestions into the revised manuscript. And we will continue updating the revised manuscript with the remaining results once the reruns are completed.

We hope these further clarifications address your remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.