acceptodds
Withdrawn from ICLR 2027

PGL-3D: Towards Progressive Geometric Learning for 3D Visual Query Localization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D Visual Query Localization (3DVQL) retrieves the latest contiguous occurrence of a queried object in an RGB–point-cloud sequence and predicts a 9-DoF cuboid for every response frame. The query is captured independently of the search sequence, so its annotated pose may differ with how the object appears in the search frames. The benchmark baseline predicts cuboids after feature modeling, leaving their geometry unused for subsequent feature refinement. We investigate whether complete intermediate cuboids can improve query and proposal representations before final decoding. To this end, we introduce the Progressive Geometric Learning for 3DVQL (PGL-3D), a predict–select–refine framework that uses complete intermediate cuboids to guide the aggregation of search evidence and update both query and proposal representations. A shared head first predicts a complete cuboid for every proposal. Query–Tube–Memory (QTM) then selects reference observations by combining proposal association, cuboid quality, frame response, and target absence, because association confidence alone establishes neither target presence nor geometric accuracy. The center, size, and orientation of each selected cuboid define soft pooling weights over the query-conditioned proposal features of its frame. The pooled memory updates both the query and the proposal representations, and the head re-predicts from the updated features. A training-only objective, ST-D9O, supervises the complete cuboid geometry at every stage by adding boundary, signed-distance, and soft-overlap terms to parameter regression. PGL-3D achieves a mean stAP of 0.2687 ± 0.0062 on 3DVQL, compared with 0.044 reported for LaF. Component ablations support the benefits of geometry-guided feature updates beyond intermediate supervision, while stage-wise analyses show improved cuboid accuracy. Replacing the geometry objective in our PROT3D reproduction with ST-D9O further improves mAO on GSOT3D from 21.63% to 25.78%. Our code and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vql, multimodal, localization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oEi3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose PGL-3D, a progressive learning framework for the 3D Visual Query Localization task. The proposed Query–Tube–Memory (QTM) module uses 9-DoF bounding boxes to refine features. The authors also propose Spatio-Temporal Differentiable 9-DoF Operator (ST-D9O), a training-only objective to supervise the geometry. The experiments on on 3DVQL and GSOT3D demonstrate improvements over the baselines.

Critical strengths
  1. The authors provide comprehensive ablation studies and analysis on the proposed modules and components to verify the effectiveness.
  2. The proposed ST-D9O has potential in other applications, as it improves PROT3D performance on GSOT3D.
Critical weaknesses
  1. Compared with existing progressive refinement methods, the reviewer consider the novelty is limited. The overall framework progressively refines query representations using intermediate predictions, which is very similar to PRVQL in 2D. The distinction between PGL-3D and PRVQL is not sufficiently clear. Many components, such as reference selection and memory aggregation, appear applicable to both 2D and 3D localization. It is not clear which components specifically address challenges unique to this new 3D task.
  2. Both QTM pooling and ST-D9O use complete 9-DoF cuboid geometry. However, it is unclear whether using full 9-DoF geometry in both components is necessary or whether simpler geometric representations could achieve similar results. It would be better to justify the design choices and provide some analysis.
Questions for authors
  1. Based on weakness 1, what are the key differences between PGL-3D and PRVQL beyond extending 2D bounding boxes to 3D cuboids? Which components are specifically designed to address challenges unique in 3D?
  2. Based on weakness 1, could the proposed QTM refinement mechanism also be applied to 2D visual query localization? Will QTM bring performance gain in 2D progressive framework?
  3. A follow-up question on 2, could the authors evaluate PRVQL-style progressive refinement on PGL-3D? This would help isolate the benefits of the proposed QTM.
  4. Based on weakness 2, could the authors provide ablations comparing different representations (for example, center-only, center-and-size, and full 9-DoF cuboids) for QTM pooling and ST-D9O supervision? This would help clarify the contribution of different geometric components and whether full 9-DoF modeling is necessary in both modules.
  5. As the framework is progressive and work on 3D where we have both images and point cloud, the reviewer is worried about the computational cost. What are the additional training and inference costs introduced by the progressive refinement stages? Could the authors provide a fair comparison of training time, inference latency, and GPU memory usage against LAF and adapted PRVQL-3D?
7 comments

We thank the reviewer for the thoughtful questions. We answer each of them below. Tables A–H are provided in this response. We will add all new results to the revised version of the paper.

Q1: Based on weakness 1, what are the key differences between PGL-3D and PRVQL beyond extending 2D bounding boxes to 3D cuboids? Which components are specifically designed to address challenges unique in 3D?

Table A summarizes how PGL-3D differs from PRVQL.

Table A. PRVQL and PGL-3D.

AspectPRVQL (2D VQL)PGL-3D (3DVQL)
Guidance for refinementAppearance knowledge pooled inside the most confident predicted 2D boxes, and spatial knowledge from attention mapsComplete 9-DoF cuboids of the selected reference proposals
Reference selectionPer-frame classification confidence: boxes above a threshold of 0.7, then the top threeFour decoupled signals: association, cuboid quality, frame response, and target absence (Eq. 4)
Evidence aggregationRoIAlign of the video features inside the selected 2D boxesSoft pooling over sparse 3D proposals, weighted by distance in the oriented, size-normalized frame of each cuboid (Eq. 5)
Feature updateQuery features by cross-attention to the appearance knowledge, and video features by re-weighting the initial features with the spatial knowledgeQuery tokens through a gated residual anchored to Q0Q^0, and proposal features through gates of box-local compatibility, temporal distance, and reference confidence (Eq. 6)
Geometry supervisionL1 and GIoU losses on 2D boxes at every stageST-D9O at every stage: parameter, boundary, SDF, and soft-overlap terms with local-axis flip marginalization

PGL-3D shares with PRVQL the principle of refining features with intermediate predictions, and Sec. 1 credits PRVQL for it. PRVQL supervises 2D boxes at each stage and pools appearance features within the selected boxes. PGL-3D explicitly supervises the full 9-DoF cuboid at each stage and uses its predicted geometry to guide the next stage of localization. During training, ST-D9O jointly constrains the center, dimensions, and orientation of the predicted cuboid. QTM uses the center, side lengths, and orientation of each selected cuboid to define a local coordinate frame in which it weights nearby 3D proposals. The geometry learned at one stage thus determines how much each proposal contributes to the evidence used for the next prediction. Three properties of 3DVQL motivate this design.

First of all, an appearance match does not imply correct geometry in 3D. The search point cloud is sparse and covers only part of the object, so a proposal can match the query while its size or orientation is wrong. Removing the cuboid-quality signal from selection raises tAP from 0.777 to 0.831 and lowers stAP0.25\mathrm{stAP}_{0.25} from 0.215 to 0.154 (original Table 4). Selection therefore needs a separate signal of geometric accuracy.

Second, the query is captured independently, so its pose cannot be transferred to the search frames. PGL-3D predicts the rotation directly in each search frame (Sec. 3.2), and QTM defines each pooling region by an oriented cuboid in the search frame (Eq. 5). With the same selection and updaters, full cuboid pooling reaches 0.266 stAP and 0.215 stAP0.25\mathrm{stAP}_{0.25}, and isotropic pooling reaches 0.250 and 0.185 (Table D, see Q4).

Finally, the benchmark paper notes that no backpropagatable 9-DoF IoU operator is available for this task. Full 9-DoF localization calls for box-level supervision that jointly accounts for center, dimensions, and orientation. ST-D9O provides this supervision at every stage by combining parameter, boundary, signed-distance, and soft-overlap terms. Adding these groups of terms successively improves stAP0.25\mathrm{stAP}_{0.25} (original Table 16).

Table B shows how stage-wise geometry supervision affects the benefit of QTM. Averaged over three seeds, QTM adds 0.036 stAP under separate parameter regression (0.150 to 0.186) and 0.090 stAP under ST-D9O supervision (0.180 to 0.270). The larger gain under ST-D9O supports the use of joint cuboid supervision together with geometry-guided refinement. Reference selection with several signals and memory updating can also be used in 2D, and Q2 discusses this point.

Table B. Factorial of QTM and ST-D9O over three seeds (mean ± standard deviation).

ConfigurationQTMST-D9OstAP, seeds 42 / 43 / 44stAPstAP0.25\mathrm{stAP}_{0.25}, seeds 42 / 43 / 44stAP0.25\mathrm{stAP}_{0.25}
Basenono0.150 / 0.147 / 0.1530.150 ± 0.0030.058 / 0.055 / 0.0610.058 ± 0.003
+ QTMyesno0.183 / 0.189 / 0.1860.186 ± 0.0030.076 / 0.082 / 0.0790.079 ± 0.003
+ ST-D9Onoyes0.178 / 0.182 / 0.1800.180 ± 0.0020.128 / 0.133 / 0.1300.130 ± 0.003
Full PGL-3Dyesyes0.266 / 0.270 / 0.2740.270 ± 0.0040.215 / 0.203 / 0.2220.213 ± 0.010

Q2: Based on weakness 1, could the proposed QTM refinement mechanism also be applied to 2D visual query localization? Will QTM bring performance gain in 2D progressive framework?

The general principle of QTM applies to 2D VQL, and PRVQL already uses it. This principle is to re-predict from features updated by the model's own intermediate predictions. The parts of PGL-3D that our ablations find important are tied to 3D cuboids. These are the oriented pooling region and the cuboid-quality signal of QTM, and the cuboid supervision of ST-D9O. In 2D, their counterparts already exist. PRVQL pools video features inside its predicted 2D boxes with RoIAlign and supervises every stage with L1 and GIoU box losses, and IoU prediction (Jiang et al., 2018) gives a 2D quality signal. A 2D version of QTM would therefore differ from PRVQL mainly in multi-signal selection and memory updating, and its gain would need to be measured on a 2D benchmark. We did not run 2D experiments.

Within 3DVQL, the closest test of this question is to remove the 3D geometry from QTM pooling (Table D). Axis-aligned pooling keeps a center and side lengths, which is the kind of box information that the RoIAlign pooling of PRVQL uses in 2D. Removing the rotation in this way lowers stAP_0.25 from 0.215 to 0.202, and removing the side lengths as well lowers it to 0.185. The rotation step is consistent in direction and not yet resolved by a single seed (see Q4). We will add this discussion to Sec. 2 of the revised version.

Q3: A follow-up question on 2, could the authors evaluate PRVQL-style progressive refinement on PGL-3D? This would help isolate the benefits of the proposed QTM.

We kept the encoder, the 3D proposals, the prediction head, ST-D9O, and the number of stages. In place of QTM, we implemented the reference selection, feature aggregation, and query–video update of PRVQL. Both models start from the same checkpoint and use the same training data and a budget of 5,000 additional updates. They are evaluated on the same 401 contiguous 40-frame windows. This short protocol differs from the main tables in training budget and frame count, so its values are comparable only within Tables C and E.

Table C. PRVQL-style refinement and QTM inside the PGL-3D pipeline (short protocol).

RefinementstAPstAP_0.25Success (%)
PRVQL-style0.24170.115079.30
QTM0.24360.130278.30

QTM raises stAP_0.25 from 0.1150 to 0.1302, a relative gain of 13.2%. The two are at parity in stAP (0.2417 and 0.2436), and QTM is lower in Success by four queries. These are single runs under a short protocol.

Original Table 2 gives a second comparison under full training. Its Intermediate supervision variant keeps three stages and both updaters, selects references by association alone, and uses the token of the selected proposal in place of cuboid pooling (original Table 11). It is the configuration in the paper closest to confidence-guided refinement. It reaches 0.246 stAP and 0.179 stAP_0.25, and full QTM reaches 0.266 and 0.215 on the same seed. These gaps equal five and 3.6 seed standard deviations.

Q4: Based on weakness 2, could the authors provide ablations comparing different representations (for example, center-only, center-and-size, and full 9-DoF cuboids) for QTM pooling and ST-D9O supervision? This would help clarify the contribution of different geometric components and whether full 9-DoF modeling is necessary in both modules.

We varied the geometry separately in the two modules.

For QTM pooling, we fixed the selection signals, the updaters, and the training objective on seed 42, and we changed only the geometry used by the pooling distance of Eq. 5. This is Table D below. Isotropic pooling uses the center and one scale, axis-aligned pooling adds the three side lengths, and full pooling adds the rotation.

Table D. Pooling geometry with identical selection signals and updaters.

Pooling geometrystAPstAP_0.25
Center and isotropic scale0.2500.185
Center and three side lengths0.2590.202
Full cuboid0.2660.215

Full pooling exceeds isotropic pooling by 0.016 stAP and 0.030 stAP_0.25, which equal four and three seed standard deviations. The side lengths account for 0.009 stAP and 0.017 stAP_0.25 of this gain, and the rotation accounts for 0.007 and 0.013. The rotation step is consistent in direction and below two seed standard deviations, so a single run does not resolve it.

For ST-D9O, the task is evaluated with 9-DoF IoU, so every variant keeps the complete 9-DoF output and the basic parameter regression. Only the geometry used by the additional ST-D9O terms changes: the center only, the center and size, or the full center, size, and rotation. These variants follow the short protocol of Q3, and the full-geometry row is the QTM run of Table C.

Table E. Geometry used by the additional ST-D9O terms (short protocol of Q3).

Geometry in the additional termsstAPstAP_0.25Success (%)
Center only0.24050.115177.56
Center and size0.24630.121378.05
Center, size, and rotation0.24360.130278.30

The full geometry gives the highest stAP_0.25, 0.0151 above center only and 0.0089 above center and size, and the highest Success. Center and size gives a slightly higher stAP (0.2463 against 0.2436). The rotation therefore matters mainly at the strict threshold. These are single runs under a short protocol.

Original Table 16 complements this analysis under full training. With the full cuboid in all terms, stAP_0.25 rises from 0.076 with separate regression to 0.118 with the parameter terms. It then rises to 0.151, 0.169, and 0.196 as the boundary, SDF, and soft-overlap terms are added.

Reviewer_Lxg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies 3D Visual Query Localization: retrieving an object’s latest contiguous appearance in an RGB–point-cloud sequence and predicting a full 9-DoF cuboid for each response frame. The central challenge is to combine instance recognition and temporal retrieval with accurate geometry despite an independently captured query, target absence, and partial observations. PGL-3D addresses this through progressive prediction and feature refinement: QTM selects reference observations and uses their predicted cuboids to update query and proposal features. A complementary training objective, ST-D9O, supervises parameter, boundary, signed-distance, and overlap agreement. The paper presents a substantial empirical advance and extensive supporting experiments.

Critical strengths
  1. Well-motivated geometric refinement. Using intermediate cuboids to update features directly connects localization with representation learning. Separating association, geometric quality, presence, and absence is sensible, and stage-wise improvements support the approach.
  2. Strong empirical results. PGL-3D achieves 0.270±0.004 stAP across three seeds, substantially outperforming reported LaF and the PRVQL-3D adaptation. Improvements also extend to stricter localization metrics.
Critical weaknesses
  1. Restricted size expressivity. Equations (24), (28), and Table 7 imply a maximum predicted aspect ratio of approximately 2.89. Repeated QTM stages do not remove this bound, potentially excluding elongated or flat cuboids. Its practical impact is not analyzed.
  2. Full-geometry benefits are not fully isolated. Disabling cuboid pooling replaces multi-proposal aggregation with a single proposal token. Thus, Table 2 does not separate the benefit of aggregation from that of predicted size and orientation. Single-run ablations also leave uncertainty around smaller gains.
  3. Reference reliability needs further analysis. Reference precision before the two QTM rounds is only 43.8% and 46.5% under the stated presence-and-IoU criterion, yet a reference is always retained. Error propagation is not quantified. The quality target also penalizes equivalent axis flips accepted by ST-D9O.
Questions for authors
  1. Does the implementation obey the derived aspect-ratio bound? How many ground-truth cuboids fall outside the representable range, and does relaxing it improve localization?
  2. With identical selection signals and update modules, how does full cuboid pooling compare with center-only/isotropic and axis-aligned pooling?
  3. How does QTM behave with incorrect references? Would skipping low-confidence updates or using a flip-invariant quality target improve selection and localization?
  4. How do the gains from temporal geometric components vary with camera-motion magnitude?
6 comments

We thank the reviewer for the precise technical questions. We answer each of them below. Tables A–F are provided in this response. We will add all new results to the revised version of the paper.

Q1: Does the implementation obey the derived aspect-ratio bound? How many ground-truth cuboids fall outside the representable range, and does relaxing it improve localization?

The implementation obeys the bound. The initial size decoder of Eq. 24 uses a log-size radius αs=0.35\alpha_s = 0.35 around the scalar query scale sˉq\bar s_q. The two refinement steps of Eq. 28 add at most 0.12 and 0.06, because αsden=0.12\alpha_s^{den} = 0.12 and βk\beta_k is 1 and then 0.5. The total log-size radius at inference is therefore 0.53. Each side lies in [0.59,1.70] sˉq[0.59, 1.70]\,\bar s_q, and the maximum aspect ratio is exp⁡(1.06)=2.886\exp(1.06) = 2.886. Each QTM round re-decodes from the fixed proposal centers and the fixed query scale with the same head (Appendix A.5), so repeated rounds keep this range.

We counted the 7,557 GT cuboids in the response frames of the 401 test queries. For 3,537 of them (46.80%), the aspect ratio exceeds 2.886. When we also apply the per-side interval around sˉq\bar s_q, 4,901 cuboids (64.85%) fall outside the representable range. The second count is larger because a cuboid with an admissible aspect ratio can still be too large or too small relative to the query scale.

We then raised the initial log-size coefficient αs\alpha_s from 0.35 to 0.70 and kept all other settings, including the refinement scales, fixed. We changed this coefficient at inference only. The total radius becomes 0.88, and the maximum aspect ratio becomes 5.812.

Table A. Representable size range (seed 42).

Size settingMax. aspect ratioOut-of-range GT cuboidsstAPstAP_0.25Success (%)
Original (αs\alpha_s = 0.35)2.8864,901 (64.85%)0.2660.21579.30
Relaxed (αs\alpha_s = 0.70)5.8122,359 (31.22%)0.2650.21379.05

The relaxed range halves the share of out-of-range GT cuboids, and localization does not improve. All differences are within one seed standard deviation. This suggests that the size range is not the main source of error at the current accuracy. In the full model, the mean frame IoU is 0.237, with a center error of 0.158 m and a rotation error of 19.3° (original Table 3). In the revised version, we will state the range and these counts in Sec. 3.2. We will also add Table A to the Limitations (Appendix E) and note there that elongated and flat objects remain a limitation of the current size parameterization.

Q2: With identical selection signals and update modules, how does full cuboid pooling compare with center-only/isotropic and axis-aligned pooling?

We fixed the four selection signals, the Query Updater, the Tube Updater, and the training objective, and we changed only the geometry used by the pooling distance of Eq. 5. Isotropic pooling uses the predicted center and one isotropic scale. Axis-aligned pooling uses the predicted center and the three predicted side lengths and ignores the rotation. Full cuboid pooling uses the center, the side lengths, and the rotation. In all three variants, the softmax of Eq. 5 runs over the proposals of each reference frame, so the variants differ only in how geometry sets the pooling weights. The comparison therefore separates the benefit of size and orientation from that of multi-proposal aggregation. Original Table 2 compares a single proposal token with full cuboid pooling, and Table B compares geometries under the same aggregation.

Table B. Pooling geometry with identical selection signals and updaters (seed 42).

Pooling geometrystAPstAP_0.25
Center and isotropic scale0.2500.185
Center and three side lengths (axis-aligned)0.2590.202
Full cuboid (center, side lengths, rotation)0.2660.215

Full cuboid pooling exceeds isotropic pooling by 0.016 stAP and 0.030 stAP_0.25. These gains equal four and three seed standard deviations (original Table 10). Of this gain, the three side lengths account for 0.009 stAP and 0.017 stAP_0.25, and the rotation accounts for 0.007 stAP and 0.013 stAP_0.25. The rotation step is consistent in direction and below two seed standard deviations, so this single run does not resolve it.

On single-run ablations in general, the factorial of QTM and ST-D9O now uses three seeds (Table C below). Its four configurations keep the same order in every seed. In the revised version, we will also describe the smaller single-run differences in the paper as consistent in direction and not resolved.

Table C. Factorial of QTM and ST-D9O over three seeds (mean ± standard deviation).

ConfigurationQTMST-D9OstAP, seeds 42 / 43 / 44stAPstAP_0.25, seeds 42 / 43 / 44stAP_0.25
Basenono0.150 / 0.147 / 0.1530.150 ± 0.0030.058 / 0.055 / 0.0610.058 ± 0.003
+ QTMyesno0.183 / 0.189 / 0.1860.186 ± 0.0030.076 / 0.082 / 0.0790.079 ± 0.003
+ ST-D9Onoyes0.178 / 0.182 / 0.1800.180 ± 0.0020.128 / 0.133 / 0.1300.130 ± 0.003
Full PGL-3Dyesyes0.266 / 0.270 / 0.2740.270 ± 0.0040.215 / 0.203 / 0.2220.213 ± 0.010

Q3: How does QTM behave with incorrect references? Would skipping low-confidence updates or using a flip-invariant quality target improve selection and localization?

Over three seeds in the 40-frame supplementary setting of Table D, 981 QTM updates used only references that fail the joint criterion of original Table 3. Over these updates, the mean change in frame IoU at the next stage is -0.0001, and 14 of them (1.43%) lose more than 0.05. QTM is therefore close to neutral on average when all references fail, and a small share of these updates degrade noticeably. Of all references that fail the criterion, 83.75% lie in target-present frames and fail only the IoU condition. Table D below gives the full statistics.

Table D. References used in QTM updates (three seeds).

StatisticValue
References used in updates7,081
References in target-present frames92.02%
Target-present share among references that fail the joint criterion83.75%
Updates in which all references fail the joint criterion981
Mean change in frame IoU at the next stage, over these updates-0.0001
Updates among these that lose more than 0.05 frame IoU14 (1.43%)

The strongest frame is always kept so that the memory is never empty. Two gates in the updaters can limit its influence. The Query Updater adds the memory through a gated residual anchored to the original query tokens Q0Q^0. The Tube Updater gates each write by reference confidence and box-local compatibility (Appendix A.5).

We tested both suggested changes on seed 42. The skip variant bypasses both updaters in a round when all three candidates fall below the 0.7 threshold. This is the case in which only the fallback frame would be kept. Its first-round references are identical to those of the original model, so its first-round precision is unchanged. The flip-invariant variant takes the minimum of the rotation distance dRd_R of Eq. 30 over the four equivalent axis flips of GflipG_{flip}, and we retrained the model with it.

Table E. Suggested changes to reference handling (seed 42). Precision changes are relative to the original QTM.

ConfigurationRef. precision change, rounds 1 / 2 (points)stAPstAP_0.25Success (%)
Original QTM0.0 / 0.00.2660.21579.30
Skip updates when only the fallback frame remains0.0 / -0.10.2650.21479.05
Flip-invariant quality target (retrained)-0.3 / -0.40.2640.21278.80

Neither change improves selection or localization, and both differences are within seed noise. We agree with the reviewer's observation about the flips. The flips in GflipG_{flip} leave a cuboid unchanged as a point set, so the evaluation IoU and the soft IoU of Eq. 31 do not depend on them. Only dRd_R does. We keep the original target. In the revised version, we will replace the rationale in Appendix A.5 with this experiment and keep the point in the Limitations.

Reviewer_DaPdOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses 3D Visual Query Localization (3DVQL). Given an independent query observation (RGB, point cloud, and an annotated 9-DoF cuboid) and an RGB–point-cloud search sequence, the model must retrieve the latest occurrence of the object and output frame-wise 9-DoF cuboids. PGL-3D is a predict–select–refine–re-predict model. A shared head predicts cuboids together with association, cuboid-quality, response and absence signals. QueryTubeMemory (QTM) combines these four signals to choose reference frames. It pools proposal features with weights from box-normalized distances to the selected cuboids, and uses the pooled memory to update both query tokens and proposal features before re-prediction (two rounds). Training uses ST-D9O, a combined cuboid objective (parameter, corner/surface, SDF and soft-occupancy terms, with axis-flip marginalization) applied at every stage. On 3DVQL, mean stAP rises from 0.044 (LaF) to 0.270 (three seeds) and Success from 46.0% to 78.8%. Swapping ST-D9O into a reproduced PROT3D on GSOT3D improves mAO from 21.63 to 25.78.

Critical strengths
  1. The experimental reporting is unusually careful. The full model is run with three seeds and all ablations branch from one declared control run. The authors state which differences are below seed noise and report results that do not favor them (several ablations have higher tAP than the full model). The stage-wise analysis of the same model (Table 3) shows center, size and rotation errors and reference precision improving across QTM rounds. This supports the claim that intermediate geometry is actually used, not just supervised.
  2. The loss transfers. Replacing only the regression objective of a reproduced PROT3D on GSOT3D gives about +4 mAO with everything else fixed. This is the clearest evidence that one of the two contributions is useful beyond this architecture.
Critical weaknesses
  1. Most of the headline gain does not come from the claimed contributions, and the external comparison is weak. Without any QTM update ("Initial stage only", Table 15), the model already reaches 0.178 stAP, four times LaF (0.044) and well above PRVQL-3D. Most of the system-level improvement therefore comes from the encoder/proposal/decoder pipeline (point painting, temporal aggregation, voting proposals, denoising refiner, GT-near warm-up, DP decoding), which is not presented as a contribution. QTM and ST-D9O account for roughly 0.18 → 0.27. Apart from numbers copied from the benchmark paper, the only baseline is the authors' own PRVQL-3D adaptation, whose stAP_0.25 of 0.025 suggests it is far from tuned. An adapted 3D tracker such as PROT3D, which the authors have already reproduced and which also refines features with intermediate predictions, would be the natural comparison for QTM.
  2. Hyperparameters are chosen without a validation split, and ablations are single runs. 3DVQL has no validation split, and the authors report one fixed configuration. However, Tables 7–8 contain many specific values (eight ST-D9O weights, refinement scales, reference threshold 0.7, path weights, peak ratios) with no account of how they were chosen. Separately, several ablation differences in stAP_0.25 (e.g., 0.199 / 0.202 / 0.207 vs. 0.215) are within one or two seed standard deviations (0.010) and come from single runs. Combined with incremental novelty (QTM extends PRVQL/PROT3D-style progressive refinement; ST-D9O combines existing geometric terms, as the authors acknowledge), this leaves the contribution less certain than the main table suggests.
Questions for authors

Clarifications that could change my assessment:

  1. How were the hyperparameters in Tables 7–8 chosen, given that there is no validation split? Were any adjusted after looking at test-set results?
  2. Can you report PROT3D (or another strong 3D tracker) adapted to 3DVQL, and a table of base pipeline / +QTM / +ST-D9O / both, each over three seeds?
  3. Reference precision is below 50% even after two rounds (Table 3). How does QTM behave on sequences where the selected references are wrong?
  4. What is end-to-end runtime, including DINOv2, at the same number of frames for all methods? (Table 19 compares 20, 30 and 40 frames and excludes feature extraction.)

Suggestion (not required for my decision): an oracle analysis (e.g., best-proposal IoU, or stAP with the ground-truth response interval) would show whether the remaining error is in retrieval or in cuboid geometry.

8 comments

We thank the reviewer for the careful reading and for the concrete list of clarifications. We answer each question below. Tables A–I are provided in this response, and we will add all new results to the revised version of the paper.

Q1: How were the hyperparameters in Tables 7–8 chosen, given that there is no validation split? Were any adjusted after looking at test-set results?

No hyperparameter was adjusted after looking at test-set results, and no test-set result was used to select a checkpoint. Appendix C.1 states this protocol, and every reported model uses the final training snapshot (original Table 6).

We set the values in original Tables 7 and 8 from the geometric normalization of the model, the bounds on its residual updates, and the function of each module. For example, the refinement scales set the largest correction that each tanh-bounded residual step of Eq. 28 can apply.

To measure how much the results depend on these values, we perturbed 13 hyperparameter groups one at a time. These are six inference settings, four ST-D9O weight groups, and three refinement scales. Each variant changes one group and keeps all other settings of the seed-42 configuration. The inference settings were changed at inference only. Variants that change a training setting were retrained with the full training protocol. Entries are stAP / stAP_0.25.

Table A. Sensitivity to inference settings (no retraining).

Inference settingLowerDefaultHigher
Reference threshold0.6: 0.263 / 0.2100.7: 0.266 / 0.2150.8: 0.267 / 0.213
Quality weight λq\lambda_q0.5: 0.261 / 0.2070.7: 0.266 / 0.2150.9: 0.264 / 0.211
Absence weight λb\lambda_b0.6: 0.265 / 0.2120.8: 0.266 / 0.2151.0: 0.262 / 0.209
Path weights0.8x: 0.264 / 0.2111.0x: 0.266 / 0.2151.2x: 0.267 / 0.214
Peak-retention ratio0.7: 0.263 / 0.2080.8: 0.266 / 0.2150.9: 0.260 / 0.205
Response boundary ratio0.6: 0.265 / 0.2140.7: 0.266 / 0.2150.8: 0.263 / 0.211

Table B. Sensitivity to training settings (each variant retrained).

Training settingLowerDefaultHigher
ST-D9O parameter weights0.8x: 0.261 / 0.2071.0x: 0.266 / 0.2151.2x: 0.264 / 0.211
ST-D9O boundary weights0.8x: 0.263 / 0.2101.0x: 0.266 / 0.2151.2x: 0.265 / 0.213
ST-D9O SDF weight0.8x: 0.264 / 0.2111.0x: 0.266 / 0.2151.2x: 0.267 / 0.214
ST-D9O soft-overlap weight0.8x: 0.260 / 0.2051.0x: 0.266 / 0.2151.2x: 0.263 / 0.210
Center refinement scale0.28: 0.262 / 0.2090.35: 0.266 / 0.2150.42: 0.264 / 0.212
Log-size refinement scale0.096: 0.264 / 0.2110.12: 0.266 / 0.2150.144: 0.265 / 0.213
Rotation refinement scale0.20: 0.261 / 0.2080.25: 0.266 / 0.2150.30: 0.263 / 0.211

Across the 26 perturbed settings, stAP stays between 0.260 and 0.267, and stAP_0.25 stays between 0.205 and 0.214. The largest drops are 0.006 in stAP and 0.010 in stAP_0.25, which equal 1.5 and 1.0 seed standard deviations of the complete model (original Table 10). In stAP_0.25, every setting lies within one seed standard deviation of the three-seed mean (0.213 ± 0.010). Three perturbed settings exceed the default in stAP. Because this analysis uses the test set, it shows stability but cannot rule out test-set bias. In the revised version, we will therefore hold out part of the training sequences as a validation split and report the configuration selected on it. We will also add this protocol and Tables A and B to Appendix C.1.

Q2: Can you report PROT3D (or another strong 3D tracker) adapted to 3DVQL, and a table of base pipeline / +QTM / +ST-D9O / both, each over three seeds?

We agree that the encoding, proposal, and decoding pipeline contributes substantially. In the revised version, we will list this pipeline as part of the system in the contribution statement and add a base row to original Table 1. One point about original Table 15 needs clarification. The "Initial stage only" model (0.178 stAP) has no QTM update, but it is trained with the complete objective of Eq. 55 (original Table 11). It therefore already contains ST-D9O. In the same way, the Separate Regression model of original Table 4 (0.183 stAP) already contains QTM. The submission did not include the configuration with neither component.

We now report all four configurations over seeds 42, 43, and 44. Switching off QTM disables both the Query Updater and the Tube Updater. Switching off ST-D9O replaces it with the separate center, size, and rotation regression of original Table 4. All other settings follow the paper. Seed 42 of the + QTM and + ST-D9O rows reproduces Separate Regression (original Table 4) and Initial stage only (original Table 15), and the full-model row matches original Table 10.

Table C. Factorial of QTM and ST-D9O over three seeds (mean ± standard deviation).

ConfigurationQTMST-D9OstAP, seeds 42 / 43 / 44stAPstAP_0.25, seeds 42 / 43 / 44stAP_0.25
Basenono0.150 / 0.147 / 0.1530.150 ± 0.0030.058 / 0.055 / 0.0610.058 ± 0.003
+ QTMyesno0.183 / 0.189 / 0.1860.186 ± 0.0030.076 / 0.082 / 0.0790.079 ± 0.003
+ ST-D9Onoyes0.178 / 0.182 / 0.1800.180 ± 0.0020.128 / 0.133 / 0.1300.130 ± 0.003
Full PGL-3Dyesyes0.266 / 0.270 / 0.2740.270 ± 0.0040.215 / 0.203 / 0.2220.213 ± 0.010

Relative to LaF (0.044 stAP), PGL-3D gains 0.226 stAP. The base pipeline accounts for 0.106 of this gain (47%), and QTM and ST-D9O account for the remaining 0.120 (53%). On the strict metric, the two components raise stAP_0.25 from 0.058 to 0.213, a factor of 3.7.

The two components are complementary. QTM alone adds 0.036 stAP, ST-D9O alone adds 0.030, and together they add 0.120. The joint gain is 1.8 times the sum of the individual gains. On stAP_0.25, the individual gains are 0.021 and 0.072, and the joint gain is 0.155. The gain of QTM grows from 0.036 to 0.090 stAP when the intermediate cuboids are trained with ST-D9O. This agrees with the design of QTM, which pools evidence according to these cuboids. The order of the four configurations is the same in every seed. Training with the QTM stages also improves the first-stage output. H0 of the full model reaches 0.229 stAP (original Table 3), and the single-stage model reaches 0.178 (original Table 15).

Q2 (continued): PROT3D adapted to 3DVQL, and the comparison for QTM.

For the external comparison, we adapted PROT3D to 3DVQL. We trained the adapted tracker with its original regression objective and with ST-D9O over seeds 42, 43, and 44, with the same adaptation, initialization, and training budget for both.

Table D. PROT3D adapted to 3DVQL (mean ± standard deviation over three seeds).

MethodstAPstAP_0.25Success (%)
PRVQL-3D (original Table 1)0.0990.02555.4
PROT3D-3DVQL, original objective0.094 ± 0.0040.022 ± 0.00354.8 ± 1.5
PROT3D-3DVQL + ST-D9O0.123 ± 0.0050.047 ± 0.00460.7 ± 1.2
PGL-3D (original Table 1)0.270 ± 0.0040.213 ± 0.01078.8 ± 0.66

PROT3D is a point-cloud tracker designed for single-object tracking over contiguous frames. 3DVQL requires retrieval from an independent query, recovery after long absences, and localization of the latest response interval. The adapted tracker reaches 0.094 stAP, close to PRVQL-3D. ST-D9O raises its stAP by 0.029 and its stAP_0.25 from 0.022 to 0.047. The stAP gain is 0.029, 0.031, and 0.027 in the three seeds. This is a second transfer result for ST-D9O, in addition to GSOT3D (original Table 5).

On the tuning of PRVQL-3D, it shares the input, the split, and the evaluator of PGL-3D (original Table 9 and Appendix C.2). The independently adapted PROT3D gives a similar result, 0.094 stAP and 0.022 stAP_0.25 against 0.099 and 0.025 for PRVQL-3D. Both adapted baselines remain below our base pipeline (0.150 stAP and 0.058 stAP_0.25, Table C), which agrees with the large contribution of the pipeline noted above. In our experiments, stAP_0.25 also depends strongly on the cuboid objective. Replacing the regression objective with ST-D9O at least doubles stAP_0.25, both for the adapted PROT3D (0.022 to 0.047) and for our base pipeline (0.058 to 0.130).

A system-level baseline cannot isolate QTM. Within the PGL-3D pipeline, the closest configuration in the paper to confidence-guided refinement is the Intermediate supervision variant of original Table 2. It keeps three stages and both updaters, selects references by association alone, and uses the token of the selected proposal in place of cuboid pooling (original Table 11). On the same seed, QTM raises stAP from 0.246 to 0.266 and stAP_0.25 from 0.179 to 0.215. These gaps equal five and 3.6 seed standard deviations. We also replace QTM with PRVQL-style refinement inside the PGL-3D pipeline, under a short single-run protocol (Table E). The two reach similar stAP (0.2417 and 0.2436), and QTM raises stAP_0.25 from 0.1150 to 0.1302.

Table E. PRVQL-style refinement and QTM inside the PGL-3D pipeline (short protocol).

RefinementstAPstAP_0.25Success (%)
PRVQL-style0.24170.115079.30
QTM0.24360.130278.30

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.