- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose PGL-3D, a progressive learning framework for the 3D Visual Query Localization task. The proposed Query–Tube–Memory (QTM) module uses 9-DoF bounding boxes to refine features. The authors also propose Spatio-Temporal Differentiable 9-DoF Operator (ST-D9O), a training-only objective to supervise the geometry. The experiments on on 3DVQL and GSOT3D demonstrate improvements over the baselines.
Critical strengths
- The authors provide comprehensive ablation studies and analysis on the proposed modules and components to verify the effectiveness.
- The proposed ST-D9O has potential in other applications, as it improves PROT3D performance on GSOT3D.
Critical weaknesses
- Compared with existing progressive refinement methods, the reviewer consider the novelty is limited. The overall framework progressively refines query representations using intermediate predictions, which is very similar to PRVQL in 2D. The distinction between PGL-3D and PRVQL is not sufficiently clear. Many components, such as reference selection and memory aggregation, appear applicable to both 2D and 3D localization. It is not clear which components specifically address challenges unique to this new 3D task.
- Both QTM pooling and ST-D9O use complete 9-DoF cuboid geometry. However, it is unclear whether using full 9-DoF geometry in both components is necessary or whether simpler geometric representations could achieve similar results. It would be better to justify the design choices and provide some analysis.
Questions for authors
- Based on weakness 1, what are the key differences between PGL-3D and PRVQL beyond extending 2D bounding boxes to 3D cuboids? Which components are specifically designed to address challenges unique in 3D?
- Based on weakness 1, could the proposed QTM refinement mechanism also be applied to 2D visual query localization? Will QTM bring performance gain in 2D progressive framework?
- A follow-up question on 2, could the authors evaluate PRVQL-style progressive refinement on PGL-3D? This would help isolate the benefits of the proposed QTM.
- Based on weakness 2, could the authors provide ablations comparing different representations (for example, center-only, center-and-size, and full 9-DoF cuboids) for QTM pooling and ST-D9O supervision? This would help clarify the contribution of different geometric components and whether full 9-DoF modeling is necessary in both modules.
- As the framework is progressive and work on 3D where we have both images and point cloud, the reviewer is worried about the computational cost. What are the additional training and inference costs introduced by the progressive refinement stages? Could the authors provide a fair comparison of training time, inference latency, and GPU memory usage against LAF and adapted PRVQL-3D?
7 comments
Q2: Based on weakness 1, could the proposed QTM refinement mechanism also be applied to 2D visual query localization? Will QTM bring performance gain in 2D progressive framework?
The general principle of QTM applies to 2D VQL, and PRVQL already uses it. This principle is to re-predict from features updated by the model's own intermediate predictions. The parts of PGL-3D that our ablations find important are tied to 3D cuboids. These are the oriented pooling region and the cuboid-quality signal of QTM, and the cuboid supervision of ST-D9O. In 2D, their counterparts already exist. PRVQL pools video features inside its predicted 2D boxes with RoIAlign and supervises every stage with L1 and GIoU box losses, and IoU prediction (Jiang et al., 2018) gives a 2D quality signal. A 2D version of QTM would therefore differ from PRVQL mainly in multi-signal selection and memory updating, and its gain would need to be measured on a 2D benchmark. We did not run 2D experiments.
Within 3DVQL, the closest test of this question is to remove the 3D geometry from QTM pooling (Table D). Axis-aligned pooling keeps a center and side lengths, which is the kind of box information that the RoIAlign pooling of PRVQL uses in 2D. Removing the rotation in this way lowers stAP_0.25 from 0.215 to 0.202, and removing the side lengths as well lowers it to 0.185. The rotation step is consistent in direction and not yet resolved by a single seed (see Q4). We will add this discussion to Sec. 2 of the revised version.
Q3: A follow-up question on 2, could the authors evaluate PRVQL-style progressive refinement on PGL-3D? This would help isolate the benefits of the proposed QTM.
We kept the encoder, the 3D proposals, the prediction head, ST-D9O, and the number of stages. In place of QTM, we implemented the reference selection, feature aggregation, and query–video update of PRVQL. Both models start from the same checkpoint and use the same training data and a budget of 5,000 additional updates. They are evaluated on the same 401 contiguous 40-frame windows. This short protocol differs from the main tables in training budget and frame count, so its values are comparable only within Tables C and E.
Table C. PRVQL-style refinement and QTM inside the PGL-3D pipeline (short protocol).
| Refinement | stAP | stAP_0.25 | Success (%) |
|---|---|---|---|
| PRVQL-style | 0.2417 | 0.1150 | 79.30 |
| QTM | 0.2436 | 0.1302 | 78.30 |
QTM raises stAP_0.25 from 0.1150 to 0.1302, a relative gain of 13.2%. The two are at parity in stAP (0.2417 and 0.2436), and QTM is lower in Success by four queries. These are single runs under a short protocol.
Original Table 2 gives a second comparison under full training. Its Intermediate supervision variant keeps three stages and both updaters, selects references by association alone, and uses the token of the selected proposal in place of cuboid pooling (original Table 11). It is the configuration in the paper closest to confidence-guided refinement. It reaches 0.246 stAP and 0.179 stAP_0.25, and full QTM reaches 0.266 and 0.215 on the same seed. These gaps equal five and 3.6 seed standard deviations.
Q4: Based on weakness 2, could the authors provide ablations comparing different representations (for example, center-only, center-and-size, and full 9-DoF cuboids) for QTM pooling and ST-D9O supervision? This would help clarify the contribution of different geometric components and whether full 9-DoF modeling is necessary in both modules.
We varied the geometry separately in the two modules.
For QTM pooling, we fixed the selection signals, the updaters, and the training objective on seed 42, and we changed only the geometry used by the pooling distance of Eq. 5. This is Table D below. Isotropic pooling uses the center and one scale, axis-aligned pooling adds the three side lengths, and full pooling adds the rotation.
Table D. Pooling geometry with identical selection signals and updaters.
| Pooling geometry | stAP | stAP_0.25 |
|---|---|---|
| Center and isotropic scale | 0.250 | 0.185 |
| Center and three side lengths | 0.259 | 0.202 |
| Full cuboid | 0.266 | 0.215 |
Full pooling exceeds isotropic pooling by 0.016 stAP and 0.030 stAP_0.25, which equal four and three seed standard deviations. The side lengths account for 0.009 stAP and 0.017 stAP_0.25 of this gain, and the rotation accounts for 0.007 and 0.013. The rotation step is consistent in direction and below two seed standard deviations, so a single run does not resolve it.
For ST-D9O, the task is evaluated with 9-DoF IoU, so every variant keeps the complete 9-DoF output and the basic parameter regression. Only the geometry used by the additional ST-D9O terms changes: the center only, the center and size, or the full center, size, and rotation. These variants follow the short protocol of Q3, and the full-geometry row is the QTM run of Table C.
Table E. Geometry used by the additional ST-D9O terms (short protocol of Q3).
| Geometry in the additional terms | stAP | stAP_0.25 | Success (%) |
|---|---|---|---|
| Center only | 0.2405 | 0.1151 | 77.56 |
| Center and size | 0.2463 | 0.1213 | 78.05 |
| Center, size, and rotation | 0.2436 | 0.1302 | 78.30 |
The full geometry gives the highest stAP_0.25, 0.0151 above center only and 0.0089 above center and size, and the highest Success. Center and size gives a slightly higher stAP (0.2463 against 0.2436). The rotation therefore matters mainly at the strict threshold. These are single runs under a short protocol.
Original Table 16 complements this analysis under full training. With the full cuboid in all terms, stAP_0.25 rises from 0.076 with separate regression to 0.118 with the parameter terms. It then rises to 0.151, 0.169, and 0.196 as the boundary, SDF, and soft-overlap terms are added.
We thank the reviewer for the thoughtful questions. We answer each of them below. Tables A–H are provided in this response. We will add all new results to the revised version of the paper.
Table A summarizes how PGL-3D differs from PRVQL.
Table A. PRVQL and PGL-3D.
PGL-3D shares with PRVQL the principle of refining features with intermediate predictions, and Sec. 1 credits PRVQL for it. PRVQL supervises 2D boxes at each stage and pools appearance features within the selected boxes. PGL-3D explicitly supervises the full 9-DoF cuboid at each stage and uses its predicted geometry to guide the next stage of localization. During training, ST-D9O jointly constrains the center, dimensions, and orientation of the predicted cuboid. QTM uses the center, side lengths, and orientation of each selected cuboid to define a local coordinate frame in which it weights nearby 3D proposals. The geometry learned at one stage thus determines how much each proposal contributes to the evidence used for the next prediction. Three properties of 3DVQL motivate this design.
First of all, an appearance match does not imply correct geometry in 3D. The search point cloud is sparse and covers only part of the object, so a proposal can match the query while its size or orientation is wrong. Removing the cuboid-quality signal from selection raises tAP from 0.777 to 0.831 and lowers stAP0.25 from 0.215 to 0.154 (original Table 4). Selection therefore needs a separate signal of geometric accuracy.
Second, the query is captured independently, so its pose cannot be transferred to the search frames. PGL-3D predicts the rotation directly in each search frame (Sec. 3.2), and QTM defines each pooling region by an oriented cuboid in the search frame (Eq. 5). With the same selection and updaters, full cuboid pooling reaches 0.266 stAP and 0.215 stAP0.25, and isotropic pooling reaches 0.250 and 0.185 (Table D, see Q4).
Finally, the benchmark paper notes that no backpropagatable 9-DoF IoU operator is available for this task. Full 9-DoF localization calls for box-level supervision that jointly accounts for center, dimensions, and orientation. ST-D9O provides this supervision at every stage by combining parameter, boundary, signed-distance, and soft-overlap terms. Adding these groups of terms successively improves stAP0.25 (original Table 16).
Table B shows how stage-wise geometry supervision affects the benefit of QTM. Averaged over three seeds, QTM adds 0.036 stAP under separate parameter regression (0.150 to 0.186) and 0.090 stAP under ST-D9O supervision (0.180 to 0.270). The larger gain under ST-D9O supports the use of joint cuboid supervision together with geometry-guided refinement. Reference selection with several signals and memory updating can also be used in 2D, and Q2 discusses this point.
Table B. Factorial of QTM and ST-D9O over three seeds (mean ± standard deviation).