- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes Parallel Tube Decoding (PTD) for MLLM-based spatio-temporal video grounding on Qwen3-VL-4B. After predicting the temporal interval as one block, it creates a frame anchor for every frame in that interval and fills all per-frame box blocks in a single forward pass. Decoupled Block Attention (DBA) lets each block see the video-query prefix but no other block. Training combines LocateAnything’s dual NTP/MTP formulation with GRPO using tIoU and GIoU − L1 rewards.
Critical strengths
S1. Decoding latency is independent of tube length, with no accuracy loss. From Fig. 2a, PTD stays at about 0.33–0.40 s from 8 to 64 frames, while Sequential Block Decoding (SBD) grows from about 0.7 s to 6.2 s. In Table 1, PTD exceeds SBD on all eight grounding metrics. This makes dense tube output practical inside a unified MLLM, and it is the paper’s most consequential result.
S2. SOTA-level results with a small, unified model. With a 4B backbone and no external detector or spatial decoder, the full model reaches VidSTG mvIoU of 38.3/32.4 (Bridge-STG: 37.2/31.3) and gains +1.5/+1.1 mvIoU on HC-STVG v1/v2. It also transfers zero-shot to temporal grounding, ReXTime, and referring VOS.
Critical weaknesses
W1. The new component is narrow, and the paper credits it with more than it does.
- Most components are prior work:
- quantized localization tokens (Pix2Seq, PaliGemma);
- box-as-one-block decoding with dual NTP/MTP training (LocateAnything, arXiv:2605.27365);
- BPS (reported by LocateAnything, uncredited in Sec. 2.3);
- GRPO with localization rewards for STVG (STVG-o1, STVG-R1; acknowledged in Sec. 2.4).
- Three of the four strategies in Contribution 1 are therefore not new. What is new is removing cross-block attention and creating frame anchors from the predicted interval. Predicting per-frame boxes in parallel is itself the standard non-MLLM STVG design, e.g., TubeDETR’s one time query per frame.
- Single-round decoding does not require DBA, yet the paper credits it. Sec. 2.2 attributes one-round decoding to DBA replacing SBD’s “causal attention across blocks.” But one round follows from every block’s input (anchor plus <mask> tokens) being known in advance. Bidirectional or causal attention across blocks over the same inputs would also finish in 1 + 1 rounds. The paper’s own within-block design shows this: six tokens are predicted jointly in one round from an anchor and five masks. Full independence between blocks is a design choice, and it is never compared against cross-block attention.
- Most of the 79× speedup comes from prior steps. It decomposes exactly from Table 1:
- 31.6 → 9.1 s (3.5×, quantization);
- 9.1 → 1.0 s (9.1×, block decoding);
- 1.0 → 0.4 s (2.5×, PTD).
- PTD’s own share is 2.5×, measured on decoding alone, since TCL excludes prefill (Sec. 2.3).
- Why consequential: the abstract and Contributions 1–3 credit PTD with gains that come largely from LocateAnything and GRPO. The new delta has to stand on its own evidence, and W2 shows it does not yet.
W2. The claim that removing cross-box dependency improves accuracy is not isolated.
- The only test of DBA changes two things at once. Table 1’s SBD vs. PTD comparison (1 SFT epoch, single run) shows +1.7/+1.2 mvIoU and +0.3/+0.4 mtIoU. Per App. D.3, SBD predicts <t_i> itself inside 7-token blocks, while PTD receives anchors for every frame of the predicted interval. SBD is therefore not “the natural ablation of Decoupled Block Attention” (Sec. 3.2). The variants also emit 14.2 vs. 17.7 boxes on average (Sec. 3.2) at nearly identical mtIoU (47.2 vs. 47.5). If SBD skips frames inside its own interval, part of the vIoU gap reflects box coverage, not box quality.
- Fig. 2c shows dependence on history, not net harm. Correcting a mislocalized box improves later boxes (as we read it, by about 0.37 IoU at distance 1 and 0.27 at distance 12). That shows correct history helps. It never compares SBD with PTD on the same boxes.
- SOTA depends on GRPO, which is applied only to PTD. Without GRPO, SFT(PTD) trails prior work on mvIoU on every benchmark:
- VidSTG: 34.9/28.7 vs. Bridge-STG 37.2/31.3;
- HC-STVG v1: 42.7 vs. STVG-o1 44.1;
- HC-STVG v2: 39.2 vs. DEViL 42.5.
- GRPO trains on the model’s own outputs, which is the standard remedy for exposure bias, so it may close the PTD–SBD gap. No GRPO run of SBD is reported.
- Why consequential: the paper claims “faster and more accurate.” The speed is established; accuracy from decoupling is not.
Questions for authors
Experiments needed:
- Please report SBD with anchors given (6-token blocks, attending to previously predicted boxes) and a cross-block bidirectional variant (still 1 + 1 rounds). Do PTD’s gains hold against both? Does SBD emit a box for every frame of its predicted interval?
- Does PTD’s advantage survive when GRPO is also applied to SBD, even at reduced scale?
Clarifications:
- Table 1’s BPS does not match BPS = T/TCL computed from the Sec. 3.2 means: SBD gives 14.2/1.0 = 14.2 (8.8 reported), and Unquantized gives 12.3/31.6 = 0.39 (0.5 reported). Is BPS averaged per sample? Please also report end-to-end latency including prefill.
- HC-STVG-v2 test labels are not public. Is the 1,901-query split the validation set? Since v1 and v2 training data are combined and v2 extends v1, please confirm that no v1 test videos appear in v2 training.
Minor (no response needed):
- NTP is never defined.
- Sec. 3.2 says the variants differ only at inference, but App. D.3 shows different training formulations.
- The App. F prompts request “object reference tokens,” which are absent from Eq. 1.
9 comments
Response to Reviewer SgS9 (2/4)
3. Single-round decoding and cross-block bidirectional attention
Single-round decoding does not require DBA, yet the paper credits it. Sec. 2.2 attributes one-round decoding to DBA replacing SBD's "causal attention across blocks." But one round follows from every block's input (anchor plus
<mask>tokens) being known in advance. Bidirectional or causal attention across blocks over the same inputs would also finish in 1 + 1 rounds. The paper's own within-block design shows this: six tokens are predicted jointly in one round from an anchor and five masks. Full independence between blocks is a design choice, and it is never compared against cross-block attention.
To address this concern, we additionally train a variant with bidirectional attention across spatial blocks. Similar to PTD, it uses frame anchors from the predicted interval and six-token spatial blocks, but allows each block to attend to other spatial blocks. Both variants decode in rounds, allowing us to examine the effect of DBA on grounding accuracy under the same decoding depth. Under the 2-epoch SFT setting, PTD improves mvIoU by points over the bidirectional variant.
Table R9. Cross-block bidirectional attention vs. PTD on VidSTG (both: 2 SFT epochs).
| Variant | Decl. mvIoU | Decl. @0.3 | Decl. @0.5 | Decl. mtIoU | Inter. mvIoU | Inter. @0.3 | Inter. @0.5 | Inter. mtIoU |
|---|---|---|---|---|---|---|---|---|
| Bidirectional blocks | 34.3 | 47.6 | 32.5 | 49.9 | 28.0 | 39.5 | 24.9 | 48.2 |
| PTD (DBA) | 34.9 | 48.5 | 33.3 | 50.0 | 28.7 | 40.2 | 25.6 | 48.2 |
To further understand this difference, we analyze the attention distribution and the effect of neighboring box predictions on localization quality.
(i) Attention shifts away from video tokens. The bidirectional variant assigns 19.4% of its attention to other spatial blocks, while attending less to video tokens compared to PTD (12.9% vs. 16.7%). However, this additional cross-block attention does not improve grounding accuracy (Table R9), suggesting that attending to other spatial blocks provides little benefit over directly using the video evidence.
(ii) Localization accuracy conditioned on neighboring boxes. We further analyze predictions on the VidSTG test set, categorizing boxes as correctly localized () or incorrectly localized (). We group frames by the correctness of neighboring predictions and compare both models on the same frames, averaging the IoU within each query and then across queries. When the preceding box is correct, the bidirectional variant performs similarly to PTD (75.0 vs. 75.3 IoU; 5,260 queries). We observe a similar trend when both adjacent boxes are correct (79.5 vs. 79.2 IoU; 4,087 queries). However, when the preceding box is incorrect, the bidirectional variant performs worse than PTD (29.1 vs. 31.6 IoU; 1,776 queries), and the gap further increases when both adjacent boxes are incorrect (21.3 vs. 26.1 IoU; 826 queries).
These results suggest that cross-block attention provides little additional benefit when neighboring boxes are correctly localized, while the bidirectional variant shows larger localization errors on frames with incorrect neighboring predictions. DBA removes these cross-block dependencies, keeping spatial predictions focused on the shared video evidence rather than other spatial blocks. Further, the independent spatial blocks can be decoded together or in smaller chunks when memory is limited.
Response to Reviewer SgS9 (3/4)
4. Novelty and attribution to prior work
Most components are prior work:
- quantized localization tokens (Pix2Seq, PaliGemma);
- box-as-one-block decoding with dual NTP/MTP training (LocateAnything, arXiv:2605.27365);
- BPS (reported by LocateAnything, uncredited in Sec. 2.3);
- GRPO with localization rewards for STVG (STVG-o1, STVG-R1; acknowledged in Sec. 2.4).
Three of the four strategies in Contribution 1 are therefore not new. What is new is removing cross-block attention and creating frame anchors from the predicted interval. Predicting per-frame boxes in parallel is itself the standard non-MLLM STVG design, e.g., TubeDETR's one time query per frame.
- We agree that quantized localization tokens, block decoding, dual NTP/MTP training, and localization-aware GRPO build on prior work, as already acknowledged in the paper. Our approach is to systematically study four decoding strategies for generative STVG, progressing from standard autoregressive and quantized token decoding to sequential block decoding, and introducing PTD to remove tube-length-dependent sequential generation.
- Our key contribution is predicting spatial boxes independently, conditioned on frame anchors constructed from the predicted temporal interval, such that decoding depth does not grow with tube length. In contrast, LocateAnything retains sequential dependencies across box blocks.
- Importantly, our additional experiments demonstrate that this formulation improves accuracy beyond simply providing temporal anchors. PTD outperforms anchored SBD by mvIoU on VidSTG, and retains gains of mvIoU when GRPO is also applied to SBD.
- While TubeDETR also predicts per-frame boxes in parallel, it relies on a dedicated detection decoder. In contrast, PTD enables parallel tube generation directly through the MLLM's language decoder, without additional prediction heads or external spatial decoders. Our contribution is therefore not parallel box prediction in general, but enabling efficient, accurate, and unified generative tube decoding within an MLLM. We have also credited LocateAnything for the BPS metric in the revised manuscript (§2.3).
5. Attribution of the 79× speedup
Most of the 79× speedup comes from prior steps. It decomposes exactly from Table 1:
- 31.6 → 9.1 s (3.5×, quantization);
- 9.1 → 1.0 s (9.1×, block decoding);
- 1.0 → 0.4 s (2.5×, PTD).
PTD's own share is 2.5×, measured on decoding alone, since TCL excludes prefill (Sec. 2.3).
We agree that the 79× decoding speedup reflects the combined benefits of quantization, block decoding, and PTD. In checking the reviewer’s calculation, we identified an incorrectly reported SBD latency. The verified mean TCL is 1.6 s, giving a decoding speedup for PTD (1.6 s to 0.4 s). We also checked the separately computed per-query BPS and grounding results and confirmed their reported values.
End-to-end latency including video encoding and prefill: To establish the practical benefit, we additionally measured end-to-end latency, including video preprocessing, encoding, and prefill. PTD reduces total latency from 4.3 s to 3.1 s, a speedup over SBD, increasing to for longer tubes (33–64 ground-truth frames). The corresponding decoding speedup increases from (1–8 frames) to (33–64 frames), as SBD's decoding time grows from 0.99 to 5.11 s while PTD's remains below 0.6 s. We have corrected the SBD latency throughout the manuscript and now explicitly distinguish PTD’s incremental gain from the cumulative speedup.
Table R10. End-to-end inference latency on VidSTG.
| Decoder | Input processing (s) | Decoding (s) | End-to-end (s) | PTD speedup (all) | PTD speedup (33–64 GT frames) |
|---|---|---|---|---|---|
| Unquantized NTP | 2.74 | 31.6 | 34.4 | 11.0× | 19.2× (94.4 vs. 4.9 s) |
| Quantized NTP | 2.75 | 9.1 | 11.9 | 3.8× | 7.7× (37.6 vs. 4.9 s) |
| SBD | 2.73 | 1.6 | 4.3 | 1.4× | 1.9× (9.5 vs. 4.9 s) |
| PTD (ours) | 2.74 | 0.4 | 3.1 | — | — |
Note: Same setting as Table 1 (2 FPS, up to 64 frames, batch size 1, single MI210 GPU).
Response to Reviewer SgS9 (1/4)
1. SBD with given frame anchors and box coverage
(a) SBD with given anchors (6-token blocks, causal across boxes).
Through Table 1, our intention was to present a progressive study of four decoding strategies for generative STVG. Following the MTP formulation in images, we first formulate Sequential Block Decoding (SBD) as a direct extension to video, where each time-indexed box is predicted as a block. We then extend this formulation to PTD, where temporal anchors are constructed from the predicted interval to enable parallel spatial decoding (temporal anchoring is part of the proposed PTD design).
To address the concern and ablate the effect of temporal anchoring separately from Decoupled Block Attention (DBA), we additionally train an anchored SBD variant. Similar to PTD, it first predicts the temporal interval and uses it to construct the frame indices for each spatial block, while retaining sequential decoding. On VidSTG, PTD still achieves a 1.6–1.9 point improvement in mvIoU over anchored SBD, showing that temporal anchoring alone does not close the gap with PTD. The remaining improvement is not explained solely by box coverage, but instead reflects improved spatial localization quality.
Table R7. SBD with given anchors vs. PTD on VidSTG (all: 1 SFT epoch).
(b) Does SBD emit a box for every frame of its predicted interval?
We analyze all VidSTG predictions and find that SBD covers every frame of its predicted interval in 98.4% of queries, while PTD ensures full coverage by design. The mean predicted interval lengths (14.2 vs. 17.7 frames) indicate that the difference in box counts primarily reflects temporal extent rather than skipped frames. Moreover, on frames where both methods predict boxes, PTD achieves higher spatial IoU (56.2 vs. 52.0) and retains a 1.6–1.9 point mvIoU advantage over anchored SBD under the same temporal anchoring procedure.
2. SOTA depends on GRPO, applied only to PTD
To address this concern, we additionally apply GRPO to SBD using the same training setting as PTD (2-epoch SFT + GRPO). As shown in Table R8 below, PTD still outperforms SBD after GRPO, with mvIoU gains of 1.9 points on declarative and 2.2 points on interrogative queries. While our SOTA results benefit from localization-aware policy optimization, applying GRPO to SBD does not close the gap with PTD.
Table R8. GRPO applied to SBD vs. PTD on VidSTG.