- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
GETR trains Qwen3-VL-8B with GRPO alone (no SFT, no tool traces) on 1,668 question–answer–evidence triplets from the LongVT-Parquet RL split. The agent receives a 256/512-frame uniform pass and two tools:
get_hint: returns up to four 30-s windows retrieved offline by matching the question against per-chunk text summaries written by Qwen3-VL-8B.crop_video: returns 8 frames from any window.
Beyond answer and format rewards, the reward includes a thresholded IoGT coverage term on the final crop and an IoU term (α = 0.2) on the first crop. Across Video-MME-Long (with subtitles), LVBench, HERBench-Lite, and CG-Bench, GETR improves the uniform-frame Qwen3-VL-8B by 3.6 points on average (4.4 on the three non-subtitle benchmarks). The paper also reports how often a crop actually reaches the annotated evidence.
Critical strengths
- Honest evidence-reach analysis. Table 2, Table 5, Fig. 4b, and the error partition in App. A.4 explicitly show how often crops reach the labeled evidence. Few papers in this area report this, and it is valuable.
- Useful negative result. Giving an untrained strong VLM a crop tool lowers accuracy (Table 1, rows "crop" and "crop + hint").
- Low supervision budget. The paper uses RL only, on 1.7k examples, and reports CIs for the duration/clue analyses (§4.4).
- The argument for IoGT over IoU for an inspecting agent is reasonable.
Critical weaknesses
W1. The key control is missing. The central claim is that rewarding where the policy looked teaches it where to look. The necessary comparison is GRPO with only R_acc + R_fmt on the same backbone, tools, hints, data, frame budgets, and steps. Without it, the gains cannot be attributed to R_iogt and R_loc. Table 4 is a pilot on a different backbone (LongVT-RL), on CG-Bench only, without intervals, and it is not a substitute. An RL baseline without tools (uniform frames only, same data) is also needed to separate "RL on this data" from "tool use".
W2. The gains may reflect protocol compliance rather than better search.
- The untrained hint+crop model leaves 72.8% of LVBench trajectories without a closed
<answer>(§4.1). - Fig. 8's before/after trajectories issue identical tool calls; the only difference is that the trained model commits to an answer.
- No crop reached the evidence for 80% of GETR's correct CG-Bench answers (§4.2).
- Hit@IoGT rises only from 7.1% to 16.8% on CG-Bench and from 2.1% to 12.5% on LVBench, and only 5.7 of the 11.3-point gain over the untrained crop policy comes from questions where a crop reached the evidence (Fig. 4b).
Against uniform frames, the +3.6 could also come from the extra crop frames. Please add (i) a uniform baseline with the same total number of frames (512 + 8 × average number of crops), and (ii) accuracy restricted to trajectories that close an answer, for all tool rows.
W3. Internal inconsistencies about what was trained. (a) §3.1–3.2 define R_loc as the IoU of the first crop with the labeled evidence, and Fig. 5's caption says GETR uses the evidence-anchored R_loc. Yet §4.3 states "Scored against the hint windows, R_loc is met almost from the first step (mean 0.99)". It presents the evidence-anchored version as a variant "trained with four hints instead of two" whose "evaluation is confounded by a training-mask error and is reported in Appendix A.3". App. A.3 contains no such evaluation. (b) §3.1 argues that hints must be returned unordered and without scores (90% vs. 8% of groups with reward variance). Yet App. A.10 ("starting from the highest-confidence candidate"), A.12 ("returns ranked windows"), Fig. 9 ("ranked by relevance"), and Table 6 ("hint windows and confidences") describe ranked, scored hints. (c) K_max = 4 in the method, but the examples show two hints. Please state exactly which reward, hint count, and hint format the reported checkpoint used, and report the evaluation of the other variant.
W4. Masking incomplete rollouts. Rollouts that do not end in exactly one closed <answer> receive neither reward nor gradient (Eq. 8). The policy is therefore never penalized for failing to answer, and the group statistics are computed over survivors only. Evaluation then adds a forced answer turn. Please report the masked fraction over training and an ablation that assigns these rollouts a zero reward instead.
W5. Train/test mismatch and limited transfer.
- Training uses 256 frames at 25,088 px; evaluation uses 512 frames at 50,176 px.
- Training videos have a median length of 3.4 min, with evidence spanning a median 7.5% of the video. On LVBench and CG-Bench the evidence spans only 0.5–0.6% (App. A.3).
The paper's own numbers show that coverage transfers poorly to long videos (final-crop recall of 11.6% on LVBench and 16.7% on CG-Bench), which undercuts the claim that GETR "learns where to look" on hour-long videos.
W6. Evaluation details.
- All multiple-choice accuracies are scored by a Qwen2.5-72B judge (App. A.7), which also extracts options from unstructured text for non-compliant baselines. Rule-based option extraction should be primary, with the judge reported separately.
- Video-MME-Long is evaluated with subtitles, where text alone reaches 60.1.
- Results are single runs.
- In Table 3, the subset intervals (about ±4) make most hint conditions statistically indistinguishable, as the authors acknowledge.
Questions for authors
- Which R_loc (hint-anchored or evidence-anchored), how many hints, and ranked or unranked hints were used for the checkpoint in Table 1? Where is the evaluation referenced as "reported in Appendix A.3"?
- What does answer-only GRPO with the same setup achieve?
- What fraction of training rollouts were masked by Eq. 8, and how did it evolve?
- How do accuracies change when MCQ answers are extracted by rule instead of by the LLM judge?