- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper uses a coding agent (Claude Code with Opus 5.0) to automatically write and refine "harnesses", programs around a frozen VLM (Qwen-3.6-27B) that decide which frames to decode under a per-question frame budget. Harnesses can call a video API (each frame counted against the budget), a free transcript API (ASR), and the VLM. Candidates are searched with MCTS plus multi-fidelity validation on a 1,000-question validation set, and the final system routes each question to one harness per budget tier (mixture-of-harness). The paper reports results on LongVideoBench, Video-MME, LVBench, Video-Holmes, Video-MMMU and MMVU, claiming SOTA and up to +11.2 points over VideoSeek with the same backbone.
Critical strengths
- The multi-fidelity MCTS makes harness search much cheaper, and Table 3 supports it: 7.8K vs 40.0K validation samples and 34.9 h vs 71.9 h compared with full-validation MCTS, with similar accuracy.
- The paper enforces a hard frame budget at the API level and evaluates on a broad set of benchmarks, with some real matched-budget gains (e.g., LongVideoBench 71.1 at 68 frames vs 65.6 for 64-frame uniform sampling).
Critical weaknesses
W1. Much of the gain appears to come from transcripts rather than better visual search. The transcript is central to the method: it is an input in Eq. 1, it costs no frames (§3.2), and member selection explicitly considers each harness's "reliance on the transcript for evidence localization" (§3.5). When subtitles are available to all methods, VidHarness stays within 0–3 points of 64-frame uniform sampling (LVBench 62.6–66.7 vs 63.5; Video-MME 78.7–83.2 vs 79.8), and the low tier is below the baseline on both. The larger margins appear mainly without subtitles, yet the paper never says whether the transcript API is disabled there, while VideoSeek has no transcript tool and relies purely on vision in that setting. The main claimed advantage over VideoSeek is exactly LVBench w/o sub (61.1 vs 54.7). Video-Holmes and Video-MMMU raise the same question. As it stands, I see no evidence that the evolved harnesses gather visual evidence better.
W2. Little improvement over uniform sampling at matched budgets. Against 64-frame uniform sampling: LVBench w/o sub 56.0 at 70 frames vs 55.5 (medium tier drops to 54.7 at 50 frames); Video-MME w/o sub 67.4 vs 68.0; Video-Holmes 59.4 vs 58.9 after 34.9 h of search. Larger gains only appear at ~126 frames, but no 128-frame uniform baseline is reported. With single runs at temperature 1.0, many reported differences (e.g., MoH vs Top-1 in Table 3) are within run-to-run noise.
W3. In-distribution validation and a weak reproduced baseline. About 80% of the validation set comes from Video-MME, LongVideoBench and Video-Holmes, the same benchmarks used for testing. Video-Holmes is tested on all 1,837 questions, so it is unclear where its validation questions come from. Even without overlap, the Video-Holmes results, including the +11.2 headline and all ablations, reflect in-distribution tuning. Moreover, the +11.2 is measured against a VideoSeek reproduction that is far below uniform sampling on Video-Holmes (49.6 vs 58.9) and slightly below it in both with-subtitle settings; against uniform sampling, the gain is +1.9 at about twice the frames.
W4. The discovered harnesses are not shown. The paper presents no final or example harness, no account of what changed across iterations, and no code release. The contribution thus reduces to a search procedure driven by a proprietary coding agent, and it is hard to tell whether it found a better visual strategy or a better way of using transcripts. "Cost-efficient" is also measured only in frames, without VLM calls, tokens or latency, which VideoSeek does report. Given that LVBench videos average about an hour, I would also be careful about long-video claims at roughly one frame every 30–60 seconds.
Questions for authors
- Is the transcript API available in the w/o-sub settings and on Video-Holmes, Video-MMMU and MMVU? Did the VideoSeek reproduction get the same transcripts?
- Could you add uniform-sampling baselines at 16/32/128 frames?
- Where do the Video-Holmes validation questions come from, and how was non-overlap verified?
- Could you show the selected harnesses and describe their frame allocation and transcript usage?
- Could you report variance over multiple runs, plus tokens and latency per question?
3 comments
Response to Reviewer pyHB [Part 2]
Q1: Transcript availability across settings and benchmarks. In the without-subtitle settings on Video-MME and LVBench, no ASR transcript is available, so the transcript API returns nothing and the harness operates on visual frames only. For Video-Holmes, Video-MMMU, and LVBench with subtitles, transcripts are available and provided to our harnesses. MMVU does not include transcripts, so the harness again operates on visual frames only. For the reproduced VideoSeek, we provide the same transcripts under identical availability conditions to ensure a fair comparison.
Q2: Uniform-sampling baseline at 128 frames. We provide the Qwen-3.6-27B uniform-sampling results at frames below, alongside VidHarness xhigh at frames.
| Method | #Frames | LongVideoBench | VideoMME (w/o sub) | VideoMME (w/ sub) | LVBench (w/o sub) | LVBench (w/ sub) |
|---|---|---|---|---|---|---|
| Qwen-3.6-27B | 128 | 70.7 | 71.2 | 82.7 | 59.5 | 67.4 |
| VidHarness (xhigh) | 126 | 72.0 | 73.1 | 83.2 | 61.1 | 66.7 |
VidHarness xhigh outperforms the -frame uniform-sampling baseline on four of the five settings, with the LVBench with-subtitle result ( versus ) being the only exception.
Q3: Video-Holmes validation set. Video-Holmes officially provides separate train and test splits. We use the official test split for evaluation and the official train split as the source of our validation candidates, so there is no overlap between the two. The official splits are publicly available in the HuggingFace repository TencentARC/Video-Holmes, and reviewers can verify the split definitions there directly.
Q5: Variance over multiple runs and token/latency measurements. Given the limited time window for the rebuttal, we are unable to run the additional repeated experiments and collect token and latency measurements in time, but we agree these are important controls and commit to including them in the revision.
Thank you for clarifying transcript availability and data separation, and for adding the 128-frame baseline. These address several concerns, but the mostly modest matched-budget gains remain difficult to assess without repeated runs and complete inference-cost measurements, so I will maintain my rating of 2
Response to Reviewer pyHB [Part 1]
We sincerely thank the reviewer for the detailed and thoughtful review. We are encouraged that the reviewer finds the multi-fidelity MCTS well-supported by Table 3 and acknowledges the matched-budget gains on LongVideoBench. Below we address each concern.
W1: Visual search ability versus transcript reliance. We would like to clarify the experimental setup, which we believe addresses the reviewer's concern. In the without-subtitle settings, no ASR transcript is available to any method, including ours. The VLM therefore has to rely entirely on visual content to answer questions. The notable gains in these settings, specifically 73.1% versus 68.0% on Video-MME without subtitles and 61.1% versus 55.5% on LVBench without subtitles, directly demonstrate that the evolved harnesses improve visual evidence gathering over uniform sampling, independent of any transcript. All methods, including VideoSeek and the uniform-sampling baseline, operate under identical transcript availability in each setting, so the comparison is controlled.
When subtitles are provided, they benefit all methods simultaneously. Because subtitle information resolves many questions directly, accuracy improves and approaches an upper bound, which naturally shrinks the gaps between approaches. The narrower margins in the with-subtitle settings therefore reflect saturation from the additional signal rather than any absence of visual search capability. The existence of two settings, one with and one without subtitles, is precisely what allows the community to measure the contribution of each modality; we believe the with-subtitle results should be read in this light rather than as evidence against our method's visual search ability.
W2: Comparison against the uniform-sampling baseline. The Qwen-3.6-27B uniform-sampling baseline is notably strong: on Video-MME without subtitles it scores 68.0%, matching or exceeding GPT-5 (67.9%); on Video-MME with subtitles it scores 79.8%, again above GPT-5 (78.1%). Critically, even VideoSeek reproduced on the same backbone falls below this baseline on Video-MME without subtitles (65.0% versus 68.0%), LVBench without subtitles (54.7% versus 55.5%), and both LVBench with-subtitle settings, confirming that this baseline is genuinely difficult to surpass. Even against this strong baseline, VidHarness achieves clear improvements: on LVBench without subtitles (61.1% at 126 frames versus 55.5% at 64 frames), on Video-MME with subtitles (83.2% at 126 frames versus 79.8% at 64 frames), on LVBench with subtitles (66.7% at 126 frames versus 63.5% at 64 frames), and on the out-of-domain MMVU (76.8% at 22 frames versus 74.4% at 50 frames). We provide the 128-frame baseline results directly in Q2 below, which shows that VidHarness xhigh outperforms matched-budget uniform sampling on four of the five benchmark settings.
W3: In-distribution validation and reproduced VideoSeek baseline. Regarding data non-overlap, our construction procedure ensures strict separation at both the question and the video level, as detailed in Q3 below. To test whether the gains generalize beyond the validation sources, we additionally evaluate on the two out-of-domain benchmarks Video-MMMU and MMVU (Table 4), which contain knowledge-intensive questions over lecture and educational videos that share no domain with the validation set. VidHarness low achieves 85.8% on Video-MMMU with only 20 frames, matching the 64-frame uniform-sampling baseline (85.3%) while using less than one-third of its frames, and 76.8% on MMVU with 22 frames, outperforming the 50-frame baseline (74.4%) by 2.4 points. These out-of-domain results show that the evolved harnesses capture reusable evidence-gathering strategies rather than patterns specific to the validation sources.
Regarding the reproduced VideoSeek: we note that VideoSeek reproduced on Qwen-3.6-27B (49.6% on Video-Holmes) falls below uniform sampling (58.9%). This suggests that VideoSeek's design is tightly coupled to GPT-5's specific instruction-following and reasoning characteristics and does not transfer cleanly to a different backbone, which is a known challenge for hand-crafted agentic pipelines and one of the motivations for automated harness evolution. We reproduce VideoSeek faithfully following its official code and prompt design, and we believe this result reflects a backbone transfer limitation of VideoSeek rather than an issue with our evaluation setup.
W4: Code and final harnesses. We upload the code to the supplementary materials; please see the Source Code section (Supplementary to All Reviewers) for a full description of all files and evolved harnesses. Regarding token cost and latency, please see the Inference Cost section (Supplementary to All Reviewers) for per-benchmark measurements and the justification for using frame count as the primary cost proxy.