acceptodds
Under review as a conference paper at ICLR 2027

VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding

Abstract

Vision-language models (VLMs) can answer questions about hour-long videos, but processing every frame is prohibitively expensive, even though the evidence for a question usually spans only a few seconds. Video agents, i.e., harness programs wrapped around a frozen VLM, address this by observing the video selectively, yet existing harnesses are hand-crafted by experts through slow build-and-test cycles. We propose VidHarness, a framework that automates harness design for cost-efficient long video understanding, in which a harness proposer iteratively evolves harnesses based on execution feedback from an evolution environment. To escape the local optima of greedy refinement, we organize the evolution as Monte Carlo tree search (MCTS), and to reduce the evaluation cost, we integrate uncertainty-aware multi-fidelity validation, which screens new harnesses on a few questions and promotes only the promising ones. Since the best harness varies with the frame budget, we further introduce a mixture-of-harness that routes each question to a harness specialized for its budget. VidHarness sets new state-of-the-art results on LongVideoBench, Video-MME, and Video-Holmes, outperforms the strongest hand-crafted video agent by up to 11.2 points, and generalizes to the knowledge-intensive benchmarks Video-MMMU and MMVU with fewer than half of the frames of uniform sampling.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Vision-Language Model, Agent Harness, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pyHBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper uses a coding agent (Claude Code with Opus 5.0) to automatically write and refine "harnesses", programs around a frozen VLM (Qwen-3.6-27B) that decide which frames to decode under a per-question frame budget. Harnesses can call a video API (each frame counted against the budget), a free transcript API (ASR), and the VLM. Candidates are searched with MCTS plus multi-fidelity validation on a 1,000-question validation set, and the final system routes each question to one harness per budget tier (mixture-of-harness). The paper reports results on LongVideoBench, Video-MME, LVBench, Video-Holmes, Video-MMMU and MMVU, claiming SOTA and up to +11.2 points over VideoSeek with the same backbone.

Critical strengths
  1. The multi-fidelity MCTS makes harness search much cheaper, and Table 3 supports it: 7.8K vs 40.0K validation samples and 34.9 h vs 71.9 h compared with full-validation MCTS, with similar accuracy.
  2. The paper enforces a hard frame budget at the API level and evaluates on a broad set of benchmarks, with some real matched-budget gains (e.g., LongVideoBench 71.1 at 68 frames vs 65.6 for 64-frame uniform sampling).
Critical weaknesses

W1. Much of the gain appears to come from transcripts rather than better visual search. The transcript is central to the method: it is an input in Eq. 1, it costs no frames (§3.2), and member selection explicitly considers each harness's "reliance on the transcript for evidence localization" (§3.5). When subtitles are available to all methods, VidHarness stays within 0–3 points of 64-frame uniform sampling (LVBench 62.6–66.7 vs 63.5; Video-MME 78.7–83.2 vs 79.8), and the low tier is below the baseline on both. The larger margins appear mainly without subtitles, yet the paper never says whether the transcript API is disabled there, while VideoSeek has no transcript tool and relies purely on vision in that setting. The main claimed advantage over VideoSeek is exactly LVBench w/o sub (61.1 vs 54.7). Video-Holmes and Video-MMMU raise the same question. As it stands, I see no evidence that the evolved harnesses gather visual evidence better.

W2. Little improvement over uniform sampling at matched budgets. Against 64-frame uniform sampling: LVBench w/o sub 56.0 at 70 frames vs 55.5 (medium tier drops to 54.7 at 50 frames); Video-MME w/o sub 67.4 vs 68.0; Video-Holmes 59.4 vs 58.9 after 34.9 h of search. Larger gains only appear at ~126 frames, but no 128-frame uniform baseline is reported. With single runs at temperature 1.0, many reported differences (e.g., MoH vs Top-1 in Table 3) are within run-to-run noise.

W3. In-distribution validation and a weak reproduced baseline. About 80% of the validation set comes from Video-MME, LongVideoBench and Video-Holmes, the same benchmarks used for testing. Video-Holmes is tested on all 1,837 questions, so it is unclear where its validation questions come from. Even without overlap, the Video-Holmes results, including the +11.2 headline and all ablations, reflect in-distribution tuning. Moreover, the +11.2 is measured against a VideoSeek reproduction that is far below uniform sampling on Video-Holmes (49.6 vs 58.9) and slightly below it in both with-subtitle settings; against uniform sampling, the gain is +1.9 at about twice the frames.

W4. The discovered harnesses are not shown. The paper presents no final or example harness, no account of what changed across iterations, and no code release. The contribution thus reduces to a search procedure driven by a proprietary coding agent, and it is hard to tell whether it found a better visual strategy or a better way of using transcripts. "Cost-efficient" is also measured only in frames, without VLM calls, tokens or latency, which VideoSeek does report. Given that LVBench videos average about an hour, I would also be careful about long-video claims at roughly one frame every 30–60 seconds.

Questions for authors
  1. Is the transcript API available in the w/o-sub settings and on Video-Holmes, Video-MMMU and MMVU? Did the VideoSeek reproduction get the same transcripts?
  2. Could you add uniform-sampling baselines at 16/32/128 frames?
  3. Where do the Video-Holmes validation questions come from, and how was non-overlap verified?
  4. Could you show the selected harnesses and describe their frame allocation and transcript usage?
  5. Could you report variance over multiple runs, plus tokens and latency per question?
3 comments
AuthorsOpenReview ↗

Response to Reviewer pyHB [Part 1]

We sincerely thank the reviewer for the detailed and thoughtful review. We are encouraged that the reviewer finds the multi-fidelity MCTS well-supported by Table 3 and acknowledges the matched-budget gains on LongVideoBench. Below we address each concern.

W1: Visual search ability versus transcript reliance. We would like to clarify the experimental setup, which we believe addresses the reviewer's concern. In the without-subtitle settings, no ASR transcript is available to any method, including ours. The VLM therefore has to rely entirely on visual content to answer questions. The notable gains in these settings, specifically 73.1%73.1\% versus 68.0%68.0\% on Video-MME without subtitles and 61.1%61.1\% versus 55.5%55.5\% on LVBench without subtitles, directly demonstrate that the evolved harnesses improve visual evidence gathering over uniform sampling, independent of any transcript. All methods, including VideoSeek and the uniform-sampling baseline, operate under identical transcript availability in each setting, so the comparison is controlled.

When subtitles are provided, they benefit all methods simultaneously. Because subtitle information resolves many questions directly, accuracy improves and approaches an upper bound, which naturally shrinks the gaps between approaches. The narrower margins in the with-subtitle settings therefore reflect saturation from the additional signal rather than any absence of visual search capability. The existence of two settings, one with and one without subtitles, is precisely what allows the community to measure the contribution of each modality; we believe the with-subtitle results should be read in this light rather than as evidence against our method's visual search ability.

W2: Comparison against the uniform-sampling baseline. The Qwen-3.6-27B uniform-sampling baseline is notably strong: on Video-MME without subtitles it scores 68.0%68.0\%, matching or exceeding GPT-5 (67.9%67.9\%); on Video-MME with subtitles it scores 79.8%79.8\%, again above GPT-5 (78.1%78.1\%). Critically, even VideoSeek reproduced on the same backbone falls below this baseline on Video-MME without subtitles (65.0%65.0\% versus 68.0%68.0\%), LVBench without subtitles (54.7%54.7\% versus 55.5%55.5\%), and both LVBench with-subtitle settings, confirming that this baseline is genuinely difficult to surpass. Even against this strong baseline, VidHarness achieves clear improvements: on LVBench without subtitles (61.1%61.1\% at 126126 frames versus 55.5%55.5\% at 6464 frames), on Video-MME with subtitles (83.2%83.2\% at 126126 frames versus 79.8%79.8\% at 6464 frames), on LVBench with subtitles (66.7%66.7\% at 126126 frames versus 63.5%63.5\% at 6464 frames), and on the out-of-domain MMVU (76.8%76.8\% at 2222 frames versus 74.4%74.4\% at 5050 frames). We provide the 128128-frame baseline results directly in Q2 below, which shows that VidHarness xhigh outperforms matched-budget uniform sampling on four of the five benchmark settings.

W3: In-distribution validation and reproduced VideoSeek baseline. Regarding data non-overlap, our construction procedure ensures strict separation at both the question and the video level, as detailed in Q3 below. To test whether the gains generalize beyond the validation sources, we additionally evaluate on the two out-of-domain benchmarks Video-MMMU and MMVU (Table 4), which contain knowledge-intensive questions over lecture and educational videos that share no domain with the validation set. VidHarness low achieves 85.8%85.8\% on Video-MMMU with only 2020 frames, matching the 6464-frame uniform-sampling baseline (85.3%85.3\%) while using less than one-third of its frames, and 76.8%76.8\% on MMVU with 2222 frames, outperforming the 5050-frame baseline (74.4%74.4\%) by 2.42.4 points. These out-of-domain results show that the evolved harnesses capture reusable evidence-gathering strategies rather than patterns specific to the validation sources.

Regarding the reproduced VideoSeek: we note that VideoSeek reproduced on Qwen-3.6-27B (49.6%49.6\% on Video-Holmes) falls below uniform sampling (58.9%58.9\%). This suggests that VideoSeek's design is tightly coupled to GPT-5's specific instruction-following and reasoning characteristics and does not transfer cleanly to a different backbone, which is a known challenge for hand-crafted agentic pipelines and one of the motivations for automated harness evolution. We reproduce VideoSeek faithfully following its official code and prompt design, and we believe this result reflects a backbone transfer limitation of VideoSeek rather than an issue with our evaluation setup.

W4: Code and final harnesses. We upload the code to the supplementary materials; please see the Source Code section (Supplementary to All Reviewers) for a full description of all files and evolved harnesses. Regarding token cost and latency, please see the Inference Cost section (Supplementary to All Reviewers) for per-benchmark measurements and the justification for using frame count as the primary cost proxy.

AuthorsOpenReview ↗

Response to Reviewer pyHB [Part 2]

Q1: Transcript availability across settings and benchmarks. In the without-subtitle settings on Video-MME and LVBench, no ASR transcript is available, so the transcript API returns nothing and the harness operates on visual frames only. For Video-Holmes, Video-MMMU, and LVBench with subtitles, transcripts are available and provided to our harnesses. MMVU does not include transcripts, so the harness again operates on visual frames only. For the reproduced VideoSeek, we provide the same transcripts under identical availability conditions to ensure a fair comparison.

Q2: Uniform-sampling baseline at 128 frames. We provide the Qwen-3.6-27B uniform-sampling results at 128128 frames below, alongside VidHarness xhigh at 126126 frames.

Method#FramesLongVideoBenchVideoMME (w/o sub)VideoMME (w/ sub)LVBench (w/o sub)LVBench (w/ sub)
Qwen-3.6-27B12870.771.282.759.567.4
VidHarness (xhigh)12672.073.183.261.166.7

VidHarness xhigh outperforms the 128128-frame uniform-sampling baseline on four of the five settings, with the LVBench with-subtitle result (66.7%66.7\% versus 67.4%67.4\%) being the only exception.

Q3: Video-Holmes validation set. Video-Holmes officially provides separate train and test splits. We use the official test split for evaluation and the official train split as the source of our validation candidates, so there is no overlap between the two. The official splits are publicly available in the HuggingFace repository TencentARC/Video-Holmes, and reviewers can verify the split definitions there directly.

Q5: Variance over multiple runs and token/latency measurements. Given the limited time window for the rebuttal, we are unable to run the additional repeated experiments and collect token and latency measurements in time, but we agree these are important controls and commit to including them in the revision.

Thank you for clarifying transcript availability and data separation, and for adding the 128-frame baseline. These address several concerns, but the mostly modest matched-budget gains remain difficult to assess without repeated runs and complete inference-cost measurements, so I will maintain my rating of 2

Reviewer_SNhsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to automatically design long-video question-answering workflows around a frozen vision-language model (VLM) under a limited frame budget. The authors use the term harness to describe an executable program that samples frames, queries transcripts, invokes the VLM, and decides when to answer. VidHarness uses a coding agent to generate and revise candidate harnesses, with execution traces and error feedback guiding subsequent designs. It explores different design branches through Monte Carlo tree search and uses multi-fidelity validation to evaluate candidates initially on a small subset of questions, then progressively evaluate promising candidates on larger subsets. At deployment, a mixture-of-harness selects a harness for one of four budget tiers according to the requested frame budget. The authors report accuracy and average frame usage on LongVideoBench, Video-MME, LVBench, Video-Holmes, and two out-of-domain benchmarks. They also compare the number of validation evaluations and evolution time across search strategies on Video-Holmes.

Critical strengths
  1. Automatically designing long-video QA workflows has practical value, and the reduction in search cost is directly supported by experiments. The framework integrates program generation, execution feedback, trace diagnosis, and tree search into an executable iterative process, providing a concrete approach to reducing the effort involved in manually designing video-agent workflows. In Table 3, multi-fidelity MCTS reduces the number of validation evaluations from 40,000 to 7,800 and evolution time from 71.9 to 34.9 hours compared with MCTS using full validation. The corresponding Video-Holmes accuracies are 59.4% and 58.4%, respectively. Under the reported experimental settings, these results support the claim of improved efficiency during workflow search.
  2. Several results are competitive with the same backbone and similar or lower average frame usage. Section 4.1 explicitly states that the authors reproduce VideoSeek using Qwen-3.6-27B. For example, on Video-MME without subtitles, VidHarness achieves 73.1% accuracy with 126 frames, compared with 65.0% with 123 frames for VideoSeek. On LVBench without subtitles, VidHarness achieves 56.0% with 70 frames, compared with 54.7% with 111.5 frames for VideoSeek. In Table 2, the low tier of VidHarness also achieves 55.5% with 21.4 frames, exceeding VideoSeek's 49.6% with 68.7 frames using the same backbone. These advantages therefore cannot simply be attributed to a stronger backbone or more frames.
Critical weaknesses
  1. The evidence that frame efficiency translates into practical inference cost efficiency remains limited. Section 3.1 explicitly formulates the task as maximizing accuracy under a frame budget, and the experiments do provide valid comparisons of frame usage and accuracy. Sections 3.1–3.2 also explicitly allow multi-turn interactions, transcript queries, and adjustable frame resolution, while stating that transcript reads do not count toward the frame budget. However, the paper does not report the actual number of VLM calls per question, visual and textual token usage, or inference latency or monetary cost. The current results therefore do not establish whether using fewer frames reduces total inference cost. This limits the scope of the cost-efficiency claim in practical deployment without invalidating the frame-budget experiments. The evolution times in Table 3 support search efficiency but do not directly address inference cost.
  2. The correspondence between some MoH results and the stability of its smaller gains remain unclear. VidHarness at the high tier is reported as 82.0 frames / 59.3% in Table 2, whereas MoH(high) is reported as 56.8 frames / 59.4% in Table 3. The former pair is identical to Top1(xhigh) in Table 3. Section 4.3 explains why MoH and top-1 have identical results at the high tier within Table 3, but it does not explain the difference between Tables 2 and 3. It is unclear whether this reflects a labeling issue or different evaluation configurations, making it difficult to assess the performance of the final deployment configuration accurately. In addition, the low-tier MoH improvement is 55.5% versus 55.2%, but the paper does not report statistics across repeated runs or empirical uncertainty estimates for this test-set difference.
Questions for authors
  1. Section 3.2 states that execution traces record model calls, frame sampling, and transcript reads. For the reported comparisons using the same backbone and similar frame counts, such as 126 versus 123 frames on Video-MME without subtitles, could these traces be used to report the actual number of model calls per question and visual and textual token usage? Could you also provide inference latency or monetary cost measured under the same execution environment? If the efficiency claim is intended to cover only frame usage and search time, please clarify this scope. These details would affect my assessment of the method's practical cost advantage.
  2. Please explain the relationship among high in Table 2 (82.0 frames / 59.3%), MoH(high) in Table 3 (56.8 frames / 59.4%), and Top1(xhigh) in Table 3 (82.0 frames / 59.3%), including the requested frame budget and selected harness for each. For the 0.3-percentage-point improvement of MoH low over top-1, are results from repeated runs or uncertainty estimates for the test-set difference available?
  3. The numerical results for VideoSeek (GPT-5) in Table 1 are shown in gray, but the caption does not explain what this denotes. Its LongVideoBench accuracy is 73.5%, exceeding VidHarness's 72.0%. I recognize that Section 4.1 already emphasizes the reproduction using the same backbone. What remains unclear is whether the statement in Section 4.2 that the method outperforms all existing baselines excludes the gray results, and which comparison scope supports the state-of-the-art claim. Clarifying this would help establish the scope of the claimed performance advantage.
2 comments
AuthorsOpenReview ↗

Response to Reviewer SNhs

We sincerely thank the reviewer for the thorough and constructive review. We are encouraged that the reviewer recognizes the practical value of automating long-video QA workflow design, finds the reduction in search cost directly supported by experiments, and acknowledges that several results are competitive with the same backbone and similar or lower average frame usage, confirming that the advantages of VidHarness cannot be attributed to a stronger backbone or more frames. Below we address each concern.

W1 & Q1: Frame efficiency and practical cost efficiency. Please see the Inference Cost section (Supplementary to All Reviewers) for per-benchmark token usage measurements and a detailed justification of why frame count is a reliable proxy for cost.

W2 & Q2: Inconsistency between Tables 2 and 3, and uncertainty estimates. We thank the reviewer for catching this inconsistency. The VidHarness high entry in Table 2 (82.0 frames / 59.3%) is an inadvertent copy of the Top-1 xhigh row from the ablation study rather than the mixture-of-harness high row; the correct entry for VidHarness high, i.e., MoH(high), should be 56.8 frames / 59.4%, consistent with Table 3. We apologize for the confusion and will correct Table 2 to ensure consistency across all tables in the revision.

For the low tier, the mixture-of-harness achieves 55.5% compared with Top-1's 55.2%. Regarding uncertainty estimates for this test-set difference: given the limited rebuttal window, we are unable to run the additional repeated evolution experiments in time, but we agree these are valuable controls and commit to including repeated-run statistics in the revision.

Q3: Gray results, and comparison with VideoSeek (GPT-5). We use gray to visually distinguish VideoSeek (GPT-5) from other baselines. A direct comparison across different backbone strengths would be misleading. The state-of-the-art claim in Sec. 4.2 therefore pertains to the same-backbone setting and explicitly excludes the gray results. We will add an explicit caption note to Table 1 to make this distinction clear in the revision.

That said, even comparing directly against VideoSeek (GPT-5), VidHarness is competitive. On Video-MME without subtitles, VidHarness xhigh achieves 73.1%73.1\% with 126126 frames, surpassing VideoSeek (GPT-5)'s 70.1%70.1\% with 60.960.9 frames. On Video-MME with subtitles, VidHarness achieves 83.2%83.2\% with 126126 frames, again exceeding VideoSeek (GPT-5)'s 81.2%81.2\% with 15.915.9 frames. These results show that VidHarness with Qwen-3.6-27B outperforms a strong proprietary system on both Video-MME settings, suggesting that the evolved harnesses contribute genuine evidence-gathering improvements beyond what backbone strength alone provides.

Thanks for your clarification and additional data. Some of my concerns have been cleaned up. I also understand that the rebuttal period is short for additional repeated-run exp.

But my main remaining concern is stiil the inference cost. The new table shows token usage for VidHarness, but not for VideoSeek, it's hard to know whether your method is better. And the claim "Img (k) accounts for the large majority of In (k) at every tier" seems too strong. For example, on Video-MME with subtitles at the low tier, only 10.2k of the 83.9k input tokens are image tokens.

I appreciate the progress made in the response, but I do not think the inference-cost question is fully resolved yet. I will therefore keep my current rating for now.

Reviewer_RwuSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VidHarness, a framework that automates video agent harness design around a frozen VLM using MCTS and multi-fidelity evaluation. Candidates are evaluated on progressively larger subsets of a development set, and selected harnesses are deployed according to the requested frame budget. The experiments report improved accuracy-frame trade-offs and reduced offline cost.

Critical strengths
  1. Practical motivation. Automating harness design could reduce substantial manual engineering effort.
  2. Multiple benchmark results and interesting ablation studies on evolution strategies.
  3. Promising efficiency results. Multi-fidelity evaluation reduces evolution time, while transfer experiments show encouraging frame savings.
Critical weaknesses
  1. The comparison with prior evolution strategies needs correction. Meta-Harness already permits archive-based, multi-candidate search rather than enforcing a single lineage. The restricted “vanilla” baseline may therefore understate existing capabilities and exaggerate the benefit of MCTS.
  2. Initialization and evolved algorithms are insufficiently described. Evolution starts from a virtual root, with initial harnesses generated by the proposer. Without the complete guidance file, initial candidates, and final harness code, the starting conditions and evolved algorithms are difficult to reproduce. Also it does not show how much gains can be achieved compared with this evolution seed.
  3. Evolution and testing split details are missing. In line 199, it is said that the valuation set QA pairs include Video-MME, LongVideoBench, and so on but it also tests these benchmarks in the experimental part. It also states that the two sets do not overlap. More details are needed here.
  4. Frame efficiency does not establish overall cost efficiency. Variable resolution, repeated VLM calls, transcript processing, and generated tokens affect actual cost. The offline evolution-time reduction is valuable, but deployment latency and token-cost are missing.
  5. Component gains need stronger controls and uncertainty estimates. Some mixture-of-harness improvements accompany increased frame usage. Repeated evolution runs, matched-cost comparisons, and confidence intervals are needed, particularly given stochastic decoding and relatively small ablation gains.
Questions for authors
  1. Does the “vanilla” baseline retain Meta-Harness’s access to archived candidates and unrestricted proposal generation? If not, how does VidHarness compare with a faithful implementation under matched search budgets?
  2. Please provide the complete guidance file, initial candidate harnesses, and final evolved programs. How much held-out improvement does evolution achieve over the initial candidates under the same setting?
  3. Which official splits were used to construct the 1,000-question development set? Is separation enforced at the question or video level, and were test results excluded from harness selection and budget-tier assignment?
  4. Can you report per-question latency, input/output tokens, VLM calls, and frame resolutions alongside accuracy? How are transcript preprocessing and retrieval costs accounted for when comparing VidHarness with the baselines?
  5. Do the MCTS, multi-fidelity, and mixture-of-harness improvements persist across repeated evolution and inference runs? Please provide uncertainty estimates and matched-cost comparisons that separate better harness selection from increased frame usage.
2 comments
AuthorsOpenReview ↗

Response to Reviewer RwuS

We sincerely thank the reviewer for the careful reading and constructive feedback. We are encouraged that the reviewer finds the problem practically motivated, acknowledges the multiple benchmark results and interesting ablation studies, and recognizes the efficiency gains from multi-fidelity evaluation and the promising generalization in the transfer experiments. Below we address each concern in detail.

W1 & Q1: Fairness of the vanilla baseline. We thank the reviewer for raising this point and would like to clarify that our vanilla baseline is a faithful reproduction of Meta-Harness. Specifically, it retains access to the full archive of all previously evaluated harnesses across all iterations, not just the most recent one, so the proposer can condition its next proposal on the complete evolution history. The description of vanilla iterative refinement as growing "a single lineage" was a simplified expression intended to contrast its topology with that of MCTS, and we will correct this in the revision to avoid any misunderstanding. All reported results for the vanilla baseline are reliable and correspond to a faithful Meta-Harness implementation under matched compute budgets, so the comparison is fair.

W2 & Q2: Code and initial candidates. Per the reviewer's suggestion, we upload the code to the supplementary materials; please see the Source Code section (Supplementary to All Reviewers) for a description of all files and evolved harnesses.

Regarding the initial candidate: initial_harness.py implements a single-pass uniform-sampling baseline (DynamicBaseline) that fits frames and transcript into the VLM context window, and its performance is exactly the Qwen-3.6-27B uniform-sampling baseline reported in Tables 1 and 2. Evolution then produces measurable gains over this seed: on LongVideoBench, VidHarness xhigh reaches 72.0%72.0\% against the seed's 65.6%65.6\% (+6.4 points); on Video-MME without subtitles, it reaches 73.1%73.1\% against 68.0%68.0\% (+5.1 points); and on Video-Holmes, it reaches 60.8%60.8\% against 58.9%58.9\% (+1.9 points). Figure 3 further shows the evolution dynamics, tracing how validation accuracy improves across iterations from the first harness to the final one, which provides a direct view of the gain attributable to evolution over the initial seed.

W3 & Q3: Validation and test split construction. As described in Sec. 3.3 and illustrated in Figure 2(a), we construct a validation set of 1,000 QA pairs from five sources: Video-MME, LongVideoBench, Video-Holmes, LongVideo-Reason, and VideoAuto-R1 (MMR-VBench). Because our test evaluation (Sec. 4.1) focuses on the challenging long-duration subsets of these benchmarks, we are careful to draw validation samples only from the complementary, shorter-duration portions and from held-out splits, and we apply two non-overlap rules at both the question and the video level.

Concretely: for Video-MME, we evaluate on the long subset at test time, so we sample validation candidates exclusively from the short and medium duration subsets. For LongVideoBench, we evaluate on the long-duration validation split at test time, so we sample validation candidates from the short and medium duration samples. For Video-Holmes, we evaluate on the official test split at test time and therefore use only the official train split as validation candidates. For LongVideo-Reason and VideoAuto-R1, we include a small number of samples whose videos do not appear in any test benchmark.

We then apply two filtering rules: (1) no QA pair in the validation set may appear in any test benchmark, and (2) no video in the validation set may appear in any test benchmark. These two rules together ensure there is no data leakage from the test sets into validation, and the validation set therefore provides an unbiased signal for harness evolution. Test results are also excluded from harness selection and budget-tier assignment. We will add these details to Sec. 3.3 in the revision.

W5 & Q5: Uncertainty estimates and matched-cost comparisons. We thank the reviewer for this important suggestion. Given the limited time window for the rebuttal, we are unable to run the additional repeated evolution experiments and matched-cost comparisons in time, but we agree these are valuable controls and commit to including them in the revision.

Reviewer_RwuSOpenReview ↗

Partially resolved

Thanks for your clarification. The response resolves some of my concerns. But the main concerns remain.

  1. Although the evolution and test sets have no overlapped videos, they do have overlapped benchmarks. The evolution set includes LongVideoBench and it achieves the largest gain on this benchmark as well. When compared with the initial harness Qwen-3.6-27B, the best configuration xhigh achieves gains with accompanied frame increase as well. The authors add comparison with 128-frame Qwen-3.6-27B baseline but the gain is small. Without multiple runs and confidence intervals, this gain is not convincing. The high setting has comparable budget to 64-frame Qwen-3.6-27B, but its gains are negligible on LVBench (w/o subtitle), and it performs even worse on VideoMME (w/o subtitle). These results cannot demonstrate its effectiveness and generalization.
  2. The ablation gains are also small, with frame budget increase. No multiple runs to ensure this small gap is convincing. Therefore, I would keep my original rating as 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Supplementary to All Reviewers

We thank all reviewers for their constructive suggestions. Several questions were raised by multiple reviewers; we address them here to avoid repetition.

Inference Cost

We adopt frame count as the primary cost proxy because visual frames are the dominant factor driving token usage and cost.

  • Each frame at 1024×10241024\times1024 pixels occupies approximately 1,0241{,}024 vision tokens; when tens or hundreds of frames are decoded per question, the visual token count dwarfs the textual component and linearly inflates the context length, the number of tokens billed per request, and the KV-cache memory footprint.
  • Raw image bytes impose a substantial data-transfer cost: a single frame is roughly 11 MB, so a request carrying 6464 frames transfers 6464 MB over the network, whereas a text-only request of equivalent semantic content transmits on the order of 1010 KB, a gap that directly translates to higher API latency and storage cost.
  • Existing high-performance serving frameworks such as vLLM and SGLang, despite supporting encoder-prefill-decoder dispatch, still process visual tokens at significantly lower throughput than text tokens, so the number of frames decoded per request remains the dominant bottleneck in end-to-end inference time.

The table below reports per-benchmark inference statistics for the low, medium, and high budget tiers of VidHarness across the five in-domain benchmarks, measuring these costs directly. All measurements are taken with the same Qwen-3.6-27B backbone under local serving. Columns: Frames average frames decoded per question; MP average megapixels decoded per question; Img (k) estimated vision tokens per question in thousands (the image token component of the input); In (k) average total input tokens per question in thousands; Out (k) average output tokens per question in thousands; Acc (%) accuracy.

The data confirm the proxy: Img (k) accounts for the large majority of In (k) at every tier, and both grow in proportion to Frames, while Out (k) remains compact.

BenchmarkTierFramesMPImg (k)In (k)Out (k)Acc (%)
LongVideoBenchlow2321.613.867.611.367.7
medium4441.324.787.114.969.1
high6862.765.4115.812.871.1
VideoMME (w/o sub)low2219.312.521.58.862.0
medium4336.322.039.512.265.9
high6757.058.481.08.167.4
VideoMME (w/ sub)low1815.710.283.96.978.7
medium3630.618.4100.09.980.3
high6655.952.493.84.982.8
LVBench (w/o sub)low2524.413.722.610.849.8
medium5047.725.442.914.354.7
high7067.063.994.611.556.0
LVBench (w/ sub)low2222.012.175.79.362.6
medium4342.221.893.312.663.5
high6966.359.9108.98.366.1

Source Code

We upload the source code to the supplementary materials. It consists of five files: (1) vid_harness.py, the main evolution loop implementing MCTS node selection, harness proposal by the coding agent, low-fidelity validation, promotion, and the final test phase (Sec. 3.4, Algorithm 1); (2) SKILL.md, the evolution guidance provided to the harness proposer as a skill file, specifying the task goal, the harness input/output protocol, the trace format, and the available APIs (Sec. 3.2); (3) initial_harness.py, the initial harness that seeds the evolution; and (4–5) final_harness_1.py, final_harness_2.py, and final_harness_3.py, which are three examples of evolved harnesses written by the harness proposer during evolution, together with the proposer's own design notes in their docstrings.

final_harness_1.py (SegmentDigestReduce) divides the video into segments, processes each with a few high-resolution frames and its local transcript slice to produce timestamped digests, and reduces all digests together with whole-video frames to produce the final answer. final_harness_2.py (DistilledTiles) performs a uniform pass and then re-renders already-decoded frames as four overlapping quadrants at higher resolution, multiplying visible detail at zero additional frame cost. final_harness_3.py (CutBudget) decodes a sparse probe grid, measures color-histogram distances between consecutive frames to locate visually dense regions, and allocates the remaining budget proportionally to visual change rather than relying on model-driven navigation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.