acceptodds
Under review as a conference paper at ICLR 2027

VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness

21 views

Abstract

LLM system capabilities depend not only on model parameters but also on the executable harnesses that organize information acquisition, tool use, reasoning, and memory. Recursive harness evolution can automate their design, but how it can reliably discover generalizable systems from limited, noisy, and repeatedly reused feedback remains unclear. We study this question in agentic video understanding, where rich multimodal evidence and coupled mechanisms make harness optimization challenging. We first introduce VideoHarness-BRSI, a recursive self-improvement framework that evolves video harnesses around a frozen vision–language model. Analyzing over 4,000 evolved candidates, we find it competitive with hand-engineered systems but identify three interconnected failure modes: validation overfitting, search-budget dilution, and the early-experience trap. We trace all three to a common root: the same imperfect validation evidence drives both harness revisions and the proposer’s search strategy. We therefore propose VideoHarness-SRSI, an evidence-governed framework in which every search decision rests on evidence that is transferable, sufficient, and gathered before the search commits. It restricts the proposer to content-free feedback, informs exploration with instrumented warm-up experiments, and develops mechanisms in isolated probe–race branches before factorial composition and joint refinement. Across four video-understanding benchmarks, VideoHarness-SRSI improves the equally weighted mean held-out score by 6.2% relative to VideoHarness-BRSI and reduces the mean validation–test gap by 37.9%. Reliable harness evolution thus depends on how search is organized and informed, not only on how well it optimizes validation performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Harness, Video Understanding, Recursive Evolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7R1ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks when an LLM proposer that repeatedly rewrites the code around a frozen model can be trusted to find designs that generalize. The setting is agentic video understanding, with Claude Opus 5 as the proposer and a frozen Qwen3-VL-8B as the executor. The authors first build a Meta-Harness-style baseline, called BRSI, with full access to validation content. Across several thousand candidates they identify three failure modes, namely validation overfitting, search-budget dilution and an early-experience trap. They explain all three with a single decomposition of the validation verdict into true gain, instance-specific fit, champion inflation and noise, given in Equation 1, and support it with dedicated estimators and propositions in Appendices E to H. They then propose SRSI, which changes only the evidence the loop acts on. The proposer receives content-free traces, a warm-up stage adds forced removal and intensity probes, fixed-template mechanism branches are allocated budget by a probe-and-race schedule, and selected branches are combined by factorial composition and joint refinement. As reported in Table 1, on Video-MME, LongVideoBench, OVO-Bench and MomentSeeker, SRSI raises the mean held-out score from 54.83 to 58.23 and reduces the mean gap between validation and test from 4.95 to 3.08. I thank the authors for a careful and ambitious study. I consider the diagnostic part of the paper to be of high quality and of practical value as a warning for work on harness optimization, which is why I rate its informativeness highly. However, the title and the main contribution concern the method, SRSI, and the current evidence does not yet show that the three-stage structure itself is responsible for the gains. Information isolation alone accounts for most of the effect, about half of the improvement comes with more frames and tool calls, and the evaluation rests on a single split that has been reused throughout. I therefore currently lean towards weak rejection.

Critical strengths

The diagnosis of why harness evolution overfits is well designed. The three nested feedback arms, Blindfolded, Trace and Multimodal, are each run at three validation fractions with three seeds as described in Appendix F.1, which allows instance-specific fit to be separated from noise. Table 11 shows that the noise share stays at roughly 0.14 to 0.16 across arms while the instance-specific share rises from 0.31 to 0.56. Table 12 adds direct evidence, since proposers that see question text quote validation questions in their code at 1.55 times the chance rate in the Trace arm and 1.27 times in the Multimodal arm. I found this the most valuable part of the paper. It gives the harness-optimization community a concrete reason to question full trace access on small validation sets. The theory is actually checked against data. The propositions make quantitative predictions that the authors test. Proposition F.1 predicts winners' held-out accuracy with a mean absolute error of 0.34 percentage points, and Proposition H.2 predicts a 0.54 probability of losing the frame-budget direction, compared with 6 of 12 runs observed. I also appreciate the transparency shown by a budget-matched comparison, re-scoring on unseen videos, and candid notes on residual leakage .

Critical weaknesses

It is not yet clear how much of the gain comes from SRSI's structure rather than isolation and extra inference budget. Table 19 shows that on LongVideoBench, BRSI with only Blindfolded feedback reaches 66.93 held-out against 67.3 for full SRSI. The "65.4 to 66.9%" result in Section 4.2 uses a 30-iteration BRSI without perception tools, whereas the Table 1 BRSI scores 66.6. Separately, Table 21 shows that on Video-MME SRSI uses 106.1 frames per question compared with 55.1, and 19.3 tool calls compared with 1.6. Under a frame cap the average gain falls from 3.40 to 1.66, and on OVO-Bench from 5.3 to 1.22, according to Table 22. Tool calls are not matched, and the capped selection draws from all evaluated candidates while deployment uses only the MERGE ledger. Because the central claim is that the staged organization of search matters, I think matched-budget ablations on all four benchmarks are needed. The generalization claims rest on a single partition per benchmark that is reused for diagnosis, design and evaluation. Each benchmark is split once and, as stated on page 15, the split is reused across all studies except the validation ladder of Appendix F. The held-out findings of Section 2 motivate each SRSI component, and Prompt 3 indicates the design was iterated. Appendix J.1 also notes that in the LongVideoBench and OVO-Bench runs a monitor evaluated validation-front members on the held-out split during search. OVO-Bench shows the largest test gain, +5.3, with little validation change, only +0.9. In addition, Finding 1, which claims "matching the strongest AKS variant", comes from a single run, as listed in Table 5, and its scores of 74.7 and 68.4 in Figure 2 exceed every other retrieval-only run in Appendix I.2, while the three retrieval-only runs in Table 14 average 66.8. For a paper about generalization from reused evidence, an evaluation of the frozen configuration on fresh splits or an untouched benchmark would make the conclusions considerably more convincing.

Questions for authors
  1. Could the authors add a "BRSI + isolation" row at the same iteration budget and tool set as SRSI on all four benchmarks, ideally with per-question paired tests? The LVB difference in Table 1, 67.3 versus 66.6 over three seeds, each with a standard deviation of about 1.0, seems hard to separate from seed variation.
  2. Would it be possible to report a comparison that matches both frames and tool calls, with the cap applied within the MERGE ledger that defines the deployed harness?
  3. Were the held-out monitor results in the LVB and OVO-Bench runs visible to the researchers before configurations, seeds or stopping points were fixed? Would the authors be able to evaluate the frozen SRSI configuration on new random splits or on an additional benchmark?
  4. For Finding 1, could the authors report the mean and confidence interval over all retrieval-only BRSI seeds against AKS-BLIP64, and explain how the run in Figure 2 was chosen?
  5. Appendix J.4 describes a pre-registered gate, but the deployed harness appears to be simply the best validation candidate after the merge step. What is the held-out score of the gate-selected harness?
  6. Several numbers appear inconsistent, and I would ask the authors to double-check them.
    • In Table 1, the clip_topk values on Video-MME, 66.21 and 63.40, do not appear attainable with 450 and 2,250 questions. For example, 298 out of 450 gives 66.22 and 1426 out of 2250 gives 63.38.
    • Figure 10 mentions 105 candidates for iterations 6 to 50, which I would expect to be 135 at three per iteration.
    • Section 2.2.3 says a frame probe "wins by ten" in one run, while Appendix H.6 describes the seed-4 probe losing by 12.
    • The mean verdicts in Appendix G.4 differ from Table 14.
4 comments
AuthorsOpenReview ↗

Official Reply to Reviewer 7R1Z (1)

We sincerely thank Reviewer 7R1Z for the careful and detailed review, and for recognizing the quality and practical value of our diagnostic analysis and its theory-to-data checks. We address each question below, including the new experiments you suggested.

Question 1: BRSI + isolation under the same budget and tool set:

We have added this comparison on all four benchmarks, with the same tool set as SRSI and comparable search budgets. BRSI with isolation schedules 150 candidates per run. SRSI has a nominal upper bound of 156 before the tie allowance in Appendix J.5, as in Table 1. Table 19 reuses separate runs from our feedback study (90 candidates per run, perception tools disabled), which is why its BRSI scores differ from those in Table 1. As in Table 1, we ran three random seeds per benchmark for the new row (12 isolation runs and 1,800 scheduled candidates in total). These results are also reported in our reply to Reviewer 3rMw (Weakness 1):

MethodVideo-MME ValVideo-MME TestLVB ValLVB TestOVO-Bench ValOVO-Bench TestMomentSeeker ValMomentSeeker TestAvg. TestAvg. Gap
BRSI72.2±1.866.5±1.270.2±0.366.6±1.072.9±2.167.2±1.723.8±0.719.0±1.854.834.95
BRSI (info. isolation)71.8±1.267.9±0.970.5±0.667.0±0.772.6±1.568.3±1.923.5±0.521.1±1.156.083.53
SRSI75.0±1.970.1±1.071.1±2.067.3±1.073.8±2.472.5±3.125.3±0.323.0±0.758.233.08

Information isolation alone improves the mean held-out benchmark score (+1.25 points) and narrows the validation–test gap (4.95 → 3.53), and the structured search of SRSI adds another +2.15 held-out points on average. Table 25 reports per-question paired held-out differences (SRSI minus BRSI with isolation), with a 95% interval for each benchmark obtained by resampling videos and runs.

Video-MMELVBOVO-BenchMomentSeeker
Δ Test+2.2+0.3+4.2+1.9
95% paired CI[+0.3, +4.1][−1.9, +2.5][+0.5, +7.8][+0.2, +3.6]

The per-benchmark paired intervals exclude zero on Video-MME, OVO-Bench, and MomentSeeker. On LVB the gain beyond isolation is +0.3 points, and its interval spans zero. The separate comparison of SRSI with the original BRSI also has an LVB interval spanning zero (+0.7 points, 95% Welch CI [−1.6, +3.0], Table 26). The 65.4% to 66.9% result in Section 4.2 of the original submission (Appendix L of the revision) is the effect of adding isolation to BRSI in the 30-iteration, perception-tools-disabled study (Table 19).

Revision: Table 1 (new row) and Section 4.1 report this comparison, Appendices P.1 and P.2 (Tables 24–26) give the protocol and intervals, and Appendix L states the setting of the 65.4% to 66.9% result.

Question 2: Matched comparison within the MERGE ledger:

We cap each SRSI run at the frames per question of the deployed BRSI harness, select within the MERGE ledger by validation accuracy only, and report the held-out difference from BRSI. The selected harnesses also use no more executor calls than BRSI (e.g., 2.03 vs. 2.43 per question on Video-MME):

Video-MMELVBOVO-BenchMomentSeeker
Δ Test (matched)+2.5+0.4+1.1+2.2
95% CI[+0.5, +4.6][−1.8, +2.6][−1.9, +4.0][+0.6, +3.8]

About half of the gain (1.55 of 3.40 points on average) survives this stricter match. Both methods run under the same tool set and the same per-question limits (60 tool calls and 12 executor calls; Table 7), so SRSI is not given a larger tool budget; its harnesses learn to use more of the same allowance. The search selects for validation performance under the common tool-call limit, so a harness can use additional CLIP [1] retrieval or shot-detection calls to gather evidence without invoking the executor. On Video-MME, the three deployed BRSI harnesses average 1.6 tool calls per question (Table 21). When we apply this tool-call cap together with the frame cap, no candidate in the existing SRSI MERGE ledger satisfies both constraints. The accuracy achievable under this joint constraint therefore remains unmeasured. We will report this explicitly and replace "about half of the gain survives" with the matched setting it refers to.

Revision: Section 4.1 and Table 2 (new column ΔJ\Delta_{\mathrm{J}}) report this matched comparison in place of "about half of the gain survives", and Appendix P.7 (Table 32) gives the intervals and the result of the joint tool-call cap.

AuthorsOpenReview ↗

Official Reply to Reviewer 7R1Z (2)

Question 3: Held-out monitor and fresh splits:

As stated in Appendix J.1, the monitor results were unreadable by the proposer and entered no decision; the seeds, candidate budgets, and stopping rules are fixed by the protocol in Appendix J rather than chosen per run. To further address this concern, we re-partition Video-MME and LongVideoBench with a new random seed, keep every other setting identical to Table 1 with the pipeline fully frozen, and re-run the evolution of both BRSI and SRSI on the new partition, so the new held-out questions are never seen during evolution. Due to the limited rebuttal period, we run two random seeds each. These results are also reported in our reply to Reviewer 8sJ1 (Question 1):

MethodVideo-MME ValVideo-MME TestLVB ValLVB Test
BRSI71.6±1.666.0±1.169.8±0.866.1±1.3
SRSI74.4±1.369.5±0.870.9±1.067.0±1.0

On the fresh split, SRSI improves over BRSI by +3.5 and +0.9 held-out points, in the same direction as Table 1 (+3.6 and +0.7). The corresponding 95% Welch CIs are [−1.04, +8.04] and [−4.42, +6.22], respectively.

Revision: Section 4.2 (Figure 4(b)) and Appendix P.4 (Table 28) report the fresh-split results; the held-out monitor is described in Appendix J.1.

Question 4: Finding 1:

Finding 1 is that recursive evolution can discover a retrieval harness that performs comparably to a hand-designed method on Video-MME. Figure 2 reports the only run conducted for the dedicated effectiveness study (Table 5). Starting from clip_topk, BRSI evolved a harness that reached 68.36% held-out accuracy using 38 frames per question on average. AKS-BLIP64 [2] reached 68.40% using 62 frames per question on average. To put this example in context, we also report the mean across four retrieval-only BRSI runs from simple seeds on Video-MME (the Figure 2 run and the three runs in Table 14). Mean held-out accuracy was 67.2% (95% CI [65.4%, 69.0%]), 1.2 percentage points below AKS-BLIP64. Each of the four evolved harnesses used at most 38 frames per question on average (Appendix P.8).

Revision: Finding 1 in Section 2.1 now reads "can discover" and reports the four-run mean, and Appendix P.8 gives the details.

Question 5: The admission gate:

The deployed harness is the validation-best candidate after MERGE; the gate was computed and recorded but did not decide deployment, so "pre-registered" was misleading and will be replaced by "recorded admission rule". On held-out data, the gate-selected harness is below the deployed one by 1.2, 0.5, 2.1, and 0.8 points on Video-MME, LVB, OVO-Bench, and MomentSeeker, so deploying by the gate would not have improved held-out accuracy.

Revision: Appendix J.4 now calls the gate the recorded admission rule, and Appendix P.9 reports this held-out comparison.

Number checks: We thank the reviewer for catching these.

(1) The clip_topk scores on Video-MME should read 66.22 and 63.42; no comparison is affected.

(2) Of the 135 scheduled candidates for iterations 6–50, 105 were evaluated; the rest failed the import check or the leak guard and were discarded.

(3) Section 2.2.3 was imprecise: as detailed in Appendix H.6, the seed-1 probe lost by 10 questions and closed the direction, while the seed-4 probe lost by 12 but a sibling at 18.3 frames won and kept the direction open; we will correct the sentence and Figure 3(c).

(4) Table 14 averages the first 87 candidates of each run, the largest budget all runs reach, whereas Appendix G.4 uses all candidates; we will state this in Appendix G.4.

Revision: We correct (1) in Table 1, (2) in Appendix H.6, (3) in Section 2.2.3 and Figure 3(c), and (4) in Appendix G.4.

References

[1] Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.

[2] Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, Qixiang Ye. Adaptive Keyframe Sampling for Long Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

AuthorsOpenReview ↗

Gentle follow up

We thank the reviewer again for their time and consideration. As the rebuttal period is approaching its end, we wanted to kindly follow up and ask whether our response has addressed your concerns or if there are any remaining questions we could clarify.

We would greatly appreciate your feedback and consideration of our additional results.

Reviewer_7R1ZOpenReview ↗

Response to Authors

Thank you for the thorough rebuttal. The matched comparison with BRSI plus isolation shows that structured search brings clear gains beyond isolation, which supports the central claim. The results on new splits, the transfer and cost results, and the corrections also address my other concerns. I have updated my assessment accordingly.

Reviewer_8sJ1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies recursive self-improvement of video-understanding harnesses built around a frozen vision-language model. The authors first introduce VideoHarness-BRSI and analyze more than 4,000 evolved candidates, identifying three recurring failure modes: validation overfitting, dilution of the search budget across a broad mechanism space, and excessive dependence on noisy early results. Based on these observations, the paper proposes VideoHarness-SRSI. The method limits the proposer to content-free feedback, performs an instrumented warm-up with controlled interventions, evolves different mechanism families in isolated probe–race branches, and combines promising mechanisms through a factorial design followed by joint refinement. Experiments with Qwen3-VL-8B-Instruct on Video-MME, LongVideoBench, OVO-Bench, and MomentSeeker show that SRSI improves the average held-out score over BRSI from 54.83 to 58.23 while reducing the validation–test gap. Roughly half of this improvement remains when the two methods are matched for the number of frames processed.

Critical strengths
  1. The failure-mode analysis is a substantial contribution. The paper does not merely report that recursive harness evolution overfits. It separates the problem into validation-specific fitting, insufficient evidence per mechanism, and path dependence caused by early noisy decisions. These claims are supported by a large collection of evolved candidates, targeted interventions, and a simple statistical model of validation verdicts.

  2. The proposed method follows directly from the diagnosis and is evaluated with useful controls. Information isolation, controlled warm-up interventions, isolated branches, and factorial composition each address a specific identified problem. The four-benchmark evaluation, warm-up and proposer ablations, unseen-video analysis, and frame-budget-matched comparison make the empirical case considerably stronger than a standard end-to-end benchmark table.

Critical weaknesses
  1. The independence of the final held-out evaluation is unclear. Several central design choices are derived from analyses involving held-out behavior on the same Video-MME and LongVideoBench partitions later used to evaluate SRSI. For example, held-out scores are used to establish the three failure modes, and the mechanism templates are described as being informed by the search-space ladder. Even if the automated proposer never sees held-out results, researcher-level adaptation to those results can still bias the final comparison. A fresh lockbox split or evaluation on benchmarks not used during method development would make the generalization claim much more convincing.

  2. The generality of the result is still somewhat limited. All primary experiments use one frozen executor, Qwen3-VL-8B-Instruct, and the harness is evolved separately for each benchmark. The proposer ablation is helpful, but it does not establish that the discovered mechanisms transfer across executor families or that a harness evolved on one dataset transfers to another without further validation-driven search. In addition, the inference-cost analysis is useful, but the total optimization cost—including proposer calls and evaluation of up to roughly 156 candidates per seed—is not foregrounded.

Questions for authors
  1. Were the SRSI stages, fixed axis templates, admission thresholds, and stopping rules finalized before inspecting held-out results from the partitions used in Table 1? If not, can the authors provide results on a completely fresh split or benchmark that was not involved in diagnosing BRSI or designing SRSI?

  2. How statistically reliable are the SRSI–BRSI improvements in the main table? In particular, the LongVideoBench improvement is small relative to the variation across three seeds. Per-benchmark paired tests or confidence intervals for the differences would help.

  3. Does a harness evolved on one benchmark improve performance on another benchmark or with a different frozen executor without being re-evolved? Such an experiment would clarify whether SRSI discovers transferable video-reasoning strategies or mainly improves benchmark-specific search.

  4. What is the total optimization cost of BRSI and SRSI in proposer tokens, executor calls, processed frames, and wall-clock or GPU time? The candidate counts are similar, but the staged procedure and evolved candidates may have different actual costs.

2 comments
AuthorsOpenReview ↗

Official Reply to Reviewer 8sJ1 (1)

We sincerely thank Reviewer 8sJ1 for the thoughtful review and for recognizing both our failure-mode analysis and how SRSI follows directly from it under controlled evaluation. We address each question below.

Question 1: Independence of the held-out evaluation:

The design of SRSI is not tailored to the held-out sets. Through large-scale experiments (over 4,000 evolved candidates), we reveal failure modes that are general to harness evolution, and each component of SRSI is designed to address one of them, not to fit particular held-out questions. We nevertheless agree that a different split is needed to establish reliability. We therefore re-partition Video-MME and LongVideoBench with a new random seed, keeping every other setting identical to Table 1 with the pipeline fully frozen, and re-run the evolution of both BRSI and SRSI on the new partition; the new held-out questions are never seen during evolution. Due to the limited rebuttal period, we run two random seeds each:

MethodVideo-MME ValVideo-MME TestLVB ValLVB Test
BRSI71.6±1.666.0±1.169.8±0.866.1±1.3
SRSI74.4±1.369.5±0.870.9±1.067.0±1.0

On the fresh split, SRSI improves over BRSI by +3.5 and +0.9 held-out points, in the same direction as Table 1 (+3.6 and +0.7). The corresponding 95% Welch CIs are [−1.04, +8.04] and [−4.42, +6.22], respectively.

Revision: Section 4.2 (Figure 4(b)) summarizes the fresh-split results, and Appendix P.4 (Table 28) reports the full scores and these Welch intervals.

Question 2: Statistical reliability:

The table gives the per-benchmark held-out differences (SRSI − BRSI) from Table 1 and their two-sided 95% Welch confidence intervals (three seeds per method).

Video-MMELVBOVO-BenchMomentSeeker
Δ Test+3.6+0.7+5.3+4.0
95% CI[+1.1, +6.1][−1.6, +3.0][−1.1, +11.7][+0.1, +7.9]

SRSI achieves higher mean held-out scores on all four benchmarks. The 95% intervals lie entirely above zero on Video-MME and MomentSeeker. OVO-Bench shows the largest observed gain (+5.3 points), though its interval is wide and includes zero. We agree with the reviewer that the LVB gain (+0.7 points) is small relative to the variation across seeds, and we will make this uncertainty explicit in the paper.

Revision: Appendix P.2 (Table 26) reports these intervals with p-values, and Section 4.1 now states that the gains on LongVideoBench are not statistically significant.

AuthorsOpenReview ↗

Official Reply to Reviewer 8sJ1 (2)

Question 3: Transfer across benchmarks and executors:

We evaluate the deployed SRSI harnesses on another benchmark, byte-identical and without re-evolution. Native and baseline scores are from Table 1:

Source → TargetTransferredNativeStrongest baseline
Video-MME → LVB66.867.362.07
LVB → Video-MME66.970.163.42
Video-MME → OVO-Bench57.072.555.36
LVB → OVO-Bench57.472.555.36
OVO-Bench → Video-MME52.070.163.42
OVO-Bench → LVB52.467.362.07

Between the two offline long-video benchmarks, Video-MME and LongVideoBench, transferred harnesses retain much of the native gain and beat every fixed baseline: when the task type is similar, an evolved harness transfers across datasets. OVO-Bench is different because it is a streaming benchmark, a substantially different task domain: each question is asked at a specific moment, the video ends at that moment, and the questions concern the current moment or trace back from it. Harnesses evolved on the offline benchmarks still beat the strongest baseline on OVO-Bench (57.0 and 57.4 vs. 55.36), though with much smaller gains than native ones, while harnesses evolved on OVO-Bench fall below every fixed baseline on Video-MME and LongVideoBench (52.0 and 52.4). The harnesses evolved for streaming differ substantially from those for offline video understanding, which must search the whole video for sparse evidence; for example, they read 47.8 frames and make 2.6 tool calls per question, against 106.1 and 19.3 on Video-MME (Table 21). This gap reflects a mismatch between task domains rather than a lack of generalizability: within the same task type, evolved harnesses transfer across datasets, and, as shown below, across executors.

Across executors, we evaluate the twelve deployed SRSI harnesses of Table 1 (four benchmarks × three seeds), byte-identical and without re-evolution, on InternVL3-8B [1], an executor from a different model family than Qwen3-VL-8B [2], together with the six fixed baselines. No harness code, prompt, or answer parser is changed, and the strongest baseline is picked on the held-out split itself, which favors the baselines. This experiment is also reported in our reply to Reviewer 3rMw (Weakness 3):

Method (InternVL3-8B executor)Video-MMELVBOVO-BenchMomentSeeker
Strongest fixed baseline62.2762.8557.1014.21
SRSI (transferred)66.7±2.063.5±0.562.1±4.416.4±1.1

The transferred harnesses beat the strongest fixed baseline on all four benchmarks, with gains of about 4.4, 5.0, and 2.2 points on Video-MME, OVO-Bench, and MomentSeeker, respectively, and a smaller margin on LVB (63.5% vs. 62.85%). SRSI thus discovers strategies that transfer across executors and across datasets of the same task type, while a different task type, such as streaming video, calls for re-evolution.

Revision: Section 4.2 summarizes both transfers, and Appendix P.5 (Tables 29 and 30) reports the full results.

Question 4: Total optimization cost:

Mean cost per seed, averaged over the four benchmarks:

BRSISRSI
Candidate budget (scheduled / upper bound)150156
Proposer input tokens (incl. cache reads)598M392M
Proposer output tokens4.7M3.3M
Executor calls (validation)104K98K
Frames processed4.1M4.3M
Evaluation GPU-hours4243

Under the same candidate budget, both methods spend nearly the same executor calls, frames, and GPU time, since evaluation dominates and its cost is set by the number of candidates. SRSI uses slightly more GPU time because its harnesses read more frames per question (4.3M vs. 4.1M frames in total) and make more perception tool calls such as CLIP retrieval (e.g., 19.3 vs. 1.6 per question for the deployed Video-MME harnesses), both of which add evaluation time. On the proposer side, SRSI uses about one-third fewer tokens, since each SRSI session is shorter because BRSI's proposer repeatedly calls the API to inspect validation videos itself, which increases token usage.

Revision: Section 4.2 summarizes the cost comparison, and Appendix P.6 (Table 31) reports the full breakdown.

References

[1] Jinguo Zhu, Weiyun Wang, Zhe Chen, et al. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv preprint arXiv:2504.10479, 2025.

[2] Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631, 2025.

Reviewer_3rMwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates validation overfitting, search-budget dilution, and early-experience traps in automated video-understanding harness evolution, and proposes VideoHarness-SRSI to address them through content-free feedback and structured, staged search. Experiments on four benchmarks with a frozen video model show improved held-out performance and smaller validation–test gaps over the baseline evolution framework, with gains partly retained under matched frame budgets.

Critical strengths

The systematic diagnosis of harness-evolution failures, supported by targeted ablations and frame-budget-matched comparisons, provides useful empirical insights into how feedback access and search organization affect held-out performance.

Critical weaknesses
  1. The necessity of the full structured pipeline is not sufficiently established. On LongVideoBench, information isolation alone accounts for approximately 80% of the improvement in the reported ablation, while the remaining stages add only 0.37 points under a different search budget. A matched comparison between the full method and BRSI with information isolation across all benchmarks is needed to establish the independent value of the more elaborate search procedure.

  2. Comparisons with stronger automated optimization methods remain limited. The main results compare SRSI against the authors’ BRSI framework and deliberately simple fixed harnesses, while the AKS comparison covers retrieval only. These experiments demonstrate improvement over the chosen baseline but provide limited evidence of an advance over existing modular agent-search or harness-optimization approaches. Evaluation with a single frozen executor also leaves cross-model generality unestablished.

Questions for authors
  1. How much does the full SRSI pipeline improve over BRSI with information isolation alone across all four benchmarks under matched search and inference budgets?

  2. How does SRSI compare with existing modular agent-search or harness-optimization methods?

  3. Can the evolved harnesses transfer to a different executor without re-running evolution?

3 comments
AuthorsOpenReview ↗

Official Reply to Reviewer 3rMw (1)

We sincerely thank Reviewer 3rMw for the constructive review and for recognizing our systematic diagnosis of harness-evolution failures, supported by targeted ablations and frame-budget-matched comparisons. We address each concern below.

Weakness 1: BRSI vs. SRSI in information isolation:

We thank the reviewer for suggesting a more comprehensive evaluation of information isolation. We first clarify two experimental settings that were not clearly described in the paper. (1) The comparison between SRSI and BRSI in Table 1 uses comparable search budgets, with 150 scheduled candidates per BRSI run and a nominal upper bound of 156 for SRSI before the tie allowance in Appendix J.5. (2) Table 19 uses separate BRSI runs from our feedback study (90 candidates per run, perception tools disabled) on the same 446 validation and 891 held-out LongVideoBench questions as Table 1. Its SRSI row comes directly from Table 1. To compare under the matched setting, we re-ran BRSI with information isolation on all four benchmarks with the same budget as Table 1, using three random seeds per benchmark as in Table 1 (12 isolation runs and 1,800 scheduled candidates in total). The results are shown below:

MethodVideo-MME ValVideo-MME TestLVB ValLVB TestOVO-Bench ValOVO-Bench TestMomentSeeker ValMomentSeeker TestAvg. TestAvg. Gap
BRSI72.2±1.866.5±1.270.2±0.366.6±1.072.9±2.167.2±1.723.8±0.719.0±1.854.834.95
BRSI (info. isolation)71.8±1.267.9±0.970.5±0.667.0±0.772.6±1.568.3±1.923.5±0.521.1±1.156.083.53
SRSI75.0±1.970.1±1.071.1±2.067.3±1.073.8±2.472.5±3.125.3±0.323.0±0.758.233.08

Information isolation alone improves the mean held-out benchmark score (+1.25 points) and narrows the validation–test gap (4.95 → 3.53), mitigating overfitting. Under the same budget, the structured search of SRSI adds another +2.15 held-out points, with gains of +2.2, +0.3, +4.2, and +1.9 on Video-MME, LVB, OVO-Bench, and MomentSeeker. The per-benchmark 95% paired intervals exclude zero on Video-MME, OVO-Bench, and MomentSeeker (Table 25).

On LVB, the reviewer's “about 80%” reading of Table 19 is correct for that comparison. Isolation raised BRSI from 65.43% to 66.93% (+1.50 points), about 80% of the distance from 65.43% to the Table 1 SRSI score of 67.3%. The remaining 0.37 points therefore compare harnesses evolved under different search conditions. In the main setting (Table 24), the LVB scores are 66.6% for BRSI, 67.0% for BRSI with isolation, and 67.3% for SRSI. The additional gain beyond isolation is +0.3 points (95% paired CI [−1.9, +2.5]).

Revision: We add the BRSI (info. isolation) row to Table 1 and summarize it in Section 4.1; Appendix P.1 (Tables 24 and 25) gives the protocol, budgets, and paired intervals, Appendix L explains the setting of Table 19 and its relation to the matched comparison, and Table 5 lists the new runs.

AuthorsOpenReview ↗

Official Reply to Reviewer 3rMw (2)

Weakness 2: Comparison with other automated optimization methods:

We thank the reviewer for this suggestion. First, BRSI follows Meta-Harness [1], a recent recursive harness optimizer, and adapts its design (an agentic coding proposer with full access to prior code, scores, and execution traces) to video understanding. VideoHarness-RSI [2], released shortly before our submission, adapts Meta-Harness to long-video understanding in the same way, recursively searching for executable context constructors around a frozen VLM. The comparison with BRSI in Table 1 is therefore already a comparison with prior evolution methods, namely Meta-Harness and its video instantiation VideoHarness-RSI. To further compare with other optimizers, we additionally evaluate SkillOpt [3], which optimizes a textual skill document with validation-gated edits, and AutoResearch [4], a greedy keep-or-revert code-editing loop. All methods share the same executor, proposer, seed harnesses, data splits, and search budget. Due to the limited rebuttal period, we run each new optimizer on Video-MME and LongVideoBench with two random seeds per benchmark. The BRSI and SRSI rows reproduce the three-seed results from Table 1.

MethodVideo-MME ValVideo-MME TestLVB ValLVB Test
SkillOpt69.8±1.365.9±1.166.3±0.863.0±1.3
AutoResearch70.4±1.665.2±1.569.3±1.065.6±0.9
BRSI (Meta-Harness)72.2±1.866.5±1.270.2±0.366.6±1.0
SRSI75.0±1.970.1±1.071.1±2.067.3±1.0

SRSI has the highest mean validation and held-out scores on both benchmarks. For the held-out gains over SkillOpt, the 95% Welch CIs are [+0.20, +8.20] on Video-MME and [−0.89, +9.49] on LVB. For the gains over AutoResearch, the corresponding intervals are [−1.71, +11.51] and [−1.39, +4.79]. SkillOpt edits only textual instructions and cannot change how evidence is gathered from the video, which limits its gains on long videos; AutoResearch keeps only edits that win their first verdict and thus falls into the early-experience trap, reaching the lowest held-out accuracy on Video-MME. The failure modes we diagnose are therefore not specific to BRSI, and the advantage of SRSI holds against other optimizers.

Revision: Section 4.2 (Figure 4(a)) and the related work (Appendix A) now discuss Meta-Harness, VideoHarness-RSI, SkillOpt, and AutoResearch, and Appendix P.3 (Table 27) reports the full results with these Welch intervals.

AuthorsOpenReview ↗

Official Reply to Reviewer 3rMw (3)

Weakness 3: Generality across executors:

We thank the reviewer for raising this point. We evaluate the twelve deployed SRSI harnesses of Table 1 (four benchmarks × three seeds), byte-identical and without re-evolution, on InternVL3-8B [5], an executor from a different model family, together with the six fixed baselines. No harness code, prompt, or answer parser is changed. The strongest baseline is picked on the held-out split itself, which favors the baselines:

ExecutorMethodVideo-MMELVBOVO-BenchMomentSeeker
Qwen3-VL-8BStrongest fixed baseline63.4262.0755.3617.98
Qwen3-VL-8BSRSI (evolved)70.1±1.067.3±1.072.5±3.123.0±0.7
InternVL3-8BStrongest fixed baseline62.2762.8557.1014.21
InternVL3-8BSRSI (transferred)66.7±2.063.5±0.562.1±4.416.4±1.1

Without re-evolution, SRSI harnesses still beat the strongest baseline on all four benchmarks: Video-MME (+4.4), OVO-Bench (+5.0), MomentSeeker (+2.2), and, by a smaller margin, LongVideoBench (63.5% vs. 62.85%). The margins are smaller than on Qwen3-VL-8B [6], so part of what SRSI evolves is a general strategy for reading video and part is tuned to the original executor. We will add these results to the appendix and state this scope in the main text.

Revision: Section 4.2 (Figure 4(c)) summarizes this transfer and states its scope, and Appendix P.5 (Table 30) reports the full results.

References

[1] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv preprint arXiv:2603.28052, 2026.

[2] Guoyang Xu, Hao Chen. VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models. arXiv preprint arXiv:2608.24302, 2026.

[3] Yifan Yang, Ziyang Gong, Weiquan Huang, et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv preprint arXiv:2605.23904, 2026.

[4] Andrej Karpathy. autoresearch. GitHub repository, https://github.com/karpathy/autoresearch, 2026.

[5] Jinguo Zhu, Weiyun Wang, Zhe Chen, et al. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv preprint arXiv:2504.10479, 2025.

[6] Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.