- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks when an LLM proposer that repeatedly rewrites the code around a frozen model can be trusted to find designs that generalize. The setting is agentic video understanding, with Claude Opus 5 as the proposer and a frozen Qwen3-VL-8B as the executor. The authors first build a Meta-Harness-style baseline, called BRSI, with full access to validation content. Across several thousand candidates they identify three failure modes, namely validation overfitting, search-budget dilution and an early-experience trap. They explain all three with a single decomposition of the validation verdict into true gain, instance-specific fit, champion inflation and noise, given in Equation 1, and support it with dedicated estimators and propositions in Appendices E to H. They then propose SRSI, which changes only the evidence the loop acts on. The proposer receives content-free traces, a warm-up stage adds forced removal and intensity probes, fixed-template mechanism branches are allocated budget by a probe-and-race schedule, and selected branches are combined by factorial composition and joint refinement. As reported in Table 1, on Video-MME, LongVideoBench, OVO-Bench and MomentSeeker, SRSI raises the mean held-out score from 54.83 to 58.23 and reduces the mean gap between validation and test from 4.95 to 3.08. I thank the authors for a careful and ambitious study. I consider the diagnostic part of the paper to be of high quality and of practical value as a warning for work on harness optimization, which is why I rate its informativeness highly. However, the title and the main contribution concern the method, SRSI, and the current evidence does not yet show that the three-stage structure itself is responsible for the gains. Information isolation alone accounts for most of the effect, about half of the improvement comes with more frames and tool calls, and the evaluation rests on a single split that has been reused throughout. I therefore currently lean towards weak rejection.
Critical strengths
The diagnosis of why harness evolution overfits is well designed. The three nested feedback arms, Blindfolded, Trace and Multimodal, are each run at three validation fractions with three seeds as described in Appendix F.1, which allows instance-specific fit to be separated from noise. Table 11 shows that the noise share stays at roughly 0.14 to 0.16 across arms while the instance-specific share rises from 0.31 to 0.56. Table 12 adds direct evidence, since proposers that see question text quote validation questions in their code at 1.55 times the chance rate in the Trace arm and 1.27 times in the Multimodal arm. I found this the most valuable part of the paper. It gives the harness-optimization community a concrete reason to question full trace access on small validation sets. The theory is actually checked against data. The propositions make quantitative predictions that the authors test. Proposition F.1 predicts winners' held-out accuracy with a mean absolute error of 0.34 percentage points, and Proposition H.2 predicts a 0.54 probability of losing the frame-budget direction, compared with 6 of 12 runs observed. I also appreciate the transparency shown by a budget-matched comparison, re-scoring on unseen videos, and candid notes on residual leakage .
Critical weaknesses
It is not yet clear how much of the gain comes from SRSI's structure rather than isolation and extra inference budget. Table 19 shows that on LongVideoBench, BRSI with only Blindfolded feedback reaches 66.93 held-out against 67.3 for full SRSI. The "65.4 to 66.9%" result in Section 4.2 uses a 30-iteration BRSI without perception tools, whereas the Table 1 BRSI scores 66.6. Separately, Table 21 shows that on Video-MME SRSI uses 106.1 frames per question compared with 55.1, and 19.3 tool calls compared with 1.6. Under a frame cap the average gain falls from 3.40 to 1.66, and on OVO-Bench from 5.3 to 1.22, according to Table 22. Tool calls are not matched, and the capped selection draws from all evaluated candidates while deployment uses only the MERGE ledger. Because the central claim is that the staged organization of search matters, I think matched-budget ablations on all four benchmarks are needed. The generalization claims rest on a single partition per benchmark that is reused for diagnosis, design and evaluation. Each benchmark is split once and, as stated on page 15, the split is reused across all studies except the validation ladder of Appendix F. The held-out findings of Section 2 motivate each SRSI component, and Prompt 3 indicates the design was iterated. Appendix J.1 also notes that in the LongVideoBench and OVO-Bench runs a monitor evaluated validation-front members on the held-out split during search. OVO-Bench shows the largest test gain, +5.3, with little validation change, only +0.9. In addition, Finding 1, which claims "matching the strongest AKS variant", comes from a single run, as listed in Table 5, and its scores of 74.7 and 68.4 in Figure 2 exceed every other retrieval-only run in Appendix I.2, while the three retrieval-only runs in Table 14 average 66.8. For a paper about generalization from reused evidence, an evaluation of the frozen configuration on fresh splits or an untouched benchmark would make the conclusions considerably more convincing.
Questions for authors
- Could the authors add a "BRSI + isolation" row at the same iteration budget and tool set as SRSI on all four benchmarks, ideally with per-question paired tests? The LVB difference in Table 1, 67.3 versus 66.6 over three seeds, each with a standard deviation of about 1.0, seems hard to separate from seed variation.
- Would it be possible to report a comparison that matches both frames and tool calls, with the cap applied within the MERGE ledger that defines the deployed harness?
- Were the held-out monitor results in the LVB and OVO-Bench runs visible to the researchers before configurations, seeds or stopping points were fixed? Would the authors be able to evaluate the frozen SRSI configuration on new random splits or on an additional benchmark?
- For Finding 1, could the authors report the mean and confidence interval over all retrieval-only BRSI seeds against AKS-BLIP64, and explain how the run in Figure 2 was chosen?
- Appendix J.4 describes a pre-registered gate, but the deployed harness appears to be simply the best validation candidate after the merge step. What is the held-out score of the gate-selected harness?
- Several numbers appear inconsistent, and I would ask the authors to double-check them.
- In Table 1, the clip_topk values on Video-MME, 66.21 and 63.40, do not appear attainable with 450 and 2,250 questions. For example, 298 out of 450 gives 66.22 and 1426 out of 2250 gives 63.38.
- Figure 10 mentions 105 candidates for iterations 6 to 50, which I would expect to be 135 at three per iteration.
- Section 2.2.3 says a frame probe "wins by ten" in one run, while Appendix H.6 describes the seed-4 probe losing by 12.
- The mean verdicts in Appendix G.4 differ from Table 14.
4 comments
Official Reply to Reviewer 7R1Z (2)
Question 3: Held-out monitor and fresh splits:
As stated in Appendix J.1, the monitor results were unreadable by the proposer and entered no decision; the seeds, candidate budgets, and stopping rules are fixed by the protocol in Appendix J rather than chosen per run. To further address this concern, we re-partition Video-MME and LongVideoBench with a new random seed, keep every other setting identical to Table 1 with the pipeline fully frozen, and re-run the evolution of both BRSI and SRSI on the new partition, so the new held-out questions are never seen during evolution. Due to the limited rebuttal period, we run two random seeds each. These results are also reported in our reply to Reviewer 8sJ1 (Question 1):
| Method | Video-MME Val | Video-MME Test | LVB Val | LVB Test |
|---|---|---|---|---|
| BRSI | 71.6±1.6 | 66.0±1.1 | 69.8±0.8 | 66.1±1.3 |
| SRSI | 74.4±1.3 | 69.5±0.8 | 70.9±1.0 | 67.0±1.0 |
On the fresh split, SRSI improves over BRSI by +3.5 and +0.9 held-out points, in the same direction as Table 1 (+3.6 and +0.7). The corresponding 95% Welch CIs are [−1.04, +8.04] and [−4.42, +6.22], respectively.
Revision: Section 4.2 (Figure 4(b)) and Appendix P.4 (Table 28) report the fresh-split results; the held-out monitor is described in Appendix J.1.
Question 4: Finding 1:
Finding 1 is that recursive evolution can discover a retrieval harness that performs comparably to a hand-designed method on Video-MME. Figure 2 reports the only run conducted for the dedicated effectiveness study (Table 5). Starting from clip_topk, BRSI evolved a harness that reached 68.36% held-out accuracy using 38 frames per question on average. AKS-BLIP64 [2] reached 68.40% using 62 frames per question on average. To put this example in context, we also report the mean across four retrieval-only BRSI runs from simple seeds on Video-MME (the Figure 2 run and the three runs in Table 14). Mean held-out accuracy was 67.2% (95% CI [65.4%, 69.0%]), 1.2 percentage points below AKS-BLIP64. Each of the four evolved harnesses used at most 38 frames per question on average (Appendix P.8).
Revision: Finding 1 in Section 2.1 now reads "can discover" and reports the four-run mean, and Appendix P.8 gives the details.
Question 5: The admission gate:
The deployed harness is the validation-best candidate after MERGE; the gate was computed and recorded but did not decide deployment, so "pre-registered" was misleading and will be replaced by "recorded admission rule". On held-out data, the gate-selected harness is below the deployed one by 1.2, 0.5, 2.1, and 0.8 points on Video-MME, LVB, OVO-Bench, and MomentSeeker, so deploying by the gate would not have improved held-out accuracy.
Revision: Appendix J.4 now calls the gate the recorded admission rule, and Appendix P.9 reports this held-out comparison.
Number checks: We thank the reviewer for catching these.
(1) The clip_topk scores on Video-MME should read 66.22 and 63.42; no comparison is affected.
(2) Of the 135 scheduled candidates for iterations 6–50, 105 were evaluated; the rest failed the import check or the leak guard and were discarded.
(3) Section 2.2.3 was imprecise: as detailed in Appendix H.6, the seed-1 probe lost by 10 questions and closed the direction, while the seed-4 probe lost by 12 but a sibling at 18.3 frames won and kept the direction open; we will correct the sentence and Figure 3(c).
(4) Table 14 averages the first 87 candidates of each run, the largest budget all runs reach, whereas Appendix G.4 uses all candidates; we will state this in Appendix G.4.
Revision: We correct (1) in Table 1, (2) in Appendix H.6, (3) in Section 2.2.3 and Figure 3(c), and (4) in Appendix G.4.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
[2] Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, Qixiang Ye. Adaptive Keyframe Sampling for Long Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Gentle follow up
We thank the reviewer again for their time and consideration. As the rebuttal period is approaching its end, we wanted to kindly follow up and ask whether our response has addressed your concerns or if there are any remaining questions we could clarify.
We would greatly appreciate your feedback and consideration of our additional results.
Response to Authors
Thank you for the thorough rebuttal. The matched comparison with BRSI plus isolation shows that structured search brings clear gains beyond isolation, which supports the central claim. The results on new splits, the transfer and cost results, and the corrections also address my other concerns. I have updated my assessment accordingly.
Official Reply to Reviewer 7R1Z (1)
We sincerely thank Reviewer 7R1Z for the careful and detailed review, and for recognizing the quality and practical value of our diagnostic analysis and its theory-to-data checks. We address each question below, including the new experiments you suggested.
Question 1: BRSI + isolation under the same budget and tool set:
We have added this comparison on all four benchmarks, with the same tool set as SRSI and comparable search budgets. BRSI with isolation schedules 150 candidates per run. SRSI has a nominal upper bound of 156 before the tie allowance in Appendix J.5, as in Table 1. Table 19 reuses separate runs from our feedback study (90 candidates per run, perception tools disabled), which is why its BRSI scores differ from those in Table 1. As in Table 1, we ran three random seeds per benchmark for the new row (12 isolation runs and 1,800 scheduled candidates in total). These results are also reported in our reply to Reviewer 3rMw (Weakness 1):
Information isolation alone improves the mean held-out benchmark score (+1.25 points) and narrows the validation–test gap (4.95 → 3.53), and the structured search of SRSI adds another +2.15 held-out points on average. Table 25 reports per-question paired held-out differences (SRSI minus BRSI with isolation), with a 95% interval for each benchmark obtained by resampling videos and runs.
The per-benchmark paired intervals exclude zero on Video-MME, OVO-Bench, and MomentSeeker. On LVB the gain beyond isolation is +0.3 points, and its interval spans zero. The separate comparison of SRSI with the original BRSI also has an LVB interval spanning zero (+0.7 points, 95% Welch CI [−1.6, +3.0], Table 26). The 65.4% to 66.9% result in Section 4.2 of the original submission (Appendix L of the revision) is the effect of adding isolation to BRSI in the 30-iteration, perception-tools-disabled study (Table 19).
Revision: Table 1 (new row) and Section 4.1 report this comparison, Appendices P.1 and P.2 (Tables 24–26) give the protocol and intervals, and Appendix L states the setting of the 65.4% to 66.9% result.
Question 2: Matched comparison within the MERGE ledger:
We cap each SRSI run at the frames per question of the deployed BRSI harness, select within the MERGE ledger by validation accuracy only, and report the held-out difference from BRSI. The selected harnesses also use no more executor calls than BRSI (e.g., 2.03 vs. 2.43 per question on Video-MME):
About half of the gain (1.55 of 3.40 points on average) survives this stricter match. Both methods run under the same tool set and the same per-question limits (60 tool calls and 12 executor calls; Table 7), so SRSI is not given a larger tool budget; its harnesses learn to use more of the same allowance. The search selects for validation performance under the common tool-call limit, so a harness can use additional CLIP [1] retrieval or shot-detection calls to gather evidence without invoking the executor. On Video-MME, the three deployed BRSI harnesses average 1.6 tool calls per question (Table 21). When we apply this tool-call cap together with the frame cap, no candidate in the existing SRSI MERGE ledger satisfies both constraints. The accuracy achievable under this joint constraint therefore remains unmeasured. We will report this explicitly and replace "about half of the gain survives" with the matched setting it refers to.
Revision: Section 4.1 and Table 2 (new column ΔJ) report this matched comparison in place of "about half of the gain survives", and Appendix P.7 (Table 32) gives the intervals and the result of the joint tool-call cap.