- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Verify-Repair-Reselect (VRR), a test-time scaling method that reuses the diagnostic evidence produced during verification to generate new candidates. The author observe that fixed-pool selectors such as LLM-as-a-Verifier rank a fixed candidate pool using LLM judgements, and therefore fail whenever every initial candidate is incorrect. VRR tackles this problem by reusing the diagnostic evidence produced during verification to generate repaired and alternative candidates beyond the initial pool, and then reselecting the final answer from the initial winner and these new candidates under the same criteria. Across two models and eight benchmarks, VRR achieves better accuracy than LAV in 9 of 16 settings.
Critical strengths
-
The experimental configuration and the prompts are disclosed in detail, which supports reproducibility.
-
The paper is carefully written and easy to follow. Its claims are appropriately measured: the authors explicitly refrain from claiming statistical significance, and they report negative outcomes.
Critical weaknesses
Novelty is limited, and the discussion of closely related prior work is insufficient.
The central research question of the paper, “Can verification also guide the construction of better candidates?”, substantially overlaps with existing test-time scaling methods that construct new candidates beyond the initial pool using information from previous candidates. These methods are neither discussed nor compared against. For example, parallel-distill-refine [1] combines LLM-as-a-judge tournament selection with Parallel-Distill-Refine, conditioning new rollouts on the summaries selected from prior attempts. In addition, Recursive Self-Aggregation [2] aggregates subsets of a candidate population into improved solutions. Considering this research, formalizing the fixed-pool ceiling carries limited novelty, and the remaining contribution would need to be established through direct comparison with these methods.
Furthermore, the paper describes its question as complementary to feedback-guided refinement methods, such as Self-Debugging. However, it is not clear that the two are complementary rather than competing. Iterative refinement methods such as Self-Debugging may achieve similar accuracy gains by themselves, without reusing ranking-oriented verification evidence. The authors should evaluate which approach yields higher accuracy at matched compute. If the approaches are truly complementary, it would be better to quantify the gain from combining them. Overall, the coverage of prior work and the quantitative comparison against it are insufficient, and the paper’s contribution is not yet well established.
Generation and verification costs are not controlled, so the source of the gains cannot be identified.
The comparison with LAV in Table 2 shares only the initial pool. Beyond that, two methods differ in several ways, including the number of total candidates, additional costs for generating criteria, and selector costs. Consequently, it is difficult to tell whether VRR's accuracy gains come from (i) generating more candidates, (ii) the collected failure evidence, or (iii) the improved verification procedure. I recommend experiments that control at least one of these factors. Especially, I’m interested in a comparison with a matched number of candidates, which I believe is essential for demonstrating the effectiveness of a test-time scaling method.
[1] Kim et al., Scaling Test-Time Compute for Agentic Coding.
[2] Venkatraman et al., Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models.
Questions for authors
-
Figure 1 appears to show the reselection pool as an expanded set containing both the initial candidates (c1–c3) and all repair candidates (r1–r3). In contrast, Algorithm 1 retains only the initial winner and the repair candidates that pass screening. Which reflects the actual behavior?
-
In Table 2, the total number of candidates seen by LAV and VRR does not appear to be matched. Do the authors have results where the number of candidates is matched?
-
How does VRR perform on agentic tasks such as SWE-bench Verified and Terminal Bench, where candidates are long trajectories rather than short programs or answers?
3 comments
Q3: How does VRR perform on agentic tasks such as SWE-bench Verified and Terminal-Bench?
A3: A direct VRR evaluation on long agentic trajectories requires an executable repair harness that can fork or continue a trajectory, execute the repaired actions in the task environment, and rerun the benchmark evaluator. Our current implementation does not yet provide this complete agentic repair interface, so we do not claim a direct VRR result on these benchmarks.
As a preliminary opportunity analysis, we treated existing frozen trajectories as a smaller initial pool plus held-out additional trajectories:
| Benchmark | Initial + held-out trajectories | Initial oracle | Expanded oracle | Gain |
|---|---|---|---|---|
| SWE-bench Verified | 2 + 1 | 408/500 | 422/500 | +2.80 points |
| Terminal-Bench | 3 + 2 | 81/89 | 82/89 | +1.12 points |
The held-out trajectory recovered 14 of 92 initially all-wrong SWE-bench pools, while the two held-out trajectories recovered 1 of 8 initially all-wrong Terminal-Bench pools. These results show that the fixed-pool ceiling also occurs for long agentic trajectories, but they are an oracle opportunity analysis—not a VRR accuracy evaluation—because the held-out trajectories were not generated from VRR diagnostics.
Thank you for the detailed response and the new experiments. However, my concerns regarding weakness 1 and 2 are not fully addressed in the rebuttal.
W1: The authors acknowledge the overlap with PDR and narrow their novelty claim to the specific integration of components. However, given this overlap, the rebuttal does not sufficiently justify why a comparison with PDR or RSA is unnecessary, so the contribution relative to prior work remains unclear.
W2: The new controls show that the variant without diagnostics performs on par with full VRR, and that independent sampling recovers most of the gain over the initial selector. This suggests that the improvement mainly comes from additional candidate generation. This means that the benefit of reusing verification evidence is not observed, further narrowing the contribution of the paper.
Thus, I will maintain the current score.
We thank the reviewer for recognizing the detailed disclosure of our experimental configuration and prompts, the clarity of the presentation, and our measured reporting of both positive and negative results.
A-W1: Our intended novelty is not the general idea of using previous attempts to generate new ones. Rather, VRR operationalizes evidence originally produced for candidate ranking within a retain–repair–screen–reselect procedure: it retains the initial winner, generates complementary alternatives from the winner and runner-up, screens them, and reselects under the original candidate-blind criteria.
This differs from Recursive Self-Aggregation [2], which aggregates candidate subsets, and from single-trajectory refinement methods such as Self-Debugging, which iteratively revise one solution. VRR maintains the incumbent as an explicit option and requires generated alternatives to compete with it during reselection. Parallel-Distill-Refine [1] is more closely related because it combines tournament selection with conditioned generation; we will discuss this overlap directly and revise our novelty claim to focus on VRR’s specific integration of ranking-oriented verification, conditional multi-role generation, screening, and incumbent-preserving reselection.
We did not perform direct reproductions of PDR or RSA and will not claim empirical superiority over them. Instead, we added controlled comparisons with independent sampling and with the same role-based generation without criterion-level diagnostic information.
A-W2: We added controls using the exact same VRR initial selector, trigger set, generation count, 4,096-token output limit, screening procedure, and final selector:
These results establish that adding generation, screening, and reselection improves upon the exact same initial selector, so the gain is not solely caused by the difference between the LAV and VRR selectors. Both role-based conditions also outperform independent sampling with the same number of generation attempts, indicating that targeted, parent-conditioned role generation provides value beyond merely adding samples.
The difference between the two role-based variants is small in this setting. We will therefore distinguish the demonstrated benefit of role-based candidate expansion from the incremental effect of the structured diagnostic ledger, for which these experiments do not establish an additional gain. Because actual token usage differs with prompt length and the number of candidates passing screening, we report the measured calls and tokens rather than describing the experiment as exactly token-matched.
A1: Algorithm 1 reflects the actual implementation. The reselection pool is
Creselect={ci∗}∪Rpass,
where ci∗ is the initial winner and Rpass contains the generated candidates that pass screening. The other initial candidates are not included in reselection. Thus, the active pool contains between two and four candidates when at least one repair is admitted.
A2: Yes. In the new controls above, every generation condition uses the same fixed 93-task trigger set and attempts exactly three additional candidates per triggered problem. All conditions also use the same 4,096-token output limit, public-test screening, and final selector.
The final admitted pool sizes may differ because candidates can fail the same screening procedure; this is an outcome of generation quality rather than a difference in the attempted candidate budget. Across both seeds, the role-based conditions achieved higher accuracy than independent sampling, while the difference between the two role-based variants was small.