acceptodds
Under review as a conference paper at ICLR 2027

REVES: REvision and VErification–Augmented Training for Test-Time Scaling

Abstract

Test-time scaling via sequential revision has emerged as a powerful paradigm for enhancing Large Language Model (LLM) reasoning. However, standard post-training methods primarily optimize single-shot objectives, creating a fundamental misalignment with multi-step inference dynamics. While recent work treats this as multi-turn reinforcement learning (RL), conventional approaches optimize over the multi-step trajectories directly, failing to further exploit the high-quality mistakes in intermediate steps that model can learn from correcting them. We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization. By converting the intermediate steps (“near-miss" answers) in the successful recovery trajectories into decoupled revision and verification prompts, our approach concentrates training on both effective answer transformation and error identification. This approach enables efficient off-policy data generation and reduces the computational overhead of long-horizon sampling compared to standard multi-turn RL. On LiveCodeBench, using publicly available test cases as feedback, we observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training. Beyond coding, our approach matches the previously reported SOTA result on circle packing. Math results under ground-truth verification further confirm improved correction ability. It also generalizes to out-of-distribution constraint-satisfaction puzzles such as n_queens and mini_sudoku, where correctness is defined entirely by problem constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3g8LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REVES to improve sequential answer revision in language models. It extracts revision and verification tasks from successful correction trajectories and trains the model using single-turn RL, with iterative data refresh. Experiments on coding, mathematics, constraint-satisfaction puzzles, and circle packing evaluate its effectiveness.

Critical strengths

I find the setting practical, especially for applications such as code debugging. Reusing incorrect intermediate answers as training inputs is a useful design that provides more direct feedback for learning to correct mistakes.

Critical weaknesses
  1. The methodological novelty appears somewhat weak. The approach mainly combines trajectory filtering, data augmentation, and revision/verification training. The paper should more clearly explain its main conceptual advance over existing self-correction methods.

  2. The presentation could be more direct. Considerable space is devoted to formal definitions and theoretical analysis before the practical training procedure is fully explained. A concrete example showing how a revision trajectory becomes training data would make the method easier to follow.

Questions for authors

How does REVES differ from existing methods for single-step improvement or joint revision and verification? What is its main insight and contribution beyond data selection and prompt construction?

Reviewer_HgCBOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

REVES trains language models for sequential revision by turning incorrect intermediate answers from successful correction trajectories into single-turn revision and verification prompts. The prompt set is refreshed as the policy changes. The paper also analyzes the bias–variance trade-off between this per-state training signal and trajectory-level credit assignment.

Critical strengths
  1. The method targets a real mismatch between single-shot training and revision-based inference. Converting the model’s own recoverable mistakes into training prompts is a clear and useful design.

  2. The empirical work is substantial. The paper compares against single-shot RL, multi-turn training, PAG, and SCoRe; it reports token and wall-clock accounting for the multi-turn comparison and ablates the state filter, prompt types, and data refresh.

Critical weaknesses
  1. The theory does not fully describe the implemented method. The objective decomposition is exact, but training fixes the state distribution and keeps only trajectories that eventually succeed.
  2. The main multi-turn comparison changes more than credit assignment. REVES uses GRPO, while the multi-turn baseline uses PPO/GAE with a critic. Matching generated tokens helps, but it does not isolate the source of the gain.
  3. The math gains depend heavily on oracle stopping. With self-confidence stopping, the improvement is much smaller; on MATH500, REVES scores 76.2 versus 78.6 for single-shot RL. The paper has not yet shown that its verification training closes this gap.
Questions for authors
  1. What fraction of incorrect states is excluded by the successful-trajectory filter, and does that fraction change substantially over training?

  2. For circle packing, what is the total candidate-evaluation budget including RL training, and does the result extend beyond the reported (n=26) instance?

Reviewer_cyjWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary The paper addresses the mismatch between single-shot RL post-training and test-time sequential revision. It decomposes the sequential-revision objective into per-state one-step recovery probabilities (Eq. 1) and proposes REVES. REVES collects intermediate incorrect answers from successful revision trajectories, turns them into single-turn revision and verification prompts, and trains on them with standard GRPO, refreshing the data every epoch. A tabular surrogate analysis characterizes when this per-state estimator beats multi-turn trajectory credit. Experiments on coding, math, puzzles, and circle packing show gains over RL, multi-turn RL, PAG, and SCoRe.

Critical strengths

The motivation is clear: shared trajectory advantages reinforce incorrect intermediate turns (Figure 2). The method is simple and cheaper than online multi-turn RL (Table 7). The baselines are strong (SCoRe, PAG, pass@k training), and the gains transfer to other revision-based test-time algorithms (Table 3). The ablations on state selection, prompt types, and continual augmentation are informative (Tables 5–6, Figure 10).

Critical weaknesses

Oracle stopping likely inflates the reported gains. With ground-truth stopping, Oracle-K counts a problem as solved if any of the K revisions is correct, which approaches pass@K and favors output diversity rather than genuine revision ability (as Appendix D itself illustrates). Under the practical SelfConf stopping rule, the results are mixed: on MATH500, REVES (76.2) underperforms RL (78.6), and on AIME24 it is slightly below SCoRe (19.6 vs. 20.0) (Table 2). I suggest reporting pass@K of independent samples for each model, so that the contribution of revision can be separated from that of sampling diversity.

The theoretical contribution is limited. Eq. (1) is a standard decomposition of a first-success objective. Theorem 4.1 is established in a tabular Markov chain with independent per-state parameters, using unnormalized REINFORCE estimators, and follows almost directly from the bias–variance decomposition of the MSE. Its relevance to LLM training, where parameters are shared across states and GRPO normalizes rewards, is asserted (§4.2) but not demonstrated.

Baseline parity and statistical reporting. Although the number of sampled responses is matched (Table 4), the Multi-Turn and PAG baselines are trained with PPO and a critic, use only two training turns, and are implemented in a different codebase (Appendix H). Their SC-4 results are also obtained under a different evaluation protocol (marked † in Table 2). Moreover, the results appear to come from a single training run per method, with no standard deviations or confidence intervals reported. Given this, the mixed coding results (e.g., Qwen2.5-3B TC-4 and Qwen3-4B CodeContest TC-4, where Multi-Turn outperforms REVES; Table 1) are difficult to interpret.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.