acceptodds
Under review as a conference paper at ICLR 2027

Learning to Verify by Doing: A Self-Renewing Policy–Verifier Loop for Sustained Self-Improvement

Abstract

Verifier-guided self-training can stall as policy improvement erodes the selection advantage of existing evaluative guidance. We show that execution feedback can renew this guidance by teaching a verifier about action outcomes and applicability conditions. We introduce Knowing–Doing Co-Evolution (KDCE), a self-renewing policy–verifier loop that learns outcome prediction and candidate ranking from budgeted executions and controlled local action comparisons, then uses uncertainty-aware selection for weighted policy self-training. The updated policy supplies experience for the next round, closing the loop. Across 8 checkpoints from 3 model families and 11 code, mathematical-reasoning, and interactive evaluations, KDCE improves family- and domain-balanced standalone performance by 0.7 percentage points over prespecified domain-specific references after 8 rounds. Matched-evidence and directional interventions support both feedback paths. Gains are heterogeneous: improved verification does not always yield policy gains or compute savings.

Primary area:
generative models
Keywords:
self-improvement+large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SDYKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The problem this paper attempts to address is that a static verifier's ability to identify good outcomes acts as a ceiling on a trained policy's ability to achieve them. The proposed method, KDCE, trains verifiers based on outcomes of a curated set of policies, then uses the updated verifier to train a new policy. This achieves very minor improvements on baselines for coding and interactive tasks, though not math.

Critical strengths

None

Critical weaknesses

This paper appears to be almost entirely written by AI. The level of AI detected is not consistent with the AI Use Statement claiming it was only used for edits, and suggests that entire sections were written by AI.

The proposed method lacks novelty, being ony the combination of existing methods. As might be expected, this leads to very minor improvements in the experiments run, which are indistinct from noise, and this is despite using more compute. Major experimental details are not provided. Overall, this gives the flavor of authors who delegated the entire paper creation process to AI.

Questions for authors

None

Reviewer_XannOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission addresses the stagnation of verifier-guided self-training that occurs when policy advancements erode the selection advantage provided by existing evaluative guidance. To overcome this bottleneck, the authors introduce Knowing–Doing Co-Evolution (KDCE), a reciprocal policy–verifier framework that leverages budgeted execution feedback—including trajectory outcomes, state–action records, and paired local edits—to continually update consequence prediction and candidate ranking. The updated verifier employs calibrated, uncertainty-aware target selection to guide weighted policy self-training, generating new experience that feeds into subsequent rounds. The paper's primary contributions include diagnosing feedback-limited self-training by isolating coverage, selection quality, and target absorption, developing the KDCE architecture for evidence-conditioned verification, and conducting directional intervention experiments that isolate both paths of the policy–verifier loop. The approach is empirically validated on 8 checkpoints across 3 model families (Qwen, Gemma, and Olmo) across 11 code, mathematical-reasoning, and interactive benchmarks.

Critical strengths
  • The paper provides a well-reasoned diagnosis for why verifier-guided self-training tends to stall—specifically, that policy advancements diminish the selection advantage (G=K−DG = K - D) of a static verifier. To address this, the Knowing–Doing Co-Evolution (KDCE) framework offers a logical mechanism by updating the verifier using budgeted execution feedback, including trajectory outcomes, state–action records, and local edit pairs. Grounding verifier updates in execution feedback rather than purely synthetic self-reflection creates a structured closed loop where updated verifier rankings provide revised targets for policy self-training. This formulation offers a clear, intuitive approach to maintaining verifier utility as policy capabilities evolve.
Critical weaknesses
  • W1. Insufficient Intuition and Explanation for Domain Heterogeneity While the paper commendably reports domain-level breakdown metrics, it lacks a satisfying explanation or theoretical intuition for the pronounced performance heterogeneity across task types. The authors observe this gap but provide little conceptual analysis as to why execution-based feedback fails to renew verifier guidance in formal mathematics compared to code or tools. It remains unclear whether this bottleneck stems from reward sparsity, the absence of granular intermediate execution state feedback in step-by-step math proofs, or fundamental limits in verifier target absorption.

  • W2. Unclear Problem Setting, Scope, and Notation Prior to Diagnostic Analyses The manuscript introduces empirical diagnostic results in Section 2.1 before establishing a clear problem setting, task scope, or formal notation. Key variables are used in Section 2.1 without proper formal definition; for instance, in Equation (1) (Kt(N)=Ct(N)Rt(N),Gt(N)=Kt(N)−DtK_t(N) = C_t(N)R_t(N), G_t(N) = K_t(N) - D_t), the notation tt appears as an iteration index without explicitly defining what constitutes a step tt (e.g., policy rollout round vs. verifier update step). Requiring the reader to parse diagnostic observations in Section 2 before the formal system architecture and operational loop are defined in Section 3 creates unnecessary friction. A concise problem formulation section prior to Section 2.1—clearly defining the trajectory space, execution feedback model, and indexing notation—would significantly improve readability.

  • W3. Dense Presentation and Need for Writing Polish The overall writing lacks polish and frequently relies on overly compressed, dense phrasing that hampers clarity. Several technical sections compress complex multi-step experimental protocols, variable definitions, and diagnostic conditions into heavily packed paragraphs without adequate structural transitions. The current draft feels unrefined in its narrative flow and requires thorough editorial polishing to make its technical mechanics accessible to a broader audience.

Questions for authors

Please see Weakness 1.

Reviewer_zGQmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KDCE uses budgeted executions and local action comparisons to update an outcome-predicting verifier, then constructs policy self-training targets through uncertainty-aware selection. The paper separates candidate coverage, recognition, and target absorption, and tests both directions of the policy–verifier loop. After eight rounds, the balanced gain over domain-specific references is 0.7 percentage points. The family-balanced geometric mean efficiency ratio is 0.99, and the authors appropriately reject a general compute-saving claim.

Critical strengths
  1. S1: Fixed reference pools, current-policy pools, standalone policy performance, and acceptance risk are carefully distinguished.
  2. S2: Identical-evidence comparisons, blocked/restored selection paths, fixed experience sources, and verifier-only failure routing provide useful mechanism tests.
Critical weaknesses
  1. W1: The headline 0.7 pp gain and identical-evidence increments of 0.2–0.6 pp are small. Five-configuration sensitivity ranges are not confidence intervals over training runs or clustered tasks. Corresponding endpoint intervals and complete subset sizes are not presented in the PDF, making stability difficult to assess.
  2. W2: Resource tables report normalized cap fractions without a reproducible absolute definition of each domain cap, execution counts, tokens, and hardware time. Equal caps need not imply equal useful consumption. Resource fairness remains difficult to verify.
  3. W3: Evidence for the distinctive components is uneven. KDCE exceeds trace-to-policy by only 0.2 pp on identical records; removing ranking improves performance by 0.1 pp, and random queries cost only 0.3 pp. Execution feedback and refreshed selection are better supported than the necessity of the full combination.
Questions for authors
  1. Q1 → W1: Please report evaluation sizes, independent training repeats, and paired intervals clustered by parent task, repository, or family, especially for the aggregate 0.7 pp and Table 18's 0.2/0.6 pp differences.
  2. Q2 → W2: Can the authors provide actual calls, tokens, GPU time, and cap definitions by domain and round, and allow simpler baselines to reallocate resources at matched useful consumption?
  3. Q3 → W3: Can outcome prediction with refreshed selection, full KDCE, and trace-to-policy be compared under identical evidence, risk, target counts, and useful compute to isolate local pairs, active queries, and ranking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.