acceptodds
Withdrawn from ICLR 2027

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes *outcome-calibrated self-guidance via trajectory balance*, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic.

Primary area:
reinforcement learning
Keywords:
Self-improvement, Reinforcement learning with verifiable rewards, Self-guidance, Distribution matching, Trajectory balance.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vqhfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FlowBalance, aiming to address two issues that arise when a reasoning model performs self-improvement based on its own on-policy trajectories: on the one hand, the terminal reward provided by a verifier is reliable but sparse; on the other hand, privileged-hindsight self-guidance can provide denser training signals, but may erroneously reinforce failed trajectories, causing self-confirmation. FlowBalance uses the frozen current policy combined with privileged context to rescore already-sampled trajectories, aggregates token-level log-probability gains into trajectory-level guidance, and calibrates its direction using a verifier-derived group-relative advantage: for trajectories with positive advantage, the guidance is retained; for trajectories with negative advantage, the guidance is reversed; and when the advantage is zero, the guidance is turned off. The method then performs exponential reweighting of a fixed reference policy based on this trajectory energy, constructs a normalized response target, and fits it via a profiled trajectory-balance objective, rather than directly adopting a token-level imitation loss. The theoretical part analyzes several properties of this target distribution, including within-group probability contrast preservation, a minimum reverse-KL characterization, monotonic control of the target reward by the verifier coefficient, and a probability-ratio correction of false-positive self-guidance on failed trajectories via sign gating. The experiments are based on Qwen3-4B and Qwen3-8B, compare against GRPO, OPSD, RLSD, and FlowRL, cover five mathematical reasoning benchmarks, and further analyze training dynamics, hyperparameter ablations, and semantic strategy diversity on AIME24. The paper reports that FlowBalance achieves higher overall average performance at both model scales and exhibits greater diversity of correct strategies in controlled diagnostics.

Critical strengths
  • The methodological framework is relatively unified, with a fairly systematic theoretical analysis of the core objective. Rather than simply stacking a verifier reward and a self-distillation loss, this paper unifies the two types of signals into a trajectory-level energy, and then completes the policy update through normalized distribution matching.

  • The experiments cover multiple model scales and evaluation dimensions. The paper conducts multi-benchmark comparisons on Qwen3-4B and Qwen3-8B, and focuses not only on final accuracy but also analyzes training speed, late-stage stability, response length, and the semantic strategy diversity of correct solutions.

Critical weaknesses
  • Insufficient empirical validation, missing direct ablations of the core mechanism, and questionable credibility of some baselines. Currently only ηA∈{5,10,15},βG∈{1,2,3}\eta_A\in\{5,10,15\},\beta_G\in\{1,2,3\} are examined, while at least three critical controls are missing: a no-guidance variant with βG=0\beta_G=0, an ungated-guidance variant with sgn⁡(A)\operatorname{sgn}(A) removed, and a no-verifier variant with ηA=0\eta_A=0. Consequently, the existing experiments make it difficult to determine whether the performance gains truly come from verifier-gated self-guidance itself, or from trajectory balance, the profiled partition, or other training designs. In addition, the OPSD [1] baseline reported in this paper is noticeably weaker than the results in the original OPSD paper. For example, this paper reports OPSD Pass@16 on AIME24 of 65.33 and 48.67 for Qwen3-4B/8B, whereas the original OPSD paper reported markedly higher results under a 12-sample evaluation setting.

  • Inadequate novelty positioning, with omissions of highly relevant prior work. One of FlowBalance's main contributions is the use of the verifier outcome to determine the direction of dense self-guidance: retaining guidance on successful trajectories, and reversing on failed trajectories the guidance that could lead to self-confirmation. However, a similar core idea has already been proposed in SGSD [2]. That work should be considered directly related work that this paper needs to discuss and compare against prominently.

Reference

[1] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. 2026. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv preprint.

[2] Jiazhen Huang, Xiao Chen, Xiao Luo, Yong Dai, Senkang Hu, and Yuzhi Zhao. 2026. Skill-conditioned gated self-distillation for llm reasoning. arXiv preprint.

Questions for authors
  1. On ablations of the core mechanism. Could the authors provide strictly controlled LLM ablations, including at least: (a) βG=0\beta_G=0, i.e., completely removing self-guidance within the same FlowBalance/trajectory-balance framework; (b) an ungated-guidance variant with sgn⁡(A)\operatorname{sgn}(A) removed; and (c) a no-verifier variant with ηA=0\eta_A=0? These experiments are essential for determining whether the performance gains truly come from verifier-conditioned gating.

  2. On the OPSD baseline. Why is the OPSD performance reported in this paper on AIME24 markedly lower than the results in its original paper, especially the large gap on Qwen3-8B? Please clearly specify the checkpoint-selection protocol, training horizon, maximum generation length, and sampling/decoding parameters adopted in this paper, as well as the concrete differences from the original OPSD setting. If re-evaluated following the evaluation and checkpoint-selection protocol of the original OPSD paper, would the results and the gap relative to FlowBalance still hold?

  3. On the relationship with SGSD and novelty. Please explicitly discuss the relationship between FlowBalance and SGSD. Both use verifier/outcome information to determine the polarity of dense self-guidance/self-distillation signals, and suppress or reverse inconsistent guidance on failed trajectories. What do the authors consider to be FlowBalance's core technical increment over SGSD?

Reviewer_eRPsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces FlowBalance, a method that combines verifier feedback with self-guidance through a trajectory-level distribution-matching objective. Experiments on mathematical reasoning report improvements over several baselines, together with analyses of training stability and solution diversity.

Critical strengths
  1. The paper is clearly written and well structured.

  2. Using verifier outcomes to calibrate self-guidance is an interesting idea with an intuitive motivation.

  3. The experiments include several relevant baselines and report results across different seeds.

Critical weaknesses
  1. The methodological contribution appears somewhat incremental, building closely on existing GRPO, OPSD, and trajectory-balance approaches. The theoretical analysis establishes properties of the proposed target, but provides limited justification for the specific combination of these components.

  2. The evaluation is limited to two sizes of the Qwen3 family on old mathematical reasoning tasks. Additional model families and more challenging benchmarks would strengthen the evidence for broader applicability.

  3. The gains over the closest baseline are relatively modest, while some other baselines perform substantially worse. More complete training details, baseline configurations, and uncertainty reporting would help establish confidence in these comparisons.

  4. The ablations mainly examine coefficient choices rather than directly isolating the proposed mechanism. A more targeted comparison of alternative ways to incorporate self-guidance would help clarify the source of the improvements.

Questions for authors

Please refer to Weaknesses

Reviewer_PBmCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FlowBalance combines verifier feedback with privileged same-model guidance in a trajectory-balance objective. A frozen snapshot of the current policy scores its own sampled responses while conditioning on training-only solutions or feedback. Clipped token-level log-probability gains relative to an initial reference policy are averaged into a trajectory score, whose contribution is retained, reversed, or disabled according to the sign of the group-relative verifier advantage. The resulting energy defines a reference-weighted target, fitted using a profiled group-level partition term.

The paper establishes several properties of this target conditional on a realized rollout group. Experiments on Qwen3-4B and Qwen3-8B report five-seed results on five mathematical reasoning benchmarks, with average gains over FlowRL of 1.04 and 1.76 percentage points. Additional analyses examine training dynamics, coefficient sensitivity, and LLM-judged strategy diversity on AIME24.

Critical strengths
  1. The proposed combination is coherent and interpretable. Verifier feedback and privileged guidance enter a single trajectory energy, and the policy is trained through one distribution-fitting objective. This provides a clear alternative to combining a policy-gradient loss with a separate token-level distillation loss. The analysis makes the effect of sign gating explicit and distinguishes target properties from guarantees about the actual training process. This clarity is useful for understanding when the guidance could help and where its protection is limited.

  2. The main accuracy evaluation provides encouraging evidence across two model scales and five seeds. FlowBalance has the highest five-benchmark average on both backbones, and the comparisons include verifier-only training, direct distillation, a hybrid, and outcome-only trajectory balance. The improvements over FlowRL are modest but appear across several benchmarks, rather than arising solely from the particularly weak OPSD runs. Reporting per-benchmark variability and acknowledging the Qwen3-4B Minerva regression make the evidence more useful than a single aggregate score.

Critical weaknesses
  1. The experiments do not isolate the paper's central contribution: outcome-calibrated privileged guidance. The coefficient sweeps never include βG=0\beta_G=0, and there is no language-model ablation that retains the guidance while removing sign gating. Comparing against FlowRL does not fully resolve this issue: the original FlowRL uses a learned partition function, whereas FlowBalance profiles a group-level intercept. GFlowRL already studies replacing the learned partition network with a rollout-based estimate, making a matched outcome-only profiled baseline particularly important. Likewise, the synthetic examples verify consequences of the chosen energy but do not show that the relevant guidance regime occurs in training. Since the gain is measured against the initial reference rather than the unprivileged current snapshot, it also contains policy drift as well as the effect of privileged context. Without these controls, the improvements cannot confidently be attributed to the proposed verifier-guidance interaction.

  2. The experimental protocol is insufficiently specified, and the supporting efficiency and diversity claims are not adequately controlled. The training corpus and privileged-context construction are not concretely identified. Table 5 explicitly leaves rollout group size and maximum response length pending release; learning rate, batch size, evaluation decoding parameters, τ\tau, and the clipping bound BB are also not specified. These choices are central to assessing both the unusually poor OPSD results and the stability comparisons. Moreover, the 1.43x acceleration measures optimizer steps, without accounting for the additional privileged scoring pass, so it does not establish a wall-clock speedup. The diversity comparison has a separate sampling issue: Appendix H.3 includes a problem only when a method generates at least two correct responses. The compared problem sets and correct-response counts can therefore differ, and the empirical Simpson estimator depends on sample size. A one-seed caveat does not resolve these confounds.

Questions for authors
  1. Additional evidence affecting the assessment: With the same profiled objective and optimizer settings, how do outcome-only training (βG=0\beta_G=0), ungated guidance, and the proposed sign-gated guidance compare? A control that removes privileged context from the scoring view would also help separate hindsight information from current-policy drift relative to the initial reference.

  2. Clarification: Please specify the training corpus, source and exact format of privileged context, rollout group size, response cap, optimizer settings, τ\tau, BB, and evaluation decoding settings. How was the step-180 reporting point chosen, and were AIME24 or any other reported benchmarks used to select coefficients or training settings?

  3. Additional evidence affecting the assessment: Does the improvement in updates-to-threshold remain an improvement in elapsed training time or total compute once privileged scoring is included? Please also clarify the metric and data split used for the AIME24 validation curves, which differ in scale from the final Pass@16 results.

  4. Additional evidence affecting the assessment: Does the diversity advantage persist on a common set of problems after matching the number of correct trajectories per problem, or using a sample-size-corrected Simpson estimate? Including FlowRL in this comparison would help establish whether the diversity benefit comes from the added guidance rather than trajectory-balance training itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.