- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FlowBalance, aiming to address two issues that arise when a reasoning model performs self-improvement based on its own on-policy trajectories: on the one hand, the terminal reward provided by a verifier is reliable but sparse; on the other hand, privileged-hindsight self-guidance can provide denser training signals, but may erroneously reinforce failed trajectories, causing self-confirmation. FlowBalance uses the frozen current policy combined with privileged context to rescore already-sampled trajectories, aggregates token-level log-probability gains into trajectory-level guidance, and calibrates its direction using a verifier-derived group-relative advantage: for trajectories with positive advantage, the guidance is retained; for trajectories with negative advantage, the guidance is reversed; and when the advantage is zero, the guidance is turned off. The method then performs exponential reweighting of a fixed reference policy based on this trajectory energy, constructs a normalized response target, and fits it via a profiled trajectory-balance objective, rather than directly adopting a token-level imitation loss. The theoretical part analyzes several properties of this target distribution, including within-group probability contrast preservation, a minimum reverse-KL characterization, monotonic control of the target reward by the verifier coefficient, and a probability-ratio correction of false-positive self-guidance on failed trajectories via sign gating. The experiments are based on Qwen3-4B and Qwen3-8B, compare against GRPO, OPSD, RLSD, and FlowRL, cover five mathematical reasoning benchmarks, and further analyze training dynamics, hyperparameter ablations, and semantic strategy diversity on AIME24. The paper reports that FlowBalance achieves higher overall average performance at both model scales and exhibits greater diversity of correct strategies in controlled diagnostics.
Critical strengths
-
The methodological framework is relatively unified, with a fairly systematic theoretical analysis of the core objective. Rather than simply stacking a verifier reward and a self-distillation loss, this paper unifies the two types of signals into a trajectory-level energy, and then completes the policy update through normalized distribution matching.
-
The experiments cover multiple model scales and evaluation dimensions. The paper conducts multi-benchmark comparisons on Qwen3-4B and Qwen3-8B, and focuses not only on final accuracy but also analyzes training speed, late-stage stability, response length, and the semantic strategy diversity of correct solutions.
Critical weaknesses
-
Insufficient empirical validation, missing direct ablations of the core mechanism, and questionable credibility of some baselines. Currently only are examined, while at least three critical controls are missing: a no-guidance variant with , an ungated-guidance variant with removed, and a no-verifier variant with . Consequently, the existing experiments make it difficult to determine whether the performance gains truly come from verifier-gated self-guidance itself, or from trajectory balance, the profiled partition, or other training designs. In addition, the OPSD [1] baseline reported in this paper is noticeably weaker than the results in the original OPSD paper. For example, this paper reports OPSD Pass@16 on AIME24 of 65.33 and 48.67 for Qwen3-4B/8B, whereas the original OPSD paper reported markedly higher results under a 12-sample evaluation setting.
-
Inadequate novelty positioning, with omissions of highly relevant prior work. One of FlowBalance's main contributions is the use of the verifier outcome to determine the direction of dense self-guidance: retaining guidance on successful trajectories, and reversing on failed trajectories the guidance that could lead to self-confirmation. However, a similar core idea has already been proposed in SGSD [2]. That work should be considered directly related work that this paper needs to discuss and compare against prominently.
Reference
[1] Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. 2026. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv preprint.
[2] Jiazhen Huang, Xiao Chen, Xiao Luo, Yong Dai, Senkang Hu, and Yuzhi Zhao. 2026. Skill-conditioned gated self-distillation for llm reasoning. arXiv preprint.
Questions for authors
-
On ablations of the core mechanism. Could the authors provide strictly controlled LLM ablations, including at least: (a) , i.e., completely removing self-guidance within the same FlowBalance/trajectory-balance framework; (b) an ungated-guidance variant with removed; and (c) a no-verifier variant with ? These experiments are essential for determining whether the performance gains truly come from verifier-conditioned gating.
-
On the OPSD baseline. Why is the OPSD performance reported in this paper on AIME24 markedly lower than the results in its original paper, especially the large gap on Qwen3-8B? Please clearly specify the checkpoint-selection protocol, training horizon, maximum generation length, and sampling/decoding parameters adopted in this paper, as well as the concrete differences from the original OPSD setting. If re-evaluated following the evaluation and checkpoint-selection protocol of the original OPSD paper, would the results and the gap relative to FlowBalance still hold?
-
On the relationship with SGSD and novelty. Please explicitly discuss the relationship between FlowBalance and SGSD. Both use verifier/outcome information to determine the polarity of dense self-guidance/self-distillation signals, and suppress or reverse inconsistent guidance on failed trajectories. What do the authors consider to be FlowBalance's core technical increment over SGSD?