acceptodds
Under review as a conference paper at ICLR 2027

Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectories

Abstract

Foundation models exhibit broad knowledge but limited task-specific reasoning, motivating post-training strategies such as RL with verifiable rewards (RLVR) and test-time scaling (TTS). While recent work highlights the role of exploration in improving pass@K, empirical evidence points to a paradox: RLVR and ORM/PRM typically reinforce existing paths rather than expanding the reasoning scope, raising the question of why exploration helps if no new patterns emerge. To reconcile this paradox, we adopt the perspective of Kim et al. (2025), viewing easy (e.g., simplifying a fraction) versus hard (e.g., discovering the some symmetry) reasoning steps as low versus high probability Markov transitions. In this tractable model, pretraining corresponds to tree-graph discovering, while post-training corresponds to CoT reweighting. We provably show that, both RLVR and ORM/PRM would favor heavily to several high-probability paths, and thereby forget rare-but-crucial CoTs. Building on this, we further prove that exploration strategies such as rejecting easy instances and KL regularization help preserve rare CoTs. Empirical simulations corroborate our theoretical results.

Primary area:
learning theory
Keywords:
Post-Training, Squeezing Effect, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_g9sUOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies distributional biases in reasoning post-training and inference scaling using a Markov-chain framework. It theoretically analyzes the tendency of RLVR and reward-guided inference to favor easy reasoning trajectories and proposes a Doob hh-transform-based process reward.

Critical strengths

The paper addresses an interesting question about reasoning diversity in post-training. The unified theoretical framework is appealing and provides an interesting perspective on the role of exploration in preserving rare reasoning trajectories.

Critical weaknesses

The main conclusion appears to depend heavily on the modeling assumptions.

Definition 2.2(iii) assumes that the probability of a valid CoT being correct is proportional to its likelihood under the base model. This directly implies that easy CoTs have higher expected rewards than hard CoTs.

Consequently, the tendency of reward-maximizing algorithms to favor easy CoTs seems to be partly built into the assumptions rather than emerging independently from the analysis.

While this may be reasonable for a simplified theoretical model, it is unclear whether the main conclusions would hold without this assumption.

Questions for authors

See weakness.

Reviewer_LphgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies distributional biases in LLM post-training through a multi-task tree-structured Markov chain framework. The authors show that RLVR and reward-guided inference can favor high-probability reasoning trajectories while suppressing rare but useful ones. They further analyze how easy-instance rejection, KL regularization, and a Doob h-transform-based process reward model (DPRM) can preserve reasoning diversity.

Critical strengths
  • First of all, the paper addresses an important question about why exploration benefits reasoning models even when post-training primarily reinforces existing reasoning trajectories.

  • What's more, I think the framework analyzes several RLVR and inference-time scaling methods under common assumptions, providing useful insights into their shared preference for high-probability reasoning paths.

  • The paper also has good theoretical contribution as the proposed DPRM uses the Doob h-transform to connect step-wise sampling with Gibbs-reweighted trajectory distributions, offering an interesting theoretical perspective on process reward design.

Critical weaknesses
  • There seems to be a potential mathematical issue in Corollary 4.5. The authors claim that DPRM and likelihood-based PRM are equivalent under BoN/BS for any positive temperature. However, Eq. (11) defines the likelihood-based PRM as

    Rlikelihoodk(ol)=E[Rk(o)∣ol],R^k_{\mathrm{likelihood}}(o_l) = \mathbb{E}[R^k(o) \mid o_l],

    whereas Eq. (14) defines DPRM as

    RDPRMk(ol)=1λlog⁡E[exp⁡(λRoutk(o))∣ol].R^k_{\mathrm{DPRM}}(o_l) = \frac{1}{\lambda}\log\mathbb{E}[\exp(\lambda R^k_{\mathrm{out}}(o)) \mid o_l].

    The proof of Corollary 4.5 argues that these two rewards induce the same ranking due to the monotonicity of exp and log. However, it seems that this argument does not hold in general. Although the exponential function is strictly increasing, it is also convex. Thus, while x>yx > y implies ex>eye^x > e^y, the inequality E[X]>E[Y]E[X] > E[Y] does not necessarily imply E[eX]>E[eY]E[e^X] > E[e^Y]. (correct me if I'm missing anything here!)

    Therefore, the claimed equivalence under BoN/BS does not follow from the stated monotonicity argument. Could the authors clarify under what additional assumptions Corollary 4.5 holds?

  • In addition, I think the modeling assumptions could be too strong. Definition 2.2 assumes that the correctness probability of a valid CoT is proportional to its likelihood. This assumption plays a central role in deriving the simplicity bias. While I thank the authors for providing motivation in Appendix E, it remains unclear whether the main conclusions hold under weaker assumptions, particularly when correctness and likelihood are not positively correlated.

  • Finally, the experiments can be further enhanced. The experiments are restricted to a small synthetic Markov chain with four layers and two states per layer, without evaluation on actual language models. Experiments across different chain sizes and branching factors, together with small-scale LLM validation, would strengthen the empirical support for the theoretical findings.

Questions for authors

Please refer to my summary of weaknesses.

Reviewer_qA5uOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies why post-training and reward-guided inference may favor common reasoning paths at the expense of rare ones. The authors use a multi-task Markov chain with a tabular softmax policy. They analyze reinforcement learning and rejection-sampling fine-tuning alongside search guided by outcome and process reward models.

The paper argues that rejecting easy instances, adding KL regularization, and using Gibbs sampling can preserve rare paths. It also introduces DPRM, a process reward formulation based on Doob's h-transform. Experiments use two synthetic tasks on an eight-state graph.

Critical strengths
  1. The paper asks a useful question: can improving average success come at the cost of losing rare solutions? Its model distinguishes a method that often works from one that is correct for a particular instance.

  2. The shared framework allows a direct comparison of training and inference strategies. Including a second task also makes it possible to study forgetting.

Critical weaknesses
  1. The DPRM proof needs a clearer state definition. Definition 4.4 uses h(o_l), which depends only on the current node. Yet the TASK1 paths in Table 1 share an endpoint and have different rewards. The boundary condition h(o_L) = exp(lambda R_out(o)) in Appendix H cannot hold for all of them. Corollary 4.5 has a separate gap: monotonicity does not imply that E[R | s] and log E[exp(lambda R) | s] rank states identically. The population reward here is not binary. Both points need to be resolved to establish the claimed equivalences.

  2. The RAFT gradient appears to omit cross-step terms. Equation (4) uses the current policy for both sampling and the log-likelihood objective. Differentiating it introduces terms involving log probabilities at other steps. Equation (8) keeps only the local term. The parameter-isolation argument in Appendix I does not explain their removal. Standard RAFT instead holds sampled data fixed during the supervised update. The authors need to specify which update they analyze and check whether the squeezing theorem holds for it.

  3. The claims about post-training are broader than the evidence supports. The experiments use a linear softmax model on an eight-state graph with two synthetic tasks. Definition E.2 also assumes that average path correctness is proportional to initial likelihood. This partly builds the preference for common paths into the setting. The authors acknowledge the abstraction's limitations, but the abstract and introduction still suggest a general explanation of LLM behavior. The claims should be narrowed or supported by analysis or experiments that relax these assumptions.

  4. Table 2 is difficult to reconcile with the sampling rule. With lambda = 0.5 and rewards in [0,1], Equation (12) allows an event to become at most exp(0.5) times as likely. Using the reported TASK2 base hard-path fraction of 1.10% gives an upper bound of about 1.81%. Soft-BoN instead reports 17.27%. This comparison assumes the same reference distribution and normalization. Please clarify which settings account for the difference.

  5. The abstract needs a clearer introduction to the problem. ORM, PRM, and CoTs appear without definitions. Terms such as “tree-graph discovering” add difficulty before the central question is clear. The abstract should explain what exploration means in this setting and state the main finding in plain language. The notation should also distinguish individual nodes from complete prefixes.

Questions for authors
  1. Does h condition on the current node or the full prefix? What additional assumption establishes the ranking equivalence in Corollary 4.5?

  2. Is the RAFT sampling distribution held fixed during each update? How does the squeezing proof change under the corresponding gradient?

  3. How was the Soft-BoN row in Table 2 computed? Please reconcile its hard-path probability with Equation (12).

  4. Which results still hold when path correctness is not proportional to initial likelihood or when the scorer observes the concrete question?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.