acceptodds
Under review as a conference paper at ICLR 2027

Beyond Final Outcomes: Process-Aware Relative Distribution Matching for LLM Reasoning

Abstract

Distribution matching has emerged as a promising paradigm for LLM reasoning by shaping probability distributions over complete trajectories to preserve diverse rewarding paths and thus improve reasoning performance. However, existing approaches remain largely trajectory-level, leaving two questions unresolved when incorporating intermediate reasoning information: where along a long reasoning trajectory should process information be measured, and how should it influence the distribution-matching objective? We first examine the former through counterfactual continuations. Building on high-entropy reasoning points as candidate decision boundaries, we find that alternative continuations from these forks exhibit greater variation in downstream verified-answer progress than those from neighboring non-fork positions, supporting their use as process-sensitive boundaries. We further observe substantial heterogeneity across segments: improving and deteriorating transitions coexist within both correct and incorrect trajectories, while a small subset of segments accounts for a disproportionate fraction of progress variation. These observations motivate distinct uses of intermediate progress in distribution matching. Progress magnitude controls credit allocation, reallocating optimization emphasis across segments while preserving the distribution-matching residual, whereas progress direction shapes the target distribution by adaptively tempering the reference-policy prior. Based on this design, we propose **P**rocess-**A**ware Relative **D**istribution **M**atching (PADM). Across trajectories sampled for the same prompt, a group-relative objective aligns relative policy probabilities with the resulting process-aware target. Theoretically, we show that exact relative matching recovers the intended target distribution, while magnitude-based segment weighting preserves this matching condition. Empirically, we evaluate PADM on six challenging mathematical reasoning benchmarks across two model scales, demonstrating consistent improvements over competitive baselines, with gains of up to 4.63 points in average accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Distribution Matching, Credit Assignment, Reinforcement Learning with Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wrSjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PADM, i.e., Process-Aware Relative Distribution Matching. It decouples the roles of intermediate progress magnitude and direction. It leverages progress magnitude to construct a residual-preserving gradient surrogate that reweights segment-level optimization, and utilizes progress direction to adaptively temper the reference policy to shape the target distribution. Meanwhile, it introduces a group-relative consistency objective across trajectories sampled for the same prompt to align policy probabilities with the target distribution. Experiments on six mathematical reasoning benchmarks demonstrate the effectiveness of PADM.

Critical strengths
  1. Through counterfactual continuation experiments, this work verifies the sensitivity of high-entropy forks to downstream answer confidence. It also highlights the coexistence of local improvements and deteriorations in both correct and incorrect trajectories, as well as the high concentration of progress in key segments.
  2. This work proposes PADM and designs a stop-gradient segment-weighting surrogate while preserving the distribution-matching residual. It also proves that the group-relative objective recovers the target distribution under exact zero matching error.
  3. Across both 1.7B and 4B model scales, PADM achieves performance gains on multiple benchmark datasets.
Critical weaknesses
  • Several existing works [1–4] have explored the same problem and proposed similar solutions, such as leveraging high-entropy tokens as reasoning forks for selective reinforcement learning, utilizing counterfactual theory for reasoning step evaluation, and measuring process progress via changes in the ground-truth answer log-likelihood to guide credit assignment. Compared to these methods, what are the unique advantages of PADM?

    [1] Beyond outcome reward: Decoupling search and answering improves LLM agents; [2] Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reas; [3] Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning; [4] Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing.

  • The definition of segment progress ΔCik\Delta C_{ik} relies on the policy model's log-likelihood evaluation of the ground-truth final answer. While this setup is viable in scenarios with deterministic, short numeric labels (such as math problems), tasks involving long-form solutions, code generation, or open-ended reasoning rarely provide a unique gold-standard y∗y^* string for direct forward log-probability computation. How does this method adapt to these broader settings? Confining applicability solely to mathematical scenarios is insufficient.

  • The paper selects the top 20% highest-entropy tokens along each trajectory with enforced minimum spacing as boundaries. However, reasoning models often produce non-semantic entropy spikes on formatting tokens or rare symbols; conversely, high-level shifts in algorithmic strategy can occur within continuous low-entropy spans. Directly equating local token entropy to a "reasoning decision boundary" lacks sufficient validation regarding its generalizability across different tasks and prompts.

  • The method introduces a downstream persistence term ∑j=k+1MiI[sign(ΔCik)(Cij−Ci,k−1)>0]\sum_{j=k+1}^{M_i} \mathbb{I}[\text{sign}(\Delta C_{ik})(C_{ij} - C_{i,k-1}) > 0]. During training, if a trajectory contains MM segments, does computing this term require running a complete forward pass targeting y∗y^* for every intermediate segment jj? Furthermore, the empirically constructed weighted gradient constitutes a non-conservative vector field, whose update direction may not strictly correspond to a scalar loss function.

  • At each training step, PADM not only generates rollouts but also dynamically calculates token entropies, filters top-20% forks, and computes ground-truth log-likelihoods and subsequent consistency terms for each segment. Compared to standard GRPO and FlowRL, what is the computational and memory overhead of PADM? Given that improvements on many datasets are less than 1%, what are the standard deviations and statistical characteristics to ensure the performance gains do not stem from randomness? Additionally, small models differ significantly from large models in reasoning divergence and entropy distributions; evaluating solely on models at or below 4B is insufficient to support the claimed effectiveness.

Questions for authors

Please see the weaknesses.

8 comments
AuthorsOpenReview ↗

Relationship to existing methods

Dear Reviewer wrSj, we appreciate your valuable comments and insightful feedback. We address each comment point by point below.

[W1] Relationship to existing methods.

We thank the reviewer for pointing out these works. In fact, our work differs fundamentally from these existing works in its distinct organization of process information within relative distribution matching. Following your comment, we have added [1,3,4] to the revised “Related Work” section and expanded the discussion of [2] to make these differences in task setting, supervision, and optimization explicit. In the following, we make a detailed comparison with these existing works and indicate the significant differences between our work and theirs.

[1] Beyond Outcome Reward (DeSA) addresses a different problem setting. DeSA addresses search-augmented question answering, where agents must learn to retrieve external evidence before answering. PADM's evaluated setting concerns verifiable mathematical reasoning without requiring external retrieval. Therefore, we study a different problem in process-aware reasoning optimization.

[2] Beyond the 80/20 Rule has a different optimization framework and usage of entropy. PADM uses entropy only to identify candidate segment boundaries, and optimization is not restricted to the selected fork tokens. Conversely, [2] restricts policy-gradient loss terms to selected high-entropy tokens. We study a process-aware relative distribution matching framework, whereas [2] uses a policy-gradient objective. We explicitly attribute the entropy-based fork insight to [2], which is already cited in our manuscript.

[3] GC2PO considers different optimization framework. Our counterfactual continuation experiments are diagnostic analyses of boundary sensitivity, and we do not construct counterfactual rewards. GC2PO incorporates counterfactual process rewards into token-level advantages and optimizes a GRPO-style clipped policy-gradient objective. PADM instead minimizes group-relative log-density matching for a process-aware target distribution, which is a framework-level distinction.

[4] Reason-KE++ has a different problem setting and framework. The task setting and optimization framework are both distinct. PADM addresses verifiable reasoning through relative distribution matching. In contrast, Reason-KE++ targets faithfulness to edited knowledge in multi-hop question answering and uses cold-start SFT followed by PPO with stage-aware rewards.

[1] Beyond outcome reward: Decoupling search and answering improves LLM agents (Wang et al., 2025).

[2] Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning (Wang et al., 2026).

[3] Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning (Wang et al., 2026).

[4] Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing (Wu et al., 2026).

AuthorsOpenReview ↗

Dependence on a verified reference answer

[W2] Dependence on a verified reference answer.

We appreciate your comment. In our current work, the central goal is to study how to achieve effective credit assignment across diverse reasoning paths leading to the same verified final answer. Mathematical reasoning is particularly well-suited to this problem because it combines verifiable final answers with multiple valid reasoning trajectories. This motivates our evaluation on six challenging mathematical benchmarks. Our current progress estimator requires a verified reference answer during training. This requirement concerns the reference-based progress estimator, and the distribution-matching framework itself does not require a unique answer string. We agree with the reviewer that the application of our current work to general reasoning tasks (e.g., with multiple answers in coding tasks) and non-verifiable reward settings is meaningful, but it is beyond the scope of the current work. Motivated by your comment, we provide a detailed discussion below to indicate a potential direction for extension and reflect this discussion in our updated manuscript in Section 4.1.

To illustrate, we provide a brief code generation example demonstrating a possible extension to tasks with multiple valid answers. Code generation admits functionally equivalent programs with different implementations. We therefore replace the single reference with a fixed set of test-passing programs and aggregate their length-normalized log-likelihoods using log-mean-exp:

\log\left( \frac{1}{G}\sum_{j=1}^{G} \exp\left( \frac{1}{|y_j^{\ast}|} \log\pi_\theta(y_j^{\ast}\mid x,z_{\leq k}) \right) \right), \qquad \Delta C_k=C_k-C_{k-1}.$$ Here, we use $G=4$, denoting the number of reference programs. Each problem's reference pool remains fixed across reasoning boundaries and rollouts. This illustration demonstrates the feasibility of constructing multi-reference coding progress. Validating its alignment with reasoning quality and its benefits for code generation remains future work. Tasks without trusted reference answers or reliable outcome verification require further design and validation and lie beyond the scope of the current study. We provide three examples from a seed-42 pool with DeepCoder coding data, with four reference programs generated by Qwen3-4B and four rollouts generated by Qwen3-1.7B-Base as below. More details are included in the revised manuscript. | **Problem** | **Outcome** | $\Delta C_1$ | $\Delta C_2$ | $\Delta C_3$ | $\Delta C_4$ | | :--- | :--- | ---: | ---: | ---: | ---: | | Pairwise bitwise-AND sum (5700-R4) | Pass | $-0.00084$ | $+0.01225$ | $-0.00243$ | $-0.02539$ | | Sequential substring reversals (6192-R2) | Fail | $+0.00204$ | $+0.00724$ | $-0.00388$ | — | | Date difference (6228-R3) | Fail | $+0.01725$ | $+0.00489$ | $-0.01839$ | — |
AuthorsOpenReview ↗

Entropy-based boundaries and the 20% threshold

[W3] Entropy-based boundaries and the 20% threshold.

We appreciate the reviewer's comments. As stated in our paper, our 20% selection ratio and the definition of reasoning forks follow the cited Beyond the 80/20 Rule [1]. Their ablation report (Figure 6) states that retaining the top-20% highest-entropy tokens provides an effective balance between exploration and training stability. We therefore adopt this empirically motivated default. The 20% ratio is not a contribution or a universally optimal choice claimed by PADM, and our segment-level optimization is not restricted to these selected tokens.

More importantly, our ablations support the empirical utility of this partitioning heuristic in our setting. Table 2 compares entropy-based partitioning with uniform and random partitioning under matched segment counts. Entropy-based partitioning achieves the highest average accuracy, supporting its effectiveness for the PADM algorithm. Our counterfactual continuation analysis (Observation I) provides complementary evidence of sensitivity to downstream answer confidence. Nevertheless, we agree that entropy is an imperfect proxy. These experiments establish practical effectiveness rather than semantic correctness or completeness of every boundary. Following your suggestion, we have revised the wording in Section 4, “Effectiveness of decision-aware segmentation,” to describe these positions as candidate process-sensitive boundaries and avoid equating them with semantically partitioned reasoning steps.

[1] Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning (Wang et al., 2026).

Reviewer_NFZqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PADM augments trajectory distribution matching with intermediate progress signals. It partitions reasoning trajectories at high-entropy positions and measures progress through changes in the length-normalized log probability of the verified answer. Progress magnitude controls how strongly segments contribute to optimization, while progress direction modifies the reference-based target. Experiments on two Qwen3 model sizes and six mathematical benchmarks show higher average accuracy than the evaluated baselines, with ablations supporting the proposed components.

Critical strengths
  • S1. Separating the strength of an update from the construction of its target is a clear and interesting way to incorporate process information into distribution matching.

  • S2. The paper connects its method to observations about where progress changes occur, and includes relevant comparisons with FlowRL and GRPO-VPS rather than only an outcome-reward baseline.

  • S3. The component and partitioning ablations provide useful evidence that the full design matters. The fixed-target analysis also clarifies what the relative matching objective is intended to recover.

Critical weaknesses
  • W1. The connection between answer-confidence changes and reasoning quality lacks independent validation. The paper defines ΔC\Delta C as a change in length-normalized ground-truth-answer log probability and explicitly treats it as a proxy. The unresolved issue is how reliably this proxy supports the process-level interpretation used in credit allocation and target shaping. Flawed reasoning can increase answer confidence, while useful exploration may temporarily reduce it. The appendix acknowledges that confidence gains and declines are not step-correctness labels; the observations and ablations establish usefulness for training, but do not measure agreement with independently assessed reasoning quality. A small quality assessment would clarify when the productive/unproductive interpretation is warranted.

  • W2. The incremental value of the proposed magnitude/direction organization is only partly established relative to the closest process-aware distribution-matching approaches. The comparisons with FlowRL and GRPO-VPS are relevant, and the −-Magnitude and −-Direction ablations already demonstrate that both proposed components contribute under the current design. Therefore, the issue is not whether these components are useful in isolation, but how much the particular organization of process information improves over alternative process-aware matching formulations. The related-work section identifies Younsi et al. (2025), Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets, as the most closely related process-aware distribution-matching approach. The paper distinguishes it mainly by noting that Younsi et al. use an explicit process reward model, whereas PADM conditions matching directly on confidence-based progress. However, no empirical comparison is provided. A representative comparison would strengthen the contribution; if a direct comparison is not meaningful because of differences in supervision, computation, objectives, or task assumptions, a more explicit explanation of those incompatibilities would also be sufficient. The fact that some ingredients have precedents does not by itself weaken the contribution, but the relative advantage of this specific organization should be positioned more clearly.

Questions for authors
  • Q1. Could a small independent assessment test how the sign and magnitude of ΔC\Delta C relate to reasoning quality? Human annotations or a suitably checked judge could examine confidence increases following flawed steps and confidence decreases during useful exploration, helping delimit the productive/unproductive interpretation rather than redefining the already specified signal.

  • Q2. The −Direction ablation already supplies a weighting-only control and supports the contribution of reference tempering. Is it intended as the representative simpler process-aware matching baseline? If an additional simplification is useful, could it specifically test removing the persistence factor and retaining only normalized absolute-progress weights with the reference target fixed, rather than repeating the existing removal of tempering?

  • Q3. Appendix A identifies Younsi et al. (2025), Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets, as particularly close. Could the authors explain why this approach was not compared, and either provide a representative comparison or specify the supervision, objective, or task differences that make one inappropriate? Any comparison should account for the PRM's additional supervision and computation.

  • Q4. Do the authors have results beyond mathematics, for example in code generation? If not, what limits or adaptations should readers expect when applying this answer-confidence signal outside mathematical reasoning?

5 comments
AuthorsOpenReview ↗

Evidence supporting the confidence-based progress signal

Dear Reviewer NFZq, we sincerely appreciate your suggestive feedback and comments, and we hope the following responses address your concerns.

[W1&Q1] Evidence supporting the confidence-based progress signal.

We thank the reviewer for raising this point. In our work, we mentioned that our choice of confidence-based progress signal is motivated by GRPO-VPS [1]. In their paper, they provide empirical support for the underlying confidence in Section 4.4, and Table 3 evaluates the progress signal against PRM800K human step annotations. The prior validation therefore supports this underlying signal family specifically for mathematical reasoning. In addition, our experiments demonstrate empirical success using confidence as a proxy within the distribution-matching framework. Following your suggestion, we have added further explanation in Section 3.1 and clarified with “confidence-increasing/decreasing” terminology where appropriate in the revised manuscript.

[1] Grpo-vps: Enhancing group relative policy optimization with verifiable process supervision for effective reasoning (Wang et al., 2026).

AuthorsOpenReview ↗

Relationship to PRM-guided GFlowNets

[W2&Q3] Relationship to PRM-guided GFlowNets.

Thank you for pointing out this work. Following your suggestion, we provide a more detailed discussion to compare our work with PRM-guided GFlowNets [1] in the updated manuscript “Related Works” section. Our algorithm's relative advantage is incorporating process information without a separate process-labeling and PRM-training pipeline. [1] generates process labels through MCTS and similarity-based augmentation, trains an auxiliary PRM, and constructs rewards from products of PRM step scores for step-level subtrajectory balance (SubTB) optimization. PADM instead combines verified terminal outcomes with policy-derived progress: magnitude reweights segment-level gradient contributions while preserving the matched distribution, and direction adaptively tempers the reference-based target. This organization allows intermediate progress to influence both optimization emphasis and the distribution being matched, and our component ablations support the contribution of each role. PADM therefore avoids MCTS-based process-label generation, separate PRM training, and auxiliary PRM inference.

Therefore, a full comparison is not feasible due to the joint variation in reward construction, optimization procedure, and objective, and would require matched computational accounting that includes both the PRM pipeline and PADM's answer-scoring cost. Our current baselines and ablations isolate PADM's contribution within the setting without a separately trained PRM.

[1] Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets (Younsi et al., 2025).

AuthorsOpenReview ↗

A simpler process-aware matching control

[Q2] A simpler process-aware matching control.

We thank the reviewer for suggesting that the existing − Direction variant provides a weighting-only control. Its comparison with full PADM is intended to isolate the contribution of direction-based reference tempering. Following your suggestion, we evaluate a further-simplified control that fixes the reference exponent, removes the persistence factor, and retains only normalized absolute-progress weights. We follow the evaluation protocol in Table 1: Avg@1 for MATH500, Minerva Math, and OlympiadBench; Avg@32 for AIME24, AIME25, and AMC23. The simplified control achieves an average accuracy of 27.25%, compared with 30.02% for − Direction and 30.87% for full PADM. Its comparison with − Direction supports the additional value of persistence-aware weighting when reference tempering is disabled. Together, these comparisons strengthen the evidence for PADM's specific organization of process information beyond basic progress-based reweighting. We have included this additional control in the revised manuscript, as shown in Table 2.

MethodMATH500 (Avg@1)Minerva Math (Avg@1)OlympiadBench (Avg@1)AIME24 (Avg@32)AIME25 (Avg@32)AMC23 (Avg@32)Avg
PADM67.6030.1528.6410.316.9841.5630.87
− Magnitude66.0022.0628.3410.626.3540.9429.05
− Direction65.6030.5131.168.854.1739.8430.02
∣ΔCik∣\lvert\Delta C_{ik}\rvert only62.4025.0024.786.676.1538.5227.25
Reviewer_MU6AOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PADM measures progress in each reasoning segment through changes in the correct answer's log-likelihood and uses process-aware relative distribution matching to adjust both segment-level training emphasis and the model's preferences over complete reasoning trajectories.

Critical strengths

1.Progress magnitude controls segment-level training emphasis, while progress direction adjusts which reasoning trajectories the model should favor, allowing these two roles to be controlled separately without training an additional process reward model. 2. Segmenting at high-entropy positions, where the model is most uncertain, yields higher average accuracy than random or uniform segmentation with the same segment count and minimum spacing, supporting the value of the chosen boundaries.

Critical weaknesses

1.The two models exceed the strongest baseline in average accuracy by 1.65 and 1.14 percentage points, respectively, but results across different training random seeds are not reported, so the stability of these gains remains unclear. 2.Table 2 removes magnitude weighting and direction-based adjustment separately but not jointly, leaving the performance of relative distribution matching without process information unknown.

Questions for authors

1.Across different training random seeds, what are PADM's mean accuracy gain over the strongest baseline and the range of variation in that gain? 2.How does accuracy change when both the magnitude weight and the exponent applied to the reference distribution are set to 1, disabling both forms of process-based adjustment? 3.Does PADM still outperform the baselines under the same GPU training-time budget after accounting for the additional answer-scoring time?

5 comments
AuthorsOpenReview ↗

Multiple training seeds

Dear Reviewer MU6A, we are grateful for your constructive feedback and insightful comments. We address all of your comments point by point below.

[W1&Q1] Multiple training seeds.

We thank the reviewer for raising the question of training-seed variability. In fact, we follow the convention to report the results of a single run. This is also routine in many popular works and our baselines, including GRPO [1], DAPO [4], as well as the closely related baselines FlowRL [2] and GRPO-VPS [3]. Within the limits of our computational resources, we have tried our best to conduct as many experiments as possible. Repeating complete RL training runs across multiple seeds incurs substantial additional computational costs and is not typically required by convention. Within our computational budget, we prioritized computational resources for PADM across two model sizes (1.7B and 4B) and six mathematical benchmarks comparisons to four baselines, covering the popular GRPO and closely related baselines FlowRL and GRPO-VPS, along with many ablation studies of process-aware components and segmentation. We believe these results provide complementary evidence of PADM's effectiveness across the evaluated configurations. We hope the reviewer understands and agrees with us on this point.

[1] Deepseekmath: Pushing the limits of mathematical reasoning in open language models (Shao et al., 2024).

[2] FlowRL: Matching reward distributions for LLM reasoning (Zhu et al., 2025).

[3] Grpo-vps: Enhancing group relative policy optimization with verifiable process supervision for effective reasoning (Wang et al., 2026).

[4] Dapo: An open-source llm reinforcement learning system at scale (Yu et al., 2025).

AuthorsOpenReview ↗

Joint ablation

[W2&Q2] Joint ablation.

We thank the reviewer for suggesting this joint ablation. Following your suggestion, we add a − Both variant that sets wik=1w_{ik}=1 and γik=1\gamma_{ik}=1, simultaneously disabling magnitude-based weighting and direction-based reference tempering while retaining the group-relative matching objective. As shown below, jointly removing both adjustments reduces the average accuracy from 30.87% to 27.39%. Full PADM improves accuracy on five benchmarks and matches this control on AIME24. The − Magnitude and − Direction variants achieve averages of 29.05% and 30.02%, respectively. Together, these results support the contributions to both process-based components and establish their combined benefit over relative matching without process-based adjustment. We have added this joint ablation study to the revised manuscript in Table 2.

MethodMATH500 (Avg@1)Minerva Math (Avg@1)OlympiadBench (Avg@1)AIME24 (Avg@32)AIME25 (Avg@32)AMC23 (Avg@32)Avg
PADM67.6030.1528.6410.316.9841.5630.87
− Magnitude66.0022.0628.3410.626.3540.9429.05
− Direction65.6030.5131.168.854.1739.8430.02
− Both62.6022.0626.7110.316.0436.6427.39
AuthorsOpenReview ↗

Accuracy under the same GPU training-time budget

[Q3] Accuracy under the same GPU training-time budget.

We thank the reviewer for pointing this out! Following your suggestion, we conducted an additional comparison under a common GPU-hour budget, using the same six-benchmark evaluation protocol as in Table 1: Avg@1 for MATH500, Minerva Math, and OlympiadBench, and Avg@32 for AIME24, AIME25, and AMC23. We set common budget caps of 130 GPU-hours for 1.7B and 140 GPU-hours for 4B, selecting the latest available checkpoint within each cap based on training cost. We include answer scoring in the training cost accounting for PADM. Within the stated caps, PADM achieves average accuracies of 30.28% and 41.33% for 1.7B and 4B, respectively, exceeding GRPO by 2.88 and 2.28 points, FlowRL by 0.69 and 2.10 points, and GRPO-VPS by 1.55 and 4.35 points. Thus, PADM's gains over these three baselines persist after accounting for answer-scoring costs, especially for the larger-scale 4B model. We revised our manuscript and incorporated the results of controlled experiments in Section 4 paragraph Computation Budget Analysis and Table 3.

ModelMethodGPU-hoursAvg
Qwen3-1.7B-BaseGRPO129.1127.40
Qwen3-1.7B-BaseFlowRL126.2229.59
Qwen3-1.7B-BaseGRPO-VPS128.6628.73
Qwen3-1.7B-BasePADM127.0830.28
Qwen3-4B-BaseGRPO124.8839.05
Qwen3-4B-BaseFlowRL129.4039.23
Qwen3-4B-BaseGRPO-VPS137.336.98
Qwen3-4B-BasePADM128.7641.33

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.