- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper propose the Attention Process Reward Model (APRM). The APRM is built upon an Attention-Fused Concept Pooler and an Edge-Biased Graph Attention Network. APRM directly evaluates hidden states and attention maps from the reasoning ´process to produce internal process rewards. We use APRM in two stages. During cold-start, APRM uses positive and negative reasoning trajectories with ranking and counterfactual losses to learn preferences over internal reasoning processes. During RL, APRM reshapes the task reward so that policy optimization is guided by both outcomes and internal attention quality.
Critical strengths
- the paper is relevant and the work is suitable for this conference
- the writing is mostly clear , and the goal of the paper is clearly defined
Critical weaknesses
- missing a problem formulation setting, confusing for the reader
- Figure 4/5 the labels in the figures are not legible
- the avg@8 gains are hard to tell if they are real gains, as some are <1%, need error bars
- the method has a lot of additional hyperparameters, e.g., ranking margin, counterfactual margin, β , and needs periodic training of APRM
- from table 3, beta seems sensitive and can make the advantage of using APRM vanish. It is also not clear if this set hyperparameter transfers across datasets, and models (sizes)
Questions for authors
- the pass@8 has more noticeable gains, do you have intuition why? possibly more diverse samples, but why from these scores? is the attention score related to entropy bonus?
- any intuition on how to choose all the hyperparameters? are they sensitive?
- why does GRPO increased token usage alot for Qwen3-1.7b-Base?
- table 3, what model is that?
5 comments
Response to Weakness 5 and Question 1&2
W5 & Q2
To further address this concern, we added experiments across model types, RL algorithms, and datasets. Specifically, we evaluated APRM on Qwen3.5-2B(instruct) with the same β value, and combined it with DAPO, a different RL algorithm. This setting covers an instruct/base model comparison, a different attention configuration, a different model scale, a different task domain, and a different RL algorithm. The results are as follows:
A. Experiment 1 (Cross-Model Validation): Qwen3.5-2B(instruct), mathematical reasoning, β=0.3
| Method | AIME Avg@8 / Pass@8 | AMC Avg@8 / Pass@8 | MATH500 Avg@8 / Pass@8 | Olympiad Avg@8 / Pass@8 | Average Avg@8 / Pass@8 |
|---|---|---|---|---|---|
| Qwen3.5-2B-Instruct | 10.42 / 20.00 | 50.31 / 77.50 | 68.23 / 85.20 | 39.32 / 61.28 | 42.07 / 61.00 |
| GRPO | 20.83 / 33.33 | 66.25 / 85.00 | 75.78 / 87.60 | 48.50 / 64.39 | 52.84 / 67.58 |
| DAPO | 21.67 / 33.33 | 72.50 / 90.00 | 79.35 / 88.60 | 52.36 / 66.91 | 56.47 / 69.71 |
| DAPO+APRM | 25.00 / 43.33 | 70.00 / 95.00 | 80.45 / 89.60 | 55.80 / 70.03 | 57.81 / 74.49 |
| Δ DAPO+APRM vs. DAPO | +3.33 / +10.00 | -2.50 / +5.00 | +1.10 / +1.00 | +3.44 / +3.12 | +1.34 / +4.78 |
B. Experiment 2 (Cross-Task Validation): Qwen3.5-2B(instruct), code generation, β=0.3 and β=0.2
| Method | LCB v5 Avg@8 / Pass@8 | HumanEval Avg@8 / Pass@8 | Average Avg@8 / Pass@8 |
|---|---|---|---|
| Qwen3.5-2B-Instruct | 10.08 / 21.15 | 54.88 / 79.88 | 32.48 / 50.51 |
| DAPO | 14.02 / 23.30 | 61.59 / 83.54 | 37.80 / 53.42 |
| DAPO+APRM, β=0.3 | 14.07 / 24.37 | 62.27 / 84.15 | 38.17 / 54.26 |
| DAPO+APRM, β=0.2 | 16.08 / 26.16 | 62.80 / 86.59 | 39.44 / 56.38 |
| Δ β=0.2 vs. DAPO | +2.06 / +2.87 | +1.22 / +3.05 | +1.64 / +2.96 |
Under the same setting, DAPO+APRM consistently improves the overall performance over DAPO across both mathematical reasoning and code generation. Specifically, it yields gains of 1.34 Avg@8 and 4.78 Pass@8 points on mathematical benchmarks, and 0.37 Avg@8 and 0.84 Pass@8 points on code benchmarks. These results suggest that provides a reasonably transferable and effective starting point across different task domains and RL settings. However, it does not necessarily represent the optimal coefficient for every task: in code generation, reducing to 0.2 leads to further improvements of 1.64 Avg@8 and 2.96 Pass@8 points over DAPO. Given the computational cost of RL experiments, we adopt a practical strategy of tuning in one setting and transferring it to others. We therefore view as a robust initial recommendation, while allowing further validation-based tuning when necessary.
Q1
APRM can affect Pass@k more than Avg@k because its marginal effect on Pass@k factorizes into an APRM-alignment term and a Pass@k sensitivity term:
Here is the per-sample success probability, is the group-centered APRM score, and denotes whether a trajectory has current or future success potential. The first factor, , captures whether APRM prefers more promising reasoning trajectories. The second factor, , is largest when is small, so the same APRM signal is amplified more on hard or boundary problems. On easy problems, is already informative because correct and incorrect rollouts coexist. On hard problems, most rollouts may fail, making outcome-based advantages nearly flat. APRM can still rank failed rollouts by internal process quality and move probability mass toward trajectories closer to success, turning some 0/k cases into at least 1/k successful cases. This differs from an entropy bonus. APRM does not directly maximize output diversity; it reweights rollouts within the same prompt by internal process quality. Thus, improved diversity or coverage is an indirect consequence of process-quality reweighting.
Response to Question 3&4 and Additional Derivation for Q1
Q3
Your question is very insightful! GRPO evaluates final-answer correctness without explicitly penalizing redundant reasoning. For a relatively small base model, longer trajectories may provide additional opportunities for exploration and verification. When these trajectories produce correct answers, their generation patterns are reinforced, potentially leading to unnecessary length expansion.
In contrast, APRM does not directly reward longer or shorter outputs; its effect depends on the correlation between response length and process quality within each prompt group. Under the same local policy-reweighting approximation, its effect on expected response length is:
A positive correlation favors longer trajectories when they exhibit higher process quality, whereas a negative correlation suppresses unproductive length expansion. Thus, APRM promotes reasoning quality rather than response length, allowing useful long reasoning while discouraging redundant deliberation.
In GRPO experiments, the optimization process tends to reinforce longer trajectories when they lead to correct answers. APRM further selects trajectories based on their internal process quality. Consequently, if longer trajectories developed during GRPO exhibit relatively higher reasoning quality, combining GRPO with APRM can reinforce these trajectories, acting as a form of implicit regularization that dynamically allocates more reasoning budget to productive exploration.
Interestingly, this behavior differs across RL algorithms. Unlike GRPO, GSPO/GPG does not exhibit the same pronounced length-growth tendency in our experiments. In this setting, APRM primarily reallocates reasoning budget according to task complexity: it favors more extensive reasoning on challenging problems such as AIME, where longer trajectories can provide greater benefits, while favoring more concise reasoning on relatively simpler benchmarks such as MATH500.
This suggests that APRM does not simply increase or decrease response length; rather, it dynamically allocates reasoning budget according to the quality of internal reasoning and the complexity of the task.
Q4
We have add the model name in the table caption to Qwen3-1.7B-Base.
Additional Derivation for Q1
For a problem , let , where indicates correctness. Then
Therefore,
This term is large for hard problems with small , and small for already-solved problems.
APRM modifies the advantage as
where is the mean APRM score within the rollout group of prompt . Under a local policy-reweighting approximation,
Thus the APRM-induced change in per-sample success probability is
Combining the two derivatives gives
Thank you again for your careful review and valuable suggestions. By incorporating your feedback, we have improved the quality of the paper. If you have any remaining concerns, we would be happy to continue the discussion.
Response to Weakness 1,2,3,4
Thank you for recognizing the paper’s fit to the conference, clarity of presentation, and well-defined goal. Your insightful comments are very important to us, and we hope the following point-by-point responses address your concerns.
In the initial submission, due to strict space constraints, we prioritized including as much substantive content as possible in the main text. As a result, some figures were relatively small and several descriptions were concise. In the revision, we make full use of the additional one-page allowance to enlarge the figures and improve readability.
We use avg@8 precisely to reduce the impact of randomness. Representative methods such as GRPO[1], DAPO[2], GSPO[3], and GPG[4] typically report fixed-seed benchmark metrics (e.g., Accuracy, Pass@K, Avg@K) rather than multi-seed confidence intervals. Our evaluation protocol therefore follows the common practice for large-scale LLM-RL experiments.
To address the concern about statistical robustness, we treat the eight sampled generations as repeated evaluations. For each generation index j, we compute accuracy over all prompts and report the mean and standard deviation across the eight generation-level accuracies.
For Pass@8, we use McNemar’s paired test over the pooled 1,244 prompts. APRM significantly improves Qwen1.7B-GRPO from 59.57% to 61.90% (+2.33 pp; 74 APRM-only solved prompts vs. 45 baseline-only solved prompts; McNemar p=0.0103). APRM also significantly improves Qwen4B-GRPO from 66.72% to 69.13% (+2.41 pp; 64 APRM-only solved prompts vs. 34 baseline-only solved prompts; McNemar p=0.0034)
[1]DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
[2]DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025
[3]Group Sequence Policy Optimization
[4]GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning, ICLR 2026
The core contribution of this paper is to introduce a new reward-modeling paradigm and demonstrate its effectiveness. Since APRM is designed to reward attention-derived internal reasoning signals, it naturally introduces several additional hyperparameters. We have provided architecture-related ablations in Appendix D.3 and D.4, the ablation of β in Table 3, and training dynamics in Figures 7 and 11.
We do not claim that the selected hyperparameter configuration is optimal. Rather, our goal is to show that APRM yields consistent improvements under a fixed and reasonably chosen setting. Other configurations may further improve performance, and we view this as an important direction for future exploration, including alternative APRM architectures and different update/training strategies. We believe this opens a valuable research space for the community.