acceptodds
Under review as a conference paper at ICLR 2027

From Rewarding What Models Say to How They Think: Attention Process Reward Model for LLM Reinforcement Learning

Abstract

Reinforcement learning (RL) has become an important component in large language models (LLMs). However, existing RL approaches focus on final outputs or textual solution trajectories, leading to coarse-grained rewards and unreliable reasoning. Such paradigms focus mainly on what the model outputs while overlooking how the model internally reasons. Therefore, can we reward how a model thinks internally rather than merely what it says? We first reveal the problem of textual reward aliasing in existing reward modeling approaches. Motivated by this, we propose the Attention Process Reward Model (APRM), which assigns internal rewards based on hidden states and attention maps to capture high-quality reasoning patterns. With only approximately 46M parameters, APRM is trained using ranking and counterfactual objectives, and its reward guides RL optimization while being periodically updated to handle distribution shifts. Integrated with representative RL methods, APRM achieved an average pass@8 gain of 2.88%. Our results demonstrate the promise of incorporating internal reasoning signals into RL for LLMs. More details in anonymous repository  https://anonymous.4open.science/r/From-Rewarding-What-Models-Say-to-How-They-Think-6B27

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Reward Model, Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XZchOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose the Attention Process Reward Model (APRM). The APRM is built upon an Attention-Fused Concept Pooler and an Edge-Biased Graph Attention Network. APRM directly evaluates hidden states and attention maps from the reasoning ´process to produce internal process rewards. We use APRM in two stages. During cold-start, APRM uses positive and negative reasoning trajectories with ranking and counterfactual losses to learn preferences over internal reasoning processes. During RL, APRM reshapes the task reward so that policy optimization is guided by both outcomes and internal attention quality.

Critical strengths
  • the paper is relevant and the work is suitable for this conference
  • the writing is mostly clear , and the goal of the paper is clearly defined
Critical weaknesses
  • missing a problem formulation setting, confusing for the reader
  • Figure 4/5 the labels in the figures are not legible
  • the avg@8 gains are hard to tell if they are real gains, as some are <1%, need error bars
  • the method has a lot of additional hyperparameters, e.g., ranking margin, counterfactual margin, β , and needs periodic training of APRM
  • from table 3, beta seems sensitive and can make the advantage of using APRM vanish. It is also not clear if this set hyperparameter transfers across datasets, and models (sizes)
Questions for authors
  • the pass@8 has more noticeable gains, do you have intuition why? possibly more diverse samples, but why from these scores? is the attention score related to entropy bonus?
  • any intuition on how to choose all the hyperparameters? are they sensitive?
  • why does GRPO increased token usage alot for Qwen3-1.7b-Base?
  • table 3, what model is that?
5 comments
AuthorsOpenReview ↗

Response to Weakness 1,2,3,4

Thank you for recognizing the paper’s fit to the conference, clarity of presentation, and well-defined goal. Your insightful comments are very important to us, and we hope the following point-by-point responses address your concerns.

W1 & W2

In the initial submission, due to strict space constraints, we prioritized including as much substantive content as possible in the main text. As a result, some figures were relatively small and several descriptions were concise. In the revision, we make full use of the additional one-page allowance to enlarge the figures and improve readability.

W3

We use avg@8 precisely to reduce the impact of randomness. Representative methods such as GRPO[1], DAPO[2], GSPO[3], and GPG[4] typically report fixed-seed benchmark metrics (e.g., Accuracy, Pass@K, Avg@K) rather than multi-seed confidence intervals. Our evaluation protocol therefore follows the common practice for large-scale LLM-RL experiments.

To address the concern about statistical robustness, we treat the eight sampled generations as repeated evaluations. For each generation index jj, we compute accuracy over all prompts and report the mean and standard deviation across the eight generation-level accuracies.

ModelDatasetBaseline Avg@8 ± stdAPRM Avg@8 ± stdΔ Avg@8
Qwen1.7B-GRPOAIME5.00 ± 4.366.25 ± 5.76+1.25 pp
Qwen1.7B-GRPOAMC2339.69 ± 4.9042.50 ± 6.81+2.81 pp
Qwen1.7B-GRPOMATH50061.42 ± 1.7961.82 ± 0.91+0.40 pp
Qwen1.7B-GRPOOlympiadBench27.32 ± 1.0428.62 ± 1.50+1.30 pp
Qwen4B-GRPOAIME10.00 ± 5.6311.67 ± 3.56+1.67 pp
Qwen4B-GRPOAMC2357.81 ± 4.3256.88 ± 2.59-0.94 pp
Qwen4B-GRPOMATH50072.65 ± 0.5573.12 ± 0.72+0.47 pp
Qwen4B-GRPOOlympiadBench41.36 ± 0.5341.12 ± 0.52-0.24 pp

For Pass@8, we use McNemar’s paired test over the pooled 1,244 prompts. APRM significantly improves Qwen1.7B-GRPO from 59.57% to 61.90% (+2.33 pp; 74 APRM-only solved prompts vs. 45 baseline-only solved prompts; McNemar p=0.0103p=0.0103). APRM also significantly improves Qwen4B-GRPO from 66.72% to 69.13% (+2.41 pp; 64 APRM-only solved prompts vs. 34 baseline-only solved prompts; McNemar p=0.0034p=0.0034)

[1]DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

[2]DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025

[3]Group Sequence Policy Optimization

[4]GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning, ICLR 2026

W4

The core contribution of this paper is to introduce a new reward-modeling paradigm and demonstrate its effectiveness. Since APRM is designed to reward attention-derived internal reasoning signals, it naturally introduces several additional hyperparameters. We have provided architecture-related ablations in Appendix D.3 and D.4, the ablation of β in Table 3, and training dynamics in Figures 7 and 11.

We do not claim that the selected hyperparameter configuration is optimal. Rather, our goal is to show that APRM yields consistent improvements under a fixed and reasonably chosen setting. Other configurations may further improve performance, and we view this as an important direction for future exploration, including alternative APRM architectures and different update/training strategies. We believe this opens a valuable research space for the community.

AuthorsOpenReview ↗

Response to Weakness 5 and Question 1&2

W5 & Q2

To further address this concern, we added experiments across model types, RL algorithms, and datasets. Specifically, we evaluated APRM on Qwen3.5-2B(instruct) with the same β value, and combined it with DAPO, a different RL algorithm. This setting covers an instruct/base model comparison, a different attention configuration, a different model scale, a different task domain, and a different RL algorithm. The results are as follows:

A. Experiment 1 (Cross-Model Validation): Qwen3.5-2B(instruct), mathematical reasoning, β=0.3

MethodAIME Avg@8 / Pass@8AMC Avg@8 / Pass@8MATH500 Avg@8 / Pass@8Olympiad Avg@8 / Pass@8Average Avg@8 / Pass@8
Qwen3.5-2B-Instruct10.42 / 20.0050.31 / 77.5068.23 / 85.2039.32 / 61.2842.07 / 61.00
GRPO20.83 / 33.3366.25 / 85.0075.78 / 87.6048.50 / 64.3952.84 / 67.58
DAPO21.67 / 33.3372.50 / 90.0079.35 / 88.6052.36 / 66.9156.47 / 69.71
DAPO+APRM25.00 / 43.3370.00 / 95.0080.45 / 89.6055.80 / 70.0357.81 / 74.49
Δ DAPO+APRM vs. DAPO+3.33 / +10.00-2.50 / +5.00+1.10 / +1.00+3.44 / +3.12+1.34 / +4.78

B. Experiment 2 (Cross-Task Validation): Qwen3.5-2B(instruct), code generation, β=0.3 and β=0.2

MethodLCB v5 Avg@8 / Pass@8HumanEval Avg@8 / Pass@8Average Avg@8 / Pass@8
Qwen3.5-2B-Instruct10.08 / 21.1554.88 / 79.8832.48 / 50.51
DAPO14.02 / 23.3061.59 / 83.5437.80 / 53.42
DAPO+APRM, β=0.314.07 / 24.3762.27 / 84.1538.17 / 54.26
DAPO+APRM, β=0.216.08 / 26.1662.80 / 86.5939.44 / 56.38
Δ β=0.2 vs. DAPO+2.06 / +2.87+1.22 / +3.05+1.64 / +2.96

Under the same β=0.3\beta=0.3 setting, DAPO+APRM consistently improves the overall performance over DAPO across both mathematical reasoning and code generation. Specifically, it yields gains of 1.34 Avg@8 and 4.78 Pass@8 points on mathematical benchmarks, and 0.37 Avg@8 and 0.84 Pass@8 points on code benchmarks. These results suggest that β=0.3\beta=0.3 provides a reasonably transferable and effective starting point across different task domains and RL settings. However, it does not necessarily represent the optimal coefficient for every task: in code generation, reducing β\beta to 0.2 leads to further improvements of 1.64 Avg@8 and 2.96 Pass@8 points over DAPO. Given the computational cost of RL experiments, we adopt a practical strategy of tuning β\beta in one setting and transferring it to others. We therefore view β=0.3\beta=0.3 as a robust initial recommendation, while allowing further validation-based tuning when necessary.

Q1

APRM can affect Pass@k more than Avg@k because its marginal effect on Pass@k factorizes into an APRM-alignment term and a Pass@k sensitivity term:

∂Passk(x)∂ηAPRM=k(1−px)k−1βCovπ(cfuture(y),s(y)−sˉx).\frac{\partial \mathrm{Pass} _k(x)}{\partial \eta _{\mathrm{APRM}}}=k(1-p _x)^{k-1}\beta\mathrm{Cov} _{\pi}(c _{\mathrm{future}}(y),s(y)-\bar{s} _x).

Here pxp _x is the per-sample success probability, s(y)−sˉxs(y)-\bar{s} _x is the group-centered APRM score, and cfuture(y)c _{\mathrm{future}}(y) denotes whether a trajectory has current or future success potential. The first factor, Covπ(cfuture(y),s(y)−sˉx)\mathrm{Cov} _{\pi}(c _{\mathrm{future}}(y),s(y)-\bar{s} _x), captures whether APRM prefers more promising reasoning trajectories. The second factor, k(1−px)k−1k(1-p _x)^{k-1}, is largest when pxp _x is small, so the same APRM signal is amplified more on hard or boundary problems. On easy problems, AbaseA _{\mathrm{base}} is already informative because correct and incorrect rollouts coexist. On hard problems, most rollouts may fail, making outcome-based advantages nearly flat. APRM can still rank failed rollouts by internal process quality and move probability mass toward trajectories closer to success, turning some 0/k cases into at least 1/k successful cases. This differs from an entropy bonus. APRM does not directly maximize output diversity; it reweights rollouts within the same prompt by internal process quality. Thus, improved diversity or coverage is an indirect consequence of process-quality reweighting.

AuthorsOpenReview ↗

Response to Question 3&4 and Additional Derivation for Q1

Q3

Your question is very insightful! GRPO evaluates final-answer correctness without explicitly penalizing redundant reasoning. For a relatively small base model, longer trajectories may provide additional opportunities for exploration and verification. When these trajectories produce correct answers, their generation patterns are reinforced, potentially leading to unnecessary length expansion.

In contrast, APRM does not directly reward longer or shorter outputs; its effect depends on the correlation between response length and process quality within each prompt group. Under the same local policy-reweighting approximation, its effect on expected response length is:

∂Eπ[L(y)]∂ηAPRM=βCovπ(L(y),s(y)−sˉx).\frac{\partial \mathbb{E} _{\pi}[L(y)]}{\partial \eta _{\mathrm{APRM}}}=\beta\mathrm{Cov} _{\pi}(L(y),s(y)-\bar{s} _x).

A positive correlation favors longer trajectories when they exhibit higher process quality, whereas a negative correlation suppresses unproductive length expansion. Thus, APRM promotes reasoning quality rather than response length, allowing useful long reasoning while discouraging redundant deliberation.

In GRPO experiments, the optimization process tends to reinforce longer trajectories when they lead to correct answers. APRM further selects trajectories based on their internal process quality. Consequently, if longer trajectories developed during GRPO exhibit relatively higher reasoning quality, combining GRPO with APRM can reinforce these trajectories, acting as a form of implicit regularization that dynamically allocates more reasoning budget to productive exploration.

Interestingly, this behavior differs across RL algorithms. Unlike GRPO, GSPO/GPG does not exhibit the same pronounced length-growth tendency in our experiments. In this setting, APRM primarily reallocates reasoning budget according to task complexity: it favors more extensive reasoning on challenging problems such as AIME, where longer trajectories can provide greater benefits, while favoring more concise reasoning on relatively simpler benchmarks such as MATH500.

This suggests that APRM does not simply increase or decrease response length; rather, it dynamically allocates reasoning budget according to the quality of internal reasoning and the complexity of the task.

Q4

We have add the model name in the table caption to Qwen3-1.7B-Base.

Additional Derivation for Q1

For a problem xx, let px=Ey∼π(⋅∣x)[c(y)]p _x=\mathbb{E} _{y\sim\pi(\cdot|x)}[c(y)], where c(y)∈0,1c(y)\in\\{0,1\\} indicates correctness. Then

Avgk(x)=px,Passk(x)=1−(1−px)k.\mathrm{Avg} _k(x)=p _x,\qquad \mathrm{Pass} _k(x)=1-(1-p _x)^k.

Therefore,

∂Passk(x)∂px=k(1−px)k−1.\frac{\partial \mathrm{Pass} _k(x)}{\partial p _x}=k(1-p _x)^{k-1}.

This term is large for hard problems with small pxp _x, and small for already-solved problems.

APRM modifies the advantage as

Ai=Aibase+β(si−sˉx),A _i=A _i^{\mathrm{base}}+\beta(s _i-\bar{s} _x),

where sˉx\bar{s} _x is the mean APRM score within the rollout group of prompt xx. Under a local policy-reweighting approximation,

πnew(y∣x)∝πold(y∣x)exp⁡(ηAPRMβ(s(y)−sˉx)).\pi _{\mathrm{new}}(y|x)\propto\pi _{\mathrm{old}}(y|x)\exp(\eta _{\mathrm{APRM}}\beta(s(y)-\bar{s} _x)).

Thus the APRM-induced change in per-sample success probability is

∂px∂ηAPRM=βCovπ(cfuture(y),s(y)−sˉx).\frac{\partial p _x}{\partial \eta _{\mathrm{APRM}}}=\beta\mathrm{Cov} _{\pi}(c _{\mathrm{future}}(y),s(y)-\bar{s} _x).

Combining the two derivatives gives

∂Passk(x)∂ηAPRM=k(1−px)k−1βCovπ(cfuture(y),s(y)−sˉx).\frac{\partial \mathrm{Pass} _k(x)}{\partial \eta _{\mathrm{APRM}}}=k(1-p _x)^{k-1}\beta\mathrm{Cov} _{\pi}(c _{\mathrm{future}}(y),s(y)-\bar{s} _x).

Thank you again for your careful review and valuable suggestions. By incorporating your feedback, we have improved the quality of the paper. If you have any remaining concerns, we would be happy to continue the discussion.

Reviewer_rSymOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Attention Process Reward Model (APRM), which incorporates internal model representations into reward modeling for LLM reinforcement learning. Motivated by the limitations of outcome-based rewards and text-based process supervision, APRM leverages hidden states and attention maps to assess reasoning quality. The proposed architecture combines an attention-fused concept pooler with a graph attention network and is trained using ranking and counterfactual objectives without explicit process-level annotations. During RL training, APRM provides an auxiliary advantage signal that is combined with the original outcome advantage, with periodic updates to adapt to policy changes. Experiments on mathematical reasoning benchmarks using Qwen3-1.7B and Qwen3-4B demonstrate improvements across several RL algorithms, particularly in Pass@8 performance.

Critical strengths
  1. Innovative Perspective on Process Reward Modeling. Instead of relying solely on final outcomes or surface-level textual rationales, this work introduces an intriguing approach that derives process supervision directly from internal attention patterns and hidden states. This presents a fresh, well-motivated angle for reward modeling within the RLVR paradigm.

  2. Sound Formulation and Mechanistic Validation (APRM). APRM couples a ranking loss with a counterfactual objective to capture informative internal features without the need for labor-intensive process annotations. Importantly, the attention-shuffle ablation provides convincing evidence that the reward model actively exploits attention dynamics rather than simply defaulting to hidden-state representations.

  3. Thorough Empirical Evaluation Across Models and RL Backbones. The experimental validation is extensive, spanning Qwen3-1.7B and 4B across key RLVR algorithms (GRPO, GSPO, and GPG). APRM yields consistent gains across standard mathematical reasoning benchmarks, especially under the Pass@8 metric, demonstrating its viability and modularity as a general-purpose reward component.

Critical weaknesses
  1. Constrained Empirical Scope and Cross-Domain Generalization. The evaluation is currently restricted to math benchmarks and dense backbones (Qwen3). Because APRM operates directly on internal attention maps and hidden activations, its inductive bias may not trivially transfer across different architectures or task domains. Expanding the benchmark suite to code reasoning and standard decision-making/agent environments (e.g., WebShop, ALFWorld) would significantly strengthen the empirical claims. Furthermore, evaluating on MoE architectures would provide valuable evidence regarding its scalability across diverse model paradigms.

  2. Incomplete Ablations on the Necessity of the Attention-GAT Architecture. APRM’s supervision remains outcome-driven: the ranking loss depends strictly on final answer validity, while the counterfactual objective merely discriminates against synthetic perturbations, which does not guarantee faithful process supervision. Although the attention-shuffle ablation confirms that the network utilizes attention signals, it does not justify why a heavyweight Attention-GAT backbone is needed over much simpler alternatives. The authors should include baseline comparisons against hidden-state pooling or hidden-state + MLP heads to demonstrate the isolated, additive contribution of graph-based attention modeling. Demonstrating these standalone gains is essential to validate the core contribution of APRM.

Questions for authors

Compatibility with DAPO. Since DAPO's dynamic sampling filters out rollout groups with identical outcome rewards, the additional I_active indicator in Eq. (11) would become unnecessary for retained groups. Have the authors considered integrating APRM with DAPO and evaluating whether it provides additional performance gains? I would like to see this comparison, as it would help establish whether APRM offers complementary benefits to a stronger RLVR training framework.

Relation to existing advantage integration methods. In Section 3.3, APRM directly adds a group-centered process score to the outcome advantage with a fixed coefficient. This is closely related to AWPO, which adaptively integrates reasoning and outcome advantages through variance-aware gating, and GDPO, which independently normalizes different reward components before combining their advantages. Could the authors clarify the similarities and differences between these approaches and APRM, particularly the motivation for using a fixed additive coefficient without variance normalization or adaptive weighting? A comparison of these integration strategies would help establish whether the proposed approach adequately handles potential scale mismatches and conflicts between process and outcome signals.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

Thank you very much for recognizing the novelty of our work, the comprehensiveness of our experiments, and our theoretical validation of the underlying mechanism. We fully incorporated your suggestions by evaluating APRM across tasks, model architectures, and RL algorithms, including its integration with DAPO. These additional experiments produced encouraging results. We respond to your questions point by point below.

W1 & Q1

Your insight regarding cross-model validation and integration with DAPO was particularly valuable. We therefore conducted two new experiments using the more advanced Qwen3.5-2B(Instruct) model. Compared with the Qwen3 family used in our original experiments, Qwen3.5 adopts a different hybrid architecture that combines gated attention with Gated DeltaNet. We evaluated APRM together with DAPO on both mathematical reasoning and code generation, thereby covering a new model architecture, a stronger RL algorithm, and a different task domain.

A. Experiment 1 (Cross-Model Validation): Qwen3.5-2B(instruct), mathematical reasoning, and DAPO

We used the same mathematical training data and training configuration as in our Qwen3 experiments. To directly test hyperparameter transferability, we reused the same APRM coefficient, β=0.3\beta=0.3, without additional tuning.

The cold-start APRM achieved high ranking quality:

methodPairwise ranking accuracy
APRM91.93%

The downstream results are:

MethodAIME Avg@8 / Pass@8AMC Avg@8 / Pass@8MATH500 Avg@8 / Pass@8Olympiad Avg@8 / Pass@8Average Avg@8 / Pass@8
Qwen3.5-2B-Instruct10.42 / 20.0050.31 / 77.5068.23 / 85.2039.32 / 61.2842.07 / 61.00
GRPO20.83 / 33.3366.25 / 85.0075.78 / 87.6048.50 / 64.3952.84 / 67.58
DAPO21.67 / 33.3372.50 / 90.0079.35 / 88.6052.36 / 66.9156.47 / 69.71
DAPO+APRM25.00 / 43.3370.00 / 95.0080.45 / 89.6055.80 / 70.0357.81 / 74.49
Δ DAPO+APRM vs. DAPO+3.33 / +10.00-2.50 / +5.00+1.10 / +1.00+3.44 / +3.12+1.34 / +4.78

These results support four conclusions:

  • Conclusion 1: DAPO is a substantially stronger and more competitive baseline than GRPO.
  • Conclusion 2: APRM integrates effectively with DAPO: DAPO+APRM further improves the average by 1.34 points on Avg@8 and 4.78 points on Pass@8. DAPO's sampling strategy allows APRM to be naturally integrated into all rollout groups, consistently contributing to performance and yielding significant improvements.
  • Conclusion 3: The positive result obtained with the directly transferred β=0.3\beta=0.3 setting provides evidence that the APRM hyperparameter is reasonably robust across RL algorithms.
  • Conclusion 4: The result shows that APRM can adapt to a different model scale, an Instruct rather than Base checkpoint, and Qwen3.5's hybrid gated-attention/Gated-DeltaNet architecture.

B. Experiment 2 (Cross-Task Validation): Qwen3.5-2B(instruct), code generation, and DAPO

To test whether APRM generalizes beyond mathematical reasoning, we conducted an additional experiment in the code domain. We sampled 3,000 problems from agentica-org/DeepCoder-Preview-Dataset. The frozen split contains 2,500 training problems and 500 validation problems. We used a training batch size of 48 and trained each RL configuration for 100 global steps. We report results on LiveCodeBench v5 and HumanEval.

The code-domain cold-start APRM also learned a meaningful ranking signal:

methodPairwise ranking accuracy
APRM75.56%
MethodLCB v5 Avg@8 / Pass@8HumanEval Avg@8 / Pass@8Average Avg@8 / Pass@8
Qwen3.5-2B-Instruct10.08 / 21.1554.88 / 79.8832.48 / 50.51
DAPO14.02 / 23.3061.59 / 83.5437.80 / 53.42
DAPO+APRM, β=0.314.07 / 24.3762.27 / 84.1538.17 / 54.26
DAPO+APRM, β=0.216.08 / 26.1662.80 / 86.5939.44 / 56.38
Δ β=0.2 vs. DAPO+2.06 / +2.87+1.22 / +3.05+1.64 / +2.96
  • Conclusion 1: These results provide direct evidence that APRM transfers to a different task domain. With β=0.2\beta=0.2, APRM improves LiveCodeBench v5 by 2.06 points on Avg@8 and 2.87 points on Pass@8. It also improves HumanEval by 1.22 points on Avg@8 and 3.05 points on Pass@8.

  • Conclusion 2: The results indicate that APRM remains effective over a transferable coefficient range, while β=0.2\beta=0.2 is better suited to this code-domain setting. Together with Experiment 1, these findings show that APRM can be integrated with more powerful RL baseline (DAPO) and adapted across model architectures, Base/Instruct checkpoints, mathematical reasoning, and code generation.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2

Q2

We further conducted the following ablations to examine the respective contributions of GAT and attention. We kept the trainable parameter counts nearly identical across architectures, thereby isolating the architectural component under investigation as closely as possible.

Specifically, we ablated the cold-start APRMs built on Qwen3.5-2B for both the code and mathematics domains introduced in Q1. All variants used the same data and training configuration. The results are as follows:

ArchitectureTrainable parametersCode ranking accuracyMath ranking accuracy (200 groups)
APRM44.251M75.56%91.93%
APRM w/o GAT44.251M75.31%90.95%
Hidden MLP44.128M71.42%84.39%

These results show that attention-based concept pooling makes a clear contribution, particularly in the mathematics domain. GAT provides a small additional numerical gain over the parameter-matched no-GAT variant. Overall, the comparison supports the importance of incorporating attention-derived representations into the reward model.

More broadly, the optimal architecture for attention-based reward modeling remains an open question. Our current experiments provide an initial validation of APRM's effectiveness, while a more comprehensive exploration of reward-model architectures is an important direction for future work.

W2

We compared our method with AWPO and GDPO. Specifically, the differences and potential connections can be summarized as follows.

(a) Our method mainly focuses on the source of the reward signal, namely how to construct rewards from attention maps and hidden states. In contrast, AWPO and GDPO primarily focus on strategies for combining multiple reward signals during policy optimization.

(b) APRM is methodologically orthogonal to AWPO and GDPO. The adaptive gating strategy in AWPO and the independent normalization strategy in GDPO can potentially be applied to APRM as alternative integration mechanisms.

(c) In this paper, we adopt a simple additive form with a fixed coefficient to keep the scientific question focused and isolate the contribution of internal reward signals. How to further exploit APRM and this broader family of internal rewards with more advanced integration strategies remains an important direction for future work.

We have added a dedicated future work section to systematically discuss the points above.

Thank you again for recognizing our work. Your comments, especially the suggestion to integrate DAPO, have greatly improved the quality of our paper and made us think more deeply about the method. If you have any further questions, we would be happy to continue the discussion and work together to further improve the paper.

Dear Reviewer rSym,

We are sincerely grateful for your thoughtful and insightful feedback, which has been invaluable in improving our manuscript. With the discussion deadline approaching, we wanted to kindly follow up and ask whether our revisions and responses have adequately addressed your concerns.

We have learned a great deal from this discussion and truly appreciate the time and effort you have devoted to sharing your insights with us.

Best regards, The Authors

Dear Reviewer rSym,

Thank you again for your time and effort. As only one day remains before the discussion deadline, we would kindly like to ask whether there are any questions that remain unresolved on your side. We would be glad to discuss them with you!

Best regards,

The Authors

Reviewer_UazuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether model-internal signals can provide useful supervision for LLM reinforcement learning. The authors propose APRM, a separate reward model that uses hidden states and attention maps with a graph attention network to assess the quality of reasoning trajectories. APRM is trained before RL and periodically updated during RL, and its scores provide an additional advantage signal for each rollout. Experiments on two Qwen3-Base models across four mathematical reasoning benchmarks show that adding APRM improves average performance compared with using the same RL methods without APRM.

Critical strengths
  1. The paper is generally clear and easy to follow.

  2. The idea of using model intrinsic signals, such as attention maps, to construct an additional reward signal for LLM reinforcement learning is interesting.

Critical weaknesses
  1. The authors provide insufficient justification for several key design choices, particularly those in the APRM architecture. The underlying insights remain unclear. The lack of explanation for these design choices also raises doubts about whether the selected components and configurations are optimal.

  2. APRM introduces an additional trainable reward model, requires a separate initial training stage, and is periodically updated during RL. Matching the RL training schedules alone does not account for these additional costs, leaving the fairness of the comparison under comparable total training budgets unclear.

  3. The statistical reliability of the reported improvements is unclear. The paper does not specify the number of independent training runs or report the mean and standard deviation across runs.

Questions for authors
  1. Textual reward aliasing is a central motivation for this work, but the paper establishes only its theoretical possibility and does not empirically demonstrate that it occurs in the evaluated models. Could the authors provide such evidence? Without it, the motivation and a key premise of the method remain insufficiently supported. 2.Regarding APRM's design:  ○ Why average the attention maps from the last K layers and multiply the result by only the final-layer hidden states? Why not first multiply each layer's attention map by its corresponding hidden states and then sum the results?  ○ Why use attention maps from only the last K layers rather than other layers?  ○ Is there a way to determine which layers' attention maps should be used before training APRM?  ○ Why project the features into a concept space and then apply a GAT? The rationale behind this overall design needs further explanation.
  2. Before alignment, base models may exhibit inconsistent question-answering behavior. This could lead to substantial differences between correct and incorrect rollouts, potentially making their attention patterns easier to distinguish. Could the authors also evaluate APRM on instruction-tuned models?
  3. Could the authors report the mean and variance of the results across multiple independent training runs?
  4. Several figures (e.g., Figure 1, 2, 5, 6) are too small to read clearly.
  5. The citation formatting throughout the paper needs correction.
5 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 and 2

Thank you for recognizing the clarity of our writing and the research interest of our work. We hope the following point-by-point responses address your concerns.

W1

Due to space constraints, the original manuscript did not fully explain the design rationale of APRM. Its architecture is guided by two complementary considerations:

  1. Reward effectiveness. APRM is designed to capture complementary signals of reasoning quality through both its architecture and training objectives. We adopt a weighted GAT to model dependencies among latent concepts. Specifically, each edge weight serves as both an additive attention bias and a multiplicative message gate, allowing the model to capture not only which concepts are related but also the strength of their interactions. The ranking loss encourages APRM to distinguish correct and incorrect reasoning trajectories, while the counterfactual loss encourages sensitivity to meaningful attention structures by contrasting original and perturbed attention patterns.

  2. Computational scalability. APRM is designed to avoid the computational overhead of directly modeling token-level interactions. We use differentiable fixed-size concept pooling to compress variable-length token representations into MM latent concepts. This bounds subsequent graph modeling to O(M2)O(M^2), independent of the original sequence length TT (several thousand tokens and O(T2T^2)), and further reduces graph propagation to O(Mk)O(Mk) through Top-kk sparsification. For the token representations, we use final-layer hidden states as the semantic foundation because they are directly projected by the output head to produce next-token predictions:p(yt∣y<t)=softmax⁡(WouthtL).p(y_t\mid y_{<t})=\operatorname{softmax}(W_{\mathrm{out}}h_t^L). We therefore fuse them with attention summaries from the last KK layers to incorporate complementary information about internal interactions without introducing the complexity of modeling hidden states across all layers.

We do not claim that this architecture is globally optimal. Rather, our goal is to establish the potential of using model-internal signals as complementary reward supervision. Exploring richer internal representations and alternative reward architectures remains an important direction for future work.

W2

We fully understand your concern. APRM introduces a new and complementary optimization space based on model-internal signals. Our primary goal is to investigate whether allocating a modest amount of additional computation to this orthogonal signal provides benefits. Accordingly, our comparisons control the RL training schedule, rollout budget, batch size, and hardware configuration, thereby isolating the contribution of the additional internal reward.

The added cost is limited and reusable. APRM contains only approximately 46M parameters, corresponding to 2.7% and 1.2% of the 1.7B and 4B backbones, respectively. Its cold-start training is a one-time cost, and the resulting checkpoint can be reused across different RL experiments with the same backbone. We therefore consider the additional cost acceptable for exploring this complementary reward paradigm.

AuthorsOpenReview ↗

Response to Weaknesses 3 and Question 1&3

W3 & Q3

Running multiple full LLM RL training seeds is computationally expensive. Many recent large-scale RL studies therefore report benchmark results under a fixed training configuration rather than extensive multi-seed evaluations. Representative methods such as GRPO[1], DAPO[2], GSPO[3], and GPG[4] typically report fixed-seed benchmark metrics (e.g., Accuracy, Pass@K, Avg@K) rather than multi-seed confidence intervals. Our evaluation protocol therefore follows the common practice for large-scale LLM-RL experiments.

To address the concern about statistical robustness, for each generation index jj, we compute accuracy over all prompts and report the mean and standard deviation across the eight generation-level accuracies.

ModelDatasetBaseline Avg@8 ± stdAPRM Avg@8 ± stdΔ Avg@8
Qwen1.7B-GRPOAIME5.00 ± 4.366.25 ± 5.76+1.25 pp
Qwen1.7B-GRPOAMC2339.69 ± 4.9042.50 ± 6.81+2.81 pp
Qwen1.7B-GRPOMATH50061.42 ± 1.7961.82 ± 0.91+0.40 pp
Qwen1.7B-GRPOOlympiadBench27.32 ± 1.0428.62 ± 1.50+1.30 pp
Qwen4B-GRPOAIME10.00 ± 5.6311.67 ± 3.56+1.67 pp
Qwen4B-GRPOAMC2357.81 ± 4.3256.88 ± 2.59-0.94 pp
Qwen4B-GRPOMATH50072.65 ± 0.5573.12 ± 0.72+0.47 pp
Qwen4B-GRPOOlympiadBench41.36 ± 0.5341.12 ± 0.52-0.24 pp

For Pass@8, we use McNemar’s paired test over the pooled 1,244 prompts. APRM significantly improves Qwen1.7B-GRPO from 59.57% to 61.90% (+2.33 pp; 74 APRM-only solved prompts vs. 45 baseline-only solved prompts; McNemar p=0.0103p=0.0103). APRM also significantly improves Qwen4B-GRPO from 66.72% to 69.13% (+2.41 pp; 64 APRM-only solved prompts vs. 34 baseline-only solved prompts; McNemar p=0.0034p=0.0034)

[1]DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

[2]DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025

[3]Group Sequence Policy Optimization

[4]GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning, ICLR 2026

Q1

A. Explanation for our design insight

  1. Last KK layers. Later layers are closer to the output decision and integrate information from preceding layers. Using a contiguous final window provides an output-relevant signal while limiting extraction cost.

  2. AˉHL\bar{A}H^L rather than ∑lAlHl\sum_l A^lH^l. Attention maps share the same token-coordinate space, whereas hidden representations evolve across layers. Applying aggregated attention to HLH^L avoids assuming that layer-wise feature spaces are aligned and keeps the representation output-relevant.

  3. Layer selection. We select the final contiguous KK layers a priori using output proximity, without training APRM or searching arbitrary layer combinations.

  4. Concept pooling before GAT. Concept pooling converts the variable-length token graph into a fixed-size concept graph, reducing graph propagation from O(T2)O(T^2) to O(M2)O(M^2), or O(Mk)O(Mk) after sparsification. This substantially reduces the additional computational and memory overhead introduced by APRM during training.

The selection, fusion, and modeling of internal signals constitute a broad design space that we leave for future work.

B. Empirical Validation of Textual Reward Aliasing

We conducted an identical-prefix continuation experiment to test whether APRM captures policy-dependent differences in continuation success beyond the prefix text. For each of 128 mathematics problems, we sampled an unfinished 512-token response prefix from the original Qwen3.5-2B, then supplied identical prompt and prefix tokens to it and its GRPO checkpoints at steps 1, 2, and 3. A fixed reward model conditioned solely on this prefix text therefore receives identical inputs across checkpoints.

The frozen APRM scored each prefix using the corresponding backbone's attention and hidden states, without observing subsequent continuations. Each backbone then sampled 16 continuations; Avg@16 was the fraction with correct final answers. For each problem and checkpoint, we computed changes in APRM score and Avg@16 relative to Base and measured their Pearson correlation.

ComparisonCorrelation rr
GRPO step 1 vs. Base0.018
GRPO step 2 vs. Base0.251
GRPO step 3 vs. Base0.112

Results and Findings. The correlations for GRPO steps 1, 2, and 3 are 0.018, 0.251, and 0.112, respectively. After centering paired changes within each checkpoint, pooling all three comparisons yields a positive correlation of r=0.133r=0.133 (p=0.036p=0.036). These results show that APRM scores exhibit a statistically significant positive association with subsequent continuation quality, despite identical prefix text across checkpoints. This provides empirical evidence that model-internal signals capture complementary, policy-dependent information that cannot be recovered from the prefix text alone.

AuthorsOpenReview ↗

Response to Question 2,4,5

Q2

Your insight regarding cross-model validation on the instruct model was particularly valuable. We therefore conducted two new experiments using the more advanced Qwen3.5-2B(Instruct) model. Compared with the Qwen3 family used in our original experiments, Qwen3.5 adopts a different hybrid architecture that combines gated attention with Gated DeltaNet. We evaluated APRM together with DAPO on both mathematical reasoning and code generation, thereby covering a new model architecture, a stronger RL algorithm, and a different task domain.

A. Experiment 1 (Cross-Model Validation): Qwen3.5-2B(instruct), mathematical reasoning

We used the same mathematical training data and training configuration as in our Qwen3 experiments. To directly test hyperparameter transferability, we reused the same APRM coefficient, β=0.3\beta=0.3, without additional tuning.

The cold-start APRM achieved a pairwise ranking accuracy of 91.93%:

methodPairwise ranking accuracy
APRM91.93%

The downstream results are:

MethodAIME Avg@8 / Pass@8AMC Avg@8 / Pass@8MATH500 Avg@8 / Pass@8Olympiad Avg@8 / Pass@8Average Avg@8 / Pass@8
Qwen3.5-2B(Instruct)10.42 / 20.0050.31 / 77.5068.23 / 85.2039.32 / 61.2842.07 / 61.00
GRPO20.83 / 33.3366.25 / 85.0075.78 / 87.6048.50 / 64.3952.84 / 67.58
DAPO21.67 / 33.3372.50 / 90.0079.35 / 88.6052.36 / 66.9156.47 / 69.71
DAPO+APRM25.00 / 43.3370.00 / 95.0080.45 / 89.6055.80 / 70.0357.81 / 74.49
Δ DAPO+APRM vs. DAPO+3.33 / +10.00-2.50 / +5.00+1.10 / +1.00+3.44 / +3.12+1.34 / +4.78

These results support the following conclusions:

  • Conclusion 1: APRM demonstrates strong generalizability, delivering consistent performance improvements across the instruction-tuned model and the more powerful RL algorithm.
  • Conclusion 2: APRM integrates effectively with DAPO: DAPO+APRM further improves the average by 1.34 points on Avg@8 and 4.78 points on Pass@8.
  • Conclusion 3: The positive result obtained with the directly transferred β=0.3\beta=0.3 setting provides evidence that the APRM hyperparameter is reasonably robust across RL algorithms.

B. Experiment 2(Cross-Task Validation): Qwen3.5-2B(instruct), code generation

To test whether APRM generalizes beyond mathematical reasoning, we conducted an additional experiment in the code domain. We sampled 3,000 problems from agentica-org/DeepCoder-Preview-Dataset. The frozen split contains 2,500 training problems and 500 validation problems. We used a training batch size of 48 and trained each RL configuration for 100 global steps. We report results on LiveCodeBench v5 and HumanEval.

On the code-generation data, the cold-start APRM achieved a pairwise ranking accuracy of 75.56%:

methodPairwise ranking accuracy
APRM75.56%
MethodLCB v5 Avg@8 / Pass@8HumanEval Avg@8 / Pass@8Average Avg@8 / Pass@8
Qwen3.5-2B10.08 / 21.1554.88 / 79.8832.48 / 50.51
DAPO14.02 / 23.3061.59 / 83.5437.80 / 53.42
DAPO+APRM, β=0.314.07 / 24.3762.27 / 84.1538.17 / 54.26
DAPO+APRM, β=0.216.08 / 26.1662.80 / 86.5939.44 / 56.38
Δ β=0.2 vs. DAPO+2.06 / +2.87+1.22 / +3.05+1.64 / +2.96
  • Conclusion 1: These results provide direct evidence that APRM transfers to a different task domain. With β=0.2\beta=0.2, APRM improves LiveCodeBench v5 by 2.06 points on Avg@8 and 2.87 points on Pass@8. It also improves HumanEval by 1.22 points on Avg@8 and 3.05 points on Pass@8.

  • Conclusion 2: The results indicate that APRM remains effective over a transferable coefficient range, while β=0.2\beta=0.2 is better suited to this code-domain setting. Together with Experiment 1, these findings show that APRM can be integrated with the more powerful RL baseline (DAPO) and adapted across model architectures, Base/Instruct checkpoints, mathematical reasoning, and code generation.

Q4 & Q5

Thank you for pointing this out. We have revised the citation format. In addition, because some figures were small due to page constraints in the initial submission, we enlarged them to improve readability.

Thank you again for your valuable comments, which have greatly helped us improve the paper. We hope the additional experiments and clarifications address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Overall Positive Feedback

The reviewers recognized the novelty and potential value of our work, particularly the innovative use of model-internal signals (attention maps and hidden states) for process reward modeling. They also appreciated the clear presentation, interesting research perspective, and comprehensive experimental evaluation. One reviewer explicitly acknowledged the soundness of our method and its mechanistic validation.

Summary of Reviewer Concerns and Our Responses

1. Further Exploration of Methodological Design

  • Reviewers' suggestions: Further explain the design choices in APRM, investigate the contributions of different architectural components, and provide empirical evidence for textual reward aliasing.
  • Our responses: Elaborated on the architectural rationale and computational efficiency, added parameter-matched ablation studies to examine the contributions of GAT, and conducted an identical-prefix experiment to empirically validate textual reward aliasing.

2. Extending APRM to More Models, Tasks, and RL Algorithms

  • Reviewers' suggestions: Explore APRM across different model architectures (including Instruct models), extend evaluation to other task domains such as code generation, and investigate its compatibility with stronger RL baselines such as DAPO.
  • Our responses: Added extensive experiments on Qwen3.5-2B-Instruct, covering mathematical reasoning and code generation, and integrated APRM with DAPO. The results demonstrate its effectiveness across different model types, task domains, and RL algorithms. We also clarified the relationship between APRM and existing advantage integration methods, including AWPO and GDPO.

3. Statistical Analysis and Hyperparameter Transferability

  • Reviewers' suggestions: Provide further statistical analysis of performance improvements and investigate the sensitivity and transferability of APRM's hyperparameters.
  • Our responses: Added mean/std reporting and statistical significance tests, clarified APRM's computational overhead, and conducted cross-model and cross-task experiments to evaluate hyperparameter transferability.

4. Understanding APRM's Performance and Reasoning Behavior

  • Reviewers' suggestions: Further explain the improvements in Pass@8, the relationship between internal rewards and output diversity, and the impact of APRM on response length.
  • Our responses: Provided additional theoretical analysis explaining how APRM reallocates probability mass according to internal reasoning quality, particularly on challenging problems, and clarified its effects on reasoning length and exploration.

Paper Revisions

  1. Improved Figure Readability: Utilized the additional page allowance to enlarge figures and improve readability.
  2. Added Problem Formulation: Introduced a dedicated section to clarify the problem setting and formulation.
  3. Expanded Experimental Evaluation: Added cross-model and cross-task experiments on Qwen3.5-2B-Instruct, covering mathematical reasoning and code generation, together with DAPO integration.
  4. Added GAT Ablation Studies: Compared APRM, APRM w/o GAT, and Hidden MLP to further investigate the contributions of the architectural components.

Overall, we carefully addressed the reviewers' suggestions through additional experiments, architectural ablations, statistical analyses, and theoretical explanations, substantially expanding the empirical validation and analysis of APRM. We also followed up with the reviewers twice to invite further discussion. No additional feedback was received before the discussion deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.