- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Anchor-Bridge Attribution Policy Optimization (ABPO), which tackles the problem of coarse sequence level credit assignment in RLVR. The paper identifies 2 types of influential tokens, Anchor and Bridge, using graph-based scores derived from attention. ABPO fuses the Anchor and Bridge scores to assign token-level advantages within a GRPO training framework.
Critical strengths
-
The method tackles an important problem of credit assignment in RLVR. Using complementary graph-based scores is a novel approach to evaluating dependencies between tokens without an additional learned critic or process reward model.
-
The results show that ABPO outperforms GRPO while also reducing the response token length. The ablation experiments are also thorough and isolate the benefit of combining the 2 scores.
Critical weaknesses
-
The paper mainly compares ABPO against GRPO, but the token level credit assignment comparisons are limited to Random and High Entropy controls, which performed similarly to GRPO. Without stronger comparisons to relevant credit assignment baselines, it is difficult to judge the effectiveness of ABPO.
-
The main results report pass@1, although Appendix D states that 16 responses are sampled per question at temperature 0.7. If only one response is used to compute pass@1, the comparison may be sensitive to sampling, particularly when the reported gains are small. Reporting average accuracy across all 16 responses would provide a more reliable comparison.
Questions for authors
Please refer to the Weaknesses section above.
3 comments
Thank you for the detailed response and additional experiments. The clarification that pass@1 averages all 16 sampled responses resolves my concern about the evaluation procedure.
The additional comparison is useful, but it remains unclear how Attention-Entropy Reweighting was adapted to your training setup. Its original paper reports improvements over DAPO, whereas it performs similarly to GRPO here. Clarifying how its weighting schedule and hyperparameters were transferred or tuned would help establish whether this is a sufficiently strong comparison.
I appreciate the additional experiments and clarifications. Given the remaining uncertainty about ABPO’s comparative advantage, I will maintain my score.
We thank the reviewer for this constructive question. Below we clarify how AttnEntropy (Low2High) was configured and why it performs similarly to GRPO in our controlled comparison.
-
Hyperparameter Transfer: We directly adopted the default configuration from the original paper (arXiv:2605.07660v1, Tables 9 & 13): normalized attention Shannon entropy at intermediate layer , batch-level Softmax () with Min-Max scaling, and a linear
Low2Highwarmup over steps (, ) using all-token-weighted normalization (without group-wise RMS renormalization). -
Controlled GRPO Backbone vs. DAPO: To isolate the effect of token-level credit assignment, all compared methods (
GRPO,AttnEntropy, andABPO) share the exact same vanilla GRPO backbone (symmetric clipping, no dynamic filtering, and identical rollout/optimization settings). In contrast, the original paper appliesAttnEntropyon top of DAPO, whose asymmetric clip-higher and dynamic filtering provide substantial orthogonal stabilization. -
Behavior Under Vanilla GRPO: Without DAPO's clip-higher or group-wise RMS advantage renormalization,
AttnEntropy's early emphasis on low-entropy tokens () tends to over-concentrate updates and suppress exploration diversity—for instance, onCrossThink-QA, policy token entropy dropped by ~80% () andpass@16fell below vanilla GRPO (0.5440vs.0.8370). By contrast,ABPOpairs structural causal-flow centrality with group-wise RMS advantage renormalization, preserving gradient scale and exploration even without DAPO's extra stabilizers.
Response to Reviewer kngk: Attention-Based Credit Comparisons and Pass@1 (Avg@16) Clarification
Thank you for recognizing the value of our complementary graph-based scores and the joint improvements in accuracy and response length. Below we present our comparison with an attention-based credit-assignment baseline and clarify how Pass@1 is computed across the 16 sampled responses.
Q1: Comparison with attention-based credit-assignment methods
We agree that comparing against established attention-based methods better contextualizes ABPO. Under the matched Table 2 setting on Qwen3-8B-Base, we compare ABPO against Attention-Entropy Reweighting. It averages
38.9—nearly identical to GRPO (39.0) and Random credit (38.9)—whereas ABPO achieves41.6(+2.7points) and leads on all four benchmarks:We also extend the comparison to the four cross-domain tasks in Table 3 (Qwen3-8B-Base, Pass@1). ABPO outperforms Attention-Entropy Reweighting across all four domains: MedQA (
68.5vs.67.0), FinQA (22.9vs.20.2), Countdown (67.5vs.62.2), and CrossThink QA (45.5vs.32.8, where Attention-Entropy's Low2High schedule suffers premature exploration collapse and drops below GRPO):Q2: Pass@1 computation across the 16 sampled responses
Thank you for pointing out this ambiguity in our notation. Our reported Pass@1 is indeed the average accuracy across all 16 sampled responses per question (often denoted Avg@16), further averaged over five independent runs.
Specifically, in each run r∈{1,…,5} at step 200, we generate K=16 responses for each benchmark question q∈{1,…,Q} at temperature 0.7 and top-p 0.95. Letting cr,q be the number of correct responses among the 16 samples, we compute the per-question accuracy cr,q/16, average across all Q questions, and report the mean over the five runs:
Pass@1(Avg@16)=51∑r=15(Q1∑q=1Q16cr,q).
Averaging over all 16 responses reduces decoding variance within each run, while the five independent runs capture run-to-run variability.
Thank you again for your constructive feedback. We have incorporated these baseline comparisons and metric clarifications into our rebuttal revision.