acceptodds
Under review as a conference paper at ICLR 2027

Where Should the Credit Lie? Attention-Induced Anchor-Bridge Attribution Policy Optimization for LLM Reasoning

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning capabilities of large language models. Methods such as GRPO provide critic-free optimization, but they assign uniform, sequence-level credit to all tokens and thus overlook the disparate roles that tokens play in the reasoning process. To address this, we leverage the model's own attention patterns to construct an attribution map that characterizes how information and contributions flow among tokens. This allows us to identify two pivotal token types: Anchor Tokens, which are frequently reused by subsequent reasoning steps, and Bridge Tokens, which connect preceding context with later reasoning. Building on these observations, we propose Anchor-Bridge Attribution Policy Optimization (ABPO), a token-level credit assignment framework guided by attribution relationships. Specifically, ABPO computes an AnchorScore and a BridgeScore for each token using the AnchorRank and BridgeTrace algorithms, respectively, and then fuses them into a SynergyScore for advantage reweighting. Experimental results across diverse reasoning tasks demonstrate that ABPO significantly and consistently outperforms the GRPO baseline and other credit assignment methods (+2.6 points on math, +7.0 points on other domains vs. GRPO). Furthermore, ABPO generates more concise reasoning chains (-3.15% tokens on math, -30.15% on other domains over GRPO), and achieves faster training convergence, highlighting its dual strengths in both effectiveness and efficiency.

Primary area:
reinforcement learning
Keywords:
Token-level Credit Assignment, Reinforcement Learning, LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kngkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Anchor-Bridge Attribution Policy Optimization (ABPO), which tackles the problem of coarse sequence level credit assignment in RLVR. The paper identifies 2 types of influential tokens, Anchor and Bridge, using graph-based scores derived from attention. ABPO fuses the Anchor and Bridge scores to assign token-level advantages within a GRPO training framework.

Critical strengths
  • The method tackles an important problem of credit assignment in RLVR. Using complementary graph-based scores is a novel approach to evaluating dependencies between tokens without an additional learned critic or process reward model.

  • The results show that ABPO outperforms GRPO while also reducing the response token length. The ablation experiments are also thorough and isolate the benefit of combining the 2 scores.

Critical weaknesses
  • The paper mainly compares ABPO against GRPO, but the token level credit assignment comparisons are limited to Random and High Entropy controls, which performed similarly to GRPO. Without stronger comparisons to relevant credit assignment baselines, it is difficult to judge the effectiveness of ABPO.

  • The main results report pass@1, although Appendix D states that 16 responses are sampled per question at temperature 0.7. If only one response is used to compute pass@1, the comparison may be sensitive to sampling, particularly when the reported gains are small. Reporting average accuracy across all 16 responses would provide a more reliable comparison.

Questions for authors

Please refer to the Weaknesses section above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kngk: Attention-Based Credit Comparisons and Pass@1 (Avg@16) Clarification

Thank you for recognizing the value of our complementary graph-based scores and the joint improvements in accuracy and response length. Below we present our comparison with an attention-based credit-assignment baseline and clarify how Pass@1 is computed across the 16 sampled responses.

Q1: Comparison with attention-based credit-assignment methods

We agree that comparing against established attention-based methods better contextualizes ABPO. Under the matched Table 2 setting on Qwen3-8B-Base, we compare ABPO against Attention-Entropy Reweighting. It averages 38.9—nearly identical to GRPO (39.0) and Random credit (38.9)—whereas ABPO achieves 41.6 (+2.7 points) and leads on all four benchmarks:

MethodCredit / Attention MechanismAIME 2024AIME 2025MATH500OlympiadBenchAvg.
GRPOUniform sequence-level advantage14.813.478.549.239.0
DAPOOutcome RLVR recipe (dynamic sampling, clip-higher)15.015.679.249.539.8
+ Random creditBudget-matched random token placement14.911.679.149.938.9
+ High-entropy creditOutput-vocabulary predictive entropy15.113.578.750.039.3
Attention-Entropy ReweightingContext-position attention entropy dynamic soft reweighting14.411.879.549.738.9
+ BridgeTraceMulti-hop path-flow mediation on attention DAG16.214.079.050.139.8
+ AnchorRankRecursive downstream attribution on attention DAG15.114.580.050.139.9
ABPOFused AnchorRank + BridgeTrace attribution17.515.581.252.341.6

We also extend the comparison to the four cross-domain tasks in Table 3 (Qwen3-8B-Base, Pass@1). ABPO outperforms Attention-Entropy Reweighting across all four domains: MedQA (68.5 vs. 67.0), FinQA (22.9 vs. 20.2), Countdown (67.5 vs. 62.2), and CrossThink QA (45.5 vs. 32.8, where Attention-Entropy's Low2High schedule suffers premature exploration collapse and drops below GRPO):

MethodMedQAFinQACountdownCrossThink QA
GRPO60.311.961.842.3
Attention-Entropy Reweighting67.020.262.232.8
+ BridgeTrace68.422.463.642.9
+ AnchorRank70.019.673.644.0
ABPO68.522.967.545.5

Q2: Pass@1 computation across the 16 sampled responses

Thank you for pointing out this ambiguity in our notation. Our reported Pass@1 is indeed the average accuracy across all 16 sampled responses per question (often denoted Avg@16), further averaged over five independent runs.

Specifically, in each run r∈{1,…,5}r \in \{1,\dots,5\} at step 200, we generate K=16K=16 responses for each benchmark question q∈{1,…,Q}q \in \{1,\dots,Q\} at temperature 0.7 and top-pp 0.95. Letting cr,qc_{r,q} be the number of correct responses among the 16 samples, we compute the per-question accuracy cr,q/16c_{r,q}/16, average across all QQ questions, and report the mean over the five runs:

Pass@1⁡  (Avg@16)=15∑r=15(1Q∑q=1Qcr,q16).\operatorname{Pass@1}\;(\text{Avg@16}) = \frac{1}{5}\sum_{r=1}^{5}\left(\frac{1}{Q}\sum_{q=1}^{Q}\frac{c_{r,q}}{16}\right).

Averaging over all 16 responses reduces decoding variance within each run, while the five independent runs capture run-to-run variability.

Thank you again for your constructive feedback. We have incorporated these baseline comparisons and metric clarifications into our rebuttal revision.

Thank you for the detailed response and additional experiments. The clarification that pass@1 averages all 16 sampled responses resolves my concern about the evaluation procedure.

The additional comparison is useful, but it remains unclear how Attention-Entropy Reweighting was adapted to your training setup. Its original paper reports improvements over DAPO, whereas it performs similarly to GRPO here. Clarifying how its weighting schedule and hyperparameters were transferred or tuned would help establish whether this is a sufficiently strong comparison.

I appreciate the additional experiments and clarifications. Given the remaining uncertainty about ABPO’s comparative advantage, I will maintain my score.

We thank the reviewer for this constructive question. Below we clarify how AttnEntropy (Low2High) was configured and why it performs similarly to GRPO in our controlled comparison.

  1. Hyperparameter Transfer: We directly adopted the default configuration from the original paper (arXiv:2605.07660v1, Tables 9 & 13): normalized attention Shannon entropy at intermediate layer l=20l=20, batch-level Softmax (τ=0.8\tau=0.8) with Min-Max scaling, and a linear Low2High warmup over Swarm=80S_{\text{warm}}=80 steps (wlow:1.0→0.0w_{\text{low}}: 1.0 \to 0.0, whigh:0.0→1.0w_{\text{high}}: 0.0 \to 1.0) using all-token-weighted normalization (without group-wise RMS renormalization).

  2. Controlled GRPO Backbone vs. DAPO: To isolate the effect of token-level credit assignment, all compared methods (GRPO, AttnEntropy, and ABPO) share the exact same vanilla GRPO backbone (symmetric clipping, no dynamic filtering, and identical rollout/optimization settings). In contrast, the original paper applies AttnEntropy on top of DAPO, whose asymmetric clip-higher and dynamic filtering provide substantial orthogonal stabilization.

  3. Behavior Under Vanilla GRPO: Without DAPO's clip-higher or group-wise RMS advantage renormalization, AttnEntropy's early emphasis on low-entropy tokens (wlow:1→0w_{\text{low}}: 1 \to 0) tends to over-concentrate updates and suppress exploration diversity—for instance, on CrossThink-QA, policy token entropy dropped by ~80% (0.5071→0.10570.5071 \to 0.1057) and pass@16 fell below vanilla GRPO (0.5440 vs. 0.8370). By contrast, ABPO pairs structural causal-flow centrality with group-wise RMS advantage renormalization, preserving gradient scale and exploration even without DAPO's extra stabilizers.

Reviewer_6WTyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author proposes ABPO, a GRPO-based method that reweights token-level advantages using the model’s attention patterns. It constructs an attention-derived graph, computes AnchorRank scores through recursive propagation and BridgeTrace scores from incoming and outgoing path flows, and combines them into token-wise multipliers. The aim is to emphasize positions carrying intermediate results or connecting reasoning stages without an additional critic or external process supervision.

The authors report improvements over GRPO, shorter responses, and faster time to selected training-reward thresholds despite higher per-step costs. Validation includes component and fusion ablations, random and entropy-based weighting controls, token-masking interventions, and a small semantic analysis of highly weighted segments.

Critical strengths
  1. Clear presentation: The paper is generally easy to follow, with intuitive illustrations of the Anchor and Bridge roles and a well-organized explanation of how the resulting scores reweight GRPO advantages.

  2. Promising empirical performance: ABPO consistently improves over GRPO across the evaluated mathematical benchmarks and additional domains, while generally producing shorter responses. These results suggest practical value for reasoning-oriented RL training.

Critical weaknesses
  1. Unclear attribution interpretation: Attention is a reasonable signal, but it is not clear why these scores identify anchors and bridges or justify stronger updates. AnchorRank accumulates scores downstream rather than directly measuring how often a token is reused. The semantic and masking analyses suggest the selected tokens matter, but do not fully support the credit-assignment interpretation.

  2. Missing matched baselines: Random and entropy weighting are useful controls, but simpler attention-based scores and existing attention-based methods are not compared. These comparisons would help show whether the graph computations are necessary, especially given the modest math gains and extra training cost.

  3. Missing cross-domain details: The largest gains come from non-math tasks, but the dataset subsets, splits, and reward checks are not clearly described. These experiments also lack the stronger baselines and weighting controls used in the math evaluation, making the gains harder to assess.

  4. Limited efficiency evaluation: The timing comparison uses a single seed, and some runs were performed on different nodes and dates. The speedup is measured at selected training-reward thresholds rather than the same test accuracy. Repeated comparisons and longer-context experiments would help show how broadly the efficiency gains hold.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response 1 to Reviewer 6WTy: Mathematical Attribution Interpretation, Cross-Domain Baselines, and Efficiency

Thank you for recognizing the clarity of our presentation and ABPO's consistent gains in accuracy and response length. Below we address attribution interpretation (W1), matched attention baselines and overhead (W2), cross-domain setup and baselines (W3), and training efficiency (W4).

W1 / Q1: Mathematical interpretation of AnchorRank, BridgeTrace, and credit assignment

We agree that AnchorRank accumulates attribution downstream rather than directly counting future token reuse. Below we show (1) why both operators isolate anchors and bridges, and (2) why stronger updates at these positions are principled.

1. Exact path-flow characterization of AnchorRank and BridgeTrace. Since Wij=AˉjiW_{ij}=\bar{A}_{ji} points from source ii to consumer jj (Wij>0  ⟹  i<jW_{ij} \gt 0 \implies i \lt j), both WW and P=WnormP=W_{\mathrm{norm}} are strictly upper triangular (WN=PN=0W^N=P^N=0), so (I−dP⊤)p=(1−d)v(I-dP^\top)p=(1-d)v expands as:

pj=(1−d)vj+d∑i<jPijpi=(1−d)∑k=0N−1dk∑i0<⋯<ik=jvi0∏r=0k−1Pir,ir+1.p_j=(1-d)v_j + d\sum_{i \lt j}P_{ij}p_i =(1-d)\sum_{k=0}^{N-1}d^k\sum_{i_0 \lt \cdots \lt i_k=j}v_{i_0}\prod_{r=0}^{k-1}P_{i_r,i_{r+1}}.

Thus pjp_j sums length-discounted incoming paths terminating at jj—identifying downstream consolidation points rather than reuse counts. Along smooth continuations, ∑i<jPijpi≈∑i<j−1Pi,j−1pi\sum_{i \lt j}P_{ij}p_i \approx \sum_{i \lt j-1}P_{i,j-1}p_i keeps pj−pj−1p_j-p_{j-1} near zero (median 0.002 vs. baseline 1440.39 in Appendix E.5, Case 3); incoming mass jumps sharply at consolidation points (up to 25.05), which ΦA\Phi_{\mathrm{A}} extracts via standardized first differences. For BridgeTrace, the finite expansion S=(I−W)−1=∑k=0N−1WkS=(I-W)^{-1}=\sum_{k=0}^{N-1}W^k factorizes all weighted paths through node ii:

bi2=Ini⋅Outi=(∑u≤iSui)(∑v≥iSiv)=∑u≤i≤v∑π∈P(u→i→v)w(π),b_i^2 = \mathrm{In}_i\cdot\mathrm{Out}_i = \left(\sum_{u\le i} S_{ui}\right)\left(\sum_{v\ge i} S_{iv}\right) = \sum_{u\le i\le v} \sum_{\pi\in\mathcal{P}(u\to i\to v)} w(\pi),

where w(π)=∏(a,b)∈πWabw(\pi)=\prod_{(a,b)\in\pi}W_{ab}, since any DAG path through ii splits uniquely into a prefix u→iu\to i and suffix i→vi\to v. With smoothing and detrending, the geometric mean bi=Ini⋅Outib_i=\sqrt{\mathrm{In}_i\cdot\mathrm{Out}_i} suppresses pure sources and sinks, peaking at multi-hop transition bottlenecks that both receive and forward context. Consistently (Table 5, 200 MATH500 traces), AnchorRank places 84.6% of top-segment gain on computations (64.3%) and final answers (20.3%), whereas BridgeTrace places 67.8% on entries (25.4%) and transitions (42.4%).

2. Why stronger updates on these tokens are justified, and how the evidence closes the loop. ABPO does not interpret Mi,tM_{i,t} as an unbiased token-level reward estimator. In critic-free RLVR with only a sequence-level advantage AiA_i, ABPO applies detached multipliers Mi,t≥1M_{i,t}\ge 1 followed by prompt-group GroupRMS normalization:

A^i,t=AiMi,t1Nvalid∑i,tmi,t(AiMi,t)2,1Nvalid∑i,tmi,tA^i,t2=1.\hat{A}_{i,t} = \frac{A_i M_{i,t}}{\sqrt{\frac{1}{N_{\mathrm{valid}}}\sum_{i,t} m_{i,t}(A_i M_{i,t})^2}}, \qquad \frac{1}{N_{\mathrm{valid}}}\sum_{i,t} m_{i,t}\hat{A}_{i,t}^2 = 1.

Because Mi,t>0M_{i,t} \gt 0, A^i,t\hat{A}_{i,t} preserves the sign of AiA_i and vanishes when Ai=0A_i=0, keeping GRPO's stationary points intact without reward bias. Under this unit-RMS budget E[A^2]=1\mathbb{E}[\hat{A}^2]=1, upweighting high-attribution hubs (Mi,t>1M_{i,t} \gt 1) causes the larger GroupRMS denominator to implicitly downweight ordinary filler (below 11), focusing finite update capacity onto outcome-decisive positions—reinforcing pivotal steps when Ai>0A_i \gt 0 and penalizing erroneous forks when Ai<0A_i \lt 0. Masking ABPO-selected tokens as attention keys (Table 1, 100 MATH500 traces) severs their outgoing flow Outt\mathrm{Out}_t and causes substantially larger disruptions than equal-count random masks (41% vs. 29% answer changes, 29% vs. 17% correctness reversals, and 0.196 vs. 0.135 nats/token log-prob drops). Crucially, budget-matched RL controls close the loop: moving the exact same multiplier increments to random (38.9) or high-entropy (39.3) positions, or using non-graph Attention-Entropy Reweighting (38.9), matches GRPO (39.0), whereas targeting Anchor and Bridge positions reaches 41.6 (Table 2) and cuts cross-domain length by 30.15% with +7.0 Pass@1 gain (Table 3, Appendix E.4).

AuthorsOpenReview ↗

Response 2 to Reviewer 6WTy: Mathematical Attribution Interpretation, Cross-Domain Baselines, and Efficiency

W2 / Q2: Graph-computation cost and matched attention-based comparison

Graph scoring adds modest per-step overhead, and under matched conditions ABPO outperforms Attention-Entropy Reweighting by +2.7 average Pass@1 points (41.6 vs. 38.9). Unlike our High-Entropy control (output-vocabulary predictive entropy), Attention-Entropy Reweighting computes normalized entropy over preceding context attention positions and applies dynamic soft token reweighting, serving as a direct non-graph attention baseline on Qwen3-8B-Base. Evaluated under the exact Table 2 protocol on DAPO-Math-17K (step-200 checkpoint, 2,048-token cap, 16 samples/question), Attention-Entropy Reweighting averages 38.9—essentially matching GRPO (39.0) and Random credit (38.9)—whereas ABPO reaches 41.6 and leads on all four benchmarks:

MethodCredit / Attention MechanismAIME 2024AIME 2025MATH500OlympiadBenchAvg.
GRPOUniform sequence-level advantage14.813.478.549.239.0
DAPOOutcome RLVR recipe (dynamic sampling, clip-higher)15.015.679.249.539.8
+ Random creditBudget-matched random token placement14.911.679.149.938.9
+ High-entropy creditOutput-vocabulary predictive entropy15.113.578.750.039.3
Attention-Entropy ReweightingContext-position attention entropy dynamic soft reweighting14.411.879.549.738.9
+ BridgeTraceMulti-hop path-flow mediation on attention DAG16.214.079.050.139.8
+ AnchorRankRecursive downstream attribution on attention DAG15.114.580.050.139.9
ABPOFused AnchorRank + BridgeTrace attribution17.515.581.252.341.6

Per-step computational cost. In Appendix E.6 (Table 9), ABPO's attribution pipeline on Qwen3-8B-Base takes 30.6 s/step (7.5% of its 406.8 s step; 8.6% on Qwen3-4B-Base), comprising 17.9 s for attention extraction/aggregation and only 12.7 s (3.1% of the step) for AnchorScore, BridgeScore, and fusion; total step time increases by +14.6% over GRPO (355.0 s) on 8B and +6.8% on 4B. On the same 8×A100 server setup, Attention-Entropy Reweighting spends 23.1 s/step on its reweighting pipeline—17.3 s for attention extraction/aggregation and 5.8 s for entropy-weight calculation—meaning ABPO's multi-hop graph computation adds only 6.9 s/step (12.7 s vs. 5.8 s) beyond direct context-attention statistics while yielding +2.7 average Pass@1 points.

W3 / Q3: Cross-domain experimental setup and additional baselines

MedQA and FinQA use their official dataset splits directly, whereas Countdown and CrossThink QA use disjoint splits sampled from their upstream pools because neither release provides a standalone official test file. Within each domain, all methods share identical splits, prompts, and final-answer verifiers (separated from training format rewards):

DatasetTrain / Val / TestSplit SourceFinal-Answer Verifier
MedQA10,178 / — / 1,273Official dataset splits (USMLE 4-option)Extracted option-set equality
FinQA6,251 / 883 / 1,147Official dataset splitsNumerical / symbolic equivalence
Countdown18,720 / — / 1,024Disjoint sampled splits (Countdown-Tasks-3to4)Target value + exact number-multiset usage
CrossThink QA9,000 / — / 1,000Disjoint sampled splits (Nemotron-CrossThink train_qa)Extracted option-set equality

Following the Section 4.1 OpenRLHF setup on Qwen3-8B-Base, we add the untrained Base model, DAPO, and Attention-Entropy Reweighting under identical domain-specific training and evaluation protocols. Across all four domains, ABPO consistently outperforms both DAPO and Attention-Entropy Reweighting; notably, on CrossThink QA, Attention-Entropy's Low2High schedule triggers premature exploration collapse and drops to 32.8 (below GRPO's 42.3):

MethodMedQAFinQACountdownCrossThink QA
Base (Qwen3-8B-Base)61.76.71.415.7
GRPO60.311.961.842.3
DAPO65.419.362.643.9
Attention-Entropy Reweighting67.020.262.232.8
+ BridgeTrace68.422.463.642.9
+ AnchorRank70.019.673.644.0
ABPO68.522.967.545.5
AuthorsOpenReview ↗

Response 3 to Reviewer 6WTy: Mathematical Attribution Interpretation, Cross-Domain Baselines, and Efficiency

W4 / Q4: Training efficiency and experimental comparability

With attribution overhead included, ABPO reaches higher held-out test accuracy in substantially less net training time on identically configured hardware. In Appendix E.6, "different dates and nodes" refers only to scheduling runs at different times across identically configured 8×A100 nodes. Within each comparison, all methods share the same training seed to align prompt order for paired comparison, whereas the main benchmark scores average five independent runs.

On held-out math benchmarks (Figure 4, Qwen3-8B-Base), ABPO reaches 40.5% average Pass@1 in 10.93 net hours (at step 100), already surpassing GRPO's step-200 accuracy of 39.0% in 19.64 net hours—a 1.80× wall-clock speedup with attribution overhead included. Finer-grained training-reward trajectories show the same pattern: ABPO reaches the 0.15 reward target in 52 steps (5.5 net hours) vs. 145 steps (14.0 net hours) for GRPO (2.56×, Table 8), and maintains a 1.90×–2.37× speedup across smoothing schemes at target 0.10 (Table 10); overall, cutting the required optimization steps far outweighs the +14.6% per-step attribution cost.

Thank you again for your constructive feedback. We have incorporated these derivations, baseline comparisons, and experimental clarifications into our rebuttal revision.

Reviewer_KWCuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ABPO, a token-level advantage reweighting method based on attention graph structures. This method combines information convergence and bridging features to adjust the update intensity of GRPO at different inference positions, without requiring an additional value model or process supervision. Experiments report accuracy improvements on multiple inference tasks, and, in some settings, shorten output and accelerate convergence.

Critical strengths
  1. For the fine-grained credit allocation problem under token-level rewards, this approach utilizes internal model signals to improve updates. This is easier to integrate into existing RLVR processes than additionally training a value model or relying on expensive process supervision, representing a clear engineering advantage.

  2. Sufficient experimental data exists on mathematical tasks. The experimental setting, employing multiple independent experiments with a fixed checkpoint (200 steps), enhances the reliability of the method.

  3. Well-controlled stochastic and entropy experiments are provided, which help examine whether the reward depends on the weight allocation position and further demonstrate that the selected position has a stronger impact on the model output.

Critical weaknesses
  1. The actual updates of ABPO depend not only on the graph score, but also on the score-to-multiplier mapping, graph normalization, and the final pruning/RMS processing. However, the current paper's definitions of these key operations are insufficient to uniquely reconstruct the complete flow from attention to training weights, and no related code has been open-sourced.

  2. The experiment does not directly compare the nearest-neighbor attention advantage reweighting method with simple graph statistics and location baselines; I believe it fails to adequately distinguish the independent gains of multi-hop propagation from general attention weighting or location effects .(page 4, lines 170-171 explicitly state: “Li et al. (2025) identified a recurring pre-planned-anchoring rhythm from attention dynamics for reweighting advantages.” )However, the experimental setup described on page 6, lines 298-307, and Tables 2-3 on page 7 primarily list comparisons of Base, GRPO, DAPO, Random, Entropy, and the authors’ own single-branch approach. While Random and Entropy preserve the number and size of increments, providing valuable control, they are not equivalent to reproducing the aforementioned nearest-neighbor methods.

Questions for authors
  1. Referring to Weaknesses 1,is there a complete formula and parameters for the two-branch mapping, the zero outgoing edge node convention, the exact objects and computation axes for clipping/RMS, and step-by-step pseudocode containing all post-processing?
  2. Referring to Weaknesses 2,Is there a comparison between nearest neighbor methods with the same backbone, data, and training budget?
2 comments
AuthorsOpenReview ↗

Response 1 to Reviewer KWCu: Weight Specification, Parameter Robustness, and Attention Baselines

Thank you for recognizing the value of internal signals and our controlled weighting experiments. Below we specify the training weights (W1) and provide matched attention-based comparisons (W2).

W1 / Q1: Complete specification and design rationale of the training weights

Both branch mappings (ΦA,ΦB\Phi_{\mathrm{A}}, \Phi_{\mathrm{B}}) extract sparse, bounded response-local deviations from raw graph scores before prompt-group RMS normalization. Below are the branch pseudocodes, defaults/axes, and design rationale.

Anchor branch (main steps).

Input: AnchorScores p, width T, group advantages A
1. w ← T · Softmax_valid(p / τ)
2. δ_1 ← 0; δ_t ← w_t − w_(t−1), t ≥ 2
3. z_t ← (δ_t − mean(δ)) / (std(δ) + ε)
4. s_t ← |z_t| · 1[|z_t| > κ]
5. M_A,t ← 1 + α · tanh(s_t / c)
6. B ← A · Detach(M_A)
7. Ahat ← B / (GroupRMS(B) + ε)

Bridge branch (main steps).

Input: smoothed, detrended Bridge residual r = MA_3(w) − MA_21(MA_3(w)), advantages A
1. z_t ← (r_t − mean(r)) / (std(r) + ε)
2. s_t ← max(|z_t| − κ, 0)
3. M_B,t ← clip(1 + α · s_t, m_min, m_max)
4. B ← A · Detach(M_B)
5. Ahat ← B / (GroupRMS(B) + ε)

Defaults and axes. On the valid graph (zero-outdegree rows of WnormW_{\mathrm{norm}} remain zero), token scores align to action tt. Fixed defaults across all runs are τ=0.5\tau=0.5, α=0.5\alpha=0.5, Anchor κ=1.04\kappa=1.04 (c=3c=3), Bridge windows (3,21)(3, 21), κ=1.0\kappa=1.0, [mmin⁡,mmax⁡]=[0.2,2.0][m_{\min}, m_{\max}]=[0.2, 2.0], and ε=10−8\varepsilon=10^{-8}. Branch mean/std are per-response. Full ABPO fuses MA,MBM^{\mathrm{A}}, M^{\mathrm{B}} via Eqs. (7)–(8) into Bi,t=AiMi,tB_{i,t}=A_i M_{i,t}; GroupRMS(B) pools all valid response tokens across the G=8G=8 rollouts without centering, ∑i,tmi,tBi,t2/∑i,tmi,t\sqrt{\sum_{i,t} m_{i,t} B_{i,t}^2 / \sum_{i,t} m_{i,t}}, before PPO ratio clipping on ρi,t\rho_{i,t}. Because thresholding selects sparse deviations, post-hoc checks show low sensitivity: sweeping α∈[0.2,1.0]\alpha\in[0.2,1.0] keeps support Jaccard at 0.994 / 1.000 (Anchor/Bridge), and κ∈[0.8,1.6]\kappa\in[0.8,1.6] keeps semantic JS divergence within 0.001–0.004. All runs reuse these fixed defaults.

Design rationale.

  • Anchor (ΦA\Phi_{\mathrm{A}}): Preprocessed AnchorScores have a high, slowly varying baseline punctuated by sharp jumps (Case Study 3, Appendix E.5: median 1440.39 vs. step-difference median 0.002, peaks up to 25.05). First-differencing removes this baseline to isolate abrupt shifts; per-response z-scoring normalizes scale; hard thresholding (∣zt∣>1.04\lvert z_t \rvert \gt 1.04, top ~30% normal tail; 29.7% selected on MATH500) keeps routine tokens at weight 1; ∣zt∣\lvert z_t \rvert captures magnitude regardless of sign; and tanh⁡(⋅/3)\tanh(\cdot/3) caps extreme spikes.
  • Bridge (ΦB\Phi_{\mathrm{B}}): Raw BridgeScores (bib_i, typically 1.7–2.3 with 8–9 peaks) exhibit positional drift. Short smoothing minus wide detrending, r=MA3(w)−MA21(MA3(w))r = \mathrm{MA}_3(w) - \mathrm{MA}_{21}(\mathrm{MA}_3(w)), suppresses point noise and local background, isolating sharp structural deviations; z-scoring, soft thresholding max⁡(∣zt∣−1,0)\max(\lvert z_t \rvert-1,0), and [0.2, 2.0] clipping produce smooth bounded multipliers. Replay on MATH500 shows window robustness: varying smoothing (3→5/73\to 5/7) or trend (21→7/41/6121\to 7/41/61) keeps semantic JS divergence at 0.008–0.028 (vs. 0.127 without detrending), and Qwen3-8B training with window 21 vs. 61 shifts avg. Pass@1 by only 0.5 (39.8 vs. 39.3, both above GRPO 39.0).
  • GroupRMS and semantics: Since multipliers (Mi,t≥1M_{i,t}\ge 1) inflate total advantage norm, GroupRMS stabilizes update scale while preserving relative weights and the sign of AiA_i. Consistently, in 200 MATH500 traces (top-12 segments/method, Appendix E.1), Anchor places 84.6% of gain on computations/answers and Bridge places 67.8% on entries/transitions.
AuthorsOpenReview ↗

Response 2 to Reviewer KWCu: Weight Specification, Parameter Robustness, and Attention Baselines

W2 / Q2: Comparisons with the closest attention-based methods

We add matched comparisons against Coupled Rhythm (Li et al., 2025) and Attention-Entropy Reweighting. Coupled Rhythm couples local attention-distance shifts with future attention influence; Attention-Entropy Reweighting applies dynamic soft weights from normalized context-position attention entropy.

2,048-token Table 2 setting (Qwen3-8B-Base, DAPO-Math-17K). Under the Table 2 protocol, Attention-Entropy Reweighting averages 38.9—matching GRPO (39.0) and Random (38.9)—whereas ABPO achieves 41.6 (+2.7) and leads all four tasks:

MethodCredit / Attention MechanismAIME 2024AIME 2025MATH500OlympiadBenchAvg.
GRPOUniform sequence-level advantage14.813.478.549.239.0
DAPOOutcome RLVR recipe (dynamic sampling, clip-higher)15.015.679.249.539.8
+ Random creditBudget-matched random token placement14.911.679.149.938.9
+ High-entropy creditOutput-vocabulary predictive entropy15.113.578.750.039.3
Attention-Entropy ReweightingContext-position attention entropy with dynamic soft reweighting14.411.879.549.738.9
+ BridgeTraceMulti-hop path-flow mediation on attention DAG16.214.079.050.139.8
+ AnchorRankRecursive downstream attribution on attention DAG15.114.580.050.139.9
ABPOFused AnchorRank + BridgeTrace attribution17.515.581.252.341.6

1,024-token Coupled Rhythm comparison (Qwen3-8B-Base, DAPO-Math-17K). Under Coupled Rhythm's reported 1,024-token cap, ABPO improves GRPO response-1024 from 34.2 to 37.6 (+3.4), matching Coupled Rhythm's average (37.6), tying MATH500 (79.7), and leading OlympiadBench (47.8 vs. 47.0):

Method (1,024-token cap)AIME 2024AIME 2025MATH500OlympiadBenchAvg.
GRPO response-10249.27.575.844.134.2
ABPO response-102410.811.979.747.837.6
Coupled Rhythm (Li et al., 2025, reported)11.412.379.747.037.6

Thank you again for your constructive feedback. We have incorporated these specification details and baseline comparisons into our rebuttal revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.