- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper starts from the observation that GRPO gives every token the same credit, while PPO with GAE passes credit back with a fixed discount; neither looks at how the model actually computed each token. The idea is to use attention for this: at each token, the authors measure how concentrated the final-layer attention is, and map it to a gate that replaces the fixed discount in GAE. When attention is concentrated, more of the later reward is passed back to that token; when it is spread out, less. They also add a small critic on top of the actor’s hidden states, which mixes the current token’s state with the attended history using the same gate, so that the critic and the credit transport are aligned. The rest of PPO is unchanged. The authors claim that this outperforms GRPO.
Critical strengths
The idea of using attention to set the discount in GAE is interesting.
Critical weaknesses
-
The writing is hard to follow. A fairly simple change (replacing the fixed discount in GAE with a per-token value computed from attention) is wrapped in a lot of new terms (architecture–credit mismatch, CCT, CompPO, Comp-GAE, TAC, transport operator, retention gate, ...), and so many sentences describe simple things in an abstract way.
-
From the shuffle and position-only controls, it is not clear that the gain really comes from attention. Shuffling the gates across tokens (so they no longer match the attention at all) still gives results close to the full method, and both are far above GRPO.
-
The average gate is around 0.6 per token, while responses go up to 16K tokens. With that much decay, the final reward basically disappears after a few dozen tokens, so it is unclear how this “transports delayed credit” through long reasoning, as the paper claims.
-
Several claims go beyond what the experiments show. For example, the abstract says the results “establish” internal computation as a useful variable for credit, although the shuffled-gate control performs close to the full method; the discussion says CCT composes with DAPO/GSPO/process rewards, but no such combination is tested; and statements like “learned values faithfully track” the targets (EV is around 0.5) or that the paper “re-evaluates” the move away from critics seem too strong for one 4B model on math with 200 training steps.
Questions for authors
Accuracy on a 500-problem set should be a multiple of 0.2%, which holds for all single-run numbers in the paper, but several seed-level values in Tables 8 and 10 (e.g., 61.5, 51.9) are not. How was ID accuracy computed for the confirmation runs?