- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors propose a method to post-train LLMs with credit assignment. The proposed approach makes use of attention scores between tokens to do dense token level credit assignment.
Critical strengths
- The work studies an important problem of credit assignment in post-training of LLMs.
- The proposed approach provides improvement over GRPO, highlighting the benefit of their credit assignment mechanism.
Critical weaknesses
- I found the paper very hard to follow. I think the introduction would benefit from providing more context. For e.g., the discussion in the 2nd para of the introduction can be moved into related works.
Since my understanding of the paper is limited, my questions are more at a high level:
- Could the authors clarify how their credit assignment mechanism based on attention scores differs from approaches that calculate advantages based on differences between state values, such as [1]? It seems that advantages computed using attention scores might not correlate with those computed using state values. I understand that approaches such as [1] are computationally expensive and that learned value networks can be inaccurate. My question is therefore not about which approach is better, but rather about how the two approaches differ in the objectives they optimize.
[1]: Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., & Roux, N. L. (2024). Vineppo: Refining credit assignment in rl training of llms. arXiv preprint arXiv:2410.01679.
Questions for authors
Please refer weaknesses
8 comments
Flow weighting improves all four base algorithms, including those whose advantages already vary within a response. The additional gains of 1.2–1.5 percentage points on the value-based methods support the complementarity of advantage estimation and flow weighting. The larger gain with GRPO is consistent with its response-wide advantage leaving more scope for token-level allocation. Even with step-level values, flow can distinguish how individual tokens participate in the group's outcome-directed paths.
As Appendix B.6 states, the graph interpretation does not establish that the resulting weighted advantage estimates a local action advantage or a causal reward contribution. The masking evidence supports the relevance of the selected positions: masking the top-20% flow tokens flips correctness in 47.4% of resamples, compared with 14.0% for count-matched random masks (Table 2a).
3) Reweighting changes the gradient, and allocation controls assess its practical value.
For a fixed checkpoint and rollout batch, let be token 's base policy-surrogate gradient, including its original loss-aggregation weight, and let contain the selected tokens. Then
=g_{\mathrm{base}}+0.5\sum_{i\in H}u_i.$$ The selected tokens' summed gradient need not be parallel to the aggregate gradient. For Qwen3-4B/1K, we summarize this deviation using the **share of the raw actor policy-surrogate gradient's squared norm that lies orthogonal to the reference gradient**: $$s_\perp=\frac{\lVert g-\alpha g_{\mathrm{ref}}\rVert_2^2}{\lVert g\rVert_2^2}, \qquad \alpha=\frac{\langle g,g_{\mathrm{ref}}\rangle}{\lVert g_{\mathrm{ref}}\rVert_2^2}.$$ | Policy-gradient comparison | Reference gradient | Orthogonal share (%) | |---|---|---:| | GRPO, all policy terms multiplied by $1.2$ | GRPO | 0.00 | | GRPO + randomly shuffled flow weights | GRPO | 3.33 | | GRPO + FlowWeaver, mean credit normalized to 1 | GRPO | 3.77 | | GRPO + FlowWeaver | GRPO | 3.78 | | PPO-GAE + FlowWeaver | PPO-GAE | 93.43 | | PPO-GAE | GRPO | 94.41 | **The orthogonal component persists after mean-credit normalization: 3.78% versus 3.77% of the squared gradient norm.** The direction change is larger when FlowWeaver is applied to PPO-GAE, with 93.43% orthogonal to the PPO-GAE reference. A direction change alone does not establish better credit assignment, as the random-weight control illustrates. The learning ablations provide the complementary evidence. In Table 3, random weighting improves the math average by at most 0.6 percentage points across the four settings. Table 13 further controls the mean credit: | Method | Mean credit per response | Math Avg. (%) | Gain over GRPO (pp) | |---|---:|---:|---:| | GRPO | 1 | 44.8 | — | | GRPO, uniform upscale within each response | $\approx 1.2$ | 44.6 | −0.2 | | FlowWeaver, normalized | 1 | 49.9 | **+5.1** | | FlowWeaver | $\approx 1.2$ | 50.4 | **+5.6** | *Table 13 setting: Qwen3-4B-Base, 8K context; averages over three training seeds.* **Most of the gain is retained at mean credit one, supporting the value of where the additional emphasis is assigned.** This controls the average loss coefficient; it does not equate the methods' gradient norms or optimization paths. **4) Additional value-based baselines provide empirical context.** To complement the objective-level explanation, we add PPO-GAE, PPO-MC, and VinePPO results on Qwen3-4B with 1K context. These supplement the AsyPPO and TEMPO comparisons already in Table 3. | Method | AIME24 | AIME25 | AMC23 | MATH500 | Olympiad | Avg. | |---|---:|---:|---:|---:|---:|---:| | GRPO | 8.4 | 5.2 | 55.1 | 74.2 | 42.8 | 37.1 | | PPO-GAE | 10.0 | 5.4 | 56.5 | 75.8 | 41.6 | 37.9 | | PPO-MC | 9.4 | 5.8 | 57.5 | 75.6 | 43.0 | 38.3 | | VinePPO | 9.8 | 6.6 | 57.0 | 76.4 | 43.7 | 38.7 | | FlowWeaver (GRPO + flow weights) | **11.8** | **7.8** | **61.7** | **77.4** | **45.7** | **40.9** | *Scores are percentages; Avg. is the unweighted mean over the five benchmarks.* All three value-based baselines improve on GRPO in this setting. FlowWeaver achieves a further **2.2–3.0 percentage points** over these baselines without a learned critic or additional continuations for value estimation. Together with the combination results above, these findings support flow weighting as a useful credit-assignment rule with both group-based and value-based advantages. We will revise Section 4.3 to make the relationship between the task objective, advantage estimation, and token weighting explicit, and include the new combination results alongside the comparison and gradient analysis. We are grateful for this question, which helps us present the method's relationship to value-based credit assignment more precisely. ---Q2: The method is relatively difficult to follow; the introduction needs more context and less related-work detail.
Thank you for taking the time to read the paper carefully and for pointing out where more context is needed. We are sorry that the presentation made the method difficult to follow; we should have introduced its building blocks more gradually. We would be very happy to clarify any part, including the background or an individual step, and to work through further examples. We agree with moving the detailed comparison in the second introduction paragraph to Related Work. Below, we start from the PPO setting, explain GRPO and information flow, and then bring these pieces together in FlowWeaver.
1) From PPO to the credit-assignment question.
In a language-model policy, generating a token is an action, and the preceding text is its state. PPO uses an advantage to indicate whether that action should become more or less likely, typically using a learned value function to estimate this signal. For a math problem, the external reward may only report whether the final answer is correct, so the training procedure must turn that final feedback into signals for the preceding token decisions.
Our main experiments use GRPO (Group Relative Policy Optimization). It retains a PPO-style clipped policy update and obtains advantages from several sampled solutions to the same problem, without a learned value function. In the outcome-only setting, it compares each solution's reward with the group's average and normalizes by the reward variation within the group. The resulting advantage is shared by all tokens in that solution.
Consider three such solutions: two reach the correct answer and one does not. Each solution, together with its prompt, is a trajectory. The two correct solutions receive positive advantages and the incorrect one a negative advantage. Within each solution, however, a key intermediate result and a repeated sentence receive the same advantage. Which token decisions should receive more emphasis from this learning signal? This is the credit-allocation question we investigate.
2) What “information flow” means here.
Attention lets a later token's representation incorporate information from earlier positions. For example, a solution may derive , use that result to compute , and report 6 as the answer. The tokens expressing the intermediate result can therefore supply information used later in the solution. We represent token positions as nodes and draw connections from earlier positions to later positions that attend to them. Chains of these connections form paths from the prompt, through intermediate tokens, to the answer.
To measure participation in these paths, imagine sending a total of one unit of traffic from the prompt toward the answer. At each position, the incoming traffic is split among onward routes according to their attention weights and downstream connections to the answer; routes that cannot reach the answer receive none. A token's flow score is the total traffic passing through it. If 0.6 of the initial unit passes through a token, its flow score is 0.6. This attention-based propagation through the constructed graph is what we mean by information flow (Section 3).
Applied to each solution separately, this is trajectory-local Flow, our prior method FlowTracer (ref. [6]). The score measures participation in attention-based paths. A high score alone does not establish that a reasoning step is correct or identify its causal contribution to the reward (Appendix B.6).
3) Why connect several solutions, and how does FlowWeaver do this?
In our three-solution example, each isolated graph leads only to its own answer. The other solutions provide alternative reasoning paths and different outcomes. FlowWeaver brings these paths into a shared graph, allowing us to assess token participation in routes toward correct or incorrect answers across the group. Two steps make this possible.
First, we need a correspondence between positions in solutions with different lengths and wording. We add up each solution's local token-flow scores in token order and rescale the total to 100%. Each token then occupies an interval on this cumulative-flow scale: a token that raises the running total from 30% to 35% covers that interval. Positions with overlapping intervals in different solutions supply one another with onward routes. Propagation can then follow a position's own routes or routes borrowed from its peers, forming the shared graph (Section 4.1). This correspondence uses accumulated flow without requiring matching words or equivalent reasoning steps.
We sincerely thank the reviewer for the thoughtful evaluation and helpful suggestions. Your questions about the relationship to value-based methods, together with your feedback on the presentation, help us explain the method more clearly. Below, we respond to each of your questions and concerns point by point.
Thank you for this thoughtful question. We appreciate the distinction you draw between empirical performance and the underlying optimization objective. The shared task goal is to improve expected outcome reward; FlowWeaver changes how token contributions are weighted in the policy surrogate used to pursue that goal. Value-based methods estimate advantages from expected returns, while FlowWeaver uses the completed trajectories' attention structure and outcomes to assign positive weights to those advantages.
1) The task objective is shared; FlowWeaver changes the token weighting in the policy surrogate.
The task objective is the expected outcome reward, J(θ)=Ex∼D,y∼π(⋅∣x;θ)[R(x,y)]. A critic estimates expected return to help construct an advantage; our flow score supplies a token weight. Neither quantity is an additional reward to be maximized.
PPO-style methods and GRPO use a clipped policy term of the form
ℓ(ri,Ai)=min(riAi,clip(ri,1−ϵ,1+ϵ)Ai),
where ri is the current-to-rollout policy probability ratio for token i, and Ai is its base advantage. FlowWeaver applies a positive weight ci to that advantage. With the weights held fixed during the update,
ℓ(ri,ciAi)=ciℓ(ri,Ai).
Thus, FlowWeaver uses the same clipped functional form, but optimizes a differently weighted surrogate. As Appendix B.6 makes explicit, this generally changes the expected gradient. We do not claim an unbiased gradient of the original task objective, identical optima, or convergence to the same policy as the base algorithm. As rollouts and credits are refreshed, the procedure optimizes a sequence of weighted surrogates; our analysis does not identify a single fixed expected-reward objective for that full procedure.
2) Advantage estimation and flow weighting provide complementary information.
The following table separates the base advantage from the weighting that FlowWeaver can apply to it. Here, PPO-MC denotes a sampled Monte-Carlo return minus a learned prefix-value baseline.
FlowWeaver builds an answer-directed graph from attention, aligns the group's responses by cumulative flow, and scores each token by its participation in paths toward its response's observed outcome (Eqs. 4–6). The top 40% of tokens receive weight 1.5 and the others weight 1 (Eqs. 7–8). These weights are held fixed with stop-gradient on each rollout batch and apply to the actor's policy term; the remaining loss terms and aggregation are retained.
A value-based advantage estimates how an action compares with the policy's expected return at its prefix; a flow score measures participation in outcome-directed paths in the sampled group's graph. They therefore need not rank tokens in the same order. For example, tokens sharing a step-level advantage can have different flow scores. Flow can supply additional resolution within that step, while the base advantage continues to determine the sign of each token's policy signal. An advantage of zero remains zero after weighting.
We test this compatibility by applying the same flow-weighting rule to each base algorithm's actor policy term. The following paired results use Qwen3-4B with 1K context:
Scores are the unweighted average over the five math benchmarks (%); all combinations use the same top-40% selection and weight of 1.5.