acceptodds
Under review as a conference paper at ICLR 2027

FlowWeaver: Weaving Reasoning Paths with Cross-Trajectory Flow for Fine-Grained Credit Assignment

Abstract

A language model may make hundreds of token-level decisions before receiving a single outcome reward, while responses sampled for the same prompt can follow different reasoning paths and reach different outcomes. Turning this cross-trajectory variation into fine-grained credit remains challenging: within-trajectory attribution can identify important tokens but evaluates each response largely in isolation, whereas cross-trajectory methods exploit outcome variation at coarser granularity or rely on shared prefixes or states. We introduce FlowWeaver, which weaves sampled reasoning paths into a shared attribution graph through cross-trajectory information flow for fine-grained credit assignment signals. We build on attention-based flow within each response, constructing paths from prompt to answer using attention weights and quantifying the share of graph propagation routed through each token. Using cumulative flow as a common coordinate, we connect sampled responses through aligned token positions, forming a shared distribution over graph paths from the prompt to different outcomes. We then condition propagation on each response's observed outcome, measuring how much of the group's flow toward that outcome passes through each token. We use these importance scores to shape token-level credit by reweighting GRPO advantages. RL experiments across mathematical reasoning, general reasoning, and agentic tasks demonstrate the effectiveness of , highlighting outcome-conditioned flow as a fine-grained signal for allocating outcome supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Reasoning Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NDDmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a method to post-train LLMs with credit assignment. The proposed approach makes use of attention scores between tokens to do dense token level credit assignment.

Critical strengths
  • The work studies an important problem of credit assignment in post-training of LLMs.
  • The proposed approach provides improvement over GRPO, highlighting the benefit of their credit assignment mechanism.
Critical weaknesses
  • I found the paper very hard to follow. I think the introduction would benefit from providing more context. For e.g., the discussion in the 2nd para of the introduction can be moved into related works.

Since my understanding of the paper is limited, my questions are more at a high level:

  • Could the authors clarify how their credit assignment mechanism based on attention scores differs from approaches that calculate advantages based on differences between state values, such as [1]? It seems that advantages computed using attention scores might not correlate with those computed using state values. I understand that approaches such as [1] are computationally expensive and that learned value networks can be inaccurate. My question is therefore not about which approach is better, but rather about how the two approaches differ in the objectives they optimize.

[1]: Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., & Roux, N. L. (2024). Vineppo: Refining credit assignment in rl training of llms. arXiv preprint arXiv:2410.01679.

Questions for authors

Please refer weaknesses

8 comments

We sincerely thank the reviewer for the thoughtful evaluation and helpful suggestions. Your questions about the relationship to value-based methods, together with your feedback on the presentation, help us explain the method more clearly. Below, we respond to each of your questions and concerns point by point.

Q1: How do attention-based credits and state-value advantages (e.g., VinePPO) differ in their rankings and optimization objectives?

Thank you for this thoughtful question. We appreciate the distinction you draw between empirical performance and the underlying optimization objective. The shared task goal is to improve expected outcome reward; FlowWeaver changes how token contributions are weighted in the policy surrogate used to pursue that goal. Value-based methods estimate advantages from expected returns, while FlowWeaver uses the completed trajectories' attention structure and outcomes to assign positive weights to those advantages.

1) The task objective is shared; FlowWeaver changes the token weighting in the policy surrogate.

The task objective is the expected outcome reward, J(θ)=Ex∼D,y∼π(⋅∣x;θ)[R(x,y)]J(\theta)=\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(\cdot\mid x;\theta)}[R(x,y)]. A critic estimates expected return to help construct an advantage; our flow score supplies a token weight. Neither quantity is an additional reward to be maximized.

PPO-style methods and GRPO use a clipped policy term of the form

ℓ(ri,Ai)=min⁡ ⁣(riAi, clip⁡(ri,1−ϵ,1+ϵ)Ai),\ell(r_i,A_i)=\min\!\left(r_iA_i,\,\operatorname{clip}(r_i,1-\epsilon,1+\epsilon)A_i\right),

where rir_i is the current-to-rollout policy probability ratio for token ii, and AiA_i is its base advantage. FlowWeaver applies a positive weight cic_i to that advantage. With the weights held fixed during the update,

ℓ(ri,ciAi)=ciℓ(ri,Ai).\ell(r_i,c_iA_i)=c_i\ell(r_i,A_i).

Thus, FlowWeaver uses the same clipped functional form, but optimizes a differently weighted surrogate. As Appendix B.6 makes explicit, this generally changes the expected gradient. We do not claim an unbiased gradient of the original task objective, identical optima, or convergence to the same policy as the base algorithm. As rollouts and credits are refreshed, the procedure optimizes a sequence of weighted surrogates; our analysis does not identify a single fixed expected-reward objective for that full procedure.

2) Advantage estimation and flow weighting provide complementary information.

The following table separates the base advantage from the weighting that FlowWeaver can apply to it. Here, PPO-MC denotes a sampled Monte-Carlo return minus a learned prefix-value baseline.

Base algorithmHow the base advantage is obtainedPolicy signal with FlowWeaver
PPO-GAEGAE combines temporal-difference residuals from a learned critic.ciAiGAEc_iA_i^{\mathrm{GAE}}
PPO-MCThe sampled return is compared with the critic's predicted value at the prefix.ciAiMCc_iA_i^{\mathrm{MC}}
VinePPOMonte-Carlo continuations estimate values at intermediate reasoning-step prefixes; value differences provide step advantages.ciAiVinec_iA_i^{\mathrm{Vine}}
GRPOThe outcome reward is normalized within a group; the resulting A^τ\hat A_\tau is shared across a response's tokens.cτ,iA^τc_{\tau,i}\hat A_\tau

FlowWeaver builds an answer-directed graph from attention, aligns the group's responses by cumulative flow, and scores each token by its participation in paths toward its response's observed outcome (Eqs. 4–6). The top 40% of tokens receive weight 1.51.5 and the others weight 11 (Eqs. 7–8). These weights are held fixed with stop-gradient on each rollout batch and apply to the actor's policy term; the remaining loss terms and aggregation are retained.

A value-based advantage estimates how an action compares with the policy's expected return at its prefix; a flow score measures participation in outcome-directed paths in the sampled group's graph. They therefore need not rank tokens in the same order. For example, tokens sharing a step-level advantage can have different flow scores. Flow can supply additional resolution within that step, while the base advantage continues to determine the sign of each token's policy signal. An advantage of zero remains zero after weighting.

We test this compatibility by applying the same flow-weighting rule to each base algorithm's actor policy term. The following paired results use Qwen3-4B with 1K context:

Base algorithmUniform weights+ FlowWeaver weightsGain (pp)
GRPO37.140.9+3.8
PPO-GAE37.939.4+1.5
PPO-MC38.339.6+1.3
VinePPO38.739.9+1.2

Scores are the unweighted average over the five math benchmarks (%); all combinations use the same top-40% selection and weight of 1.5.

Flow weighting improves all four base algorithms, including those whose advantages already vary within a response. The additional gains of 1.2–1.5 percentage points on the value-based methods support the complementarity of advantage estimation and flow weighting. The larger gain with GRPO is consistent with its response-wide advantage leaving more scope for token-level allocation. Even with step-level values, flow can distinguish how individual tokens participate in the group's outcome-directed paths.

As Appendix B.6 states, the graph interpretation does not establish that the resulting weighted advantage estimates a local action advantage or a causal reward contribution. The masking evidence supports the relevance of the selected positions: masking the top-20% flow tokens flips correctness in 47.4% of resamples, compared with 14.0% for count-matched random masks (Table 2a).

3) Reweighting changes the gradient, and allocation controls assess its practical value.

For a fixed checkpoint and rollout batch, let uiu_i be token ii's base policy-surrogate gradient, including its original loss-aggregation weight, and let HH contain the selected tokens. Then

=g_{\mathrm{base}}+0.5\sum_{i\in H}u_i.$$ The selected tokens' summed gradient need not be parallel to the aggregate gradient. For Qwen3-4B/1K, we summarize this deviation using the **share of the raw actor policy-surrogate gradient's squared norm that lies orthogonal to the reference gradient**: $$s_\perp=\frac{\lVert g-\alpha g_{\mathrm{ref}}\rVert_2^2}{\lVert g\rVert_2^2}, \qquad \alpha=\frac{\langle g,g_{\mathrm{ref}}\rangle}{\lVert g_{\mathrm{ref}}\rVert_2^2}.$$ | Policy-gradient comparison | Reference gradient | Orthogonal share (%) | |---|---|---:| | GRPO, all policy terms multiplied by $1.2$ | GRPO | 0.00 | | GRPO + randomly shuffled flow weights | GRPO | 3.33 | | GRPO + FlowWeaver, mean credit normalized to 1 | GRPO | 3.77 | | GRPO + FlowWeaver | GRPO | 3.78 | | PPO-GAE + FlowWeaver | PPO-GAE | 93.43 | | PPO-GAE | GRPO | 94.41 | **The orthogonal component persists after mean-credit normalization: 3.78% versus 3.77% of the squared gradient norm.** The direction change is larger when FlowWeaver is applied to PPO-GAE, with 93.43% orthogonal to the PPO-GAE reference. A direction change alone does not establish better credit assignment, as the random-weight control illustrates. The learning ablations provide the complementary evidence. In Table 3, random weighting improves the math average by at most 0.6 percentage points across the four settings. Table 13 further controls the mean credit: | Method | Mean credit per response | Math Avg. (%) | Gain over GRPO (pp) | |---|---:|---:|---:| | GRPO | 1 | 44.8 | — | | GRPO, uniform upscale within each response | $\approx 1.2$ | 44.6 | −0.2 | | FlowWeaver, normalized | 1 | 49.9 | **+5.1** | | FlowWeaver | $\approx 1.2$ | 50.4 | **+5.6** | *Table 13 setting: Qwen3-4B-Base, 8K context; averages over three training seeds.* **Most of the gain is retained at mean credit one, supporting the value of where the additional emphasis is assigned.** This controls the average loss coefficient; it does not equate the methods' gradient norms or optimization paths. **4) Additional value-based baselines provide empirical context.** To complement the objective-level explanation, we add PPO-GAE, PPO-MC, and VinePPO results on Qwen3-4B with 1K context. These supplement the AsyPPO and TEMPO comparisons already in Table 3. | Method | AIME24 | AIME25 | AMC23 | MATH500 | Olympiad | Avg. | |---|---:|---:|---:|---:|---:|---:| | GRPO | 8.4 | 5.2 | 55.1 | 74.2 | 42.8 | 37.1 | | PPO-GAE | 10.0 | 5.4 | 56.5 | 75.8 | 41.6 | 37.9 | | PPO-MC | 9.4 | 5.8 | 57.5 | 75.6 | 43.0 | 38.3 | | VinePPO | 9.8 | 6.6 | 57.0 | 76.4 | 43.7 | 38.7 | | FlowWeaver (GRPO + flow weights) | **11.8** | **7.8** | **61.7** | **77.4** | **45.7** | **40.9** | *Scores are percentages; Avg. is the unweighted mean over the five benchmarks.* All three value-based baselines improve on GRPO in this setting. FlowWeaver achieves a further **2.2–3.0 percentage points** over these baselines without a learned critic or additional continuations for value estimation. Together with the combination results above, these findings support flow weighting as a useful credit-assignment rule with both group-based and value-based advantages. We will revise Section 4.3 to make the relationship between the task objective, advantage estimation, and token weighting explicit, and include the new combination results alongside the comparison and gradient analysis. We are grateful for this question, which helps us present the method's relationship to value-based credit assignment more precisely. ---

Q2: The method is relatively difficult to follow; the introduction needs more context and less related-work detail.

Thank you for taking the time to read the paper carefully and for pointing out where more context is needed. We are sorry that the presentation made the method difficult to follow; we should have introduced its building blocks more gradually. We would be very happy to clarify any part, including the background or an individual step, and to work through further examples. We agree with moving the detailed comparison in the second introduction paragraph to Related Work. Below, we start from the PPO setting, explain GRPO and information flow, and then bring these pieces together in FlowWeaver.

1) From PPO to the credit-assignment question.

In a language-model policy, generating a token is an action, and the preceding text is its state. PPO uses an advantage to indicate whether that action should become more or less likely, typically using a learned value function to estimate this signal. For a math problem, the external reward may only report whether the final answer is correct, so the training procedure must turn that final feedback into signals for the preceding token decisions.

Our main experiments use GRPO (Group Relative Policy Optimization). It retains a PPO-style clipped policy update and obtains advantages from several sampled solutions to the same problem, without a learned value function. In the outcome-only setting, it compares each solution's reward with the group's average and normalizes by the reward variation within the group. The resulting advantage is shared by all tokens in that solution.

Consider three such solutions: two reach the correct answer and one does not. Each solution, together with its prompt, is a trajectory. The two correct solutions receive positive advantages and the incorrect one a negative advantage. Within each solution, however, a key intermediate result and a repeated sentence receive the same advantage. Which token decisions should receive more emphasis from this learning signal? This is the credit-allocation question we investigate.

2) What “information flow” means here.

Attention lets a later token's representation incorporate information from earlier positions. For example, a solution may derive x=3x=3, use that result to compute 2x=62x=6, and report 6 as the answer. The tokens expressing the intermediate result can therefore supply information used later in the solution. We represent token positions as nodes and draw connections from earlier positions to later positions that attend to them. Chains of these connections form paths from the prompt, through intermediate tokens, to the answer.

To measure participation in these paths, imagine sending a total of one unit of traffic from the prompt toward the answer. At each position, the incoming traffic is split among onward routes according to their attention weights and downstream connections to the answer; routes that cannot reach the answer receive none. A token's flow score is the total traffic passing through it. If 0.6 of the initial unit passes through a token, its flow score is 0.6. This attention-based propagation through the constructed graph is what we mean by information flow (Section 3).

Applied to each solution separately, this is trajectory-local Flow, our prior method FlowTracer (ref. [6]). The score measures participation in attention-based paths. A high score alone does not establish that a reasoning step is correct or identify its causal contribution to the reward (Appendix B.6).

3) Why connect several solutions, and how does FlowWeaver do this?

In our three-solution example, each isolated graph leads only to its own answer. The other solutions provide alternative reasoning paths and different outcomes. FlowWeaver brings these paths into a shared graph, allowing us to assess token participation in routes toward correct or incorrect answers across the group. Two steps make this possible.

First, we need a correspondence between positions in solutions with different lengths and wording. We add up each solution's local token-flow scores in token order and rescale the total to 100%. Each token then occupies an interval on this cumulative-flow scale: a token that raises the running total from 30% to 35% covers that interval. Positions with overlapping intervals in different solutions supply one another with onward routes. Propagation can then follow a position's own routes or routes borrowed from its peers, forming the shared graph (Section 4.1). This correspondence uses accumulated flow without requiring matching words or equivalent reasoning steps.

Reviewer_VmzeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FlowWeaver, a credit assignment method that constructs a shared attribution graph from reasoning paths to model how outcome-conditioned information flows through each token. It uses token-level importance scores derived from this graph to refine credit assignment by reweighting GRPO advantages. Experiments across mathematical reasoning, general reasoning, and agentic tasks demonstrate FlowWeaver’s effectiveness, highlighting outcome-conditioned information flow as a fine-grained signal for allocating outcome-based supervision.

Critical strengths
  1. The paper addresses an important bottleneck in reinforcement learning for large language models: the coarse-grained nature of outcome-based supervision, which provides limited guidance for assigning credit to individual tokens or reasoning steps.

  2. The proposed method is presented clearly and intuitively, with a well-structured exposition that makes the overall framework, its key components, and their interactions easy to follow. The paper coherently explains the methodological design and clearly motivates each component within the broader framework. This clarity enables readers to readily grasp the underlying rationale of the approach and understand how its components work together to achieve fine-grained token-level credit assignment. Overall, the well-organized presentation enhances the accessibility of the proposed method and facilitates a comprehensive understanding of its technical contributions.

  3. Results look promising.

Critical weaknesses
  1. The paper would benefit from more detailed case studies and qualitative analyses of credit assignment quality. For example, the authors could compare the token-level credit scores assigned by FlowWeaver with those produced by competing methods to illustrate how different approaches distribute credit across reasoning steps and identify critical tokens. Additionally, where available, comparisons with expert annotations or other forms of ground-truth supervision could provide a more direct assessment of the accuracy and interpretability of the assigned credits. Such analyses would help clarify whether FlowWeaver consistently assigns higher credit to tokens that meaningfully contribute to successful outcomes, beyond merely evaluating its impact on the final scalar reward or overall performance.

  2. Although the authors provide a preliminary analysis of computational overhead (Appendices D.8 and C.1, Table 18), the practical efficiency of FlowWeaver warrants further investigation. The reported attribution pipeline, including attention extraction, accounts for 1.45–2.17% of training-step time on mathematical reasoning tasks and 0.33–0.68% on agentic tasks under the measured configurations. The 8K-sequence benchmark also reports peak GPU memory allocation of 8.848 GiB and sampled CPU process PSS of 5.722 GiB with two groups in flight. While these results suggest relatively modest overhead in the evaluated settings, the quadratic scaling of dense graph storage with sequence length raises concerns about scalability to longer reasoning trajectories and larger workloads. Further evaluation of computational costs across varying sequence lengths, group sizes, and model scales would help establish whether FlowWeaver remains efficient in more demanding settings. Moreover, relating the additional computational costs to the magnitude of performance gains over competing methods would provide a clearer assessment of its practical benefits.

Questions for authors

Please refer to the Weaknesses section, as they are also my concerns and questions for the authors. Any additional clarifications would be welcome.

5 comments

We sincerely thank the reviewer for the careful evaluation and constructive suggestions. We particularly appreciate your questions about the quality of token-level credit assignment and the computational costs of the method. Below, we respond to each of your questions and concerns point by point.

Q1: Does FlowWeaver assign meaningful token-level credit beyond improving final performance? Can case studies, method comparisons, and annotation agreement support this?

Thank you for suggesting a more direct analysis of credit assignment quality. We agree that examining where credit is assigned would make the method's behavior more interpretable. We have added annotation-based evaluations and three side-by-side case studies, comparing how FlowWeaver and competing methods allocate a limited token budget. These analyses examine whether the rankings reach annotated errors, avoid redundant derivations, and distinguish important expressions within a reasoning step.

1) Comparison against error and redundancy annotations.

We evaluate complete FlowWeaver and competing rankings on the same focal trajectories at token budgets of 5%, 10%, 20%, and 40%. On SHARP-MATH and SHARP-GSM8K, error-span hit rate measures whether at least one selected token falls inside an annotated error span. On ReasonEval Perturbation, selected-token redundancy measures the fraction of selected tokens that belong to annotated redundant steps. These annotations provide concrete reference points for evaluating credit placement.

Table R2-A(a). Error-span hit rate (%) ↑ at different token budgets.

Ranking methodSHARP-MATH 5%10%20%40%SHARP-GSM8K 5%10%20%40%
FlowWeaver (ours)32.3447.3378.9092.6520.5630.0042.6070.40
Entropy8.3615.1628.4751.146.4510.5917.6734.83
Surprisal11.8019.0930.7751.2213.1618.2724.8838.39
Qwen2.5-Math PRM26.1945.0966.6186.1216.2427.3644.2466.20
Qwen2.5-Math PRM800K25.4743.5468.8487.5114.6625.3142.9466.28
Math-Shepherd19.4132.8352.5977.1513.3222.5838.8664.67
ReasonEval-7B19.1031.7951.3573.6915.3825.2540.6264.33
Uniform random5.1310.1220.1140.105.2410.2320.2140.21

Budgets are fractions of eligible reasoning tokens; bold marks the highest hit rate in each column. PRM800K denotes the Qwen2.5-Math-7B verifier trained on PRM800K; Qwen2.5-Math PRM denotes Qwen2.5-Math-PRM-7B.

FlowWeaver has the highest error-span hit rate in seven of the eight dataset–budget combinations. At the smallest budget, 5%, it reaches 32.34% of error spans on SHARP-MATH and 20.56% on SHARP-GSM8K, compared with 26.19% and 16.24% for the strongest competing method. The exception is SHARP-GSM8K at 20%, where FlowWeaver reaches 42.60%, below Qwen2.5-Math PRM (44.24%) and the PRM800K-trained verifier (42.94%).

Table R2-A(b). Selected-token redundancy (%) ↓ on ReasonEval Perturbation.

Ranking method5% budget10% budget20% budget40% budget
FlowWeaver (ours)6.717.588.479.04
Entropy14.6413.9713.2912.27
Surprisal13.3013.2812.8912.22
Qwen2.5-Math PRM10.3910.179.449.83
Qwen2.5-Math PRM800K9.149.389.8610.46
Math-Shepherd10.7610.7911.7411.18
Uniform random11.0311.0311.0311.03
Step-length control10.6510.6911.0210.55

Redundancy is the percentage of selected tokens belonging to annotated redundant steps; bold marks the lowest value in each column.

FlowWeaver allocates the smallest share of its selected budget to redundant steps at all four budgets. At 5%, this share is 6.71%, compared with 9.14% for the strongest baseline and 11.03% for uniform random selection. Together, the two evaluations show that the ranking can reach annotated errors while placing less emphasis on redundant content in the respective datasets. Complete FlowWeaver uses peer trajectories and observed outcomes, while the PRMs score the focal reasoning steps; these results compare the methods under their respective inputs. The labels assess error localization and redundancy, rather than supplying a continuous ground-truth importance score for every token.

2) Case studies show where the methods place emphasis within and across steps.

The following examples compare the selected tokens at a 40% token budget. PRM step scores are assigned to the tokens within each step, so tokens in the same step receive the same score. Comparing the selected positions makes the resulting allocation visible on a common token scale.

Table R2-B. Three case studies of token-budget allocation.

PhenomenonExcerpt from the response (intervening text omitted)FlowWeaver selectionPRM selection
A local error inside a long step“14 minutes.” … “6 (singing) + 18 (practicing) + 30 (tantrums) = 14 minutes.” The correct sum is 54.Selects both erroneous occurrences of 14, in the opening answer and the final sum, covering all 4 annotated error tokens.All three PRMs spend their budget within the long explanatory step containing several subcalculations. Equal scores within that step spread selection across correct calculations and explanatory text, while the opening 14 is missed. Reported error-token coverage is 21.8% for each PRM.
An existing error carried into later results“First, we'll use the distributive property …” … “Combine like terms:” “x⁴ − 4x² + 4” … “the expanded product is x⁴ − 4x² + 4.” The correct result is x⁴ + 4.Concentrates on the erroneous 4x² terms repeated in the simplified expression and final answer, covering all 8 annotated error tokens (100%). Selects only 7 tokens in the earlier distributive-property explanation.All three PRMs select the entire earlier distributive-property explanation (40 tokens). Reported coverage of the later annotated errors is 34.6%, 11.5%, and 0%, respectively; more of the budget goes to the method explanation and local multiplication steps.
Correct but unnecessary formula restatement“Using the property of logarithms that log_b(x^y) = y*log_b(x) …” … “Since log_4(4) = 1, this becomes n/d = log_4(8).” Both steps are manually annotated as redundant.Selects only 4 of the 82 tokens in these two steps. More of its budget goes to the target 4^(n/d) = 8 and the use of 4¹ = 4 and 4² = 16 to bound the exponent.Select 42, 31, and 22 of the 82 tokens, respectively. All three PRMs select the entire “Since log_4(4) = 1 …” step, retaining a correct but redundant derivation.

PRM results are ordered as Math-Shepherd, Qwen2.5-Math-7B trained on PRM800K, and Qwen2.5-Math-PRM-7B. Error-token coverage is distinct from the error-span hit rate in Table R2-A(a). In the third case, 82 is the total number of tokens in the two redundant steps.

The first case illustrates why resolution within a step matters: identifying a long step containing an error still leaves the selection budget spread over its correct and incorrect content. The second shows a further distinction across steps: FlowWeaver emphasizes the erroneous expressions as they reappear in later results, while the PRM rankings allocate more tokens to the earlier procedural explanation. The third separates correctness from necessity: a valid derivation can repeat information that is already sufficient for the solution. These cases make the differences in allocation concrete and help interpret the broader annotation-based results. The comparison concerns the tested conversion of PRM step scores into token rankings.

For the first two cases, emphasizing an error has a specific learning interpretation. A high flow score determines where to emphasize the trajectory's outcome signal; the original advantage supplies its sign. On an unsuccessful trajectory with a negative advantage, selecting an erroneous token strengthens the negative learning signal at that position. Thus, the error cases assess whether failure-related emphasis reaches the relevant expressions, while the redundancy case examines whether the budget avoids unnecessary derivations.

3) Masking provides complementary evidence of outcome relevance.

The annotation comparisons complement the intervention in Table 2 of the submission. At the same 20% mask size, masking FlowWeaver's top-ranked tokens changes final correctness in 47.4% of resamples, compared with 23.9% for trajectory-local Flow and 14.0% for random selection. Restricting the comparison to positions on which FlowWeaver and local Flow disagree, masking tokens entering the top set produces 24.1% correctness flips, versus 12.9% for tokens leaving it, at equal per-response mask counts (Table 2b). These flips count changes in both directions and therefore measure outcome sensitivity.

The annotation results, case studies, and masking experiment provide complementary evidence about the placement and relevance of the assigned credits. They support the interpretation that FlowWeaver concentrates supervision on meaningful positions, while leaving the stronger question of each token's causal reward contribution open. We will include these comparisons and case studies in the revised credit-assignment analysis.


Q2: How do FlowWeaver's time and memory costs scale with sequence length, group size, and model size? Do its gains over competing methods justify the overhead?

Thank you for recognizing the modest overhead in our evaluated settings and for suggesting a broader scalability analysis. We extend the measurements to exact sequence lengths of 16K and 32K and report actual policy updates in agentic runs using these context limits. The results show how execution optimizations reduce memory requirements with measurable runtime tradeoffs. We also clarify the remaining quadratic storage cost and relate the attribution computation to the gains in our training experiments.

1) Blockwise computation and bounded scheduling reduce memory pressure at longer sequence lengths.

After sampling, we replay completed trajectories, reconstruct attention in query/head blocks from selected layers' Q/K tensors, and accumulate the layer/head average directly. Fixed-capacity buffers and reuse of attention/flow storage reduce temporary allocations and full-matrix copies. We then solve complete prompt groups, retaining only the token weights and releasing their graphs. A memory reservation covers every group undergoing replay, waiting, or being solved, so the graph budget controls the entire attribution pipeline.

Table R2-C(a). Memory–runtime tradeoff for Q/K-to-flow graph export.

Exact total lengthReference GPU stage memory (GiB)Optimized GPU stage memory (GiB)Memory reductionOptimized construction time (s)Construction-time increase
16K4.9222.07357.9%0.449+29.4%
32K11.8366.14548.1%1.593+20.4%

The comparison preserves the graph construction and covers attention reconstruction from four layers of Q/K and flow-graph export. These are stage-specific memory and timing measurements; model replay, joint solving, and policy optimization are outside this stage. Total length includes the prompt, with 1K = 1,024 tokens.

At 32K, the optimized export uses 48.1% less stage memory, taking 1.593 seconds, with a 20.4% increase in construction time. This provides a measured option for reducing GPU memory demand while retaining the same graph construction.

We also extend the 8K group-scheduling analysis in Appendix C.1 to longer sequences. The following comparison processes the same four complete groups, each containing eight trajectories, using either full-batch graph accumulation or a pipeline with at most two groups in flight.

Table R2-C(b). CPU graph residency and total attribution time for four groups of eight trajectories.

Exact total lengthFull-batch CPU PSS (GiB)Bounded-pipeline CPU PSS (GiB)Total attribution-time increase
16K34.29917.971+4.38%
32K130.25566.080+8.84%

PSS is sampled process proportional set size. Attribution time covers replay, graph export, and CPU solving; the increases are paired comparisons between schedules and exclude generation and policy optimization. Both schedules retain the same scores and weights assigned to the top 40% of tokens.

At 32K, bounded scheduling reduces CPU PSS by 49.3% with an 8.84% increase in total attribution time. This directly demonstrates that controlling graph accumulation can substantially reduce memory demand without changing the resulting credits. In a separate GPU benchmark at 32K and G=8G=8, planning and solving take 479.34 ms, including 151.54 ms for the two-pass solve; the graph-plus-processing peak is 32.250 GiB, of which approximately 32 GiB is the graph itself. This supports the value of bounded solver workspace while identifying graph storage as the dominant cost in that benchmark.

2) Sparse correspondence and group scheduling control distinct scaling costs; dense native graphs retain a quadratic term.

Cross-trajectory correspondence already has an exact sparsity bound. Appendix B.1, Eq. 16 shows that two trajectories with nkn_k and nℓn_\ell positive-flow tokens have at most nk+nℓ−1n_k+n_\ell-1 nonzero correspondence entries, constructed by a linear-time sweep. For group size GG and sequence length TT, all pairwise correspondences therefore require O(G2T)O(G^2T) storage. Keeping these correspondences separate from the native within-trajectory graphs gives a graph representation of size O(GT2+G2T)O(GT^2+G^2T) and avoids materializing a dense (GT)×(GT)(GT)\times(GT) joint matrix.

Scheduling controls the number of these groups resident at once. For RR total trajectories per update and at most CC groups in flight, native-graph residency is bounded by O(CGT2)O(CGT^2), instead of full-batch accumulation at O(RT2)O(RT^2); sparse correspondences add O(CG2T)O(CG^2T). At fixed C,G,TC,G,T, additional prompt groups can be processed in further rounds without increasing the graph-residency budget. Total computation and rollout data still grow with the workload.

Reviewer_vH3kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed FlowWeaver, which leverages the attribution graph across trajectories to enable token-level credit assignment. The evaluation has been conducted on mathematical reasoning and agentic tasks. Multiple baselines have been evaluated and compared.

Critical strengths
  1. The paper has been written well and is easy to read, especially the illustrative figure of the framework.
  2. Extensive baselines have been evaluated on two domains, covering both mathematical reasoning and agentic tasks.
Critical weaknesses
  1. My fundamental concern is about the design of this credit assignment method. The token attribution is constructed through the LLM's attention connectivity. Thus, the gradients of the outcome reward will naturally backpropagate through the same attention attribution by standard gradient descent. Could the authors clarify the essential difference between their credit assignment method and standard gradient descent?

  2. If the proposed credit assignment method works similarly to standard gradient descent, could the proposed method be just some gradient normalisation method? According to the design of the proposed method, it seems that the major difference from GRPO is that such information flow is deliberately constructed through a sparse connection (or a more skewed distribution). In such a case, could the authors try to compare with some naive baselines that just enable sparse gradient descent? On the other hand, if the proposed method achieves better performance than the naive gradient descent method, what is the major contribution that enables a better information flow?

  3. In Table 3, since the PRM is a very close baseline, could the authors compare with more PRMs, including Math-Shepherd[1], PRM800K[2].

  4. Also, in Table 3, the performance gain is relatively marginal compared to baselines. Could the authors report multiple random seeds and show the CI for the proposed method and baselines?

[1] Wang, Peiyi, et al. "Math-shepherd: Verify and reinforce llms step-by-step without human annotations." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[2] Lightman, Hunter, et al. "Let's verify step by step." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors

Please refer to weaknesses.

11 comments

We sincerely thank the reviewer for the careful evaluation and for recognizing the clarity of the presentation and the breadth of the experiments. We appreciate the opportunity to clarify the mechanism and empirical value of our credit-assignment approach. Below, we respond to each of your questions and concerns point by point.


Q1: How does attention-based credit assignment differ from standard backpropagation through the same attention connections?

Thank you for identifying this point. Attention does participate in the original backward computation. FlowWeaver uses this same observable for a different calculation: it constructs token credit weights, which are then used in a policy surrogate optimized by standard backpropagation. The distinction is between computing the parameter derivative of a given training term and deciding its relative weight within the response.

1) The same token can be both context for a later prediction and an earlier generation decision.

Consider a solution that first produces an intermediate conclusion and later uses it to derive the answer. Let xix_i be a token in that conclusion and xtx_t a later answer token. When differentiating the training term for xtx_t, the earlier token xix_i is part of the given prefix. Gradients can pass through attention into its representation, teaching the model how to use that context when predicting xtx_t.

FlowWeaver asks a further question after the trajectories and outcomes have been observed: how much emphasis should the training term for generating xix_i itself receive? It examines the token's participation in outcome-directed paths in the group's attribution graph and uses that importance to weight the policy term at xix_i. That term concerns the decision to generate xix_i from the prefix preceding it. The original advantage continues to determine whether the decision is encouraged or discouraged.

AspectAttention in standard backpropagationAttention in FlowWeaver's attribution
Question answeredGiven a token's training term, how should the parameters change?Given the completed trajectories and their outcomes, which token terms should receive more emphasis?
Role of the earlier token in the exampleContext used to predict the later token xtx_t.A generation decision whose own policy term receives a credit weight.
Use of attentionPart of the chain rule through the full neural computation, together with upstream derivatives, projections, and other layers.Connectivity for a graph analysis using paths, cross-trajectory correspondence, and outcome conditioning.
Direct outputA parameter-gradient vector for each training term.A scalar credit weight for each token, held fixed during the update.

Backpropagation through the representation of xix_i does not differentiate through the discrete sampling operation that produced it or automatically assign a larger coefficient to its own policy term. GRPO already supplies that term with an outcome advantage; FlowWeaver changes its relative emphasis within the response. Shared parameters can still make a later token's loss affect earlier token probabilities indirectly. This distinction concerns the training coefficients, not an absence of such parameter sharing.

2) Reweighting changes the combination of gradients throughout the model.

For a fixed checkpoint and rollout batch, let ui=∇θℓiu_i=\nabla_\theta\ell_i be the gradient of token ii's original policy-surrogate term, including its advantage, clipping, and loss-aggregation weight. With the credit weights held fixed,

g_{\mathrm{FW}}=\sum_i c_i u_i =g_0+(\gamma-1)\sum_{i\in H}u_i,$$ where $H$ is the selected set and $\gamma=1.5$ in our default configuration. Every $u_i$ already includes the complete backward computation through attention. FlowWeaver changes the coefficients in their sum. Because token gradients can have different directions and partially cancel, their reweighted sum need not be parallel to the original sum. Deriving the coefficients from attention does not impose that parallelism (Appendix B.6). The same reasoning applies within attention. For example, at the value representation $V_j^{(l)}$ of position $j$ in layer $l$, $$\frac{\partial L_{\mathrm{FW}}}{\partial V_j^{(l)}} =\sum_i c_i\frac{\partial\ell_i}{\partial V_j^{(l)}}.$$ Several later token terms can contribute gradients to this position. Changing their relative weights can change the direction of their sum, even with the same forward computation and local Jacobians. At an individual output-logit position, positive weighting simply scales that position's gradient; as contributions from different positions combine inside attention, the resulting direction can change. The output head's parameter gradient also aggregates positions and can therefore change direction.

We examined this directly using paired gradients at a shared checkpoint on the same rollouts. The comparison covers the full policy model, the output head, and attention parameters at different depths:

Table R1-A. Gradient relationship to the unweighted GRPO reference.

Weighting ruleFull-model norm ratioFull-model cosineOutput-head cosineShallow attention cosineMiddle attention cosineDeep attention cosine
GRPO111111
Uniform scaling by κ>0\kappa>0 (analytic control)κ\kappa11111
Random upweighting1.3098740.9855480.9866930.9875580.9861760.986151
Gradient-Norm1.4206760.9918800.9941790.9926390.9916040.992376
Answer-Gradient1.2290970.9891340.9878410.9918900.9878450.991920
Sparse GRPO1.8920030.7568640.7348890.6917160.7396770.778110
GMTS2.5007170.9999180.9999580.9999580.9999310.999710
FlowWeaver, mean credit normalized to 11.0005320.9808260.9837040.9829190.9793950.982999
FlowWeaver1.2387010.9809110.9836210.9831680.9795330.983266

Qwen3-4B-Base, 1K context. Measurements compare raw actor policy-surrogate gradients before the optimizer and global gradient clipping. Norm ratios and cosines are relative to GRPO; attention columns summarize shallow, middle, and deep layers.

FlowWeaver's full-model gradient remains closely aligned with GRPO, with a cosine of 0.980911 and a norm ratio of 1.238701. After mean-credit normalization, the norm ratio is nearly one (1.000532), while the directional deviation persists (cosine 0.980826). Matching mean credit does not in general match gradient norms; the near-unit norm ratio is an observation in this experiment. The normalized variant's shallow, middle, and deep attention cosines are 0.982919, 0.979395, and 0.982999, respectively, showing that the measured changes cover different depths as well as the output head. These results support the mechanism above: the credit rule changes how token contributions combine throughout the backward computation, beyond a single scalar rescaling of the GRPO gradient. A different direction alone does not establish a better allocation: random upweighting also changes direction, and Sparse GRPO produces a substantially larger deviation. We address allocation quality through the learning and attribution comparisons in Q2.

FlowWeaver therefore uses standard backpropagation to optimize an explicitly weighted surrogate. Appendix B.6 characterizes the resulting gradient shift; it does not claim that the weights recover causal token contributions or preserve the unweighted GRPO gradient. We will make the distinction between the two uses of attention explicit in Section 4.3, with the example and comparison above.


Q2: Could gradient normalization or sparse updates explain the gains? How does flow compare with simple sparse baselines, and what makes it effective?

We appreciate the request for both direct baselines and an explanation of what makes the allocation useful. There are two design choices to examine: how tokens are selected, and how their training terms are weighted after selection. We have completed additional comparisons addressing both gradient-based selection and sparse token updates, and use the attribution analyses to examine why the selected positions matter.

1) Direct comparisons with gradient-based selection and sparse token updates.

The additional runs use Qwen3-4B-Base on DAPO Math, a total context limit of 1K tokens, and 64 prompts with eight responses per prompt in each round. Each of the four added methods was trained for 600 steps with three seeds, giving twelve additional training runs.

Gradient-Norm and Answer-Gradient use the same per-response selection fraction and amplification rule as FlowWeaver, changing the importance signal. Sparse GRPO and GMTS select across the valid response tokens in each actual packed training microbatch and use the same compensation rule, differing only in the selection signal. The comparison is organized as follows:

Table R1-B. Additional baselines and training results.

MethodToken-selection signalSelected / remaining policy-term weightsMath Avg. (%)
GRPOUniform credit1 / 137.1 ± 0.3
Gradient-NormOutput-head gradient norm; top 40% per response1.5 / 136.9 ± 0.8
Answer-GradientGradient × activation attribution from the generated answer to input embeddings; top 40% per response1.5 / 138.1 ± 0.8
Sparse GRPORandom scores; top KK per packed microbatchN/KN/K / 036.6 ± 1.5
GMTSHtdtH_t d_t; top KK per packed microbatchN/KN/K / 036.4 ± 0.9
FlowWeaverCross-trajectory, outcome-conditioned flow; top 40% per response1.5 / 140.9 ± 0.3

Math Avg. is the unweighted average, reported across three training seeds.

For both sparse methods, let MM be the set of valid response-token positions under the original loss mask in a packed training microbatch, excluding prompt and padding positions. With N=∣M∣>0N=|M|>0, K=⌈0.4N⌉K=\lceil0.4N\rceil, and S⊆MS\subseteq M the KK selected positions,

\widetilde\ell_t=w_t\ell_t^{\mathrm{PPO}},\qquad \frac{1}{N}\sum_{t\in M}w_t=1.$$ Here $\ell_t^{\mathrm{PPO}}$ is the base PPO-style policy-surrogate term used in GRPO, before sparse weighting. Sparse GRPO selects by random scores; GMTS ranks by $H_t d_t$, where $H_t$ is the policy entropy at position $t$ and $d_t=\left|\partial\ell_t^{\mathrm{PPO}}/\partial\log\pi_t\right|$. Thus both sparse rules have mean token weight exactly one within each packed microbatch, without a fixed retention quota for each response. This coefficient normalization does not constrain the parameter-gradient norm. FlowWeaver and the two gradient-based upweighting rules have mean credit approximately 1.2. **FlowWeaver improves Math Avg. by 3.8 percentage points over GRPO and by 2.8 points over the strongest added baseline, Answer-Gradient.** With the same top-40% selection budget and 1.5 / 1 weighting rule, Gradient-Norm and Answer-Gradient change the mean score relative to GRPO by −0.2 and +1.0 points, respectively. These comparisons support the value of FlowWeaver's selection signal beyond the tested gradient-based alternatives under this training protocol. Sparse GRPO and GMTS have mean scores 0.5 and 0.7 points below GRPO, respectively; the tested sparse variants also do not reproduce FlowWeaver's gain. They directly test sparsity in token-loss contributions, while their aggregate parameter gradients can still be dense. FlowWeaver retains the base contribution of every token and adds emphasis at selected positions. Relative to FlowWeaver, the sparse methods differ in selection signal, selection scope, and weighting rule, so these comparisons assess the complete methods rather than isolating the effect of retaining all token terms. Answer-Gradient is a particularly useful comparison: differentiating the generated answer with respect to its input embeddings already captures dependencies through multiple layers. Its attribution measures answer sensitivity within a trajectory. FlowWeaver measures token participation in an auxiliary path distribution that also incorporates peer trajectories and their observed outcomes. The 2.8-point margin shows a benefit over this answer-directed gradient attribution baseline in the tested setting. The component analyses below examine the roles of cross-trajectory correspondence and outcome conditioning within FlowWeaver. **2) Allocation controls and component analyses explain what the flow rule adds.** The normalization control in Table 13 separates mean credit from its allocation across tokens. Uniformly increasing GRPO's coefficients does not reproduce the gain, while normalizing FlowWeaver back to mean credit one retains most of its improvement. This supports the importance of the relative allocation. The control matches average loss coefficients, rather than parameter-gradient norms. The attribution itself has three relevant steps. Attention supplies within-response connectivity for prompt-to-answer paths. Cumulative flow provides a correspondence through which different responses contribute routes to a shared graph. Conditioning that graph on the response's observed outcome then measures token participation in paths toward that outcome. GRPO already uses the group's rewards to construct its advantage; FlowWeaver additionally uses the group's trajectory structure and outcomes to determine the token coefficients. The existing analyses test these choices separately: | Mechanism tested | Controlled comparison | Observation | |---|---|---| | Relative credit allocation | Uniform scaling versus flow weights, with matched mean credit (Table 13). | Flow-guided allocation remains effective after mean-credit normalization. | | Cross-trajectory correspondence | Flow progress versus token progress, step progress, and embedding similarity (Table 11b); independent prefix continuations (Table 1). | Flow correspondence gives better training results and matches prefixes with more similar continuation success probabilities. | | Outcome conditioning | Conditioned versus unconditioned propagation on the same shared graph, across exchange strengths (Table 11a). | Conditioning improves the average score at every tested exchange strength. | | Which positions receive emphasis | Top-flow versus bottom-flow selection at the same fraction and multiplier (Table 8b). | The high-flow selection performs better despite the identical weight budget. | These comparisons support the particular correspondence, conditioning, and selection rules used in FlowWeaver. They also show why the contribution needs to be assessed beyond the fact that attention is used as an input signal.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely appreciate the reviewers’ thoughtful feedback and their participation throughout the discussion. We were pleased that they highlighted the following strengths.

  • The paper addresses an important problem in LLM post-training. Reviewer Vmze and NDDm emphasized the challenge of assigning credit to individual reasoning decisions when supervision comes only from the final outcome.
  • The framework is intuitive and clearly illustrated. Reviewer Vmze appreciated the explanation of how the components fit together, and Reviewer vH3k singled out the overview figure.
  • The experiments show gains across a broad evaluation. Reviewer vH3k highlighted the range of baselines across reasoning and agentic tasks. Reviewer Vmze found the results promising, and NDDm recognized the gains over GRPO.

Our contribution. FlowWeaver introduces outcome-conditioned, cross-trajectory flow as a signal for token-level credit assignment. A shared graph connects responses to the same prompt and scores tokens by their participation in paths to the observed outcome. The signal helps localize errors in reasoning traces and guides training by reweighting token losses. Experiments across Qwen and Llama-based models show gains on mathematical reasoning, general reasoning, and agentic tasks. Ablations examine the roles of cross-trajectory alignment, outcome conditioning, and token selection.

We also thank the reviewers for the questions they raised and for their continued engagement. We addressed each question point by point and ran additional experiments to evaluate token attribution, compare against the suggested baselines, and assess scalability.

Reviewer vH3k

  1. How does flow weighting differ from standard backpropagation? FlowWeaver assigns a weight to each token loss before backpropagation. Our gradient analysis showed that this changes the direction of the GRPO update.
  2. Could normalization or sparsity explain the gains? Most of the gain persisted after normalizing the weights, and flow-based selection outperformed sparse and gradient-based controls.
  3. How does FlowWeaver compare with other process reward models? It outperformed Math-Shepherd and both additional Qwen verifiers, including one trained on PRM800K.
  4. Are the gains consistent across seeds? The confidence intervals we reported across training seeds placed FlowWeaver ahead of both GRPO and trajectory-local FlowTracer in each math configuration.
  5. Does flow simply favor tokens that receive more attention? Flow and raw attention ranked different tokens, and direct attention weighting yielded little benefit. We will make the token-loss reweighting explicit in the paper.
  6. Why should high-flow tokens receive more weight? Peer responses reveal alternative paths to successful and failed answers. Masking high-flow tokens changed correctness more often than random masking, and conditioning on outcomes improved learning in ablations.
  7. Could the same rationale apply to other attention-based models? We identified sequential generation with outcome feedback as a plausible setting for extension and left broader validation to future work.

Reviewer Vmze

  1. Do the token rankings identify meaningful reasoning decisions? Side-by-side case studies and comparisons with annotations showed that FlowWeaver often pinpointed errors within a step and gave less weight to redundant reasoning.
  2. How does attribution cost scale relative to the gains? We reported timing and memory across configurations alongside task gains. Longer-context runs demonstrated complete policy updates, while dense graph storage remained quadratic in sequence length.

Reviewer NDDm

  1. How does the objective differ from value-based credit assignment? A value-based method estimates a signed advantage. FlowWeaver reweights its contribution to the policy loss at each token. Combining the two improved learning in our comparisons.
  2. How could the paper be easier to follow? We will introduce GRPO and flow through a worked example and move the detailed literature comparison to Related Work.
  3. How was the VinePPO comparison run? We described how reasoning steps were segmented, how continuations were sampled to estimate prefix values, and how training budgets were matched.
  4. Which tasks benefit most? Relative gains over GRPO were larger on AIME than on MATH500, and the largest agentic gain over FlowTracer was on full ALFWorld.
  5. How could flow complement newer critic-based methods? We outlined how it could weight token advantages from Le Critique or refine credit within the action chunks used by Trust the Critic More.

We are sincerely grateful to all the reviewers for the time and care they devoted to the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.