- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Language model agents are prone to distribution mismatch in long-horizon, multi-turn tasks, which can impair their reasoning and instruction-following abilities. To address this issue, the paper proposes Group GFlowNet for Agentic RL (G$^2$ARL). Built on GFlowNets, G$^2$ARL incorporates group-level flow reallocation to align trajectories with their corresponding targets and coverage-guided dynamic rebalancing to reuse successful experiences. Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate its effectiveness.
Critical strengths
- The paper addresses a practically relevant challenge, and the experimental results demonstrate the effectiveness of the proposed method.
Critical weaknesses
- The presentation would benefit from greater precision and conciseness, as well as a clearer explanation of the key differences from FlowRL [1].
- The paper introduces pairwise residuals to replace the partition function but provides insufficient justification for this design. It also lacks theoretical analysis and empirical validation of whether this substitution preserves the learning mechanism of standard GFlowNets.
- The paper lacks an analysis of the training budget, making it difficult to assess the advantages of the proposed method.
[1] Zhu X, Cheng D, Zhang D, et al. Flowrl: Matching reward distributions for llm reasoning. In ICLR, 2027.
Questions for authors
-
In Eq.(2), the language-model policy and reward function have different units, warranting further justification. Moreover, why does the equality hold?
-
G²ARL leverages GFlowNets' ability to preserve diversity, but how does this capability contribute to solving practical tasks within the proposed framework?
-
What exactly does the UV metric measure?
3 comments
Thank you for the clarification. However, several important concerns remain. In addition, reference [1] contains a bibliographic error: FlowRL was published at ICLR 2026.
Presentation and differences from FlowRL. The paper introduces many mathematical symbols, but it defines and without using them, and it does not define . Descriptions of the benchmarks and some implementation details are scattered across the Related Work, G2ARL, and Experiments sections. Although G2ARL differs from FlowRL in four components, these appear to combine existing techniques. Moreover, Table 3 indicates that G2ARL’s performance depends heavily on the replay component, yet its relevance to addressing sparse rewards in long interactions appears unclear.
Training budget. In the additional budget experiments, how is the number of new trajectories determined for a given group size? Why do G2ARL and GiGPO collect the same number of new trajectories?
Thank you for the follow-up. We clarify the contribution, replay mechanism and trajectory accounting below.
1. Contribution beyond FlowRL and the role of replay.
Our contributions are outcome-dependent pair weighting and composition-dependent replay. The distribution-matching target, contrastive residual and coverage-selection rule build on prior work, which we credit explicitly.
Sparse terminal rewards can leave a current group with only failures even after the task has produced a successful trajectory. Replay re-executes and verifies stored same-task successes, then inserts eligible ones into that group. This restores success–failure comparisons. The pair weights determine how strongly those comparisons enter the loss. For example, one success in a group of 16 creates 15 success–failure pairs. With weights (WW, LL, WL) = (1, 0.3, 3), these pairs receive 58.8% of the total pair coefficient, versus 12.5% with equal weights. These are coefficient shares, not gradient shares.
The ALFWorld-1.5B ablation at β = 8 separates the two contributions:
| Pairwise objective | Replay off | Replay on |
|---|---|---|
| Equal weights (Var-TB) | 55.47% | 66.31% |
| Outcome-dependent weights | 65.71% | 80.18% |
Weighting adds 13.87 percentage points with replay fixed and 10.24 points without replay. Replay benefits both objectives. Each cell is one training run, so we do not claim a statistically established interaction.
Relative to FlowRL, our objective uses summed trajectory log-probabilities, terminal-score differences and outcome-weighted pair residuals; its shared normalizer cancels. FlowRL uses length-normalized log-probabilities, group-standardized scores and a learned normalizer. By canceling the learning head, our method improves the training stability. However, such design requires groups with good quality and learning signals, that's what outcome-depended weights do. Our replay rule also depends on group composition: it supplies successes to all-failure groups, decreases injection into mixed groups during training, and leaves all-success groups unchanged. These implementation differences are now distinguished from the two contribution claims.
Replay cannot guarantee the first success or provide step-level credit assignment. Equal-score FlowRL groups may still have nonzero gradients; they lack a success–failure score contrast. Our ablations support weighting and replay within our pipeline.
2. How new trajectories are counted.
The reported 150-update runs correspond to 2,400 task groups:
| Group size | Task groups | New trajectories |
|---|---|---|
| 8 | 2,400 | 19,200 |
| 16 | 2,400 | 38,400 |
G²ARL and GiGPO have the same count in these controls because both the group size and fresh task-group count are matched.
3. Notation and organization.
We define n_i = n(τ_i) and n_j as invalid-command counts, c as a candidate stored trajectory, and S as the subset selected so far. We link the success/failure counts n_W and n_L directly to their use in the linear-time loss decomposition. Algorithm 1 collects the update procedure, while the experimental setup and its appendix collect benchmark and implementation details. We also state explicitly that R is an unnormalized probability mass and derive Eq. (2) by taking the logarithm of the target definition. We will improve the notation in revised paper.
Thanks for your advice. Can you clarify why reference [1] is cited as “ICLR 2027”? FlowRL was published at ICLR 2026, as cited in our submission. Please confirm which version you refers to.
W1. Presentation and differences from FlowRL
We will simplify Section 3, consolidate the update procedure in Algorithm 1, and make the following differences explicit. FlowRL’s stated target belongs to the same family as ours at ρ=1, but its training objective differs:
These differences are analyzed in Appendix C.5. In particular, FlowRL’s length normalization introduces a trajectory-length factor into its zero-residual condition.
W2. Why pairwise residuals preserve the GFlowNet target
A TB residual is ri=logZx+qi. Subtracting two same-task residuals gives ri−rj=qi−qj, eliminating the shared partition function exactly.
With connected comparisons over the target support, zero pairwise loss implies πθ=π∗=Rx/Zx (Proposition 3). On the trajectory-prefix tree, this policy induces conserved flow with terminal flow Rx (Appendix A.2). With equal weights and independent on-policy samples, the expected pairwise gradient is (G−1)/G times the expected TB gradient.
We also isolated the substitution under TB’s training settings:
The task-paired success difference is −0.94. This substitution removes the learned normalization term; it does not by itself establish a performance gain. Separately, class weighting improves success from 66.31% to 80.18% with replay unchanged (Table 3).
W3. Training budget
Original training compute and environment interactions are not matched. We added controls in both directions:
All runs use 150 updates; the full pool contains 3,553 tasks. G²ARL retains an advantage with the same group size, training pool, and new-trajectory count. Even use much more smaller training pool, G²ARL still give better performance.
Q1. Units and the equality in Eq. (2)
Rx is an unnormalized target probability mass, not the raw score y. Both πθ and Rx are dimensionless, and their logarithms are measured in nats.
Taking logarithms of Eq. (1) gives logRx=ρlogπref+βy−λn. Therefore, by definition, qi=Di−βyi+λni=logπθ(τi∣x)−logRx(τi). We will state this derivation explicitly.
Q2. How diversity helps practical tasks
During training, the buffer retains different successful trajectories for later comparisons. Replacing coverage selection with reward-prioritized replay reduces ALFWorld success from 80.18% to 60.48%, and UV from 7.414 to 5.106 (Table 3).
At inference, G²ARL produces more distinct successful command sequences at comparable success rates:
With verifier-selected best-of-16, G²ARL also reaches 94.68% on ALFWorld-1.5B versus 89.74% for the strongest baseline. These results show practical multi-sample utility, although they do not isolate diversity as the cause of the gains.
Q3. What UV measures
UV counts distinct successful environment-command sequences per task, then averages over tasks solved at least once and over evaluation runs. It excludes generated reasoning text and uses exact command-sequence matching.
With 16 samples per task, 1≤UV≤16. Different query wording or action orders can count separately, so UV measures command-sequence diversity, not necessarily distinct semantic strategies. ScienceWorld’s main evaluation uses one sample per task, making UV equal to 1 on solved tasks; it is therefore omitted from Table 2.