acceptodds
Under review as a conference paper at ICLR 2027

G2ARL: GROUP GFLOWNETS FOR AGENTIC REINFORCEMENT LEARNING

13 views

Abstract

Many agentic tasks admit multiple successful trajectories, yet reward maximization does not explicitly preserve their diversity. GFlowNets match a reward-proportional trajectory distribution, but a sparse reward over long interactions makes such a distribution difficult to learn. We introduce Group GFlowNet for Agentic RL (G²ARL), which combines group level flow reallocation with coverage-guided dynamic rebalancing for multi-turn language model agents. The flow reallocation loss learns relative probability allocation from pairwise trajectory comparisons, while dynamic gating and submodular replay retain verified successes selected for coverage and return them to groups that have none. Across ALFWorld, WebShop, and ScienceWorld, G²ARL achieves the highest success among all retrained baselines in every main benchmark setting. Across benchmark settings, G²ARL achieves 4.2 to 20.7 percentage points higher success than the strongest retrained comparator, while also producing shorter solutions with fewer repeated actions and loops.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
GFlownet, submodular optimization, agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_PazkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Language model agents are prone to distribution mismatch in long-horizon, multi-turn tasks, which can impair their reasoning and instruction-following abilities. To address this issue, the paper proposes Group GFlowNet for Agentic RL (G$^2$ARL). Built on GFlowNets, G$^2$ARL incorporates group-level flow reallocation to align trajectories with their corresponding targets and coverage-guided dynamic rebalancing to reuse successful experiences. Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate its effectiveness.

Critical strengths
  1. The paper addresses a practically relevant challenge, and the experimental results demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. The presentation would benefit from greater precision and conciseness, as well as a clearer explanation of the key differences from FlowRL [1].
  2. The paper introduces pairwise residuals to replace the partition function but provides insufficient justification for this design. It also lacks theoretical analysis and empirical validation of whether this substitution preserves the learning mechanism of standard GFlowNets.
  3. The paper lacks an analysis of the training budget, making it difficult to assess the advantages of the proposed method.

[1] Zhu X, Cheng D, Zhang D, et al. Flowrl: Matching reward distributions for llm reasoning. In ICLR, 2027.

Questions for authors
  1. In Eq.(2), the language-model policy πθ(⋅)\pi_{\theta}(\cdot) and reward function Rx(⋅)R_x(\cdot) have different units, warranting further justification. Moreover, why does the equality Di−βyi+λni=log⁡πθ(τi∣x)Rx(τi)D_i - \beta y_i + \lambda n_i = \log \frac{\pi_{\theta}(\tau_i \mid x)}{R_x(\tau_i)} hold?

  2. G²ARL leverages GFlowNets' ability to preserve diversity, but how does this capability contribute to solving practical tasks within the proposed framework?

  3. What exactly does the UV metric measure?

3 comments

Thanks for your advice. Can you clarify why reference [1] is cited as “ICLR 2027”? FlowRL was published at ICLR 2026, as cited in our submission. Please confirm which version you refers to.

W1. Presentation and differences from FlowRL

We will simplify Section 3, consolidate the update procedure in Algorithm 1, and make the following differences explicit. FlowRL’s stated target belongs to the same family as ours at ρ=1\rho=1, but its training objective differs:

ComponentFlowRLG²ARL
Log-probabilitiesLength-normalizedSummed over generated tokens
ScoresGroup-standardizedTerminal scores without group standardization
Partition functionLearned log⁡Zϕ\log Z_\phiCancels in pairwise differences
Comparisons and replayNo class-weighted pairs or our replay mechanismClass-weighted pairs and composition-dependent success injection

These differences are analyzed in Appendix C.5. In particular, FlowRL’s length normalization introduces a trajectory-length factor into its zero-residual condition.

W2. Why pairwise residuals preserve the GFlowNet target

A TB residual is ri=log⁡Zx+qir_i=\log Z_x+q_i. Subtracting two same-task residuals gives ri−rj=qi−qjr_i-r_j=q_i-q_j, eliminating the shared partition function exactly.

With connected comparisons over the target support, zero pairwise loss implies πθ=π∗=Rx/Zx\pi_\theta=\pi^*=R_x/Z_x (Proposition 3). On the trajectory-prefix tree, this policy induces conserved flow with terminal flow RxR_x (Appendix A.2). With equal weights and independent on-policy samples, the expected pairwise gradient is (G−1)/G(G-1)/G times the expected TB gradient.

We also isolated the substitution under TB’s training settings:

ALFWorld-1.5BSuccess (%)UV
TB with learned log⁡Z\log Z54.417.296
Same settings, group centering instead of learned log⁡Z\log Z53.467.324

The task-paired success difference is −0.94. This substitution removes the learned normalization term; it does not by itself establish a performance gain. Separately, class weighting improves success from 66.31% to 80.18% with replay unchanged (Table 3).

W3. Training budget

Original training compute and environment interactions are not matched. We added controls in both directions:

ALFWorld-1.5BTraining poolGroup sizeNew trajectoriesSuccess by training run (%)
G²ARLFull819,20086.02
GiGPOFull819,20069.44 / 69.81
G²ARL64 tasks819,20087.21 / 81.44 / 76.98
GiGPO64 tasks819,20063.54
G²ARL64 tasks1638,40089.31 / 85.80
GiGPOFull1638,40078.56 / 77.74

All runs use 150 updates; the full pool contains 3,553 tasks. G²ARL retains an advantage with the same group size, training pool, and new-trajectory count. Even use much more smaller training pool, G²ARL still give better performance.

Q1. Units and the equality in Eq. (2)

RxR_x is an unnormalized target probability mass, not the raw score yy. Both πθ\pi_\theta and RxR_x are dimensionless, and their logarithms are measured in nats.

Taking logarithms of Eq. (1) gives log⁡Rx=ρlog⁡πref+βy−λn\log R_x=\rho\log\pi_{\mathrm{ref}}+\beta y-\lambda n. Therefore, by definition, qi=Di−βyi+λni=log⁡πθ(τi∣x)−log⁡Rx(τi)q_i=D_i-\beta y_i+\lambda n_i=\log\pi_\theta(\tau_i|x)-\log R_x(\tau_i). We will state this derivation explicitly.

Q2. How diversity helps practical tasks

During training, the buffer retains different successful trajectories for later comparisons. Replacing coverage selection with reward-prioritized replay reduces ALFWorld success from 80.18% to 60.48%, and UV from 7.414 to 5.106 (Table 3).

At inference, G²ARL produces more distinct successful command sequences at comparable success rates:

ALFWorld-7B, temperature 1.3Success (%)UV
GiGPO88.499.146
G²ARL89.5912.622

With verifier-selected best-of-16, G²ARL also reaches 94.68% on ALFWorld-1.5B versus 89.74% for the strongest baseline. These results show practical multi-sample utility, although they do not isolate diversity as the cause of the gains.

Q3. What UV measures

UV counts distinct successful environment-command sequences per task, then averages over tasks solved at least once and over evaluation runs. It excludes generated reasoning text and uses exact command-sequence matching.

With 16 samples per task, 1≤UV≤161\leq\mathrm{UV}\leq16. Different query wording or action orders can count separately, so UV measures command-sequence diversity, not necessarily distinct semantic strategies. ScienceWorld’s main evaluation uses one sample per task, making UV equal to 1 on solved tasks; it is therefore omitted from Table 2.

Thank you for the clarification. However, several important concerns remain. In addition, reference [1] contains a bibliographic error: FlowRL was published at ICLR 2026.

Presentation and differences from FlowRL. The paper introduces many mathematical symbols, but it defines nWn_W and nLn_L without using them, and it does not define ni,nj,c,Sn_i, n_j, c, S. Descriptions of the benchmarks and some implementation details are scattered across the Related Work, G2ARL, and Experiments sections. Although G2ARL differs from FlowRL in four components, these appear to combine existing techniques. Moreover, Table 3 indicates that G2ARL’s performance depends heavily on the replay component, yet its relevance to addressing sparse rewards in long interactions appears unclear.

Training budget. In the additional budget experiments, how is the number of new trajectories determined for a given group size? Why do G2ARL and GiGPO collect the same number of new trajectories?

Thank you for the follow-up. We clarify the contribution, replay mechanism and trajectory accounting below.

1. Contribution beyond FlowRL and the role of replay.

Our contributions are outcome-dependent pair weighting and composition-dependent replay. The distribution-matching target, contrastive residual and coverage-selection rule build on prior work, which we credit explicitly.

Sparse terminal rewards can leave a current group with only failures even after the task has produced a successful trajectory. Replay re-executes and verifies stored same-task successes, then inserts eligible ones into that group. This restores success–failure comparisons. The pair weights determine how strongly those comparisons enter the loss. For example, one success in a group of 16 creates 15 success–failure pairs. With weights (WW, LL, WL) = (1, 0.3, 3), these pairs receive 58.8% of the total pair coefficient, versus 12.5% with equal weights. These are coefficient shares, not gradient shares.

The ALFWorld-1.5B ablation at β = 8 separates the two contributions:

Pairwise objectiveReplay offReplay on
Equal weights (Var-TB)55.47%66.31%
Outcome-dependent weights65.71%80.18%

Weighting adds 13.87 percentage points with replay fixed and 10.24 points without replay. Replay benefits both objectives. Each cell is one training run, so we do not claim a statistically established interaction.

Relative to FlowRL, our objective uses summed trajectory log-probabilities, terminal-score differences and outcome-weighted pair residuals; its shared normalizer cancels. FlowRL uses length-normalized log-probabilities, group-standardized scores and a learned normalizer. By canceling the learning head, our method improves the training stability. However, such design requires groups with good quality and learning signals, that's what outcome-depended weights do. Our replay rule also depends on group composition: it supplies successes to all-failure groups, decreases injection into mixed groups during training, and leaves all-success groups unchanged. These implementation differences are now distinguished from the two contribution claims.

Replay cannot guarantee the first success or provide step-level credit assignment. Equal-score FlowRL groups may still have nonzero gradients; they lack a success–failure score contrast. Our ablations support weighting and replay within our pipeline.

2. How new trajectories are counted.

The reported 150-update runs correspond to 2,400 task groups:

Group sizeTask groupsNew trajectories
82,40019,200
162,40038,400

G²ARL and GiGPO have the same count in these controls because both the group size and fresh task-group count are matched.

3. Notation and organization.

We define n_i = n(τ_i) and n_j as invalid-command counts, c as a candidate stored trajectory, and S as the subset selected so far. We link the success/failure counts n_W and n_L directly to their use in the linear-time loss decomposition. Algorithm 1 collects the update procedure, while the experimental setup and its appendix collect benchmark and implementation details. We also state explicitly that R is an unnormalized probability mass and derive Eq. (2) by taking the logarithm of the target definition. We will improve the notation in revised paper.

Reviewer_RxVROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

It is a new approach to work reinforcement learning because of agentic environment. The main idea is to collect a useful trajectories and form a group to learn.

Critical strengths

The results are promising.

Critical weaknesses

The crucial problem is to find a set of useful of trajectories, and form a group to learn. The cost and the search may be substantial.

Questions for authors

The authors do not mention how to form a group ?

1 comment

W1. Trajectory search and computational cost

Selection reuses trajectories already sampled and scored during training; there is no separate trajectory-search stage. Buffer selection runs on the CPU only when capacity is exceeded, without additional model forward passes or environment interactions.

MeasurementCost
CPU selection, worst-case benchmark0.17–3.36 seconds/update
Replay off → on, first 20 updates298.2 → 364.6 seconds/update (+22.3%)
Full ALFWorld-1.5B training, G²ARL42.7 / 48.2 GPU hours
Full ALFWorld-1.5B training, GiGPO23.1 GPU hours

Selection timings are medians over 15 repetitions on separate CPU hardware. Full-training costs use different training stacks and GPU counts; G²ARL samples 38,400 new trajectories versus GiGPO’s 19,200.

Replay requires re-executing stored commands and rescoring them under the current policy. In the second ALFWorld-1.5B run, injected trajectories amount to 5.9% of newly sampled trajectories.

Q1. How groups are formed

Each group initially contains GG trajectories sampled by the current policy for the same task, at temperature 1.0. We use G=16G=16 on ALFWorld and WebShop, and G=32G=32 on ScienceWorld. Verified successes enter that task’s buffer before group rebalancing.

Sampled groupRebalancing ruleGroup weight
All-successNo injectionα\alpha
All-failureReplace up to ⌊G/2⌋\lfloor G/2\rfloor members with stored same-task successes1 if a success is injected; otherwise 0
MixedInject with probability p(u)=0.75−0.50up(u)=0.75-0.50u; request ⌈0.25G⌉\lceil0.25G\rceil to ⌊0.5G⌋\lfloor0.5G\rfloor replacements uniformly, retaining at least one failure1

Here u∈[0,1]u\in[0,1] is training progress. We use α=0\alpha=0 on ALFWorld and WebShop-7B, and α=0.25\alpha=0.25 on WebShop-1.5B and ScienceWorld.

ScienceWorld additionally stores positive-score failures from the task’s top decile until a success is stored. All-failure groups refilled only with these samples receive weight 0.25.

Replay candidates are drawn uniformly without replacement, re-executed, and rescored. We check their stored endpoints and reconfirm stored successes; stale candidates are excluded. Accepted candidates replace existing members, so group size remains GG.

We will consolidate these rules in Algorithm 1, Appendix A.7, with a reference from Section 3.3.

Reviewer_ZS4TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces G2ARL, a GFlowNet-based reinforcement learning method for multi-turn language model agents. The approach combines a pairwise flow reallocation loss, which emphasizes comparisons between successful and failed trajectories, with coverage-guided replay that introduces previously successful trajectories into training groups. Experiments on ALFWorld, WebShop, and ScienceWorld with Qwen2.5 models at 1.5B and 7B scales show consistent improvements over seven retrained baselines. At 1.5B, G2ARL outperforms GiGPO by 17.9, 20.7, and 7.6 percentage points across the three benchmarks, respectively. The method also produces shorter trajectories with fewer repeated actions.

Critical strengths

Strong experimental results across multiple environments - the evaluation covers three different agentic benchmarks, two model scales, and seven baselines, including PPO, GRPO, GiGPO, and several GFlowNet-based methods. G2ARL achieves substantial improvements in success rates, particularly on ALFWorld and WebShop at 1.5B. The improvements are also accompanied by shorter successful trajectories, fewer repeated actions, and fewer loops, which are important properties for practical agentic systems.

Solid theoretical motivation and analysis. The proposed pairwise objective avoids learning an explicit partition function while emphasizing success-failure comparisons. The authors establish conditions under which the objective recovers the intended reward-proportional distribution and analyze how pairwise weights affect the optimization landscape. The additional analysis of TB, SubTB, and FlowRL provides useful insights into why existing GFlowNet objectives may struggle with sparse rewards and homogeneous trajectory groups.

Thorough ablations and experimental controls. The paper examines the contributions of the pairwise loss, replay mechanism, pair weights, and replay selection strategy. In particular, replacing the proposed loss with Var-TB reduces ALFWorld success from 80.1880.18% to 66.3166.31%, while disabling replay reduces it to 65.7165.71%. The authors also evaluate multiple decoding temperatures, provide paired confidence intervals, and verify that the improvements persist when training on the full ALFWorld dataset. These additional experiments make the empirical findings more convincing.

Critical weaknesses

Limited methodological novelty. Several components build on existing ideas, including group-centered GFlowNet objectives, contrastive trajectory comparisons, and submodular replay. While their combination works well, the contribution of the proposed weighting scheme is less clear. In Appendix Table 15, using only success-failure pairs achieves 77.3477.34% success, compared with 76.8076.80% for the full objective. This suggests that the within-class constraints may not be necessary for the observed improvements. A more direct comparison with related contrastive GFlowNet methods would help clarify the novelty.

Training budget and statistical robustness. G2ARL uses groups of 16 or 32 trajectories, while the policy-gradient baselines use groups of 8. It is unclear whether the methods are compared under equivalent environment-interaction and computational budgets. Additionally, most configurations are evaluated using a single training run, and the reported bootstrap intervals do not account for training randomness. For ALFWorld-7B, the 4.2-point improvement over GiGPO has a confidence interval that includes zero.

Insufficient evidence for improved distribution matching and diversity. Although preserving diverse successful trajectories is a central motivation, the experiments mainly demonstrate improvements in success rate and action efficiency. There is no direct evaluation of how well the learned distribution matches the reward-proportional target. Moreover, solution diversity does not consistently improve. On WebShop-1.5B, G2ARL achieves a UV score of 6.30, compared with 7.33 for GiGPO and 11.31 for Flow-GRPO.

Questions for authors

1)Appendix Table 15 shows that using only success-failure pairs performs similarly to the full objective. What is the practical benefit of retaining within-class constraints? How does G2ARL compare with existing contrastive GFlowNet approaches when using the same replay mechanism?

2)Can the authors clarify whether the total number of environment interactions and training compute are matched across methods? How does G2ARL perform with the same group size as the policy-gradient baselines? Could the authors also provide additional training-seed results, particularly for WebShop or ALFWorld-7B?

3)Distribution matching and diversity. Can the authors provide direct evidence that G2ARL better approximates the reward-proportional trajectory distribution? Evaluating probability-ratio errors or solution diversity at comparable success rates could help clarify whether the gains come from distribution matching or primarily from more effective optimization.

2 comments

W1 & Q1. Novelty and within-class constraints

Our two contributions are class-dependent pair weighting and composition-dependent injection. With replay held fixed, weighting improves ALFWorld success from 66.31% to 80.18% (Table 3). Under TB on WebShop, replacing fixed replay with composition-dependent injection, including zero-weighting all-failure groups without stored successes, improves success from 13.34% to 16.73%.

The comparison with success–failure-only weighting is:

BenchmarkSuccess: (1, 0.3, 3)Success: (0, 0, 3)UV: (1, 0.3, 3)UV: (0, 0, 3)
ALFWorld-1.5B, original Table 1576.8077.346.6086.900
WebShop-1.5B84.47182.32110.4909.174
ScienceWorld-1.5B66.7960.412——

ScienceWorld’s deployed-weight result is from original Table 2. Success–failure-only weighting is competitive on ALFWorld. Within-class weights additionally support learning from all-success groups on WebShop and ScienceWorld, and partial-score failure groups on ScienceWorld. On WebShop, the gains are 2.150 in success and 1.315 in UV.

For the contrastive comparison:

ALFWorld-1.5BSuccess (%)UV
Stable-GFlowNet64.606.376
Our (0, 0, 3) variant77.7648.059
G²ARL, original Table 3 row (2)80.187.414

Our method give better performance compared to Stable-GFlowNet

W2 & Q2. Budgets and training randomness

Total environment interactions and compute are not matched. We added controls with the same group size and number of newly sampled trajectories:

ALFWorld-1.5BTraining poolGroup sizeNew trajectoriesSuccess by training run (%)
G²ARLFull819,20086.02
GiGPOFull819,20069.44 / 69.81
G²ARL64 tasks819,20087.21 / 81.44 / 76.98
GiGPO64 tasks819,20063.54
G²ARL64 tasks1638,40089.31 / 85.80
GiGPOFull1638,40078.56 / 77.74

All runs use 150 updates; the full pool contains 3,553 tasks. G²ARL retains an advantage in the group-8 comparisons with the same training pool. Even use much more smaller training pool, G²ARL still give better performance.

Additional training seed results are:

BenchmarkG²ARL success (%)GiGPO success (%)
ALFWorld-7B92.79 / 90.60 / 85.7788.61
WebShop-1.5B84.92 / 84.47164.19 / 68.821

W3 & Q3. Distribution matching and diversity

At ρ=1\rho=1, the target is the optimum of KL-regularized reward maximization. Our probability-ratio diagnostic shows closer within-success matching than GiGPO, but not GRPO or TB; it does not establish universally better target matching.

At the same temperature and comparable success rates, the diversity results are:

ALFWorld-7B, temperature 1.3Success (%)UV
GiGPO88.499.146
G²ARL, β=8\beta=889.2412.013
G²ARL, deployed β=128\beta=12889.5912.622

Under similar success rate, Our G²ARL yield more diverse solution, higher UV.

The training logs also support better finite-budget optimization: weighted and equal-weight runs have cumulative update lengths within 0.28%, but final-window on-policy success is 90.07% versus 66.61%, and test success is 80.18% versus 66.31%. The evidence therefore supports improved learning progress and diversity in specific settings, rather than a universal distribution-matching advantage.

Thak you for your additional experiments. Some of them satisfied my previous concerns.

Reviewer_x8pBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses post-training of multi-turn LLM agents with sparse terminal rewards. It proposes G2ARL, a GFlowNet-style method that targets a reference-tilted, reward-proportional trajectory distribution instead of pure reward maximization, motivated by the loss of trajectory diversity and of success-failure contrast as GRPO-style groups become homogeneous during training. The method has two parts: a class-weighted pairwise residual loss over same-task trajectories that cancels the partition function, upweights success-failure pairs, and reduces to Var-TB/VarGrad under equal weights; and a coverage-guided rebalancing scheme that keeps verified successes through submodular selection and injects them into all-failure groups and, with decaying probability, into mixed groups. LLM baselines are covered extensively: PPO, GRPO, GiGPO, Flow-GRPO, FlowRL, TB and SubTB are all retrained from shared Qwen2.5-1.5B/7B-Instruct LoRA starting points on ALFWorld, WebShop and ScienceWorld, where G2ARL reports 4.2 to 20.7 point success gains over the strongest comparator, shorter and less repetitive solutions.

Critical strengths

The paper is well motivated. It identifies a concrete and increasingly relevant failure mode of group-based agent RL: as the policy improves, groups become all-success or all-failure and lose the contrast that carries the reward signal. The pairwise loss is clean and well connected to prior work. It needs no learned partition function, it has a closed form that is linear in the group size, and it reduces exactly to Var-TB under uniform weights.

Critical weaknesses

First of all, the technical novelty is limited, because the method is largely an assembly of existing components, as the authors themselves cite (Related Work section). The pairwise residual, in which the partition function cancels, is the contrastive balance condition of EP-GFlowNets. Stable-GFlowNet already pairs contrastive trajectory balance with high-reward replay anchors, the same loss-plus-replay design in a single-turn setting. With equal weights the loss reduces exactly to Var-TB and VarGrad, and its target is already characterized for GVPO. The replay selection rule is adopted unchanged from RapTB, group filtering follows DAPO, and success injection follows self-imitation, RePO and ExGRPO. GFlowVLM has also already applied GFlowNet objectives to ALFWorld agents. The genuinely new element is the class-dependent pair weighting, and it is thin. The paper’s own theory shows the weights leave the fixed point unchanged, and keeping only success-failure pairs performs as well as the full scheme. With binary rewards, that variant is exactly the reward-contrast mask of Stable-GFlowNet, the closest prior work, from which the paper claims to differ.

Besides, the conceptual contribution of “distribution matching” is overstated. With ρ = 1, the target is exactly the optimum of KL-regularized reward maximization. At the deployed β of 64 or 128, it places essentially all its mass on the success class, with the conditional over successes simply inherited from the reference policy. The contrast with reward maximization drawn in the introduction is therefore much weaker than presented, and the diversity motivation is not clearly reflected in the paper’s own diversity metric. The theoretical results restate that the zero set of a connected graph-Laplacian quadratic form does not depend on its edge weights, in tabular coordinates. They do not explain why the weighting helps under a parameterized policy.

The paper lacks some clarity. The main text is dense with notation and heavily qualified sentences. Results are often phrased as readings relative to other ablation rows, which forces the reader to reconstruct the comparison. Essential mechanisms are scattered rather than stated in one place: how groups are gated, how stored trajectories are re-executed and re-scored, how the injection probability is scheduled, and the ScienceWorld partial-score fallback. Rows across tables also mix different reward scales, training pools and dispersion conventions, so it is hard to see at a glance what is being compared. The limitations section is minimal and does not engage with the method’s assumptions.

Last but not least, the theory does not explain where the gains come from. Its main result is that the optimum of the pairwise objective is the same whatever the pair weights and however often each pair is sampled, which covers the effect of replay. Yet the design choices the ablations credit with the improvement are exactly these: the class weights, turning replay on or off, and coverage-based versus reward-prioritized selection (Table 3). By the paper’s own result all these variants share the same optimum, so their differences must come from optimization dynamics under the LoRA parameterization, which the authors explicitly leave unanalyzed. The theory therefore justifies the target rather than the method, and the mechanism behind the reported gains remains unexplained.

Questions for authors
  1. How would the pairwise loss extend beyond binary verifiable rewards to graded scores, learned reward models or rubric-based judges in non-verifiable domains? In those settings the success/failure partition, and hence the class weights, is no longer defined. Would the weights become a function of score differences, and would the target-preservation property remain meaningful under noisy rewards?

  2. Could flow reallocation be applied at the step level rather than the trajectory level? For example, sub-trajectories that pass through a shared anchor state, as in GiGPO, could be compared to obtain denser credit assignment. How would such a variant relate to sub-trajectory balance without a learned state flow?

  3. Does the learned distribution provide benefits beyond single-sample success? Examples include stronger test-time scaling through best-of-n or voting, or better exploration when the trained policy is used as the starting point for further RL on new tasks. Would annealing β during training offer a useful way to trade off exploration early against exploitation late?

  4. Could the replay buffer be shared across related tasks to transfer successful strategies, or seeded with demonstrations or trajectories from other models to bootstrap tasks that never produce a success on-policy? How would such a buffer operate in stochastic or non-resettable environments such as live web interaction, where stored trajectories cannot be deterministically re-executed and verified?

3 comments

Thanks for your advice.

Weakness1: novelty and contribution of G2ARL

Our two main contributions are:

  1. Class-dependent pair weighting. We weight success–success, failure–failure, and success–failure pairs. The pairwise loss preserves the same zero-loss target under connected comparisons. The group weights change the optimization geometry (Propositions 3–4). With replay held fixed, these weights improve ALFWorld success from 66.31% with equal weights to 80.18% (Table 3).

  2. Composition-dependent injection. Unlike fixed-ratio replay, we inject available same-task successes into all-failure groups, anneal injection into mixed groups, and skip injection into all-success groups. Under TB on WebShop, replacing fixed replay with this rule, including zero-weighting all-failure groups without stored successes, improves success from 13.34% to 16.73%.

We ported Stable-GFlowNet’s loss and fixed-ratio anchors into our pipeline:

MethodALFWorld-1.5B success (%)UV
Stable-GFlowNet port64.60 ± 0.676.376
G²ARL80.18 ± 0.507.414 ± 0.142

The comparison with success–failure-only weighting is:

Benchmark / runSuccess: (1, 0.3, 3)Success: (0, 0, 3)UV: (1, 0.3, 3)UV: (0, 0, 3)
ALFWorld, original Table 1576.8077.346.6086.900
ALFWorld, additional experiment80.93580.1196.33227.3692
WebShop, additional experiment84.47182.32110.4909.174
ScienceWorld, additional experiment66.7960.412——

All models are 1.5B; success is in percent. ScienceWorld’s deployed-weight result is from original Table 2.

We agree that (0, 0, 3) is competitive on ALFWorld. The deployed weights perform better in the reported WebShop and ScienceWorld comparisons, but we do not claim universal superiority. On WebShop, the task-paired gains are 2.150 in success and 1.315 in UV.

Finally, binary success labels do not imply binary rewards: Eq. (1) includes reference-policy probabilities and invalid-action penalties. In our port, Stable-GFlowNet’s reward-contrast mask retains 91.58% of success–success and 97.90% of failure–failure pairs, (0, 0, 3) excludes both. They are therefore different objectives in our setting. We will revise the contribution statement to make these distinctions and limitations explicit.

Weakness2: motivation and diveristy performance

At ρ = 1, our target is exactly the optimum of KL-regularized maximization of βy − λn, as Eq. (6) states. We will revise the abstract and introduction accordingly. Our contribution concerns how we optimize this target through class-weighted pairwise residuals and composition-dependent replay.

For binary scores, large β strongly favors success, and the conditional distribution over successful trajectories is proportional to π_ref exp(−λn). It therefore inherits the reference policy’s relative probabilities, adjusted for invalid actions; it does not introduce a separate diversity objective.

At the same higher temperature and comparable success rates, the original ALFWorld-7B results show:

MethodTemperatureSuccess (%)UV
GiGPO1.388.499.146
G²ARL1.389.5912.622

UV counts distinct successful command sequences among 16 samples per solved task. These results demonstrate greater sampled solution diversity in similar success rate, indicating our method finds more solutions.

For distribution matching, our direct distribution-matching diagnostic establish consistently closer matching on sucess pool:

Mean absolute pairwise residual on a shared success-only pool ↓G²ARLGiGPO
ALFWorld-1.5B18.0625.41
WebShop-1.5B108.10164.20

G²ARL improves this diagnostic over GiGPO, it measures within-success relative-probability matching.

Weakness3: clarity

In the revision, we will state each mechanism in one place, remove repeated qualifications from the main text, and give absolute values before differences in every comparison.

Weakness4: how those modules are combined

We will add Algorithm 1 to Appendix A.7 and cite it in Section 3.3. Each update:

  1. Sample and verify trajectories; store successes per task using coverage selection. ScienceWorld also stores partial-score samples until a success is found.
  2. Set group weights by composition. Refill all-failure groups from the buffer, inject into mixed groups with decreasing probability, and leave all-success groups unchanged.
  3. Re-execute replay candidates, verify endpoints and stored successes, and rescore under the current policy; discard stale entries.
  4. Apply the weighted pairwise loss and replay penalty, clip gradients, and update the policy.

Weakness5: training pool, beta, and "dispersion"

We agree. We don't really understand what "dispersion" means, can you clarify this? To the best of our understanding, we choose the random seed experiments to test the "dispersion". The table below brings the relevant ALFWorld-1.5B results together, with their settings. All use the same evaluation split (134 tasks), temperature 0.4, and 16 rollouts per task.

Method / sourceβTraining poolSuccess (%)Meaning of ±
G²ARL, Table 16464 tasks87.55 ± 0.17Decoding seeds of the two-training-run mean
G²ARL, Table 186464 tasks87.55 ± 2.48Two training runs
G²ARL, Table 3 row (2)864 tasks80.18 ± 0.50Eight evaluation runs
G²ARL, Table 1764Full split (3,553 tasks)87.34 ± 2.73Two training runs
GiGPO, Table 17—64 tasks63.54 ± 1.10Decoding seeds
GiGPO, Tables 17–18—Full split (3,553 tasks)69.62 ± 0.26Two training runs

The first two rows summarize the same trained models using different dispersion units. β is the GFlowNet reward-scale parameter and does not apply to GiGPO. We will add these settings and explicit dispersion labels to the relevant captions, rather than leaving readers to reconcile Tables 1, 3, 17, 18, and 20.

Weakness6: why loss with same optimal solution give different performance

The same optimum does not imply the same training dynamics under LoRA and a finite update budget. Weights change gradient contributions; replay and selection change which trajectories contribute.

Four runs on the same machine show distinct trajectories in effective LoRA weight space:

VariantSampled path lengthFinal displacement from base
Deployed (1, 0.3, 3)20.290711.6621
Equal weights19.922810.9119
Success–failure only (0, 0, 3)20.903312.3491
Replay disabled21.378312.9716

Distances use effective LoRA updates 2BA, sampled from the base through checkpoints every 10 updates. Path length alone does not rank optimization quality.

The controlled comparison in Table 3 provides stronger evidence: deployed and equal weights have cumulative parameter-update lengths within 0.28%, but deployed weights produce more aligned consecutive updates in the final 50 steps (0.8157 vs. 0.7801) and higher on-policy success (90.07% vs. 66.61%), followed by higher test success (80.18% vs. 66.31%).

Question1

The pairwise residual already supports graded scores. ScienceWorld uses normalized terminal scores y∈[−1,1]y\in[-1,1], together with a separate success flag for class weighting. It therefore tests graded rewards, but not settings without a success/failure partition.

Without that partition, one possible extension is a symmetric score-dependent rule, such as wij=ε+f(∣yi−yj∣)w_{ij}=\varepsilon+f(|y_i-y_j|), where ε>0\varepsilon>0 and ff is nonnegative. This would emphasize larger score differences while retaining all comparisons. We have not evaluated this extension.

For fixed scores and weights held constant during differentiation, Proposition 3 still applies if the comparison graph remains connected: changing the weights preserves the zero-loss target π∗∝πrefρexp⁡(βy−λn)\pi^\ast\propto\pi_{\mathrm{ref}}^\rho\exp(\beta y-\lambda n). This includes fixed scores from a learned reward model or rubric-based judge.

Under noisy judging, target preservation means preserving the target defined by those scores, not recovering a noise-free or human-preferred target. Large β\beta can amplify scoring errors. If scores are resampled at each update, the fixed-target guarantee does not directly apply, especially when weights depend on the same noisy scores. Our current evidence concerns environment-provided scores and success flags.

Question2

Yes, for terminal continuations sharing the same history hh: the common prefix and state flow F(h)F(h) cancel in their pairwise ratio, yielding a contrastive subtrajectory objective without a learned state flow. These comparisons preserve the target under Proposition 3’s connectivity assumptions.

A shared observation, however, is insufficient. Different histories can have different continuation values, leaving a correction log⁡Γ(h′)−log⁡Γ(h)\log\Gamma(h')-\log\Gamma(h). Arbitrary nonterminal subtrajectories likewise require endpoint-flow terms. Thus, this cancellation does not replace general SubTB. Our fixed-potential SubTB analysis illustrates the limitation: at valid, nonterminal branches, its zero-loss allocation matches the target only when all actions have equal continuation values (Proposition 8(ii)).

Question3

With a verifier, G²ARL retains benefits beyond one sample. Each cell reports G²ARL / strongest baseline pass@kk (%):

Benchmarkk=1k=1k=2k=2k=4k=4k=8k=8k=16k=16
ALFWorld-1.5B87.55 / 69.6290.63 / 76.3292.75 / 80.7794.05 / 85.8594.68 / 89.74
ALFWorld-7B92.79 / 88.6194.47 / 91.5195.80 / 93.5396.75 / 94.8097.20 / 95.71
WebShop-1.5B84.92 / 64.1985.98 / 70.0386.62 / 74.0987.12 / 77.9087.60 / 81.07
WebShop-7B82.15 / 71.4882.77 / 77.8083.18 / 81.7583.62 / 84.2984.00 / 86.20

G²ARL leads at every reported budget in three settings, although the margins narrow. On WebShop-7B, the baseline overtakes it at larger budgets. These results support higher multi-sample success, not uniformly stronger scaling.

Question4

Yes. A shared pool, demonstrations, or trajectories from other models can bootstrap tasks with no on-policy successes. Candidates must be validated and rescored on the receiving task; comparisons remain within that task because partition functions are task-specific. Under connected comparison support, the zero-loss target does not depend on the candidates’ source.

Stochastic environments require fresh validation rather than assuming replay reproduces an outcome. In non-resettable environments, stored outcomes can support replay, but their continued validity cannot be guaranteed as the environment changes. Such use would extend our current method: its replay validation and theoretical guarantees assume deterministic, resettable environments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

summary of reviews

We thank the reviewers for their feedback. They recognized the practical relevance and effectiveness of the method (Pazk), its motivation and clean pairwise loss (x8pB), strong results and thorough ablations (ZS4T), and promising performance (RxVR). We address their questions as follows.

  • What contributes beyond prior work (Pazk, x8pB, ZS4T). We clarify that our contributions are outcome-dependent pair weights and composition-dependent replay. Replay restores missing success–failure comparisons using verified past successes; weighting strengthens these comparisons. On ALFWorld-1.5B, weighting improves success from 66.31% to 80.18% with replay fixed, and from 55.47% to 65.71% without replay. We distinguish these designs from FlowRL, Var-TB and Stable-GFlowNet. Within-class comparisons help on WebShop but are not uniformly better on ALFWorld.
  • Why the pairwise objective preserves the target (Pazk, x8pB). Subtracting same-task trajectory-balance residuals cancels the shared normalizer exactly. Under common positive support and connected comparisons, zero pairwise loss recovers the reward-proportional distribution and conserved prefix-tree flow. We clarify the reward mass, Eq. (2), the equivalence to KL-regularized reward maximization at ρ = 1, and the distinction between target preservation and convergence under LoRA. Removing the normalizer alone gives no performance gain in our control.
  • Training budgets and randomness (Pazk, ZS4T). With the same training pool, group size 8 and 19,200 new trajectories, G²ARL reaches 86.02% success versus GiGPO’s 69.44% and 69.81%. We define new trajectories as group size times sampled task groups, before filtering or replacement. Replay execution and rescoring cost extra; the reported timing control measures 22.3% overhead. Additional training runs separate training variation from decoding variation; the 7B results vary more than the 1.5B results.
  • Distribution matching and practical diversity (ZS4T, Pazk, x8pB). We add probability-ratio diagnostics on shared successful-trajectory pools and define UV as the number of distinct successful command sequences per solved task. On ALFWorld-7B at temperature 1.3, UV is 12.622 versus GiGPO’s 9.146 at comparable success. Verifier-selected best-of-16 reaches 94.68% versus 89.74% on ALFWorld-1.5B. These benefits depend on the setting and do not establish diversity as the cause of the gains.
  • Group construction and presentation (RxVR, Pazk, x8pB). Algorithm 1 consolidates sampling, storage, composition-based injection, verification and rescoring. We define the missing symbols, point class counts to their use in the loss decomposition, consolidate benchmark details, and explain how replay reuses sparse successes without guaranteeing the first success. FlowRL is consistently cited as ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.