- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Experience-Guided Action Selection (EGAS), an extension of GiGPO for training LLM agents. EGAS maintains an experience pool of executed (task, state, action) tuples with time-decayed success and failure counts, modeled as Beta posteriors over task success. During a subset of rollouts, the policy samples an initial action. If the pool suggests a historically supported alternative may do better, the policy samples K−1 additional candidates and executes the one with the highest posterior mean. Only one action is executed per state, so no extra environment branches are needed.
The pool also shapes learning in two ways. GiGPO advantages are reweighted per state group by the estimated potential improvement. An auxiliary PPO-style objective trains unexecuted candidates using pairwise preferences, estimated from the pool, between each candidate and the executed action.
Across ALFWorld, WebShop, and Sokoban with Qwen3 1.7B/4B models, EGAS reports gains of 1.7–7.9pp in held-out success over GiGPO. It also reaches GiGPO’s best-observed performance with 7.3–25% fewer training trajectories.
Critical strengths
The core idea is simple and well motivated.
The ablation suite is fairly thorough. It covers component ablations, intervention allocation, candidate-set size, and auxiliary weight. The retrieved-action ablation in Appendix F is a nice design-justifying experiment.
The paper is clearly written and the method is easy to follow.
Critical weaknesses
No confidence intervals or significance testing. None of the main results, efficiency metrics, or ablations report confidence intervals. I understand these experiments are expensive, but the gains are small enough that some evidence of statistical significance is necessary. Each checkpoint is evaluated on 128 episodes (256 for WebShop). At a success rate of about 55%, the binomial standard error of a single run is about 4.4pp. A difference between two independent runs therefore has a 95% interval of roughly ±12pp from evaluation sampling alone. Even the largest headline gain (+7.9pp on ALFWorld-1.7B) is not distinguishable from zero on that basis, and seed variance adds to it: Appendix G reports EGAS seed SDs of 2.9pp on ALFWorld-1.7B and 4.7pp on Sokoban. Multi-seed results are given only for EGAS, not for GiGPO or other baselines, and the main table and ablations appear to be single runs. Multi-seed means with confidence intervals for both EGAS and GiGPO would substantially strengthen the paper. This is the main thing that would change my assessment.
Reliance on exact matching limits the scope of the method. As I understand Appendix C.4, a candidate is eligible for selection only if its normalized (task, observation string, action string) key exactly matches a pool entry with sufficient support. There is no semantic matching. This is plausible for ALFWorld and Sokoban, whose action spaces are small, enumerable sets of commands or moves. It seems much less likely for free-text actions such as WebShop’s search[...] queries, where selection may effectively operate only over click[...] actions. Keys also include the task identifier, so the pool helps only when the same task is resampled in a later iteration. The paper should also state explicitly that the method targets deterministic environments with discrete, string-matchable observations and a finite set of repeated training tasks.
The “learn without executing” framing is somewhat overstated. Every pool entry with a success estimate exists because a trajectory containing that exact state-action pair was executed in an earlier iteration. The method avoids executing a candidate again now. It does not evaluate truly unexecuted actions: unsupported candidates receive only a prior-based signal. Framing the pool explicitly as a tabular Monte Carlo critic estimated from past rollouts would make both the contribution and its limits clearer.
Overall, I think this is a good paper with a sensible, practical idea, and I lean toward weak reject in its current form. My main concern is the lack of statistical evidence for gains of this size. I would be happy to raise my score if the authors report multi-seed results with confidence intervals for both EGAS and the GiGPO baseline.
Questions for authors
Can you report results over multiple seeds, with confidence intervals, for both GiGPO and EGAS? This applies to the main results (Table 1, Figures 3–4) and ideally the key ablations (Figure 7).
3 comments
Thank you for running the additional experiments. I appreciate the effort.
The paper’s contribution is primarily empirical. For a contribution of this kind, we need evidence that the gains are statistically distinguishable (or at least plausibly so). Its really hard for me to justify raising my score for an empirical paper at ICLR without that evidence.
From your new results, EGAS only outperforms GiGPO in one setting (ALFWorld-4B). For the remaining settings, the intervals on the difference are wide and include zero. The consistent positive direction across settings is encouraging, but I find it hard to recommend acceptance without stronger evidence. I’d encourage running additional seeds.
For now I’m keeping my score, but I’d be glad to revisit it if additional seeds establish the gains clearly.
Thank you for pointing out the limitations of our statistical evidence. We agree that, although EGAS achieved higher cross-seed mean performance in all reported settings, the confidence intervals for the performance differences were still wide with only three training seeds, providing insufficient evidence to establish statistically significant improvements across settings.
To address this concern, we added three training seeds for both GiGPO and EGAS, increasing the number of runs per method in each setting from three to six. We now report the complete per-seed results, means, standard deviations, and the mean performance differences between EGAS and GiGPO with their 95% confidence intervals. The updated analysis shows that the confidence intervals for the differences are entirely above zero for all seven reported metrics. All seven comparisons also remain statistically significant at the 0.05 level after applying two-sided Welch t-tests and Holm correction for multiple comparisons. These additional results provide stronger statistical support for the mean performance improvements of EGAS in the reported settings. The detailed results are provided below.
Table 1: Complete results across six training seeds
| Setting / Metric | Method | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Seed 5 |
|---|---|---|---|---|---|---|---|
| ALFWorld 1.7B / Success | GiGPO | 50.00 | 60.16 | 55.47 | 49.21 | 49.21 | 52.34 |
| ALFWorld 1.7B / Success | EGAS | 60.16 | 64.84 | 59.38 | 61.71 | 60.16 | 64.84 |
| ALFWorld 4B / Success | GiGPO | 64.84 | 62.50 | 63.28 | 60.16 | 61.71 | 60.93 |
| ALFWorld 4B / Success | EGAS | 71.10 | 70.30 | 73.40 | 70.30 | 71.10 | 71.88 |
| WebShop 1.7B / Score | GiGPO | 80.66 | 76.39 | 77.68 | 73.40 | 74.05 | 73.82 |
| WebShop 1.7B / Score | EGAS | 80.20 | 80.90 | 77.10 | 81.20 | 81.60 | 82.01 |
| WebShop 1.7B / Success | GiGPO | 62.89 | 49.61 | 51.56 | 54.29 | 52.73 | 54.68 |
| WebShop 1.7B / Success | EGAS | 64.50 | 65.20 | 61.80 | 66.40 | 65.23 | 63.28 |
| WebShop 4B / Score | GiGPO | 86.19 | 86.91 | 87.72 | 88.76 | 84.33 | 84.91 |
| WebShop 4B / Score | EGAS | 89.60 | 88.40 | 88.20 | 90.34 | 92.31 | 92.10 |
| WebShop 4B / Success | GiGPO | 78.91 | 73.83 | 80.08 | 75.00 | 71.48 | 71.48 |
| WebShop 4B / Success | EGAS | 80.50 | 79.30 | 79.70 | 82.42 | 82.81 | 83.98 |
| Sokoban VL-4B / Success | GiGPO | 85.94 | 76.56 | 81.25 | 78.68 | 78.25 | 78.03 |
| Sokoban VL-4B / Success | EGAS | 80.50 | 85.90 | 89.80 | 89.06 | 88.71 | 89.73 |
Table 2: Cross-seed means, standard deviations, and 95% confidence intervals for each method
Each method has six training runs in each setting. Mean ± SD denotes the cross-seed mean and sample standard deviation. The two-sided 95% confidence interval for each method's mean is computed using Student's t method with five degrees of freedom.
| Setting / Metric | GiGPO Mean ± SD | GiGPO 95% CI | EGAS Mean ± SD | EGAS 95% CI |
|---|---|---|---|---|
| ALFWorld 1.7B / Success | 52.73 ± 4.36 | [48.15, 57.31] | 61.85 ± 2.44 | [59.29, 64.41] |
| ALFWorld 4B / Success | 62.24 ± 1.69 | [60.47, 64.01] | 71.35 ± 1.17 | [70.12, 72.57] |
| WebShop 1.7B / Score | 76.00 ± 2.83 | [73.03, 78.97] | 80.50 ± 1.78 | [78.64, 82.37] |
| WebShop 1.7B / Success | 54.29 ± 4.60 | [49.46, 59.12] | 64.40 ± 1.63 | [62.69, 66.12] |
| WebShop 4B / Score | 86.47 ± 1.68 | [84.71, 88.23] | 90.16 ± 1.77 | [88.30, 92.02] |
| WebShop 4B / Success | 75.13 ± 3.66 | [71.28, 78.98] | 81.45 ± 1.89 | [79.47, 83.43] |
| Sokoban VL-4B / Success | 79.79 ± 3.38 | [76.24, 83.33] | 87.28 ± 3.62 | [83.49, 91.08] |
Table 3: Mean differences (EGAS − GiGPO), 95% confidence intervals, and significance tests
We use training runs as the statistical units, with n = 6 per method in each setting, and define the mean difference as Δ = Mean(EGAS) − Mean(GiGPO). We apply independent-samples, two-sided Welch t-tests and compute the approximate 95% confidence intervals for the differences using Welch–Satterthwaite degrees of freedom. We also apply Holm correction to the seven comparisons reported in this table.
| Setting / Metric | Mean Difference Δ | 95% CI for Δ | Two-sided p-value | Holm-adjusted p-value |
|---|---|---|---|---|
| ALFWorld 1.7B / Success | +9.12 | [4.40, 13.84] | 0.00219 | 0.01223 |
| ALFWorld 4B / Success | +9.11 | [7.21, 11.01] | 0.00000195 | 0.00001364 |
| WebShop 1.7B / Score | +4.50 | [1.38, 7.62] | 0.01011 | 0.01629 |
| WebShop 1.7B / Success | +10.11 | [5.28, 14.94] | 0.00204 | 0.01223 |
| WebShop 4B / Score | +3.69 | [1.47, 5.91] | 0.00411 | 0.01629 |
| WebShop 4B / Success | +6.32 | [2.39, 10.25] | 0.00630 | 0.01629 |
| Sokoban VL-4B / Success | +7.50 | [2.99, 12.00] | 0.00407 | 0.01629 |
Thank you again for encouraging us to conduct this additional validation. We will include the complete results and statistical methodology in the revised manuscript. We hope this additional evidence addresses your main concern regarding the statistical reliability of the main results.
Response to Reviewer s8XY
As suggested, we added main results for both GiGPO and EGAS using seeds 0, 1, and 2, together with three-seed component ablations on ALFWorld with Qwen3-1.7B.
We report individual-seed results, Mean ± SD, and approximate 95% confidence intervals computed using Student’s t method, with training seeds as the statistical units. All metrics use a 0–100 scale.
Main results
EGAS achieves higher cross-seed means on all reported metrics, with the SDs and confidence intervals showing the associated variability and uncertainty.
Key component ablations
Full EGAS outperforms all five ablation variants in each seed, providing further support for the consistent direction of the component contributions.
We thank the reviewer for pointing out these clarity issues. We will revise the relevant sections to make the methodological descriptions more precise and clearly state the scope of our claims and the method’s limitations.