- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GAUGE, a policy optimization method that controls updates based on executable actions rather than textual outputs alone. It decomposes textual policy changes into changes in action selection and changes in how the same action is expressed. GAUGE estimates action probabilities by pooling equivalent expressions and uses action- and argument-level changes to weight updates. Validation includes fixed-history diagnostics on WebShop and experiments with Qwen3-4B on ALFWorld, WebShop, Sokoban, and TIR Math.
Critical strengths
-
The paper provides a theoretical distinction between changes in executable actions and changes in their textual expression. The exact KL decomposition gives a motivation for controlling updates in action space, making the contribution conceptually meaningful beyond a particular weighting rule.
-
The experiments cover four different environments and include several relevant baselines, including GSPO and DAPO. The reported gains on ALFWorld, WebShop, and Sokoban provide encouraging evidence that the approach can benefit multiple types of agent tasks.
Critical weaknesses
-
GAUGE changes action-probability estimation, ratio bounding, and semantic gating simultaneously. The current comparisons do not isolate each component’s contribution to the performance gains. Please add controlled ablations, particularly a text-level objective with the same bounding and gating rules, and a GAUGE variant without slot-level gating. I would consider raising my score if additional experimental evidence and explanations adequately address this concern.
-
GAUGE uses a selected set of candidate actions to measure policy changes. Important policy changes may be missed if relevant actions are not included in the candidate set. Please explain how candidate actions are selected, especially in open-ended tasks such as search and code generation, and evaluate the performance impact of increasing or decreasing the number of candidate actions.
-
GAUGE requires scoring additional candidate actions, so the computational cost may be higher than that of the baselines. Please report the additional training cost and compare performance under the same compute budget to assess whether the gains justify this cost.
-
Proposition 5 assumes that actions in the interaction history are represented as executable actions. If the implementation retains previous reasoning text and raw outputs, different textual representations of the same action may still affect subsequent decisions. Please explain how interaction histories are constructed and clarify whether the experimental pipeline satisfies the proposition’s assumptions.
Questions for authors
Please see weaknesses.
16 comments
Rebuttal for W1 by Authors (2/2)
Let denote a batch gradient under rule , and let be its mean over the same 64 batches. We define
The table reports both raw covariance-trace ratios and . Unlike raw variance, is unchanged if every gradient is multiplied by the same nonzero constant. Mean-gradient norm ratios relative to IS are 1.000, 0.975, and 0.946 for IS, bounded-only, and full GAUGE. Dividing each raw variance ratio by the square of its mean-norm ratio gives the normalized values below. These are norms of the mean gradients, not means of individual gradient norms.
| Update rule | Relative mean-gradient displacement | Raw gradient variance / IS | Normalized gradient variance / IS | Success (%) | Paired gain vs. IS, 95% interval (points) |
|---|---|---|---|---|---|
| Unbounded action IS | 0 | 1.00 | 1.000 | 74.4 ± 1.0 | Reference |
| Bounded ratio, no gate | 0.049 | 0.85 | 0.894 | 74.8 ± 0.8 | +0.4 [−0.1, 0.9] |
| Full GAUGE | 0.082 | 0.76 | 0.849 | 75.5 ± 0.6 | +1.1 [0.1, 2.1] |
Results and analysis. Full GAUGE reduces raw gradient variance by 24% and scale-normalized variance by 15.1%. The latter reduction cannot be explained by uniform gradient shrinkage alone. Its 8.2% displacement is measured relative to same-support IS. The mean-gradient cosine to this reference is 0.998.
The main text reports the 15.1% normalized-noise reduction and 8.2% displacement in Section 4, Question 2 (p. 10); definitions, raw and normalized variance, and paired intervals are in Appendix B.2, Table 13 (pp. 20–21).
The success intervals use the three paired training-seed differences and a two-sided Student- interval with 2 degrees of freedom. They do not use the 64 gradient batches or individual evaluation tasks as independent training runs. Full GAUGE improves success by 1.1 points over IS, with a 95% interval of [0.1, 2.1], and by 0.7 point over bounded-only, with an interval of [0.2, 1.2]. Bounded-only exceeds IS by 0.4 point, but its interval [−0.1, 0.9] includes zero, so that particular improvement is not statistically resolved.
Relaxing constraints reduces displacement and raises noise. Removing the gate reduces mean-gradient displacement from 8.2% to 4.9%, while normalized variance rises from 0.849 to 0.894 and success falls from 75.5% to 74.8%. Removing the remaining ratio bound recovers the IS reference, with zero relative displacement and normalized variance of 1.000. The further success change to 74.4% is a point-estimate decrease; the bounded-only versus IS interval crosses zero. The controls therefore show the trade-off directly: a smaller displacement comes with noisier updates.
Experimental setup—individual seed check. We inspect the individual WebShop runs behind the three-seed summaries, keeping the paired initialization, task schedule and 200-update protocol unchanged. This checks whether the mean advantage comes from only one favorable seed.
| Method | Seed 1 | Seed 2 | Seed 3 |
|---|---|---|---|
| GSPO | 71.7 | 72.3 | 73.2 |
| Original GAUGE candidate estimator | 74.3 | 75.0 | 75.7 |
| Pure text bounding and gating | 72.7 | 73.1 | 73.5 |
| GAUGE without slot energy | 74.1 | 75.0 | 75.6 |
| Full GAUGE | 74.9 | 75.5 | 76.1 |
Results and analysis. Every paired seed favors full GAUGE over GSPO, so the average advantage is not produced by just one favorable run. Intervals use differences between runs sharing a seed, rather than treating repeated evaluation tasks as independent training runs.
The main-text summary is in Section 4, Question 2, Table 3 (p. 10), with the three-seed endpoint row also in Table 2 (p. 9); the individual WebShop runs are in Appendix B.1, Table 11 (p. 20).
Manuscript changes: The main-text method in Sections 3.2–3.3 (pp. 7–8) now separates slot energy, the trust gate and ratio bounding. Section 4, Question 2, Table 3 (p. 10) adds the requested pure-text and no-slot controls alongside individual removals; Table 2 (p. 9) provides the three-seed GAUGE summary. Appendix B.1–B.2, Tables 10–13 (pp. 20–21) supplies the exact controls, individual seeds and scale-normalized gradient measurements. These additions make the action-level contribution distinguishable from generic attenuation.
Rebuttal for W2 by Authors (1/5)
[W2] Candidate expansion improves coverage, with modest learning gains
“Please explain how candidate actions are selected, especially in open-ended tasks such as search and code generation, and evaluate the performance impact of increasing or decreasing the number of candidate actions.”
Response. Missing candidates can hide policy changes. The construction is explicit, and the tested budget comparisons show limited observed learning impact under the studied selection rules. (1) Search and TIR alternatives come from frozen-old-policy responses, with deduplication, parser-based grouping and common-string scoring. (2) In WebShop, increasing the budget from 8 to 64 raises retained mass from 92.8% to 97.0% and success from 74.6% to 75.5%. (3) The TIR and tool-free studies support candidate construction or applicability.
When an environment exposes legal actions, we enumerate them. For search, complete responses from a frozen old policy supply distinct queries; only parser-equivalent strings are grouped, without merging paraphrases by semantic similarity. TIR uses the parser-based construction below, not equality of program execution outputs.
Experimental setup. The WebShop calibration uses 512 fixed histories from 128 tasks, balanced between search and click states. At each history, 512 old-policy proposals construct the expanded pool, and both policies then generate 2,048 fresh evaluation responses. Candidate sets and estimates are frozen before evaluation.
Results and analysis. The original estimator retains 95.1% of the observed action mass, with 4.9% outside support and mean empirical TV of 0.081. Expanded actions and verified renderings retain 97.0%, with 3.0% outside support and TV of 0.045. With expanded actions but canonical renderings, omission is also 3.0% and TV is 0.060. Action and rendering expansion separately improve probability calibration on the audited histories; their learning impact is assessed by the paired training comparisons below.
The main text presents the calibration and matched-learning comparison in Section 4, Question 1, Fig. 4 (pp. 9–10); the full probability table and audit protocol are in Appendix A.1, Table 5 (p. 17).
Experimental setup—frozen gradient comparison. We use 128 prespecified histories with native nonzero advantages, grouped into 16 matched batches of eight histories. The reference scores up to 256 actions and eight verified renderings per action. Advantages and auxiliary settings are held fixed, with no optimizer updates.
Results and analysis. Relative displacement from the larger finite-pool gradient decreases from 0.091 to 0.034 after expansion. Thus the probability improvement is accompanied by closer update agreement with that finite reference. The main text discusses the candidate-budget effect in Section 4, Question 1 (p. 10); the matched backward protocol and reference are specified in Appendix A.2, Table 7 (p. 18).
Experimental setup. We also compare the three estimators directly in training, holding the ratio bound, gate, reference KL, optimizer, initialization, and task order fixed. Each branch runs for 200 updates under three paired seeds and is evaluated on the same 1,024 held-out tasks.
Results and analysis. The original estimator reaches 75.0 ± 0.7% success versus 72.4 ± 0.8% for GSPO, so most of the advantage is already present before expansion. Expansion adds 0.5 point, with a paired-seed 95% interval of [0.3, 0.7]; the original estimator's gain over GSPO is 2.6 points, with an interval of [2.4, 2.8]. Within these matched runs, expansion preserves the original method's advantage and provides a modest additional improvement.
The main text reports this direct estimator-to-learning comparison in Section 4, Question 1, Fig. 4 (pp. 9–10); Appendix A.1, Table 5 (p. 17) gives the three estimator variants and their matched training results.
We then vary the number of retained actions with all other settings fixed:
Distinguishing the two K=16 settings. The original K=16 estimator uses the original candidate-construction rule and canonical renderings. The budget sweep uses the expanded candidate generator with verified natural renderings, including at K=16. Thus, the two K=16 settings have the same action-count cap but can contain different actions and rendering sets. Within the budget sweep, candidate construction and rendering rules are held fixed, and only the action budget changes.
| Candidate budget: expanded generator + verified renderings | Retained action mass | Outside-support frequency | Mean empirical TV | WebShop success after 200 updates (%) |
|---|---|---|---|---|
| Fixed 8 | 92.8% | 7.2% | 0.105 | 74.6 ± 0.8 |
| Fixed 16 | 96.0% | 4.0% | 0.061 | 75.1 ± 0.7 |
| Fixed 32 | 96.8% | 3.2% | 0.049 | 75.4 ± 0.6 |
| Fixed 64 | 97.0% | 3.0% | 0.045 | 75.5 ± 0.6 |
| Adaptive, at most 64 | 96.4% | 3.6% | 0.053 | 75.5 ± 0.6 |
Rebuttal for W1 by Authors (1/2)
Thank you for recognizing the distinction between executable actions and their textual forms. We add the requested component controls, candidate-budget comparisons and matched-compute results. We also check the history assumption in Proposition 5 and test a canonical-history implementation that satisfies it.
These are new controlled comparisons, reported separately from the original endpoint table. The canonical-history variant is distinguished from the original raw-history implementation.
Shared experimental setting. We use Qwen3-4B in ARL-Arena and three paired training seeds. Paired runs start from the same initialization and use the same task order, rollout group size, reference KL, optimizer, and evaluation tasks. Training lasts 200 updates except in compute-limited comparisons. WebShop evaluation uses 1,024 held-out tasks. Training tables report mean ± standard deviation across seeds. Frozen-history diagnostics use task-bootstrap intervals, while training comparisons use intervals over paired seed differences.
[W1] Action scoring and slot control add gains beyond generic constraints
Response. Yes. Both action-level scoring and slot control add gains beyond generic update attenuation. (1) On WebShop, full GAUGE exceeds the pure text bounding-and-gating control by 2.4 points, with a paired 95% interval of [1.9, 2.9]. (2) Removing slot energy reduces success by 0.6 point, with an interval of [0.2, 1.0]. (3) The component pattern is consistent across ALFWorld, WebShop and Sokoban.
Experimental setup. The pure text control uses a text-level score and text-ratio drift, retaining GAUGE’s smooth bounding and gating functions. It has no parser-derived action or slot signal. Its gate temperature is chosen on training validation data to match GAUGE’s average attenuation. A second control changes only the score to a text score while retaining the action/slot-derived weights. This isolates quotient scoring from the choice of trust weights.
Removing slot energy sets its coefficient to zero and retains action-level gating. Removing the gate sets it to one and retains bounded ratios. Removing ratio bounding restores the unbounded action ratio and retains the gate. All auxiliary losses are identical.
Results and analysis. Full GAUGE improves WebShop success by 2.4 points over the pure text control, with a paired-seed 95% interval of [1.9, 2.9]. Against the control retaining GAUGE weights but using a text score, the gain is 2.0 points, with an interval of [1.8, 2.2]. Removing slot energy lowers success by 0.6 points, with an interval of [0.2, 1.0]. These are improvements beyond those obtained by generic attenuation alone.
These requested controls appear in the main text at
Section 4, Question 2, Table 3 (p. 10); removal definitions and complete comparisons are inAppendix B.1, Table 10 (p. 20).The results also separate the roles of the components. Bounding and gating improve the text baseline, so some stabilization is generic. Executable-action scoring provides a further improvement, and slot information adds a smaller but consistent benefit on these structured-action environments. We do not assume that these effects add linearly or that a slot gate is useful when no reliable semantic slot exists.
(1) Gradient noise falls after accounting for update size
Experimental setup. To test whether lower gradient variance merely reflects smaller updates, we additionally evaluate unbounded action IS, bounded-only, and full GAUGE on the same finite support and the same 64 independently sampled rollout batches. All model parameters enter the policy-gradient vectors. The diagnostic isolates the advantage-weighted policy term, excluding identical auxiliary gradients; training retains those auxiliary losses unchanged. The associated learning comparison uses the same three paired WebShop seeds and 200 updates.