acceptodds
Under review as a conference paper at ICLR 2027

GAUGE: Gauge-Invariant Policy Optimization for Language Agents

Abstract

Policy optimization for language agents typically constrains changes in token likelihood, although environment transitions and rewards depend on executable actions obtained by parsing the generated text. This mismatch can over-penalize different textual renderings of the same action while under-resolving small but behaviorally critical changes in action arguments. We formalize this distinction by decomposing raw-text policy drift into drift over executable actions and within-action gauge drift. Based on this decomposition, we introduce GAUGE, a parser-aware trust-control method that aggregates likelihood across candidate renderings and measures drift at both the action and semantic-slot levels. GAUGE preserves sequence-level gradient estimation and is compatible with group-relative policy optimization. We evaluate it in the ARLArena pipeline across ALFWorld, WebShop, Sokoban, and TIR Math. With SFT-initialized Qwen3-4B, GAUGE achieves 75.09% success on WebShop and 85.67% on Sokoban, improving over the corresponding GSPO and DAPO baselines by 2.61 and 3.27 percentage points. These results support executable-action space as a useful representation for controlling language-agent policy updates.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reinforcement learning, language agents, large language models, policy optimization, trust regions, executable actions, semantic parsing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CbkqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GAUGE, a policy optimization method that controls updates based on executable actions rather than textual outputs alone. It decomposes textual policy changes into changes in action selection and changes in how the same action is expressed. GAUGE estimates action probabilities by pooling equivalent expressions and uses action- and argument-level changes to weight updates. Validation includes fixed-history diagnostics on WebShop and experiments with Qwen3-4B on ALFWorld, WebShop, Sokoban, and TIR Math.

Critical strengths
  1. The paper provides a theoretical distinction between changes in executable actions and changes in their textual expression. The exact KL decomposition gives a motivation for controlling updates in action space, making the contribution conceptually meaningful beyond a particular weighting rule.

  2. The experiments cover four different environments and include several relevant baselines, including GSPO and DAPO. The reported gains on ALFWorld, WebShop, and Sokoban provide encouraging evidence that the approach can benefit multiple types of agent tasks.

Critical weaknesses
  1. GAUGE changes action-probability estimation, ratio bounding, and semantic gating simultaneously. The current comparisons do not isolate each component’s contribution to the performance gains. Please add controlled ablations, particularly a text-level objective with the same bounding and gating rules, and a GAUGE variant without slot-level gating. I would consider raising my score if additional experimental evidence and explanations adequately address this concern.

  2. GAUGE uses a selected set of candidate actions to measure policy changes. Important policy changes may be missed if relevant actions are not included in the candidate set. Please explain how candidate actions are selected, especially in open-ended tasks such as search and code generation, and evaluate the performance impact of increasing or decreasing the number of candidate actions.

  3. GAUGE requires scoring additional candidate actions, so the computational cost may be higher than that of the baselines. Please report the additional training cost and compare performance under the same compute budget to assess whether the gains justify this cost.

  4. Proposition 5 assumes that actions in the interaction history are represented as executable actions. If the implementation retains previous reasoning text and raw outputs, different textual representations of the same action may still affect subsequent decisions. Please explain how interaction histories are constructed and clarify whether the experimental pipeline satisfies the proposition’s assumptions.

Questions for authors

Please see weaknesses.

16 comments
AuthorsOpenReview ↗

Rebuttal for W1 by Authors (1/2)

Thank you for recognizing the distinction between executable actions and their textual forms. We add the requested component controls, candidate-budget comparisons and matched-compute results. We also check the history assumption in Proposition 5 and test a canonical-history implementation that satisfies it.

These are new controlled comparisons, reported separately from the original endpoint table. The canonical-history variant is distinguished from the original raw-history implementation.

Shared experimental setting. We use Qwen3-4B in ARL-Arena and three paired training seeds. Paired runs start from the same initialization and use the same task order, rollout group size, reference KL, optimizer, and evaluation tasks. Training lasts 200 updates except in compute-limited comparisons. WebShop evaluation uses 1,024 held-out tasks. Training tables report mean ± standard deviation across seeds. Frozen-history diagnostics use task-bootstrap intervals, while training comparisons use intervals over paired seed differences.


[W1] Action scoring and slot control add gains beyond generic constraints

“Please add controlled ablations, particularly a text-level objective with the same bounding and gating rules and a GAUGE variant without slot-level gating.”

Response. Yes. Both action-level scoring and slot control add gains beyond generic update attenuation. (1) On WebShop, full GAUGE exceeds the pure text bounding-and-gating control by 2.4 points, with a paired 95% interval of [1.9, 2.9]. (2) Removing slot energy reduces success by 0.6 point, with an interval of [0.2, 1.0]. (3) The component pattern is consistent across ALFWorld, WebShop and Sokoban.

Experimental setup. The pure text control uses a text-level score and text-ratio drift, retaining GAUGE’s smooth bounding and gating functions. It has no parser-derived action or slot signal. Its gate temperature is chosen on training validation data to match GAUGE’s average attenuation. A second control changes only the score to a text score while retaining the action/slot-derived weights. This isolates quotient scoring from the choice of trust weights.

Removing slot energy sets its coefficient to zero and retains action-level gating. Removing the gate sets it to one and retains bounded ratios. Removing ratio bounding restores the unbounded action ratio and retains the gate. All auxiliary losses are identical.

MethodALFWorld success (%)WebShop success (%)Sokoban success (%)
GSPO78.1 ± 1.172.4 ± 0.882.2 ± 1.0
Text score + text-based bounding and gating78.8 ± 1.073.1 ± 0.483.0 ± 1.1
Text score + unchanged GAUGE action/slot weights79.2 ± 0.973.5 ± 0.683.5 ± 1.0
GAUGE without slot energy80.3 ± 0.874.9 ± 0.885.1 ± 0.9
GAUGE without the trust gate80.1 ± 0.974.8 ± 0.885.0 ± 0.9
GAUGE without ratio bounding80.4 ± 0.975.0 ± 0.785.2 ± 1.0
Full GAUGE80.7 ± 0.875.5 ± 0.685.7 ± 0.9

Results and analysis. Full GAUGE improves WebShop success by 2.4 points over the pure text control, with a paired-seed 95% interval of [1.9, 2.9]. Against the control retaining GAUGE weights but using a text score, the gain is 2.0 points, with an interval of [1.8, 2.2]. Removing slot energy lowers success by 0.6 points, with an interval of [0.2, 1.0]. These are improvements beyond those obtained by generic attenuation alone.

These requested controls appear in the main text at Section 4, Question 2, Table 3 (p. 10); removal definitions and complete comparisons are in Appendix B.1, Table 10 (p. 20).

The results also separate the roles of the components. Bounding and gating improve the text baseline, so some stabilization is generic. Executable-action scoring provides a further improvement, and slot information adds a smaller but consistent benefit on these structured-action environments. We do not assume that these effects add linearly or that a slot gate is useful when no reliable semantic slot exists.

(1) Gradient noise falls after accounting for update size

Experimental setup. To test whether lower gradient variance merely reflects smaller updates, we additionally evaluate unbounded action IS, bounded-only, and full GAUGE on the same finite support and the same 64 independently sampled rollout batches. All model parameters enter the policy-gradient vectors. The diagnostic isolates the advantage-weighted policy term, excluding identical auxiliary gradients; training retains those auxiliary losses unchanged. The associated learning comparison uses the same three paired WebShop seeds and 200 updates.

AuthorsOpenReview ↗

Rebuttal for W1 by Authors (2/2)

Let gmg_m denote a batch gradient under rule mm, and let gˉm\bar g_m be its mean over the same 64 batches. We define

Vm=tr⁡Cov⁡^(gm)∥gˉm∥2,Rm=VmVIS.V_m=\frac{\operatorname{tr}\widehat{\operatorname{Cov}}(g_m)} {\|\bar g_m\|^2}, \qquad R_m=\frac{V_m}{V_{\mathrm{IS}}}.

The table reports both raw covariance-trace ratios and RmR_m. Unlike raw variance, VmV_m is unchanged if every gradient is multiplied by the same nonzero constant. Mean-gradient norm ratios relative to IS are 1.000, 0.975, and 0.946 for IS, bounded-only, and full GAUGE. Dividing each raw variance ratio by the square of its mean-norm ratio gives the normalized values below. These are norms of the mean gradients, not means of individual gradient norms.

Update ruleRelative mean-gradient displacementRaw gradient variance / ISNormalized gradient variance / ISSuccess (%)Paired gain vs. IS, 95% interval (points)
Unbounded action IS01.001.00074.4 ± 1.0Reference
Bounded ratio, no gate0.0490.850.89474.8 ± 0.8+0.4 [−0.1, 0.9]
Full GAUGE0.0820.760.84975.5 ± 0.6+1.1 [0.1, 2.1]

Results and analysis. Full GAUGE reduces raw gradient variance by 24% and scale-normalized variance by 15.1%. The latter reduction cannot be explained by uniform gradient shrinkage alone. Its 8.2% displacement is measured relative to same-support IS. The mean-gradient cosine to this reference is 0.998.

The main text reports the 15.1% normalized-noise reduction and 8.2% displacement in Section 4, Question 2 (p. 10); definitions, raw and normalized variance, and paired intervals are in Appendix B.2, Table 13 (pp. 20–21).

The success intervals use the three paired training-seed differences and a two-sided Student-tt interval with 2 degrees of freedom. They do not use the 64 gradient batches or individual evaluation tasks as independent training runs. Full GAUGE improves success by 1.1 points over IS, with a 95% interval of [0.1, 2.1], and by 0.7 point over bounded-only, with an interval of [0.2, 1.2]. Bounded-only exceeds IS by 0.4 point, but its interval [−0.1, 0.9] includes zero, so that particular improvement is not statistically resolved.

Relaxing constraints reduces displacement and raises noise. Removing the gate reduces mean-gradient displacement from 8.2% to 4.9%, while normalized variance rises from 0.849 to 0.894 and success falls from 75.5% to 74.8%. Removing the remaining ratio bound recovers the IS reference, with zero relative displacement and normalized variance of 1.000. The further success change to 74.4% is a point-estimate decrease; the bounded-only versus IS interval crosses zero. The controls therefore show the trade-off directly: a smaller displacement comes with noisier updates.

Experimental setup—individual seed check. We inspect the individual WebShop runs behind the three-seed summaries, keeping the paired initialization, task schedule and 200-update protocol unchanged. This checks whether the mean advantage comes from only one favorable seed.

MethodSeed 1Seed 2Seed 3
GSPO71.772.373.2
Original GAUGE candidate estimator74.375.075.7
Pure text bounding and gating72.773.173.5
GAUGE without slot energy74.175.075.6
Full GAUGE74.975.576.1

Results and analysis. Every paired seed favors full GAUGE over GSPO, so the average advantage is not produced by just one favorable run. Intervals use differences between runs sharing a seed, rather than treating repeated evaluation tasks as independent training runs.

The main-text summary is in Section 4, Question 2, Table 3 (p. 10), with the three-seed endpoint row also in Table 2 (p. 9); the individual WebShop runs are in Appendix B.1, Table 11 (p. 20).


Manuscript changes: The main-text method in Sections 3.2–3.3 (pp. 7–8) now separates slot energy, the trust gate and ratio bounding. Section 4, Question 2, Table 3 (p. 10) adds the requested pure-text and no-slot controls alongside individual removals; Table 2 (p. 9) provides the three-seed GAUGE summary. Appendix B.1–B.2, Tables 10–13 (pp. 20–21) supplies the exact controls, individual seeds and scale-normalized gradient measurements. These additions make the action-level contribution distinguishable from generic attenuation.

AuthorsOpenReview ↗

Rebuttal for W2 by Authors (1/5)

[W2] Candidate expansion improves coverage, with modest learning gains

“Please explain how candidate actions are selected, especially in open-ended tasks such as search and code generation, and evaluate the performance impact of increasing or decreasing the number of candidate actions.”

Response. Missing candidates can hide policy changes. The construction is explicit, and the tested budget comparisons show limited observed learning impact under the studied selection rules. (1) Search and TIR alternatives come from frozen-old-policy responses, with deduplication, parser-based grouping and common-string scoring. (2) In WebShop, increasing the budget from 8 to 64 raises retained mass from 92.8% to 97.0% and success from 74.6% to 75.5%. (3) The TIR and tool-free studies support candidate construction or applicability.

When an environment exposes legal actions, we enumerate them. For search, complete responses from a frozen old policy supply distinct queries; only parser-equivalent strings are grouped, without merging paraphrases by semantic similarity. TIR uses the parser-based construction below, not equality of program execution outputs.

Experimental setup. The WebShop calibration uses 512 fixed histories from 128 tasks, balanced between search and click states. At each history, 512 old-policy proposals construct the expanded pool, and both policies then generate 2,048 fresh evaluation responses. Candidate sets and estimates are frozen before evaluation.

Results and analysis. The original estimator retains 95.1% of the observed action mass, with 4.9% outside support and mean empirical TV of 0.081. Expanded actions and verified renderings retain 97.0%, with 3.0% outside support and TV of 0.045. With expanded actions but canonical renderings, omission is also 3.0% and TV is 0.060. Action and rendering expansion separately improve probability calibration on the audited histories; their learning impact is assessed by the paired training comparisons below.

The main text presents the calibration and matched-learning comparison in Section 4, Question 1, Fig. 4 (pp. 9–10); the full probability table and audit protocol are in Appendix A.1, Table 5 (p. 17).

Experimental setup—frozen gradient comparison. We use 128 prespecified histories with native nonzero advantages, grouped into 16 matched batches of eight histories. The reference scores up to 256 actions and eight verified renderings per action. Advantages and auxiliary settings are held fixed, with no optimizer updates.

Results and analysis. Relative displacement from the larger finite-pool gradient decreases from 0.091 to 0.034 after expansion. Thus the probability improvement is accompanied by closer update agreement with that finite reference. The main text discusses the candidate-budget effect in Section 4, Question 1 (p. 10); the matched backward protocol and reference are specified in Appendix A.2, Table 7 (p. 18).

Experimental setup. We also compare the three estimators directly in training, holding the ratio bound, gate, reference KL, optimizer, initialization, and task order fixed. Each branch runs for 200 updates under three paired seeds and is evaluated on the same 1,024 held-out tasks.

Results and analysis. The original estimator reaches 75.0 ± 0.7% success versus 72.4 ± 0.8% for GSPO, so most of the advantage is already present before expansion. Expansion adds 0.5 point, with a paired-seed 95% interval of [0.3, 0.7]; the original estimator's gain over GSPO is 2.6 points, with an interval of [2.4, 2.8]. Within these matched runs, expansion preserves the original method's advantage and provides a modest additional improvement.

The main text reports this direct estimator-to-learning comparison in Section 4, Question 1, Fig. 4 (pp. 9–10); Appendix A.1, Table 5 (p. 17) gives the three estimator variants and their matched training results.

We then vary the number of retained actions with all other settings fixed:

Distinguishing the two K=16 settings. The original K=16 estimator uses the original candidate-construction rule and canonical renderings. The budget sweep uses the expanded candidate generator with verified natural renderings, including at K=16. Thus, the two K=16 settings have the same action-count cap but can contain different actions and rendering sets. Within the budget sweep, candidate construction and rendering rules are held fixed, and only the action budget changes.

Candidate budget: expanded generator + verified renderingsRetained action massOutside-support frequencyMean empirical TVWebShop success after 200 updates (%)
Fixed 892.8%7.2%0.10574.6 ± 0.8
Fixed 1696.0%4.0%0.06175.1 ± 0.7
Fixed 3296.8%3.2%0.04975.4 ± 0.6
Fixed 6497.0%3.0%0.04575.5 ± 0.6
Adaptive, at most 6496.4%3.6%0.05375.5 ± 0.6
Reviewer_nHpQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that trust control for language agents should be defined over executable actions rather than raw-text tokens. It formalizes a parser-induced decomposition of policy drift into behavioral drift (over executable actions) and within-action gauge drift (over textual renderings), and introduces GAUGE, a parser-aware trust-control method that pools likelihoods across parser-equivalent renderings and measures drift at both the action and semantic-slot levels. GAUGE plugs into GRPO-style training and shows endpoint improvements across ALFWorld, WebShop, Sokoban, and TIR Math, along with diagnostics that empirically reveal the predicted mismatch between textual and executable drift.

Critical strengths
  1. The paper identifies an important mismatch between textual policy updates and changes in the actions actually executed by an environment. Its decomposition of the text-level KL divergence into action-level and conditional realization terms provides a clear mathematical explanation for why text-level updates may not accurately reflect behavioral changes. This is the paper's strongest contribution because it offers a useful conceptual perspective on policy optimization for language agents, beyond a purely empirical modification to an existing optimizer.

  2. The proposed method reports improvements over recent baselines on ALFWorld, WebShop, and Sokoban, suggesting that accounting for executable-action structure may be practically useful in multiple settings.

Critical weaknesses
  1. The theoretical formulation is not sufficiently validated by the finite-support implementation. The paper's central motivation relies on distinguishing executable actions from their textual realizations, but the practical method estimates action-level quantities over a finite candidate set. The paper does not sufficiently establish how well this candidate set covers the relevant executable actions and textual realizations, or how sensitive the method is to missing candidates. Consequently, it remains unclear how closely the implemented objective approximates the theoretical motivation, and whether the reported benefits persist when candidate coverage changes. This is a central concern because the reliability of the proposed approach depends on the correspondence between the theoretical decomposition and its practical estimation.

  2. The experiments do not isolate the source or establish the robustness of the reported gains. The proposed framework combines several mechanisms, including candidate-action scoring, semantic-slot drift estimation, a trust gate, and bounded importance weights, but the evidence does not adequately isolate their individual contributions or establish that the full design is necessary. Moreover, the reported results lack sufficient multi-seed uncertainty analysis to determine whether the improvements are robust, and the additional scoring cost is not characterized well enough to assess the method under comparable computational budgets. These limitations make it difficult to attribute the gains specifically to the proposed action-level formulation rather than to generic update constraints or additional computation.

Questions for authors
  1. How are the finite candidate-action sets constructed, and what evidence demonstrates that they adequately cover the executable actions relevant to the policy updates? Can the authors quantify the sensitivity of GAUGE to candidate-set size or coverage, including cases where relevant actions or textual realizations are omitted?

  2. Can the authors provide controlled ablations that separately remove the action-level scoring, semantic-slot drift mechanism, trust gate, and bounded importance weights? In particular, does GAUGE still outperform the strongest baseline when generic update constraints are retained but the action-level formulation is removed? Are the reported gains consistent across multiple random seeds, with uncertainty estimates?

  3. What additional computation is required by candidate-action scoring and the other proposed mechanisms? Do the performance improvements remain under matched wall-clock time, GPU-hours, or total scoring-computation budgets?

17 comments
AuthorsOpenReview ↗

Rebuttal for W1 by Authors (1/8)

Thank you for recognizing the action-level perspective and KL decomposition. We address the three main concerns below: how well the finite candidate pool represents the policy, which components produce the gains, and whether the gains remain at equal compute.

The new training comparisons use the expanded-candidate implementation and are reported separately from the original single-run results.


Shared setting. Agent-task experiments use Qwen3-4B in ARL-Arena, with the same initialization, task splits, reference policy, rollout group size, optimizer, and evaluation protocol within each comparison. Training comparisons use three paired seeds and 200 updates unless a compute budget determines the stopping point. WebShop evaluation uses 1,024 held-out tasks. Tables report mean ± standard deviation across training seeds. Paired-seed intervals describe training variation; task-bootstrap intervals describe uncertainty in frozen-history diagnostics. These are kept separate.

Correspondence to the review. W1 addresses Critical Weakness 1 and Question 1. W2 addresses the component-attribution and multi-seed concerns in Critical Weakness 2 and Question 2. W3 addresses the computational-cost concern in Critical Weakness 2 and Question 3.


[W1: Weakness 1 / Question 1] The finite-support identity is exact, and expansion improves calibration

“It remains unclear how closely the implemented objective approximates the theoretical motivation, and whether the reported benefits persist when candidate coverage changes.”

Response. The KL identity is exact for a shared, fixed finite pool. Independent calibration measures how well that pool approximates unrestricted generation; gradient and learning comparisons test the practical effect. (1) Independent WebShop calibration quantifies the discrepancy: mean empirical TV changes from 0.081 to 0.045 after expansion. (2) Matched training retains the original method's advantage, while ordinary budget sweeps show limited observed learning impact. (3) At fixed K=32, omitting a high-probability alternative changes probabilities and gradients more than omitting a low-probability alternative.

(1) Expanding actions and renderings improves independent calibration

Legal actions are enumerated when an environment exposes them. For open-ended actions, complete responses from the frozen old policy supply alternative parsed actions. The sampled rollout action is retained for the training update. Renderings are pooled only when the production parser extracts the same executable action. The old and current models score the same selected strings. Rewards and evaluation outputs are never used to construct candidates.

Experimental setup. The WebShop audit uses 512 histories from 128 tasks, equally divided between search and click states. At each history, 512 old-policy proposal responses construct the expanded pool. Candidate sets and estimates are frozen before 2,048 independent evaluation responses are generated from each checkpoint. The main pair is step 120/130; steps 80/90 and 90/190 test earlier and larger-drift comparisons. Outside actions and parser failures remain in the denominator.

Log-ratio errors use the same 432 histories with at least 20 target observations under each policy. Targets are fixed from construction data. The remaining histories stay in coverage and TV statistics; they are not silently removed from the audit.

We also compare the three estimators directly in training, holding the ratio bound, gate, reference KL, optimizer, initialization, and task order fixed. Each branch runs for 200 updates under three paired seeds and is evaluated on the same 1,024 held-out tasks.

EstimatorRetained action massOutside-support frequencyMean empirical TVMean absolute action log-ratio errorWebShop success (%)
Original GAUGE candidate estimator95.1%4.9%0.0810.11275.0 ± 0.7
Expanded actions, canonical renderings97.0%3.0%0.0600.08575.2 ± 0.6
Same expanded actions, verified renderings97.0%3.0%0.0450.05875.5 ± 0.6

Results and analysis. TV is computed against independent parsed-output frequencies, including a residual category for outside actions. The original estimator already retains 95.1% of observed action mass. Expansion improves mean empirical TV from 0.081 to 0.045, a paired reduction of 0.036 with a task-bootstrap 95% interval of [0.030, 0.042]. This is an absolute improvement of 0.036; its effect on training is assessed separately rather than inferred from TV alone. The second and third rows share their action set, so the further improvement isolates rendering coverage. Probability columns use step 130; the success column reports a separate matched three-seed, 200-update comparison under the shared training protocol.

AuthorsOpenReview ↗

Rebuttal for W1 by Authors (2/8)

The main text presents the calibration and matched-learning comparison in Section 4, Question 1, Fig. 4 (pp. 9–10); the full probability table and audit protocol are in Appendix A.1, Table 5 (p. 17).

The original estimator reaches 75.0 ± 0.7% success versus 72.4 ± 0.8% for GSPO, so most of the advantage is already present before expansion. Expansion adds 0.5 point, with a paired-seed 95% interval of [0.3, 0.7]; the original estimator's gain over GSPO is 2.6 points, with an interval of [2.4, 2.8]. Within these matched runs, expansion preserves the original method's advantage and provides a modest additional improvement.

The main text reports this direct estimator-to-learning comparison in Section 4, Question 1, Fig. 4 (pp. 9–10); Appendix A.1, Table 5 (p. 17) gives the three estimator variants and their matched training results.

The expanded estimator also improves calibration on the replication pairs: mean TV changes from 0.073 to 0.039 for steps 80/90 and from 0.096 to 0.061 for steps 90/190. The larger-drift pair is less accurate, but none of these comparisons is presented as a failure of the original training method.

The replication pairs are introduced in Section 4, Question 1 (p. 9); their individual calibration values are in Appendix A.1, Table 6 (p. 17).


(2) Larger pools improve gradient agreement, with modest learning gains

Experimental setup. We construct nested candidate sets using old-policy rankings, then vary the action budget while retaining the same rendering rule. The sampled target is retained in every gradient and training comparison. Frozen gradient checks use native nonzero advantages on 128 prespecified histories, grouped into 16 matched eight-history batches. Their reference is a larger finite pool of up to 256 actions and eight renderings per action; it is not labeled an exact full-policy gradient. Zero-advantage groups remain in separate rollout coverage statistics.

The learning study changes only the candidate rule and runs each setting for 200 updates under the same three paired seeds.

Distinguishing the two K=16 settings. The original K=16 estimator uses the original candidate-construction rule and canonical renderings. The budget sweep uses the expanded candidate generator with verified natural renderings, including at K=16. Thus, the two K=16 settings have the same action-count cap but can contain different actions and rendering sets. Within the budget sweep, candidate construction and rendering rules are held fixed, and only the action budget changes.

Candidate budget: expanded generator + verified renderingsMean empirical TVMean gradient cosine to larger poolRelative gradient displacementWebShop success (%)
Fixed 80.1050.99400.12174.6 ± 0.8
Fixed 160.0610.99800.07575.1 ± 0.7
Fixed 320.0490.99920.04475.4 ± 0.6
Fixed 640.0450.99950.03475.5 ± 0.6
Adaptive, at most 640.0530.99900.05075.5 ± 0.6

Results and analysis. Reducing the budget to 8 gives a modest performance loss: 74.6% versus 75.5% at 64 candidates, a difference of 0.9 point. The original 16-action estimator also retains the main benefit over GSPO, as the direct estimator comparison shows. Fixed 32 and fixed 64 differ by 0.1 point, with a paired-seed 95% interval of [−0.6, 0.8]. Relative gradient displacement is reported along with cosine to expose magnitude changes; on the same 16 batches, displacement is 0.091 for the original estimator and 0.034 for the expanded 64-action estimator. The evidence supports limited budget sensitivity in the tested range, rather than complete insensitivity or exact full-policy gradient recovery.

The main text discusses the budget-dependent learning effect in Section 4, Question 1 (p. 10); full coverage, gradient and learning results are in Appendix A.2, Tables 7–8 (p. 18).

The adaptive setting starts at 16 actions and expands to 32 or 64 if an independent monitoring sample has more than 5% outside-support outcomes. It uses 23.6 actions on average. Final evaluation samples remain separate from monitoring, and unresolved low-coverage histories are retained in reporting.

AuthorsOpenReview ↗

Rebuttal for W1 by Authors (3/8)

(3) Missing high-probability actions causes larger probability and gradient errors

Experimental setup. A budget sweep does not isolate which alternatives are missing. We therefore add a separate frozen diagnostic on 128 eligible search histories from 64 tasks in the main cohort. Eligibility requires at least 33 distinct parseable actions in the preconstructed expanded pool, capped at 64, and a recorded nonzero native advantage. Selection uses construction data and recorded advantages before held-out outputs are examined. This conditional stress cohort is reported separately from the balanced main cohort.

For each history, we retain a baseline pool of 32 actions, including the sampled target. Within that baseline pool, we identify the highest- and lowest-probability non-target actions using the frozen old-policy scores on the expanded pool. A single reserve action, ranked highest among those outside the baseline pool, is fixed in advance. The low-probability omission arm replaces the lowest-ranked non-target action with that reserve; the high-probability omission arm replaces the highest-ranked non-target action with the same reserve. Every arm therefore has exactly 32 distinct actions and retains the sampled target. The other actions and their verified renderings remain unchanged. “High” and “low” refer to old-policy candidate scores, not correctness labels.

All strings in the common expanded pool are scored once under both frozen models; the variants apply different fixed masks to those same scores. No new generation or optimizer steps are taken for the perturbations. Coverage and TV use the same independent step-130 evaluation outputs. The gradient comparison uses 16 matched eight-history batches with identical advantages and auxiliary settings. Its reference is the unmodified 32-action pool on this stress cohort, rather than the larger finite pool used in the preceding table.

Pool perturbation, all K=32Mean retained action massMean empirical TVRelative gradient displacement vs. unmodified K=32
Unmodified pool96.6%0.0520
Omit one low-probability alternative96.3%0.0560.014
Omit one high-probability alternative92.1%0.1010.112

Results and analysis. TV increases by 0.004 after low-probability omission and by 0.049 after high-probability omission. The paired excess increase for high versus low omission is 0.045, with a task-bootstrap 95% interval of [0.034, 0.056]. Thus equal candidate counts do not imply equal approximation quality: losing a high-probability alternative changes both calibration and the update more strongly. This supports monitoring retained mass and candidate composition. It is not evidence that final training performance is robust to deliberate high-probability omissions. The sampled target is protected, so unsupported-target cases are outside this particular diagnostic.

This fixed-size omission diagnostic is summarized in Section 4, Question 1 (p. 10); cohort selection, replacement rules and the full table are in Appendix A.3, Table 9 (pp. 18–19).

Reviewer_o5kgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GAUGE, a policy optimization method for language agents. The main idea is to measure policy changes over executable actions rather than only generated text. Different textual expressions can lead to the same action, while small changes in action arguments can lead to different outcomes. GAUGE groups parser-equivalent expressions and uses action-level and slot-level probability changes to control policy updates. The authors provide a theoretical decomposition of textual policy drift and evaluate the method on four agent tasks using Qwen3-4B.

Critical strengths
  1. The motivation is clear. The difference between generated text and executable actions is relevant to policy optimization for language agents.
  2. The theoretical analysis helps explain why text-based trust control may over-penalize formatting changes or miss changes in important action arguments.
  3. Experiments across four agent tasks show promising results compared to existing policy optimization methods.
Critical weaknesses
  1. My main concern is the bias introduced by the candidate set and the modified update weights. Although the authors mention this issue, it is unclear whether the bias is large enough to affect training. Can the authors provide more experiments to study this and explain how to reduce the bias?
  2. It is unclear how the candidate set is designed, especially for TIR Math. How are alternative tool calls or programs generated? For non-tool-use tasks, such as general math reasoning, defining these candidates seems even harder. I think this may limit the generality of the method.
Questions for authors

See weakness

14 comments
AuthorsOpenReview ↗

Rebuttal for W1 by Authors (3/3)

(3) Omitted mass and within-pool error bound the probability error

Probability-error guarantee. Let pp denote the unrestricted action distribution, CC the candidate set, and qq the estimator supported on CC. Write ε=p(Cc)\varepsilon=p(C^c) for omitted action mass and η=TV(p(⋅∣C),q)\eta=\mathrm{TV}(p(\cdot\mid C),q) for error within the retained actions. Then

TV(p,q)≤ε+(1−ε)η\mathrm{TV}(p,q) \leq \varepsilon + (1-\varepsilon)\eta

This bound explains why action coverage alone is insufficient: even with small ε\varepsilon, incomplete rendering estimates can leave η\eta large. For a reward in [0,1][0,1], the same expression bounds the error in expected reward at that fixed decision. Gradients and learning are evaluated separately.

Proof idea. At a fixed history, freeze the candidate set and suppose it has positive probability under pp. Split the TV sum into candidates and noncandidates. The sum of absolute probability differences outside the set is the missing mass because qq is zero there. Inside the set, write pp as its retained mass times its conditional distribution and apply the triangle inequality. Dividing the total absolute error by two gives the TV bound against the full action law, not only against a larger finite probe. The main text states the result in Section 3.3, Eq. (25) (p. 8); the proof and fixed-decision reward implication are in Appendix F.2, Proposition 11 (p. 26).

We also add an independent calibration certificate. Because qq is zero outside CC, aggregating every outside action into one residual category preserves TV exactly. With KK frozen candidates, nn independent evaluation responses, and failure probability α\alpha,

TV(p,q)≤min⁡{1,D^n+(K+1)log⁡2+log⁡(1/α)2n}\mathrm{TV}(p,q) \leq \min\{1,\widehat{D}_n + \sqrt{\frac{(K+1)\log 2 + \log(1/\alpha)}{2n}}\}

where D^n\widehat{D}_n is empirical TV on candidates plus the residual category. The certificate includes missing actions through the residual category; the table reports empirical TV.

Calibration guarantee and proof idea. Conditional on the frozen construction, the evaluation counts are independent draws on the candidates plus one residual category. Combining the exact residual identity with a finite-alphabet concentration bound and the triangle inequality gives the displayed upper bound with probability at least 1−α1-\alpha for that fixed history-policy pair. All outside actions remain represented in the residual category. Thus the certificate does not assume that a finite probe is the whole policy, and the empirical TV entries remain distinct from the confidence bound. The main text points to this audit in Section 4, Question 1 (pp. 9–10); the identity and certificate are proved in Appendix F.3, Proposition 12, Eqs. (35)–(36) (p. 27).


Manuscript changes: In the main text, Section 3.1 (p. 6) now defines the support-conditioned estimator and frozen construction procedure. Section 4, Question 1, Fig. 4 (pp. 9–10) separates probability calibration from the effect on learning, and Eq. (25) (p. 8) states the omitted-mass bound. Appendix A.1–A.2, Tables 5–8 (pp. 17–18) supplies the full audit, gradient and budget comparisons; Appendix F.2–F.3 (pp. 26–27) proves the full-law error bound and independent calibration certificate.

AuthorsOpenReview ↗

Rebuttal for W2 by Authors (1/3)

[W2] Bounding and gating reduce gradient noise while changing the update

“Can the authors provide more experiments to study this and explain how to reduce the bias?”

Response. Bounding and gating deliberately change the same-support IS update. Relaxing these constraints reduces the displacement, but increases gradient noise. (1) Full GAUGE has 8.2% mean-gradient displacement relative to same-support IS. (2) Scale-normalized gradient variance is 15.1% lower, and WebShop success is 1.1 points higher, with a paired 95% interval of [0.1, 2.1]. (3) Removing the gate reduces displacement to 4.9%; removing the remaining ratio bound recovers the IS reference.

This section isolates weight-induced displacement; candidate approximation is evaluated separately in W1.

Experimental setup. We hold histories, candidate sets, rendering likelihoods, and advantages fixed. Each rule is evaluated on the same 64 independently sampled rollout batches at a fixed policy pair. The gradient diagnostic includes all model parameters but isolates the advantage-weighted policy term; identical auxiliary gradients are excluded from this diagnostic and retained unchanged in training. We compute the trace of the empirical covariance across batches. Training uses the same three paired WebShop seeds and 200-update protocol described above.

Bounding and gating make the weights less concentrated.

Update ruleMean weightWeight varianceESS/N
Unbounded action IS1.040.3476.1%
Bounded ratio, no gate1.010.1984.3%
Full GAUGE0.970.1387.9%

Results and analysis. Bounding alone reduces weight variance from 0.34 to 0.19; the full rule reduces it to 0.13, while ESS/N rises from 76.1% to 87.9%. The sample weights are less concentrated. We next test actual gradient noise, because weight dispersion alone does not establish lower policy-gradient variance.

The main text explains the weight-induced update trade-off in Section 4, Question 2 (p. 10); the weight-concentration statistics themselves are reported in Appendix B.2, Table 12 (p. 21).

Here ESS is (∑iwi)2/∑iwi2(\sum_i w_i)^2/\sum_i w_i^2. These pooled weight statistics describe weight concentration; they do not substitute for a gradient-variance measurement.

AuthorsOpenReview ↗

Rebuttal for W2 by Authors (2/3)

GAUGE lowers normalized gradient noise and improves success.

Let gmg_m denote a batch gradient under rule mm, and let gˉm\bar g_m be its mean over the same 64 batches. We define

{\|\bar g_m\|^2}, \qquad R_m=\frac{V_m}{V_{\mathrm{IS}}}.$$ The table reports both raw covariance-trace ratios and $R_m$. Unlike raw variance, $V_m$ is unchanged if every gradient is multiplied by the same nonzero constant. Mean-gradient norm ratios relative to IS are 1.000, 0.975, and 0.946 for IS, bounded-only, and full GAUGE. Dividing each raw variance ratio by the square of its mean-norm ratio gives the normalized values below. These are norms of the mean gradients, not means of individual gradient norms. | Update rule | Relative mean-gradient displacement | Raw gradient variance / IS | Normalized gradient variance / IS | Success (%) | Paired gain vs. IS, 95% interval (points) | |---|---:|---:|---:|---:|---:| | Unbounded action IS | 0 | 1.00 | 1.000 | 74.4 ± 1.0 | Reference | | Bounded ratio, no gate | 0.049 | 0.85 | 0.894 | 74.8 ± 0.8 | +0.4 [−0.1, 0.9] | | Full GAUGE | 0.082 | **0.76** | **0.849** | **75.5 ± 0.6** | **+1.1 [0.1, 2.1]** | **Results and analysis.** Full GAUGE reduces raw gradient variance by 24% and scale-normalized variance by **15.1%**. The latter reduction cannot be explained by uniform gradient shrinkage alone. Its 8.2% displacement is measured relative to same-support IS. The mean-gradient cosine to this reference is 0.998. The main text reports the 15.1% normalized-noise reduction and 8.2% displacement in `Section 4, Question 2 (p. 10)`; definitions, raw and normalized variance, and paired intervals are in `Appendix B.2, Table 13 (pp. 20–21)`. The success intervals use the three paired training-seed differences and a two-sided Student-$t$ interval with 2 degrees of freedom. They do not use the 64 gradient batches or individual evaluation tasks as independent training runs. Full GAUGE improves success by 1.1 points over IS, with a 95% interval of [0.1, 2.1], and by 0.7 point over bounded-only, with an interval of [0.2, 1.2]. Bounded-only exceeds IS by 0.4 point, but its interval [−0.1, 0.9] includes zero, so that particular improvement is not statistically resolved. **Relaxing constraints reduces displacement and raises noise.** Removing the gate reduces mean-gradient displacement from 8.2% to 4.9%, while normalized variance rises from 0.849 to 0.894 and success falls from 75.5% to 74.8%. Removing the remaining ratio bound recovers the IS reference, with zero relative displacement and normalized variance of 1.000. The further success change to 74.4% is a point-estimate decrease; the bounded-only versus IS interval crosses zero. The controls therefore show the trade-off directly: a smaller displacement comes with noisier updates. On a fixed sampling distribution, let $\rho$ be the unbounded ratio, $w$ the modified weight, $A$ the advantage, and $s=\nabla\log q(a\mid h)$ the score. Then $$\left\|\mathbb E[wAs]-\mathbb E[\rho As]\right\| \leq \mathbb E\left[|w-\rho|\,|A|\,\|s\|\right].$$ If advantage magnitudes and score norms are bounded by $A_{\max}$ and $G$, this is at most $A_{\max}G\mathbb E|w-\rho|$. This inequality relates weight differences to update displacement on the declared support. The reported displacement comes from the gradient comparison. **Update-displacement guarantee.** The two updates use the same sampling law, action support and score vector. Subtracting them leaves the expectation of $(w-\rho)As$; the norm-of-expectation inequality and triangle inequality give the displayed bound. Bounded advantages and scores give its simpler form. Therefore the expected update approaches same-support IS whenever the weighted difference vanishes in this integrable sense. This is a bound on the intentional change of update weights, separate from W1’s probability approximation. It is stated in the main text at `Eq. (26) (p. 8)` and proved in `Appendix F.4, Proposition 13 (p. 27)`. Increasing action and rendering coverage addresses candidate approximation, a separate issue. On a fixed finite support, enlarging the ratio range and weakening the gate approaches unbounded IS. The table tests component removals; continuous adjustment of the range or gate temperature is an analytical limiting argument here, not an additional parameter-sweep result. The learning comparisons guide the choice of constraints rather than assuming that minimum displacement always gives the best performance.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close soon, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.