acceptodds
Under review as a conference paper at ICLR 2027

Pairwise Workflow Policy Optimization for Tool-Using Language Agents

Abstract

Tool-using language agents can automate complex tasks, but reinforcement learning treats different execution orders as distinct experiences even when they produce identical outcomes, introducing noise unrelated to task quality. We introduce Pairwise Workflow Policy Optimization (PWPO), which verifies that swapping two adjacent tool calls preserves their outputs and resulting state, then optimizes the summed probabilities of both complete execution histories. In deterministic environments with restorable states, this aggregation preserves the expected on-policy gradient and removes variation within verified pairs; our analysis specifies when this reduction offsets additional computation. Across AppWorld-Atomic, ALFWorld, and WebShop under matched rollout budgets, PWPO achieves the highest means among evaluated methods on all 16 main metrics, improving AppWorld-Atomic task completion over G2PO by 2.32–2.64 percentage points. These results support learning from verified workflow alternatives while retaining the agent's sequential deployment interface.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, tool-using agents, reinforcement learning, policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_avbbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce Pairwise Workflow Policy Optimization (PWPO) in this paper. The goal of this method is to improve training stability. For a rollout, certified reordering of two adjacent tool calls is obtained. A reordering is certified only if all the interaction outcomes are the before and after the swap. For these swaps, a total probability is calculated using the current policy and a frozen policy. These together form the importance sampling ratio for the standard LOO advantage. The authors further establish that the PWPO update has lower variance and that PWPO can sometimes be computationally better than an ordinary estimator.

Experiments on instruct based models on 3 agentic datasets show that a model trained with PWPO performs better than techniques like G2PO and Seq-LOO. Further analysis shows that PWPO improves task completion and reduces variance. The authors also show that the variance shifts towards deeper layers in the network

Critical strengths

Improving training stability by reducing variance is an important task especially when post-training of large models is involved. Using a simple property like swapping tool calls can be applied to a large variety of problems. The authors establish that equivalent outcomes can result in different learning signals and can introduce noise during training.

Although, the budget is sufficiently large, the authors prove that PWPO can be computationally better than ordinary estimators. They also show experimental results where it can be seen that training with PWPO results in lower variance. At the end, they also show an interesting result where the variance is pushed towards deeper layers of the network.

Critical weaknesses

The experiments use the dated Qwen2.5 family instruct models on fairly dated datasets. Even if newer models were used, a big part that is lacking is not using thinking models. Most of the community has moved towards thinking models as they are more practical, especially in agentic tasks. PWPO in its current formulation may not work for reasoning models as the thinking traces might not make sense causally after swapping. Moreover, longer thinking traces will increase rescoring and differentiation costs and might also decrease statistical gains obtained by PWPO at the same time.

The paper presentation is lacking. They use a lot of non standard terms like 'transcripts' and 'cells'. While these terms are defined, they make the paper very difficult to follow. It also seems that notation is used excessively and sometimes in places where plain language would explain things much better (for example, in 'Certifying an adjacent swap' in Section 4.1). It would be much better if the authors dedicated more space towards 4.3 and 4.4 rather than defining their method, which can be summarized as a new importance sampling method that takes into account swappable tool calls.

Questions for authors

Have the authors considered using PWPO with reasoning models? If so, how will PWPO work with the thinking traces in the trajectories generated by the models and what are the changes if any?

5 comments
AuthorsOpenReview ↗

Response to Reviewer avbb (1/2)

Thank you for taking the time to review our work and for your helpful comments on reasoning models and presentation. Below, we explain the main idea and present the thinking-model results in response to your questions.

W1. Older instruct models, reasoning-trace compatibility, and rescoring costs.

Response.{\color{green}\mathbf{Response.}} To address your reasoning-model concern, we tested PWPO on Qwen3.5-9B in thinking mode with the same AppWorld-Atomic interface. The original comparisons use a shared Qwen2.5 backbone across agent-RL baselines. The additional experiment extends the evaluation to a thinking policy. Across the main and supplementary experiments, the paper covers four environments, including ScienceWorld.

BudgetMethodGPU-hoursCalls (k)TGC Test-NTGC Test-C
170k callsG2PO382169.732.0 ± 1.126.2 ± 1.0
170k callsSeq-LOO367169.830.7 ± 1.225.0 ± 1.0
170k callsFrozen-weight Pair305169.831.3 ± 1.125.6 ± 1.0
170k callsPWPO304169.931.5 ± 1.125.8 ± 1.0
382 GPU-hoursG2PO382169.732.0 ± 1.126.2 ± 1.0
382 GPU-hoursFrozen-weight Pair382213.532.2 ± 1.026.5 ± 0.9
382 GPU-hoursPWPO382213.932.8 ± 1.027.0 ± 0.9
PWPO−G2POTest-N TGC differenceTest-C TGC difference
170k calls−0.5 [−2.0, 1.0]−0.4 [−1.8, 1.0]
382 GPU-hours+0.8 [−0.7, 2.3]+0.8 [−0.6, 2.2]

These runs demonstrate certified-pair training with a thinking policy. PWPO reaches 32.8/27.0 Test-N/Test-C TGC at 382 GPU-hours. The paired completion intervals span zero.

PWPO thinking-trace diagnosticValue
Thinking tokens per rollout, mean / P90641 / 1,185
Truncation rate1.8%

The fixed-checkpoint comparison measures how thinking affects coverage, variance and cost, using 256 blocks per mode:

ModeAccepted coverageMedian 4w0w14w_0w_1Full-gradient variance ratioComplete cost ratioCost–variance ratio
Thinking off39.0%0.420.79 [0.74, 0.85]1.16 [1.12, 1.20]0.92 [0.84, 1.00]
Thinking on31.5%0.280.84 [0.78, 0.90]1.24 [1.19, 1.29]1.04 [0.94, 1.16]

Each mode is normalized to its own Seq-LOO. Thinking retains 16% variance reduction. Section 4.4 requires more than 0.24/1.24≈19.40.24/1.24\approx19.4\\% reduction to offset 24% extra cost, explaining the measured cost–variance ratio of 1.04. This connects the observed rescoring cost to the theoretical budget criterion. Q1 below explains how the thinking traces are handled.


W2. Terminology, intuition, and the contribution beyond an importance ratio.

Response.{\color{green}\mathbf{Response.}} You can read a transcript as the complete interaction trajectory, and a cell as one trajectory or a certified equivalent pair, together with the sampled swap boundary.

The central insight is that equivalent execution orders can produce different learning signals. Two independent calls AA and BB can lead to the same outcome in either order. Their histories, A!→!BA\\!\rightarrow\\!B and B!→!AB\\!\rightarrow\\!A, still have different policy probabilities and gradients. PWPO learns their combined probability and averages out the variation caused by choosing between these equivalent orders. The importance ratio follows from this choice of training event.

Section 4.3 expresses this directly:

=\mathbb E\\!\left[\nabla_\theta\log p_\theta(\tau)\mid C,j,x\right].$$ Certification establishes a shared reward for the two histories. The boundary label gives a valid partition even when candidate swaps overlap. Full-history rescoring supplies the probability weights for their gradients. Conditional marginalization provides the statistical foundation [1]. PWPO makes it usable for tool-using language policies by constructing verifiable, reward-preserving trajectory events. Section 4.4 asks when the variance reduction offsets the added cost. Figure 5 shows where removable order sensitivity is concentrated as a workflow progresses. Its horizontal axis is position within the trajectory, with model parameters held fixed throughout the analysis. [1] Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications.
AuthorsOpenReview ↗

Response to Reviewer avbb (2/2)

Q1. How does PWPO handle thinking traces, and what changes are required?

Response.{\color{green}\mathbf{Response.}} PWPO handles thinking traces by treating thinking text + one atomic call + its observation as an indivisible block. The original thinking tokens and IDs are preserved. Exchanges with argument, reference or termination dependencies are rejected. Both accepted histories therefore retain the original text and pass the execution checks.

All generated tokens contribute to the likelihood, including thinking, submission and incomplete final prefixes. Each reordered history is rendered through the actual chat template, and its suffix is rescored. Qwen3.5's template uses the final output from earlier responses as conversation history [1]. Thinking tokens still contribute their probabilities at the time they are generated.

Proposition 1 applies to these extended records under fixed support, symmetric certification and reward preservation. The certificate checks executable dependencies and the shared outcome. Policy probabilities determine the weights of the two complete histories. A reordered trace with lower policy probability receives a smaller conditional weight, limiting its contribution to the averaged gradient.

The measured tradeoff is:

Mode, same frozen backboneBlocks / rollouts per blockCoverageMedian 4w0w14w_0w_1Full-gradient variance ratioComplete cost ratioCost–variance ratio
Thinking off256 / 839.0%0.420.79 [0.74, 0.85]1.16 [1.12, 1.20]0.92 [0.84, 1.00]
Thinking on256 / 831.5%0.280.84 [0.78, 0.90]1.24 [1.19, 1.29]1.04 [0.94, 1.16]

With thinking enabled, 31.5% of rollouts admit a certified pair and full-gradient variance falls by 16%. The experiment shows that the averaging mechanism remains effective with thinking traces. The cost measurements include rescoring and differentiating the longer histories.

[1] Qwen3.5-9B Model Card.

AuthorsOpenReview ↗

Follow-up on your concerns

Thank you again for highlighting the questions about reasoning models and presentation. Our replies include Qwen3.5-9B thinking-mode experiments showing 16% lower full-gradient variance, explain how thinking traces are preserved and rescored, and quantify the resulting cost tradeoff.

We would welcome your feedback and would be happy to discuss any remaining questions or provide further clarification and supporting evidence while the discussion is open. If these results and explanations address your concerns, we would be grateful if you could reconsider your rating.

Reviewer_Bh7HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PWPO reduces execution-order variation by aggregating the probabilities of a sampled tool-use trajectory and a certified adjacent-call reordering. Certification checks outputs, endpoint state, and reward/termination consistency. The method constructs boundary-labeled cells, rescores complete histories, and optimizes a clipped cell-level objective. The paper gives on-policy variance and cost analyses and evaluates several agent benchmarks.

Critical strengths
  1. The technical increment is the construction of usable training information from certified tool-execution equivalence. Boundary-labeled cells, symmetric certification, singleton fallback, and full reordered-history suffix scoring address concrete obstacles to preserving the learning target. The marginalization principle is established, but correctly operationalizing it for history-sensitive tool policies has value.
  2. The evidence goes beyond improvement over an untrained model: strong RL baselines, probability-weighting controls, certification audits, and an AppWorld-Atomic/7B compute-matched gain of 2.19/2.16 points over G2PO support usefulness of the certified-pair training framework in its stated setting.
Critical weaknesses
  1. The contribution should be separated from unproven refinements. PWPO’s 0.48–0.99-point advantage over Frozen-weight Pair has four paired confidence intervals containing zero, and their reported on-policy variance diagnostics match. The evidence supports certified probability-weighted pairing, but does not independently establish the necessity of exact cell-mass clipping.
  2. The practical scope is restricted by deterministic complete-state restoration and additional interaction costs. GPU matching increases tool calls from 169.6k to 213.9k and is demonstrated in only one setting. The theorem concerns on-policy gradient estimation rather than general clipped-update convergence; snapshot/certification and scoring artifacts are therefore important for assessing reuse.
Questions for authors
  1. Should the main empirical conclusion be narrowed to probability-weighted certified pairing, or is there evidence that distinguishes exact cell-level clipping from Frozen-weight Pair beyond the current intervals?
  2. How does the method perform under matched total tool-interaction budgets, and can the authors release the snapshot, certification, and suffix-rescoring code with independently checkable audit cases?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Bh7H (1/2)

We appreciate the time you devoted to our paper and your constructive comments on both the theory and experiments. Below, we address your questions about certified pairing, clipping and resource costs.

W1. Probability-weighted pairing and the role of cell clipping.

Response.{\color{green}\mathbf{Response.}} The central empirical result is the benefit of certified probability-weighted pairing. The variance guarantee follows from this averaging step. Frozen-weight Pair isolates the additional effect of cell-level clipping.

Table 8 gives the following paired TGC differences, with 95% CIs in brackets:

PWPO minus comparatorTest-NTest-C
G2PO+2.58 [1.31, 3.84]+2.64 [1.72, 3.57]
Seq-LOO+3.37 [2.05, 4.70]+3.60 [2.54, 4.65]
Uniform-Pair+1.78 [0.55, 3.02]+1.44 [0.49, 2.39]
Frozen-weight Pair+0.99 [−0.06, 2.04]+0.72 [−0.12, 1.55]

Both weighted estimators achieve the same full-gradient variance reduction:

EstimatorFull-gradient variance ratioComplete cost ratioCost–variance ratio
Frozen-weight Pair0.79 [0.74, 0.85]1.16 [1.13, 1.19]0.92 [0.84, 0.99]
PWPO0.79 [0.74, 0.85]1.16 [1.13, 1.19]0.92 [0.84, 0.99]

Their equality at θold\theta_{\rm old} follows from the conditional-score identity. Both estimators remove the same within-pair execution-order variance. Table 8 shows positive paired intervals against Seq-LOO and Uniform-Pair, identifying the benefit of certified probability-weighted pairing.


W2. Deterministic restoration, interaction overhead, and the scope of the theorem.

Response.{\color{green}\mathbf{Response.}} Theorem 1 addresses the resource tradeoff you raise. For relative overhead κ\kappa and variance reduction Δ\Delta, the large-budget benefit requires

ΔVSeq>κ1+κ.\frac{\Delta}{V_{\rm Seq}}>\frac{\kappa}{1+\kappa}.

The theorem quantifies the precision of on-policy gradient estimation at a fixed checkpoint. The training comparisons below assess the clipped optimizer in practice. Section 4.1 specifies complete deterministic restoration for certification. Deployment uses the original sequential policy.

The requested interaction-budget test counts rollout, failed certification attempts, both replay orders, and training rechecks:

Constraint, AppWorld-Atomic/7BMethodTotal calls (k)GPU-hoursTGC Test-N / Test-C
Original GPU-matched, Table 7(a)G2PO169.6383.124.40 / 20.14
Original GPU-matched, Table 7(a)PWPO213.9381.726.59 / 22.30
New 170k-call capG2PO169.638324.40 / 20.14
New 170k-call capPWPO169.930424.20 / 19.95
New 170k-call comparisonTest-N TGC differenceTest-C TGC difference
PWPO−G2PO, 95% CI−0.20 [−1.4, 1.0]−0.19 [−1.3, 0.9]

Under the original GPU budget, PWPO improves Test-N/Test-C TGC by 2.19/2.16 points. Under the total-call cap, the paired differences are close to zero and their intervals span zero. These comparisons show how the practical gain depends on the resource budget.

Independent native-state rechecks also test the restoration contract:

EnvironmentAccepted pairs checkedOutput/state/suffix violationsInjected negatives detected
AppWorld-Atomic1000/100 in each check24/24
ALFWorld1000/100 in each check24/24
WebShop1000/100 in each check24/24
ScienceWorld1000/100 in each check24/24

The audits support the restoration contract in all four tested adapters. This contract also gives a concrete criterion for applying PWPO to another environment.

AuthorsOpenReview ↗

Response to Reviewer Bh7H (2/2)

Q1. Is there evidence distinguishing cell-level clipping from Frozen-weight Pair?

Response.{\color{green}\mathbf{Response.}} The additional diagnostics address your question about clipping behavior. For fixed cells and advantages, the unclipped objectives coincide for every θ\theta. The two rules apply clipping before or after probability-weighted aggregation. Their observed branch disagreements are:

Diagnostic windowClipping-branch disagreement
θ=θold\theta=\theta_{\rm old}0.0%
Iterations 10–204.8%
Iterations 70–8010.2%
Iterations 130–14013.1%

These percentages use nonzero-advantage two-member cells as the denominator. Each rollout batch makes two AdamW steps, one per minibatch. The second step evaluates likelihoods after the first parameter update.

The corresponding task-completion differences are:

PWPO−Frozen-weight PairTest-N TGCTest-C TGC
Original rollout-matched, Table 8+0.99 [−0.06, 2.04]+0.72 [−0.12, 1.55]
New 170k-call cap+0.25 [−0.8, 1.3]+0.20 [−0.7, 1.1]

The paired intervals include zero in both budget settings. Table 8 supports the performance benefit of certified probability-weighted pairing, and the branch statistics show how clipping changes the optimization rule.


Q2. Matched total interactions and independently checkable implementation evidence.

Response.{\color{green}\mathbf{Response.}} Your requested total-interaction comparison uses a 170k-call cap and the last complete update checkpoint below it. The reported scores and paired differences are:

MethodCalls (k)Certification/replay (k)GPU-hoursTGC Test-NTGC Test-C
G2PO169.60.038324.40 ± 0.6020.14 ± 0.24
Seq-LOO169.80.036823.05 ± 0.9118.90 ± 0.70
Frozen-weight Pair169.750.530523.95 ± 0.9019.75 ± 0.72
PWPO169.951.030424.20 ± 0.8519.95 ± 0.70
PWPO minus comparatorTest-N TGC, 95% CITest-C TGC, 95% CI
G2PO−0.20 [−1.4, 1.0]−0.19 [−1.3, 0.9]
Seq-LOO+1.15 [−0.1, 2.4]+1.05 [−0.2, 2.3]
Frozen-weight Pair+0.25 [−0.8, 1.3]+0.20 [−0.7, 1.1]

The learning curves show the same pattern. Entries are Test-N/Test-C TGC at the nearest complete checkpoint below each call cap:

Call cap (k)G2POSeq-LOOFrozen-weight PairPWPO
5016.8 / 13.216.0 / 12.516.5 / 12.816.7 / 13.0
10020.4 / 16.619.1 / 15.519.8 / 16.120.0 / 16.3
15023.0 / 18.921.8 / 17.722.6 / 18.422.8 / 18.6
17024.40 / 20.1423.05 / 18.9023.95 / 19.7524.20 / 19.95

The implementation audit uses an independent native-state export, checks the unchanged suffix, and tests known failures:

EnvironmentIndependent accepted-pair checksOutput/state/suffix violationsInjected negatives detected
AppWorld-Atomic1000/100 in each check24/24
ALFWorld1000/100 in each check24/24
WebShop1000/100 in each check24/24
ScienceWorld1000/100 in each check24/24

Appendix A.1.3 specifies the scoring rule. Every generated suffix token is reevaluated under its reordered history, and the common prefix retains its gradient contribution. This accounts for history dependence throughout the complete trajectory. The independent rechecks then verify the outputs, state and final outcome of the accepted exchange.

Reviewer_apJZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Pairwise Workflow Policy Optimization (PWPO) for tool-using language agents. It verifies whether exchanging two adjacent tool calls preserves their outputs and resulting state, then optimizes the combined probability of the two complete trajectories. The authors establish an on-policy variance-reduction guarantee, analyze its computational cost, and evaluate the method on AppWorld-Atomic, ALFWorld, WebShop, and ScienceWorld.

Critical strengths

The paper identifies a useful source of variation in agent training: execution orders can be equivalent for the environment while producing different policy gradients. I appreciate the care taken with the formulation. Boundary labels handle overlapping transcript sets, reordered suffixes are rescored, and failed or rejected trajectories remain in the training distribution. The analysis also accounts for shared leave-one-out baselines. I did not find a clear error in the main theoretical arguments.

The experimental controls are well chosen. Seq-LOO, Uniform-Pair, and Frozen-weight Pair help separate the effects of pairing, probability weighting, and clipping placement. Table 8 supports improvements over Seq-LOO and Uniform-Pair on all four completion metrics. The compute-matched comparison is also useful, although PWPO requires approximately 26% more tool calls than G2PO at comparable GPU time. The authors appropriately distinguish atomic from native AppWorld results and evaluate BEACON with feedback-matched controls.

Critical weaknesses

W1: The gradient measurements are insufficiently specified. Section 3 and Appendix B.5 record 64 score coordinates for each of 640 trajectory pairs, but do not adequately explain how these coordinates are constructed. The specific frozen model, checkpoint, and controlled environment for this preliminary study are also unclear. This concerns the measured vectors before PCA, rather than the visualization.

Separately, Appendix B.3 calls the object measured in Tables 7(b)/11 a “measured gradient representation” without defining its construction. It is therefore unclear which parameters or representation the reported variance ratio of 0.77 and cost–variance ratio of 0.91 concern. If a subset or projection is used, its relative improvement need not represent the full gradient. The depth analysis specifies 1,600 cells and 28 bins but omits the parameter-to-bin mapping. Table 10 likewise lacks a complete finite-model specification sufficient to reproduce the enumeration. These omissions limit the interpretation of the numerical evidence without invalidating the theoretical guarantee.

W2: Exact certification needs more concrete implementation details. The paper provides a careful contract and reports no observed replay faults or pair violations. It also explains that rechecks examine outputs, pair-end states, and rewards and termination after the full continuation. However, the environment-specific state fields, serialization and comparison procedures, and relationship between the certifier and recheck remain insufficiently documented. If both checks share an incomplete state representation, agreement alone would not establish completeness. I am not claiming that this occurred; adapter specifications and representative accepted and rejected exchanges would make this central assumption easier to assess.

W3: Some comparison details remain incomplete. Table 4 provides shared settings, and Appendix B.4 states that development-search allocations are matched, but the search spaces, numerical budgets, and final method-specific configurations are not fully reported. B.3 specifies five seeds for 1.5B and at least three for 7B/14B; Table 9 further specifies five/three runs for the PWPO audit. Exact counts for the remaining 7B/14B comparisons and the number of evaluation replicates are not fully enumerated. This is a lower-priority reproducibility concern, not evidence of unfair comparisons.

Questions for authors

Q1: Please address the missing specifications in W1–W3, particularly the separate gradient representations, identifiable checkpoints, and original number of independent task blocks used for variance estimation and bootstrap intervals. The budgeted counts of 32/38 blocks do not establish that original sample size.

Q2: How often do memberwise and cell-level clipping produce different updates? Their unclipped objectives coincide for fixed cells and advantages, while all four PWPO–Frozen-weight Pair intervals in Table 8 include zero. The authors acknowledge this limitation; clipping statistics would help clarify the additional contribution. Please also specify the minibatch loop and gradient accumulation, since Algorithm 1 shows an abstract update per epoch while Table 4 specifies batches of 128 trajectories and minibatches of 64.

4 comments
AuthorsOpenReview ↗

Response to Reviewer apJZ (1/3)

Thank you for the time you spent reviewing our paper and for your thoughtful comments on measurement and reproducibility. We address your questions below with additional experiments and implementation details.

Brackets denote reported confidence intervals. Completion scores are percentages.

W1. Gradient representations, full-gradient interpretation, and finite-model specification.

Response.{\color{green}\mathbf{Response.}} The 64-dimensional score you ask about is a fixed random projection of the complete transcript score over all trainable parameters:

\qquad s(\tau)=R^\top g(\tau), \qquad R\in\mathbb R^{d_\theta\times64}.$$ The Rademacher projection uses seed 2026 and the same matrix for all 640 pairs. Cosines, norms and order-noise statistics are computed in this 64-dimensional space. PCA produces the two-dimensional display. | Diagnostic | Representation and checkpoint | |---|---| | Preliminary / Figure 2 | Projected scores from frozen initial `Qwen/Qwen2.5-7B-Instruct` | | Tables 7(b)/11 | The same 64-dimensional projection at a fixed AppWorld-Atomic/7B reference checkpoint | | Figure 5 | The same reference checkpoint as Tables 7(b)/11. Bins 1–28 map to `model.layers.0`–`model.layers.27`, respectively | To address your full-gradient concern, the new AppWorld-Atomic/7B measurement covers **all trainable parameters**, before gradient-norm clipping and AdamW. The estimators share 256 independent task blocks, each containing $N=8$ rollouts. For 128 independent block pairs, $$D_e=\tfrac12\lVert G_e^{(1)}-G_e^{(2)}\rVert_2^2, \qquad \mathbb E[D_e]=\operatorname{tr}\operatorname{Cov}(G_e).$$ This identity estimates variance directly over the full parameter vector. | Estimator | Full-gradient variance / Seq-LOO | Complete cost / Seq-LOO | Cost–variance product / Seq-LOO | |---|---|---|---| | Seq-LOO | 1.00 | 1.00 | 1.00 | | Uniform-Pair | 0.88 [0.83, 0.94] | 1.16 [1.13, 1.19] | 1.02 [0.94, 1.12] | | Frozen-weight Pair | 0.79 [0.74, 0.85] | 1.16 [1.13, 1.19] | 0.92 [0.84, 0.99] | | PWPO | 0.79 [0.74, 0.85] | 1.16 [1.13, 1.19] | 0.92 [0.84, 0.99] | PWPO achieves **21% lower full-gradient variance** and an 8% lower cost–variance product. This complements the projected variance ratio of 0.77 in Tables 7(b)/11. Figure 2 examines projected score geometry, and Figure 5 locates removable variance across model depths. The finite-model enumeration reproduces Table 10: | Setting / estimator | Squared bias / $V_{\rm Seq}$ | Centered variance / $V_{\rm Seq}$ | MSE / $V_{\rm Seq}$ | |---|---:|---:|---:| | Certified pairs / Uniform-Pair | 0.034 | 0.842 | 0.876 | | Certified pairs / Frozen-weight Pair | 0.000 | 0.742 | 0.742 | | Certified pairs / PWPO | 0.000 | 0.742 | 0.742 | | All singletons / each of the four estimators | 0.000 | 1.000 | 1.000 | For certified pairs, probability weighting preserves the mean and reduces both variance and MSE. Uniform-Pair has squared bias 0.034, which contributes to its MSE of 0.876. The weighted estimators' zero bias agrees with Proposition 1 and Appendix D.
AuthorsOpenReview ↗

Response to Reviewer apJZ (2/3)

W2. Complete-state certification and independence of the recheck.

Response.{\color{green}\mathbf{Response.}} To address your concern about shared state representations, the additional recheck uses native environment state exports and an independent source-level read/write inventory. It operates independently of the certifier's state projection. The checks cover both swap directions, associated outputs, pair-end state, and reward/status after the unchanged suffix.

Environment, 7BAccepted pairs checkedDirection / output / state / suffix violationsInjected negative cases detected
AppWorld-Atomic1000 / 0 / 0 / 024/24
ALFWorld1000 / 0 / 0 / 024/24
WebShop1000 / 0 / 0 / 024/24
ScienceWorld1000 / 0 / 0 / 024/24

Each violation count has denominator 100. AppWorld checks cover database records, relations, IDs and clocks. ALFWorld checks cover objects, containment, inventory and task flags. WebShop checks include session, page, selection and purchase state. ScienceWorld checks include objects, simulation time, progress, randomness and termination.

The injected cases include read/write and reference dependencies, hidden counters, and premature termination. A write/read exchange is rejected when the read output changes, even if the final database state matches. Ordinary rejection retains the original singleton. Failed original-order replay halts the affected update. Together, the native-state rechecks and injected failures test both accepted exchanges and rejection of known dependencies.


W3. Search allocations, method-specific configurations, seeds, and evaluation repeats.

Response.{\color{green}\mathbf{Response.}} Each method receives eight random-search trials, each lasting five iterations. An iteration uses 16 tasks with eight rollouts each. This gives 640 rollouts per trial and 5,120 tuning rollouts per method. Configurations are selected by development performance.

Search parameterCandidate values
Actor learning rate5×10−7,10−6,2×10−65\times10^{-7},\\ 10^{-6},\\ 2\times10^{-6}
PPO clipping width0.1,0.2,0.30.1,\\ 0.2,\\ 0.3
PPO critic learning rate5×10−6,10−5,2×10−55\times10^{-6},\\ 10^{-5},\\ 2\times10^{-5}

The final-run controls and reported replication counts are:

ItemRecorded setting and scope
Rollout-matched training19,200 original trajectories per method, with matched task interface, terminal reward and information
Shared optimizationAdamW, learning rate 10−610^{-6}, clip width 0.2, gradient norm 1.0, one epoch, minibatch 64
Training runsFive seeds at 1.5B and at least three at 7B/14B (Appendix B.3)
PWPO certification auditExactly five runs at 1.5B and three at 7B/14B (Table 9)
EvaluationOne rollout per task per replicate, with shared decoding seeds. AppWorld pairing preserves scenario variants

This gives each method the same search allocation in trials and original rollouts. Grouping parameters follow each baseline's definition. Table 9 provides the exact run counts for the PWPO certification audit.

AuthorsOpenReview ↗

Response to Reviewer apJZ (3/3)

Q1. Identifiable gradient measurements and the original number of independent task blocks.

Response.{\color{green}\mathbf{Response.}} The original Tables 7(b)/11 use M=128M=128 independent task blocks, N=8N=8 rollouts per block, and 10,000 paired block-level bootstrap resamples. Their gradient representation is the fixed 64-dimensional Rademacher projection (seed 2026) at the shared AppWorld-Atomic/Qwen2.5-7B-Instruct reference checkpoint. The sample and budget counts have different roles:

QuantityMeaningValue
MM in original Tables 7(b)/11Independently sampled task blocks128
Bootstrap in original Tables 7(b)/11Paired block-level resamples10,000
MM in the new full-gradient measurementIndependently sampled task blocks256
NNIndependent rollouts within each task block8
Block pairs in the new measurementIndependent units for the squared-gradient-difference statistic128
KK in original Table 11Blocks averaged under the diagnostic budgetPWPO 32, Seq-LOO 38

The new measurement gives the following separate estimates:

EstimatorFull-gradient variance ratioComplete cost ratioCost–variance ratio
Frozen-weight Pair0.79 [0.74, 0.85]1.16 [1.13, 1.19]0.92 [0.84, 0.99]
PWPO0.79 [0.74, 0.85]1.16 [1.13, 1.19]0.92 [0.84, 0.99]

Thus, 32/38 are budgeted averages KK, 128 is the original sample size MM, and 256 is the separate full-gradient experiment's sample size.


Q2. How often do the clipping rules differ, and how are minibatches accumulated?

Response.{\color{green}\mathbf{Response.}} You are right that the unclipped objectives coincide. With wold(u)=pold(u)/Mold(C)w_{\rm old}(u)=p_{\rm old}(u)/M_{\rm old}(C) and ru=pθ(u)/pold(u)r_u=p_\theta(u)/p_{\rm old}(u),

=\frac{M_\theta(C)}{M_{\rm old}(C)}=\rho_C.$$ The two rules apply clipping before or after averaging. At $\theta_{\rm old}$, their gradients coincide. The following comparison holds parameters, AdamW moments, cells and advantages fixed: | Window | Cell clipping | Weighted member clipping | Branch disagreement | |---|---:|---:|---:| | $\theta=\theta_{\rm old}$ | 0.0% | 0.0% | 0.0% | | Iterations 10–20 | 1.8% | 5.9% | 4.8% | | Iterations 70–80 | 5.4% | 13.8% | 10.2% | | Iterations 130–140 | 7.2% | 18.6% | 13.1% | The table reports branch frequencies among two-member cells with nonzero advantage. The active threshold is $r>1.2$ for positive advantages and $r<0.8$ for negative advantages. Task performance is assessed separately in Table 8, where all four PWPO–Frozen intervals include zero. Each 128-trajectory batch produces two minibatches of 64. Each minibatch executes `zero_grad`, forward/backward, gradient-norm clipping at 1.0, and one AdamW step. Gradient accumulation is one minibatch, giving 300 optimizer steps over 150 iterations. The behavior policy, cells, old probabilities and LOO advantages remain fixed across both steps. Current likelihoods are recomputed for each minibatch. The first step starts at $\theta_{\rm old}$. The second evaluates the updated policy, where the clipping rules can differ. For the update comparison, both methods start from identical parameters and optimizer states. An update is classified as different when the relative L2 discrepancy between their full-parameter AdamW updates exceeds $10^{-4}$. Update frequency uses all paired optimizer-step comparisons as its denominator. Branch frequency uses nonzero-advantage two-member cells.
AuthorsOpenReview ↗

Follow-up on your concerns

Thank you again for your careful review and the time you have devoted to our paper. Our replies provide a full-gradient measurement showing 21% lower variance, independent certification checks across four environments, and details of the score projection, sampling, tuning budget and minibatch loop.

We would welcome your thoughts on these results and would be happy to discuss any remaining questions or provide further clarification and supporting evidence while the discussion is open. If our responses address your concerns, we would be grateful if you could consider revisiting your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.