- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce Pairwise Workflow Policy Optimization (PWPO) in this paper. The goal of this method is to improve training stability. For a rollout, certified reordering of two adjacent tool calls is obtained. A reordering is certified only if all the interaction outcomes are the before and after the swap. For these swaps, a total probability is calculated using the current policy and a frozen policy. These together form the importance sampling ratio for the standard LOO advantage. The authors further establish that the PWPO update has lower variance and that PWPO can sometimes be computationally better than an ordinary estimator.
Experiments on instruct based models on 3 agentic datasets show that a model trained with PWPO performs better than techniques like G2PO and Seq-LOO. Further analysis shows that PWPO improves task completion and reduces variance. The authors also show that the variance shifts towards deeper layers in the network
Critical strengths
Improving training stability by reducing variance is an important task especially when post-training of large models is involved. Using a simple property like swapping tool calls can be applied to a large variety of problems. The authors establish that equivalent outcomes can result in different learning signals and can introduce noise during training.
Although, the budget is sufficiently large, the authors prove that PWPO can be computationally better than ordinary estimators. They also show experimental results where it can be seen that training with PWPO results in lower variance. At the end, they also show an interesting result where the variance is pushed towards deeper layers of the network.
Critical weaknesses
The experiments use the dated Qwen2.5 family instruct models on fairly dated datasets. Even if newer models were used, a big part that is lacking is not using thinking models. Most of the community has moved towards thinking models as they are more practical, especially in agentic tasks. PWPO in its current formulation may not work for reasoning models as the thinking traces might not make sense causally after swapping. Moreover, longer thinking traces will increase rescoring and differentiation costs and might also decrease statistical gains obtained by PWPO at the same time.
The paper presentation is lacking. They use a lot of non standard terms like 'transcripts' and 'cells'. While these terms are defined, they make the paper very difficult to follow. It also seems that notation is used excessively and sometimes in places where plain language would explain things much better (for example, in 'Certifying an adjacent swap' in Section 4.1). It would be much better if the authors dedicated more space towards 4.3 and 4.4 rather than defining their method, which can be summarized as a new importance sampling method that takes into account swappable tool calls.
Questions for authors
Have the authors considered using PWPO with reasoning models? If so, how will PWPO work with the thinking traces in the trajectories generated by the models and what are the changes if any?
5 comments
Response to Reviewer avbb (2/2)
Q1. How does PWPO handle thinking traces, and what changes are required?
PWPO handles thinking traces by treating thinking text + one atomic call + its observation as an indivisible block. The original thinking tokens and IDs are preserved. Exchanges with argument, reference or termination dependencies are rejected. Both accepted histories therefore retain the original text and pass the execution checks.
All generated tokens contribute to the likelihood, including thinking, submission and incomplete final prefixes. Each reordered history is rendered through the actual chat template, and its suffix is rescored. Qwen3.5's template uses the final output from earlier responses as conversation history [1]. Thinking tokens still contribute their probabilities at the time they are generated.
Proposition 1 applies to these extended records under fixed support, symmetric certification and reward preservation. The certificate checks executable dependencies and the shared outcome. Policy probabilities determine the weights of the two complete histories. A reordered trace with lower policy probability receives a smaller conditional weight, limiting its contribution to the averaged gradient.
The measured tradeoff is:
| Mode, same frozen backbone | Blocks / rollouts per block | Coverage | Median | Full-gradient variance ratio | Complete cost ratio | Cost–variance ratio |
|---|---|---|---|---|---|---|
| Thinking off | 256 / 8 | 39.0% | 0.42 | 0.79 [0.74, 0.85] | 1.16 [1.12, 1.20] | 0.92 [0.84, 1.00] |
| Thinking on | 256 / 8 | 31.5% | 0.28 | 0.84 [0.78, 0.90] | 1.24 [1.19, 1.29] | 1.04 [0.94, 1.16] |
With thinking enabled, 31.5% of rollouts admit a certified pair and full-gradient variance falls by 16%. The experiment shows that the averaging mechanism remains effective with thinking traces. The cost measurements include rescoring and differentiating the longer histories.
[1] Qwen3.5-9B Model Card.
Follow-up on your concerns
Thank you again for highlighting the questions about reasoning models and presentation. Our replies include Qwen3.5-9B thinking-mode experiments showing 16% lower full-gradient variance, explain how thinking traces are preserved and rescored, and quantify the resulting cost tradeoff.
We would welcome your feedback and would be happy to discuss any remaining questions or provide further clarification and supporting evidence while the discussion is open. If these results and explanations address your concerns, we would be grateful if you could reconsider your rating.
Response to Reviewer avbb (1/2)
Thank you for taking the time to review our work and for your helpful comments on reasoning models and presentation. Below, we explain the main idea and present the thinking-model results in response to your questions.
Response. To address your reasoning-model concern, we tested PWPO on Qwen3.5-9B in thinking mode with the same AppWorld-Atomic interface. The original comparisons use a shared Qwen2.5 backbone across agent-RL baselines. The additional experiment extends the evaluation to a thinking policy. Across the main and supplementary experiments, the paper covers four environments, including ScienceWorld.
These runs demonstrate certified-pair training with a thinking policy. PWPO reaches 32.8/27.0 Test-N/Test-C TGC at 382 GPU-hours. The paired completion intervals span zero.
The fixed-checkpoint comparison measures how thinking affects coverage, variance and cost, using 256 blocks per mode:
Each mode is normalized to its own Seq-LOO. Thinking retains 16% variance reduction. Section 4.4 requires more than 0.24/1.24≈19.4 reduction to offset 24% extra cost, explaining the measured cost–variance ratio of 1.04. This connects the observed rescoring cost to the theoretical budget criterion. Q1 below explains how the thinking traces are handled.
Response. You can read a transcript as the complete interaction trajectory, and a cell as one trajectory or a certified equivalent pair, together with the sampled swap boundary.
The central insight is that equivalent execution orders can produce different learning signals. Two independent calls A and B can lead to the same outcome in either order. Their histories, A!→!B and B!→!A, still have different policy probabilities and gradients. PWPO learns their combined probability and averages out the variation caused by choosing between these equivalent orders. The importance ratio follows from this choice of training event.
Section 4.3 expresses this directly:
=\mathbb E\\!\left[\nabla_\theta\log p_\theta(\tau)\mid C,j,x\right].$$ Certification establishes a shared reward for the two histories. The boundary label gives a valid partition even when candidate swaps overlap. Full-history rescoring supplies the probability weights for their gradients. Conditional marginalization provides the statistical foundation [1]. PWPO makes it usable for tool-using language policies by constructing verifiable, reward-preserving trajectory events. Section 4.4 asks when the variance reduction offsets the added cost. Figure 5 shows where removable order sensitivity is concentrated as a workflow progresses. Its horizontal axis is position within the trajectory, with model parameters held fixed throughout the analysis. [1] Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications.