- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes PAIR, a framework for synthesizing intermediate reasoning from expert action trajectories to train language agents. It warms up a reasoning generator using the target policy’s own trajectories, then optimizes it through reinforcement learning with rewards for action compatibility and consistency with the target policy’s distribution. The completed trajectories are used for supervised fine-tuning. Experiments across deep research, computer use, and web navigation compare PAIR with action behavior cloning and existing reasoning-reconstruction methods, while ablations examine the training stages and reward components. The paper also investigates its utility as an initialization for subsequent agentic reinforcement learning.
Critical strengths
-
Practically important problem. Learning from expert action trajectories without access to their reasoning could expand the available training data for language agents, particularly when demonstrations come from human interactions or proprietary models.
-
Extensive empirical evaluation. The experiments cover three agent domains and multiple target models, complemented by ablations of the reward components and training stages, comparisons across expert sources, and evaluations of subsequent reinforcement learning. This provides useful evidence of the method’s practical effectiveness.
Critical weaknesses
-
The abstract states: “While action data is abundant in environment logs (e.g., human interaction traces in computer-use settings), the underlying reasoning is inherently latent and difficult to collect.” However, the proposed method does not actually resolve this issue. The expert’s original reasoning remains inaccessible; what is reconstructed is reasoning from . Even “reconstruction” may be an inaccurate description, since the target policies used in the paper are small, open-weight models whose reasoning traces are readily available. The core contribution therefore appears to be optimizing the reasoning generated by itself.
-
Based on the current results, PAIR seems more appropriately viewed as a mid-training method. The generalizability and practical utility of its synthesized trajectories warrant further discussion. For example, can complete trajectories produced by an RG trained for one target policy also benefit other models? In my view, the warm-up data are generated by , the RG is initialized from and then warmed up, and the RL objective aligns the generated reasoning with the expert actions and ’s distribution. Consequently, using these reconstructed trajectories to fine-tune another model may primarily amount to a more effective form of distillation from $\pi_\theta$.
Questions for authors
- Figure 2 states that “reasoning is detailed, so that it can be supported by the policy’s distribution.” What justifies this connection between reasoning detail and compatibility with the policy’s distribution? Could short, generic reasoning also be highly probable under the target policy?
- How does PAIR compare with PRR and REER under matched total computational budgets, including generator training and data synthesis? Appendix A describes the baselines as training-free, although their reconstructed trajectories are also used for policy SFT. Could you clarify this distinction and report the actual costs?
- The reward ablations also change the length of reconstructed reasoning. Can you separate the benefits of distribution consistency from those of reasoning length?
- line 316-317 typo: "(18.1 -¿ 58.3)"
4 comments
Q3: The reward ablations also change the length of reconstructed reasoning. Can you separate the benefits of distribution consistency from those of reasoning length?
First, we do not explicitly reward or control reasoning length during training—as noted above, longer traces are simply a natural side effect of aligning outputs with the base model's natural thinking style. More importantly, longer reasoning alone does not lead to better downstream performance:
-
As shown in our ablation (Sec. 4.8), using alone actually produces longer reasoning than (119 vs. 98 tokens), yet gives much worse downstream performance (24.0 vs. 35.0) because the long traces are often wrong or irrelevant without .
-
Similarly, baselines like PRR and REER also generate full-length reasoning traces (rather than short 32-token shortcuts), yet PAIR still clearly outperforms them because its reasoning is much better aligned with the student policy's distribution via .
| Reward Setting | Action Acc | Reasoning length | Downstream Performance |
|---|---|---|---|
| only | 30% | 119 | 24.0 |
| + | 97% | 98 | 35.0 |
Q4: line 316-317 typo
Thank you very much for your careful reading and for catching this formatting typo! We have fixed it in the revised paper.
Thank you for your detailed responses. I have no further questions, and I will maintain my original score.
We are glad to hear that we have resolved your questions. We appreciate your constructive feedback and positive assessment on our work. Thank you very much!
Dear Reviewer J5HS:
Thank you for your careful review and constructive feedback. We address your concerns about our misleading statement and missing analysis below:
W1: "Reconstruct reasoning" is an inaccurate description in this paper.
Thanks for the helpful feedback. We agree that “reconstruct” may be misleading, and have replaced it with “synthesize” throughout the paper. We would also like to clarify the following two points:
Synthesizing learnable reasoning rather than recovering original reasoning. Our goal is to effectively train agents using action-only trajectories, such as human interaction logs or outputs from closed-source models. While collecting experts’ original reasoning is the most direct approach, it is often infeasible or prohibitively expensive, limiting the utility of abundant action data. This is precisely what motivates PAIR.
Optimizing the reasoning generated by the policy itself. Yes, your understanding is correct: a core contribution of our work is to improve the reasoning generated by the policy π itself. Although reasoning traces from π are readily available, they are often insufficient for solving challenging tasks. Our key insight is to leverage external expert actions to elicit stronger reasoning that enables the policy to tackle these tasks.
W2: The reasoning is aligned with πθ, so fine-tuning another model using these reasoning seems to distill from πθ. The generality of the reasoning for other models is unknown
We apologize for the confusion. In fact, πθ is the only model of interest in this work, and it is also the student model to be trained on the data synthesized by RG. We use different notations to distinguish the various roles (e.g. πθ, qϕ) of the same model throughout the paper, but our method does not rely on a separate teacher or distillation model.
Overall, our method can be viewed as a post-training approach that improves the base model πθ using action-only trajectories.
Q1: What justifies the connection between reasoning detail and compatibility with the policy’s distribution? Could short, generic reasoning also be highly probable under the target policy?
We apologize for the misunderstanding. In fact, we do not explicitly encourage more detailed reasoning. Rather, by aligning synthesized reasoning with the policy’s natural distribution, our method tends to produce more detailed traces instead of abrupt or overly short ones. This is because the reasoning style of the target policy is detailed. We have updated Figure 2 to clarify this point: “The reasoning is aligned with the policy’s distribution, which naturally leads to more detailed reasoning.”
Our experiments further support this observation. In Sec. 4.7, we show that short, generic shortcuts (e.g., simply restating the action or providing vague filler thoughts) receive substantially lower RDC scores because they deviate from the base policy’s natural reasoning distribution. In Sec. 4.8, our ablation study further shows that adding RDC increases the average reasoning length from 32 tokens (with RAC alone) to 98 tokens.
Q2: Compare computational budgets between PAIR and the training-free baselines.
First, we clarify the terminology in Appendix A. By “training-free,” we only meant that PRR and REER do not train a dedicated Reasoning Generator (RG). All methods undergo the exact same final policy SFT on the synthesized data, with identical SFT costs. To avoid confusion, we have replaced “training-free” with “generator-training-free” throughout Appendix A.
Second, following your suggestion, we added a detailed comparison of computational costs (in GPU-hours) for RG training and data synthesis in Appendix A, reporting the total cost at the full 15k-data scale. Since PRR directly generates reasoning through prompting without RG training, its per-sample synthesis cost is comparable to PAIR’s, but its final performance is substantially lower (27.0 vs. 34.6). In contrast, REER performs iterative searches at each step, resulting in much slower inference. At 15k samples, REER incurs a higher total cost than PAIR (376.3 vs. 284.0 GPU-hours) while achieving lower performance (30.8 vs. 34.6). Importantly, PAIR’s RG training is a one-time cost that can be amortized over larger datasets, making PAIR increasingly computationally efficient as the data scale grows.