acceptodds
Under review as a conference paper at ICLR 2027

Uncovering Latent Reasoning from Action Trajectories for Long-Horizon Agents

Abstract

Training Large Language Model (LLM) agents for long-horizon tasks requires high-quality trajectory data that tightly interleaves reasoning and action. While action data is abundant in environment logs (e.g., human interaction traces in computer-use settings), the underlying reasoning is inherently latent and difficult to collect. In this paper, we introduce PAIR, a framework designed to synthesize complete training trajectories by uncovering latent reasoning directly from observable action sequences. The key idea is to train a reasoning generator via reinforcement learning through a novel reward mechanism: an action compatibility reward that maximizes alignment between the synthesized reasoning and subsequent ground-truth actions, paired with a distribution consistency reward that constrains the reasoning to remain within the target policy's natural distribution, preventing trivial shortcuts. Across diverse domains, including deep research, computer use, and web navigation, PAIR consistently outperforms action behavior cloning and state-of-the-art reasoning reconstruction techniques. Furthermore, PAIR bootstraps agent performance on unfamiliar tasks and serves as a strong initialization for subsequent agentic RL, demonstrating that action-only trajectories can serve as a powerful supervisory signal for long-horizon agent training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, agent, reasoning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_J5HSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PAIR, a framework for synthesizing intermediate reasoning from expert action trajectories to train language agents. It warms up a reasoning generator using the target policy’s own trajectories, then optimizes it through reinforcement learning with rewards for action compatibility and consistency with the target policy’s distribution. The completed trajectories are used for supervised fine-tuning. Experiments across deep research, computer use, and web navigation compare PAIR with action behavior cloning and existing reasoning-reconstruction methods, while ablations examine the training stages and reward components. The paper also investigates its utility as an initialization for subsequent agentic reinforcement learning.

Critical strengths
  • Practically important problem. Learning from expert action trajectories without access to their reasoning could expand the available training data for language agents, particularly when demonstrations come from human interactions or proprietary models.

  • Extensive empirical evaluation. The experiments cover three agent domains and multiple target models, complemented by ablations of the reward components and training stages, comparisons across expert sources, and evaluations of subsequent reinforcement learning. This provides useful evidence of the method’s practical effectiveness.

Critical weaknesses
  1. The abstract states: “While action data is abundant in environment logs (e.g., human interaction traces in computer-use settings), the underlying reasoning is inherently latent and difficult to collect.” However, the proposed method does not actually resolve this issue. The expert’s original reasoning remains inaccessible; what is reconstructed is reasoning from πθ\pi_\theta. Even “reconstruction” may be an inaccurate description, since the target policies used in the paper are small, open-weight models whose reasoning traces are readily available. The core contribution therefore appears to be optimizing the reasoning generated by πθ\pi_\theta itself.

  2. Based on the current results, PAIR seems more appropriately viewed as a mid-training method. The generalizability and practical utility of its synthesized trajectories warrant further discussion. For example, can complete trajectories produced by an RG trained for one target policy also benefit other models? In my view, the warm-up data are generated by πθ\pi_\theta, the RG is initialized from πθ\pi_\theta and then warmed up, and the RL objective aligns the generated reasoning with the expert actions and πθ\pi_\theta’s distribution. Consequently, using these reconstructed trajectories to fine-tune another model may primarily amount to a more effective form of distillation from $\pi_\theta$.

Questions for authors
  1. Figure 2 states that “reasoning is detailed, so that it can be supported by the policy’s distribution.” What justifies this connection between reasoning detail and compatibility with the policy’s distribution? Could short, generic reasoning also be highly probable under the target policy?
  2. How does PAIR compare with PRR and REER under matched total computational budgets, including generator training and data synthesis? Appendix A describes the baselines as training-free, although their reconstructed trajectories are also used for policy SFT. Could you clarify this distinction and report the actual costs?
  3. The reward ablations also change the length of reconstructed reasoning. Can you separate the benefits of distribution consistency from those of reasoning length?
  4. line 316-317 typo: "(18.1 -¿ 58.3)"
4 comments

Dear Reviewer J5HS:

Thank you for your careful review and constructive feedback. We address your concerns about our misleading statement and missing analysis below:

W1: "Reconstruct reasoning" is an inaccurate description in this paper.

Thanks for the helpful feedback. We agree that “reconstruct” may be misleading, and have replaced it with “synthesize” throughout the paper. We would also like to clarify the following two points:

  • Synthesizing learnable reasoning rather than recovering original reasoning. Our goal is to effectively train agents using action-only trajectories, such as human interaction logs or outputs from closed-source models. While collecting experts’ original reasoning is the most direct approach, it is often infeasible or prohibitively expensive, limiting the utility of abundant action data. This is precisely what motivates PAIR.

  • Optimizing the reasoning generated by the policy itself. Yes, your understanding is correct: a core contribution of our work is to improve the reasoning generated by the policy π\pi itself. Although reasoning traces from π\pi are readily available, they are often insufficient for solving challenging tasks. Our key insight is to leverage external expert actions to elicit stronger reasoning that enables the policy to tackle these tasks.

W2: The reasoning is aligned with πθ\pi_\theta, so fine-tuning another model using these reasoning seems to distill from πθ\pi_\theta. The generality of the reasoning for other models is unknown

We apologize for the confusion. In fact, πθ\pi_\theta is the only model of interest in this work, and it is also the student model to be trained on the data synthesized by RG. We use different notations to distinguish the various roles (e.g. πθ\pi_\theta, qϕq_\phi) of the same model throughout the paper, but our method does not rely on a separate teacher or distillation model.

Overall, our method can be viewed as a post-training approach that improves the base model πθ\pi_\theta using action-only trajectories.

Q1: What justifies the connection between reasoning detail and compatibility with the policy’s distribution? Could short, generic reasoning also be highly probable under the target policy?

We apologize for the misunderstanding. In fact, we do not explicitly encourage more detailed reasoning. Rather, by aligning synthesized reasoning with the policy’s natural distribution, our method tends to produce more detailed traces instead of abrupt or overly short ones. This is because the reasoning style of the target policy is detailed. We have updated Figure 2 to clarify this point: “The reasoning is aligned with the policy’s distribution, which naturally leads to more detailed reasoning.”

Our experiments further support this observation. In Sec. 4.7, we show that short, generic shortcuts (e.g., simply restating the action or providing vague filler thoughts) receive substantially lower RDCR_{DC} scores because they deviate from the base policy’s natural reasoning distribution. In Sec. 4.8, our ablation study further shows that adding RDCR_{DC} increases the average reasoning length from 32 tokens (with RACR_{AC} alone) to 98 tokens.

Q2: Compare computational budgets between PAIR and the training-free baselines.

First, we clarify the terminology in Appendix A. By “training-free,” we only meant that PRR and REER do not train a dedicated Reasoning Generator (RG). All methods undergo the exact same final policy SFT on the synthesized data, with identical SFT costs. To avoid confusion, we have replaced “training-free” with “generator-training-free” throughout Appendix A.

Second, following your suggestion, we added a detailed comparison of computational costs (in GPU-hours) for RG training and data synthesis in Appendix A, reporting the total cost at the full 15k-data scale. Since PRR directly generates reasoning through prompting without RG training, its per-sample synthesis cost is comparable to PAIR’s, but its final performance is substantially lower (27.0 vs. 34.6). In contrast, REER performs iterative searches at each step, resulting in much slower inference. At 15k samples, REER incurs a higher total cost than PAIR (376.3 vs. 284.0 GPU-hours) while achieving lower performance (30.8 vs. 34.6). Importantly, PAIR’s RG training is a one-time cost that can be amortized over larger datasets, making PAIR increasingly computationally efficient as the data scale grows.

methodTraining costInference cost (5k/10k/15k data)Total costfinal performance
PRR-23.1/46.2/72.572.527.0
REER-124.0/248.2/376.3376.330.8
PAIR205.625.8/51.7/78.4284.034.6

Q3: The reward ablations also change the length of reconstructed reasoning. Can you separate the benefits of distribution consistency from those of reasoning length?

First, we do not explicitly reward or control reasoning length during training—as noted above, longer traces are simply a natural side effect of RDCR_{DC} aligning outputs with the base model's natural thinking style. More importantly, longer reasoning alone does not lead to better downstream performance:

  • As shown in our ablation (Sec. 4.8), using RDCR_{DC} alone actually produces longer reasoning than RDC+RACR_{DC} +R_{AC} (119 vs. 98 tokens), yet gives much worse downstream performance (24.0 vs. 35.0) because the long traces are often wrong or irrelevant without RACR_{AC} .

  • Similarly, baselines like PRR and REER also generate full-length reasoning traces (rather than short 32-token shortcuts), yet PAIR still clearly outperforms them because its reasoning is much better aligned with the student policy's distribution via RDCR_{DC} .

Reward SettingAction AccReasoning lengthDownstream Performance
RDCR_{DC} only30%11924.0
RDCR_{DC} + RACR_{AC}97%9835.0

Q4: line 316-317 typo

Thank you very much for your careful reading and for catching this formatting typo! We have fixed it in the revised paper.

Thank you for your detailed responses. I have no further questions, and I will maintain my original score.

We are glad to hear that we have resolved your questions. We appreciate your constructive feedback and positive assessment on our work. Thank you very much!

Reviewer_4HpGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Policy Compatible Action Conditioned Inverse Reasoning (PAIR), an approach that reconstructs missing reasoning traces from action-only trajectories, to support supervised finetuning for long-horizontal agents. PAIR trains a reasoning generator (RG) in two stages: warm-up stage using target policy's own reasoning trajectories, and RL stage that generates reasoning maximizing action consistency reward and compatibility reward. Experiments across deep research, compute use and web navigation shows improvements over action behavior cloning and existing reasoning reconstruction baselines.

Critical strengths
  1. The paper has practical impacts on improving long-horizon agents which involves a lot of action taking. The work addresses a realistic data bottleneck.
  2. Empirical results demonstrate consistent gains across different domains.
Critical weaknesses
  1. The paper did not provide analysis on the faithfulness of reconstructed reasoning traces. It measures quality through downstream tasks, but it remains unclear whether the reasoning traces are truly relevant to the improvement of action accuracy, or some reward hacking behaviors are responsible for the improvement.
  2. The distribution consistency reward encourages reasoning traces that are likely under the target policy, but high likelihood does not necessarily translate to the correctness of reasoning, especially when the policy itself is weak. Some reward that judges the quality of the reasoning traces is more preferred.
Questions for authors
  1. One of my central concerns is how generalizable the reasoning generator is across different domains. Although Appendix C includes results on unseen trajectories, it would be interesting to evaluate on some agentic tasks from heterogenous domains (e.g. tau-bench, E-bench).

  2. I wonder at stage 1 of training, instead of asking the policy model to generate its own reasoning trajectories, what if you warm-up with a strong model like GPT, Claude to construct the reasoning trajectories? What would be the quality difference after stage 2 training?

  3. To systematically assess the quality of reconstructed thoughts, I'd suggest picking some agent training data with both human-labeled thoughts and actions, then remove the human thoughts and compare the reconstructed thoughts with ground truths.

5 comments

Dear Reviewer 4HpG:

Thank you for your careful review and constructive feedback. We address your concerns about missing analysis and evaluation below:

W1: Missing analysis on the faithfulness of reconstructed reasoning traces

Thank you for raising this question. In Appendix D, we directly evaluate reasoning quality along five rubric-based dimensions: Problem Deconstruction, Logical Consistency, Comprehensiveness of Analysis, Presentation Clarity, and Factual Grounding. As shown in the quantitative results below, PAIR demonstrates strong and well-rounded reasoning quality, consistently outperforming PRR and REER across all five dimensions.

MethodProblem DeconstructionLogical ConsistencyComp. of AnalysisPresentation ClarityFactual Grounding
PRR3.33.22.94.03.7
REER3.63.64.13.74.0
PAIR4.04.64.55.04.7
GLM-4.74.85.04.84.55.0

W2: RDCR_{DC} does not guarantee the correctness of reasoning

Thanks for the question. RDCR_{DC} alone is insufficient to elicit correct reasoning. When combined with RACR_{AC}, the two rewards jointly enable effective reasoning synthesis. Specifically, RACR_{AC} aligns the inferred action Pπ(at∣…,zt)P_{\pi}(a_t \mid \dots, z_t) with the ground-truth expert action ata_t, thereby implicitly encouraging reasoning ztz_t that leads to correct actions.As shown in our ablation results, the two rewards are complementary, and both are important for effective reasoning synthesis.

Reward SettingAction AccReasoning lengthDownstream Performance
RDCR_{DC} only30%11924.0
RDCR_{DC} + RACR_{AC}97%9835.0

Q1: How generalizable the reasoning generator is across different domains

Thanks for this great suggestion! We have added the suggested cross-domain evaluation, including τ\tau-bench. Specifically, we directly applied the RG trained exclusively on Deep Research tasks in our main experiments, without any further tuning, to generate reasoning for Retail-3I [1] and SAP-9k [2], which are training datasets for retail and BFCL-style tool-use tasks, respectively. We then trained a Qwen3-4B agent on the synthesized data.As shown below, the resulting agent achieves clear improvements across all three benchmarks:

Modelτ\tau-bench retailτ2\tau^2-bench retailBFCL-v4 multi-turn
Qwen3-4B60.057.957.7
Qwen3-4B after training62.762.364.1

[1] Wang, Ziyi, et al. "Trajectory2task: Training robust tool-calling agents with synthesized yet verifiable data for complex user intents." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026. [2] Tian, Zichen, et al. "SAP: State-Guided Data Synthesis with Argument Provenance for Multi-Turn Tool Use." Findings of the Association for Computational Linguistics: EMNLP 2026. 2026.

Q2: What if warm-up with strong models like GPT/Claude to construct reasoning traces?

Thank you for this question! Our paper focuses on training a student model directly on action-only trajectories, assuming no access to reasoning from stronger external models anywhere in our pipeline. Moreover, strong proprietary models like GPT and Claude hide their internal reasoning traces, which cannot be used for warm-up. In fact, our main experiments on Deep Research tasks use action-only trajectories from GPT-5.5, and PAIR achieves clear improvements relying on Stage 1 warm-up with its own reasoning.

Q3: Compare generated reasoning with human thoughts

Following your suggestion, we added this comparison in Appendix K. Specifically, we sampled trajectories with human-written thoughts from OPeRA [1], a dataset of real human online shopping trajectories. We masked the original human thoughts and used our RG to synthesize reasoning for the corresponding actions. A representative example is shown below.

Human Thought:

I added the product to cart a bit earlier, now I'm going to the cart to check it.
----------------------------------------------------------------
Synthesized Reasoning:

I have been reviewing the Nature Made Ashwagandha product page, and I now want to return to the full Shopping Cart. The navigation provides an explicit Cart link to the cart view, so I will click that control. The sidebar currently records two of the Ashwagandha bottles, but the full cart provides the broader item-management context I opened earlier. This click is navigation only, rather than an adjustment through the sidebar’s quantity buttons or a new purchase through the product page. I will use the current cart view to inspect its state rather than assume that the navigation count explains every item detail.

These examples illustrate that human-written thoughts and our synthesized reasoning provide different but complementary perspectives on the same actions. Human notes often reflect personal interests, browsing habits, and broader goals, whereas our generated reasoning focuses more directly on the current page context, preceding steps, and immediate action rationale. For example, when navigating to the shopping cart, the human note expresses the intention to check an item, while our synthesized reasoning explicitly explains the transition from the product page to the cart and the purpose of this navigation. This demonstrates how our generated reasoning can complement human annotations by providing a more explicit account of the page context and action-level logic.

[1] Wang, Ziyi, et al. “OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation.” Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026.

Dear Reviewer 4HpG:

Thanks for your time and effort in reviewing this paper. As the rebuttal period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving this paper.

Reviewer_jBg5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoT is essential for SFT on demo trajectories. The paper presents, PAIR, a method to generate CoT from demo trajectories containing only the action labels. PAIR learns a reasoning generator RG by maximizing self-supervised rewards. The rewards are computed from the likelihoods of the SFT model itself. The reward has two components, R_AC and R_DC. R_AC rewards action consistency while R_DC rewards distributional consistency. The paper compares to baselines including ABC, PRR, and REER. PAIR empirically improves model's reasoning performance and achieve SOTA results.

Critical strengths
  1. Interesting empirical findings. For example, in section 4.7, PAIR achieves performance close to distilling real CoT from GLM-4.7 and by learning from GPT5.5 PAIR outperforms GLM-4.7. It implies that stronger expert capability matters more than the access of CoT for PAIR. This finding lifts the contribution of this paper.
  2. Relatively clear demonstration and writing.
Critical weaknesses
  1. Missing ablations/explanation on potential reward hacking issues.

    a. R_DC might be rewarding the student model to hallucinate a reasoning z_t that is irrelevant or incorrect. In other words, the reasoning generator can still hack R_DC reward by hallucinating an illogical trace. This is not a major concern that could lead to rejection of the paper but I encourage the authors to design & run ablations.

    b. The paper applies tanh to wrap both R_AC and R_DC. This trick has been used in quiet-star and rlpr. But we still need ablations to see why PAIR benefits from tanh wrapping here. Similarly, the paper does not explain the normalization of R_AC by |a|. Explaining these implementation choices could improve writing clarity a lot.

  2. Missing computation-matched comparisons. PAIR involves more training phases than the baselines. I encourage the authors to add compute-matched experiments.

  3. Missing discussion of similar recent approaches (e.g. LeAct). "LeAct" was not public until July 2026. This research field has been progressing exponentially fast. I encourage the authors to closely follow recent works (even within a few months before this submission).

Questions for authors
  1. How should we interpret R_DC? It seems to me that R_DC cannot prevent leaking action info. The reasoning generator can still rub the exact action into the reasoning z_t like "Casablanca immediately comes to mind". It might be rewarding the student model to hallucinate a reasoning z_t that is irrelevant or incorrect. In other words, the reasoning generator can still hack R_DC reward by hallucinating an illogical trace.
  2. R_AC depends heavily on the frozen pi model. Does model's reasoning capability upper bound the quality of R_AC?
4 comments

Dear Reviewer jBg5:

Thank you for your careful review and constructive feedback. We address your concerns about missing experiments and interpretation below.

W1.a & Q1: R_DC might hallucinate a reasoning that is incorrect or leaks action.

Your concern is valid. Let us reiterate the complementary roles of the two rewards: RACR_{AC} aligns the action inferred by the student with the ground-truth action, while RDCR_{DC} regularizes the reasoning to follow the student’s original reasoning distribution. Our ablation studies in Secs. 4.7 and 4.8 demonstrate that RDCR_{DC} is critical for preventing RACR_{AC} from exploiting shortcuts.

Nonetheless, we agree that our model may still produce incorrect reasonings. Regarding the example “Casablanca immediately comes to mind,” we believe such hallucinations are inherent limitations of LLMs, which is bounded by the capabilities encoded in their pretrained weights.

W1.b: Missing ablations about reward normalization

Thanks for raising these questions. We have added the following clarifications and corresponding ablations in Appendix H:

  • Why apply tanh wrapping? Both RACR_{AC} and RDCR_{DC} are based on log-probabilities, which are unbounded. Extreme values may lead to excessively large critic losses during PPO training, potentially destabilizing the training process. Applying tanh bounds both rewards within (-1, 1), helping stabilize training.

  • Why normalize RACR_{AC} by |a|? RACR_{AC} accumulates information gain token by token, which would otherwise introduce a bias toward longer action sequences. Dividing by |a| removes this length bias, preventing sequence length from confounding the reward.

W2: Missing computation-matched comparisons

Thanks for the suggestion. We have added a detailed comparison of computational costs (in GPU hours) in Appendix A, as summarized below. PRR generates reasoning directly using an optimized prompt without additional training. Its inference cost is thus comparable to PAIR’s, but its final performance is substantially lower (27.0 vs. 34.6). In contrast, REER iteratively searches for better reasoning at each step, incurring much higher computational costs per sample. For 15k samples, REER’s total cost is even higher than PAIR’s (376.3 vs. 284.0 GPU hours), while achieving lower performance (30.8 vs. 34.6). These results demonstrate that PAIR generates high-quality reasoning data more efficiently and effectively.

methodTraining costInference cost (5k/10k/15k data)Total costfinal performance
PRR-23.1/46.2/72.572.527.0
REER-124.0/248.2/376.3376.330.8
PAIR205.625.8/51.7/78.4284.034.6

W3: Missing discussion of related work

Thanks for pointing this out. LeAct is indeed highly relevant to our work. We have cited it and added a detailed discussion of its contributions in the revised Related Work section.

Q2: Does the capability of π\pi upper bound the quality of RACR_{AC}?

Thanks for the question! In practice, RACR_{AC} checks whether a given reasoning trace is consistent with the subsequent action. While generating reasoning from a question can be challenging, we find that evaluating this consistency is relatively easy for modern LLMs across different model sizes. To verify this, we computed RACR_{AC} scores on the same batch of generated reasoning using Qwen3.5 models of different sizes as the frozen policy π\pi: 4B, 35B-A3B, and 122B-A10B. As shown below, the Pearson correlation coefficients between their RACR_{AC} scores are consistently high, indicating strong agreement in their judgments across a wide range of model sizes.

This suggests that as long as π\pi is a well-pretrained base model, its reasoning capability is unlikely to be a bottleneck for RACR_{AC} quality.

4B35B-A3B122B-A10B
4B1.000.760.73
35B-A3B0.761.000.84
122B-A10B0.730.841.00

Dear Reviewer jBg5: Thanks for your time and effort in reviewing this paper. As the rebuttal period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving this paper.

Thanks for the clarifications and answers. I keep my positive review.

We appreciate your constructive feedback and positive assessment on our work. Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.