acceptodds
Under review as a conference paper at ICLR 2027

Studying from Trajectories: Turning Agent Experience into Stepping-Stone Tasks

Abstract

Continually improving agents must decide what to learn from experience and how to practice it. We define studying as identifying knowledge and skills worth learning from trajectories and practicing them through targeted exercises. Stage implements this process by turning lessons from trajectories into single-turn stepping-stone tasks. These tasks let the agent practice individual components of complex behavior with focused feedback, without repeating the full interaction. New interaction experience guides each study iteration, allowing practice to evolve with the agent. On AppWorld, Stage improves avg@4 by 15.3 percentage points over the base model on Test-Challenge, demonstrating substantial gains from learning through studying and interaction. Further analysis suggests that studying strengthens knowledge and local decision making, while interaction helps the agent apply and combine these capabilities within longer workflows. Alternating these complementary phases turns experience into focused practice that supports more effective action and continued improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents, LLMs, Continual Learning, Synthetic Tasks Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ryAxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed "STAGE", a method to automatically curate smaller/simpler training tasks from a target models' rollout, perform a round of RL to improve the model on these targeted capabilities, and finally perform standard multi-turn RL. The authors show that under a similar training budget this outperforms directly performing multi-turn RL, which indicates the importance of improving the model prior before RL. The authors mainly experimented on APPWorld with 126 training tasks and conducted numerous analysis such as learning from mostly failed trajectories or iterative using STAGE.

Critical strengths
  • The authors showed positive result on APPWorld where STAGE(ours, self) outperforms simple multit-turn RL on Test-Challenge.
  • Interesting analysis including training with mostly failed experiences (section 4.3, which I believe mimics the idea of offline RL) and running STAGE for multiple iterations (section 5).
Critical weaknesses
  1. The overall idea seems relatively simple: the authors aim to give the agent a stronger prior (L84) to improve the effectiveness of standard multi-turn RL. A common approach to achieving this is to perform SFT on high-quality synthetic or human-annotated data that targets the missing or weak capabilities of the base model. The proposed "studying" method, which generates tasks targeting these weaknesses based on existing trajectories, seems conceptually very similar. It feels like the authors are introducing a new term or concept for an approach that is already common in LLM agent post-training.

  2. The proposed method is evaluated only on AppWorld. It is therefore unclear whether the method generalizes to other benchmarks or domains, such as coding or web/computer-use.

  3. Why does SFT degrade performance so substantially (Table 1), especially when the training data comes from the same benchmark? This is quite surprising and could indicate issues with the experimental setup, training data, or implementation in Section 4.

  4. STAGE results in Table 1 seems to be unfair. Specifically, without accessing any stronger LLMs STAGE (ours, self) underperforms multi-turn RL in Test-Normal. It is only when STAGE uses a stronger model (Qwen3.8-27B) for data generation it outperforms multi-turn RL. I believe this is not an apple-to-apple comparison.

  5. The training pool consists of only 126 tasks. I believe this is a relatively weak setup for RL (especially for multi-turn agent tasks used in this work), and the conclusions about the effectiveness of baseline methods such as direct GRPO/RL may not be reliable. With such a limited number of training tasks, it is possible that the gains of the proposed method (STAGE) over standard RL methods would diminish if more training data were available.

Questions for authors

please see weaknesses

6 comments

Dear Reviewer ryAx,

We appreciate your thoughtful review and recognition of our efforts. We hope that the following responses help clarify the reviewer’s questions:


W1.

Thank you for raising this important point. We clarify the definitions of studying and stepping-stone tasks, and how STAGE differs from existing approaches.

1. What are studying and stepping-stone tasks?

  • Studying is not a new optimization algorithm, but a dedicated learning phase that identifies what to learn from agent trajectories and provides focused opportunities to practice it.
  • Stepping-stone tasks are single-turn QA exercises that isolate specific knowledge or decision-making skills from trajectories, allowing agents to practice them with direct feedback without replaying the full interaction.

For example, when an agent accidentally deletes existing note content using the update_note API, STAGE identifies the underlying concept (the API replaces the entire note) and generates a QA exercise requiring the agent to preserve unchanged content.

2. How does STAGE differ from existing approaches?

We acknowledge the connections to prior work and do not claim that practicing individual skills or learning from QA is novel in itself.

  • Curriculum learning: While curriculum learning organizes or generates learning tasks, STAGE focuses on first discovering what needs to be learned from successful and failed agent trajectories.
  • Skill learning: Unlike context-based skill reuse methods, STAGE converts extracted knowledge and skills into targeted exercises for updating model parameters.
  • Synthetic QA fine-tuning: Rather than directly generating QA from trajectories, STAGE explicitly separates what to learn (concept extraction) from how to practice (concept-conditioned QA generation).

3. What is the key contribution?

STAGE introduces a structured learning process:

Trajectory → Concept (what to learn) → Stepping-stone task (how to practice) → Single-turn RL → Multi-turn RL

The central idea is that agents should not rely solely on trial-and-error interaction, but can also benefit from explicitly studying the knowledge and skills revealed by their experience. Single-turn RL develops these targeted capabilities, while subsequent multi-turn RL learns to apply and combine them in complete workflows.


W2/W5. Generalization Beyond AppWorld and Larger-Scale Training

We thank the reviewer for raising concerns about generalization beyond AppWorld and the limited training pool of 126 tasks. To address both concerns, we conducted additional experiments on τ²-bench, following the experimental setup of AReaL-SEA [1]. Specifically, we use 800 tasks from the Airline training split for training, the remaining Airline tasks for ID evaluation, and the Retail domain for OOD evaluation.

ID test (airline)OOD test (retail)
Qwen3.5-4B50.778.7
Multi-turn RL61.578.7
Studying only55.277.5
STAGE (Studying + Multi-turn RL)63.580.0

STAGE consistently outperforms multi-turn RL, achieving improvements of +2.0 points on Airline and +1.3 points on Retail. Although the gains are modest, these results suggest that the benefits of combining studying with interaction extend beyond AppWorld and persist with a substantially larger training pool. We will incorporate these additional experimental results and the corresponding discussion into the camera-ready version to further strengthen the empirical evidence for STAGE's generalization and scalability.


[1] From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents, Gao et al., 2026

W3. SFT Experiments

We thank the reviewer for pointing out this issue. We used the standard SFT implementation from prime-rl, and the original setting (LR = 2e-5) substantially improved training performance but degraded test performance. This prompted us to investigate the sensitivity of SFT to the learning rate.

We conducted additional experiments with lower learning rates and found that SFT achieves substantial test improvements when properly tuned, as shown below.

Train setID testOOD test
pass@4avg@4pass@4avg@4pass@4avg@4
Qwen-4B53.233.355.434.451.627.9
SFT (lr: 2e-5)72.263.951.829.928.513.2
SFT (lr: 1e-5)70.663.760.140.051.627.9
SFT (lr: 5e-6)69.163.170.847.655.435.0
STAGE (ours)60.349.873.252.164.743.2

These results suggest that the original degradation was primarily due to learning-rate sensitivity rather than an inherent limitation of SFT. Importantly, STAGE still outperforms the best-tuned SFT baseline by 4.5 and 8.2 percentage points in ID and OOD avg@4, respectively.

We acknowledge that our original SFT baseline was insufficiently tuned and that its results and corresponding discussion should be revised. We will update the manuscript accordingly. We appreciate the reviewer for identifying this important issue.


W4. Fairness of Comparison and Stronger Generator

We thank the reviewer for raising this concern. We acknowledge that generator capability affects STAGE's performance. Our primary goal is to investigate whether studying can complement trial-and-error learning, rather than claim that self-studying consistently outperforms multi-turn RL. Indeed, self-studying outperforms multi-turn RL on Test-Challenge (+5.0 avg@4), but underperforms on Test-Normal.

We use a stronger 27B generator to compensate for the 4B learner's limited task-generation capability. However, using a stronger model alone does not guarantee comparable improvements, as supported by the following comparisons:

  • Table 1: SFT using 27B-generated demonstrations, as well as world-modeling and self-reflection ([1], [2]) baselines constructed using 27B-generated trajectories, all perform below STAGE.
  • Table 8: Training on 27B-generated step-level actions from 4B trajectories also fails to achieve comparable downstream improvements.

These results suggest that how knowledge from a stronger model is transformed into learning opportunities matters, rather than simply having access to stronger supervision. Nevertheless, we acknowledge that these comparisons do not fully isolate the effect of teacher capability, and we will clarify this limitation in the revised manuscript.


[1] Agent Learning via Early Experience, Zhang et al., 2025
[2] Agentic Critical Training, Liu et al., 2026

Reviewer_ryAxOpenReview ↗

Response

Thanks for the detailed response. I believe my main concerns are still not addressed. Since these concerns affects not only the novelty but also the main experiment results, I am inclined to keep my current score and suggest substantial revision.

W1. ...The proposed "studying" method, which generates tasks targeting these weaknesses based on existing trajectories, seems conceptually very similar. It feels like the authors are introducing a new term or concept for an approach that is already common in LLM agent post-training.

I remain confused what is the contribution. You acknowledged that "the connections to prior work and do not claim that practicing individual skills or learning from QA is novel in itself". In this case what is the novelty of "Trajectory → Concept (what to learn) → Stepping-stone task (how to practice) → Single-turn RL"? My understanding is its a common practice for researchers to analyze error patterns of a given model and then convert that to training data (for SFT or RL). I believe the proposed method cannot meaningfully differentiate itself from this common approach.

W2. The proposed method is evaluated only on AppWorld.

I believe this needs to be included in the main result, as evaluating on more than one benchmark is essential to ensure the proposed method can generalize. While the new results on tau2bench is positive, it (1) omits main baselines mentioned in the paper such as world modeling and self-reflection; and (2) it only outperforms direct RL by less than 2 points - are the results statistically significant?

W3. Why does SFT degrade performance so substantially (Table 1), especially when the training data comes from the same benchmark? This is quite surprising and could indicate issues with the experimental setup, training data, or implementation in Section 4.

In the updated result table the SFT baseline increased from 34.4 (as also in Table 1) to 47.6 in ID test score, an absolute increment of 13.3 points. I believe this already causes much concern (as I originally indicated) about the rest of the baselines included in Table 1. For example, world modeling and self-reflection is also SFT based method. Since the proposed method STAGE (w/ stronger generator) only outperform these baselines by less than 5 point, it is also possible that slightly tuning the LR for these baseline can also outperform STAGE.

Reviewer_d9ouOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STAGE, a framework that enables targeted learning from agent interaction trajectories. The authors argue that conventional multi-turn reinforcement learning relies on task-level rewards, which provide limited supervision for identifying specific knowledge gaps or local decision-making errors. To address this issue, STAGE identifies knowledge and skills worth learning (concepts) from both successful and failed trajectories and transforms them into single-turn question-answering exercises, referred to as stepping-stone tasks. These tasks undergo solvability filtering, difficulty refinement, and diversity-based selection before being used for single-turn RL. The model then undergoes multi-turn RL to learn how to apply and combine the acquired capabilities within complete workflows. This process can be repeated iteratively to support continual improvement through experience. The authors evaluate STAGE on AppWorld. On Test-Challenge, STAGE improves the avg@4 score of Qwen3.5-4B from 27.9% to 43.2%, outperforming standard multi-turn RL, world modeling, self-reflection, and experience-reuse methods such as Trace2Skill, ReMe, and ACE.

Critical strengths
  1. The paper addresses an important and practical problem. By transforming local knowledge gaps and decision-making failures identified in interaction trajectories into focused training tasks, STAGE provides more direct learning feedback. This is a promising direction, particularly for tasks where successful trajectories are scarce or full environment interactions are expensive.
  2. The experimental results are strong, with attention to training compute costs. STAGE achieves substantial improvements on AppWorld, including under approximately matched compute budgets against multi-turn RL.
  3. The paper provides useful analyses of learning from failed trajectories and knowledge transfer. These experiments help demonstrate the potential of extracting useful learning signals from unsuccessful interactions and transferring locally practiced concepts to multi-turn agent behavior.
Critical weaknesses
  1. The independent contribution of the core methodological innovation is not sufficiently validated. The paper emphasizes automatically identifying what to learn from trajectories and transforming the resulting concepts into targeted stepping-stone tasks. However, the experiments primarily compare STAGE against methods with predefined auxiliary learning objectives. A more direct trajectory-to-QA baseline is missing, where single-turn QA tasks are generated directly from the same trajectories without explicit concept extraction and then used in an otherwise identical training pipeline. Without this comparison, it remains unclear whether concept extraction itself is essential to the observed improvements. This weakens the evidence supporting exploration for learning, which appears to be one of the paper's central contributions.
  2. The attribution of training improvements is confounded by differences in learning signals, raising concerns about the fairness of the baseline comparisons. Although STAGE and multi-turn RL are evaluated under approximately matched compute budgets, their training objectives and reward distributions differ substantially. STAGE explicitly selects single-turn QA tasks with intermediate success rates, making it easier to obtain informative, non-uniform rewards. In contrast, multi-turn RL on complete tasks may encounter many rollouts in which all attempts fail, resulting in little or no useful advantage signal. Consequently, part of STAGE's performance gains may come from denser and easier-to-optimize learning signals rather than from trajectory-driven knowledge discovery itself.
  3. The transferability of the acquired knowledge and skills remains unclear. Many of the generated concepts are closely tied to AppWorld-specific API semantics, data schemas, and action constraints. The observed improvements may therefore partly reflect the memorization of environment-specific knowledge in model parameters, rather than the acquisition of generalizable decision-making skills. Although the OOD results demonstrate generalization to unseen AppWorld tasks, these tasks still share the same underlying applications and API environment. The paper does not sufficiently investigate generalization to modified APIs, unseen applications, or changing environment rules. As a result, the claims regarding transferable skill acquisition and continual improvement require further validation.
Questions for authors
  1. Is explicit concept extraction necessary? Could the authors include a baseline that generates QA tasks directly from trajectories, while keeping the generator, training dataset size, solvability filtering, and RL compute budget consistent? This would help determine whether explicitly identifying what to learn provides additional benefits beyond standard trajectory-based training data generation.
  2. Have the authors evaluated the factual correctness of the automatically generated QA tasks? For tasks involving API semantics, action prerequisites, or causal attribution, could the reference answers be verified through environment execution, rule-based checks, or human annotation? In particular, how do the authors ensure that the situations constructed in modified tasks remain consistent with the actual environment constraints?
  3. How much of STAGE's improvement comes from a more favorable reward distribution? Could the authors report the proportion of groups with non-zero advantages, the reward distributions, and the proportion of effective training samples for single-turn RL and multi-turn RL? Would STAGE still demonstrate substantial gains when compared against standard local QA tasks or other dense-reward auxiliary objectives with matched difficulty distributions?
  4. To what extent do the generated concepts represent environment-specific knowledge versus generalizable decision-making skills? A breakdown of the generated concepts and their contributions could help distinguish environment-specific knowledge memorization from the acquisition of transferable skills.
4 comments

Dear Reviewer d9ou,

Thank you for your deep understanding of our work, and appreciate the interesting question you raised. We would like to address the concerns in the comments below:


W1/Q1. Effect of Exploration for Learning

We thank the reviewer for this important suggestion. We agree that the independent contribution of explicit concept extraction, the core of exploration for learning, should be more clearly validated.

To address this, we conducted an additional ablation (STAGE w/o concept extraction) that generates QA tasks directly from the same trajectories, without explicitly identifying concepts. We kept the generator, training dataset size, solvability filtering, and RL compute budget identical to STAGE for a controlled comparison.

ID testOOD test
pass@4avg@4pass@4avg@4
Qwen3.5-4B55.434.451.627.9
STAGE73.252.164.743.2
STAGE w/o concept extraction64.942.459.737.5

Key findings:

  • Direct trajectory-to-QA generation is beneficial, improving OOD avg@4 from 27.9% to 37.5%, even without concept extraction.
  • Explicit concept extraction provides substantial additional gains, improving ID and OOD avg@4 by 9.7%p and 5.7%p, respectively, over direct QA generation.

These results demonstrate that STAGE's improvements are not solely attributable to generating single-turn QA tasks from trajectories. Rather, explicitly identifying what to learn before generating how to practice contributes substantially to downstream agent performance, providing direct empirical support for exploration for learning.


Q2. Quality of QA Tasks

We thank the reviewer for raising this important concern. Unlike unconstrained synthetic task generation, STAGE grounds its QA tasks in observed interaction trajectories and requires supporting evidence, including for modified scenarios. However, we agree that trajectory grounding alone does not guarantee factual correctness.

To assess the quality of generated tasks, we conducted an additional LLM-as-a-judge analysis, separately evaluating question and reference-answer correctness.

QuestionAnswerBoth
Iter 198.3%86.0%86.0%
iter 291.3%76.3%75.0%
iter 393.0%81.0%79.3%
Reasoning ErrorsEnvironment Understanding ErrorsQuestion Construction ErrorsInsufficient Answers
iter147.6%33.3%14.3%4.8%
iter226.7%53.3%9.3%10.7%
iter327.4%53.2%8.1%11.3%

Key observations:

  • Question quality remains high, while reference-answer correctness is relatively lower.
  • Error analysis reveals that reasoning and environment-understanding errors account for the majority of identified errors, suggesting that reference-answer verification is particularly important.
  • Joint correctness is highest in Iter 1 and decreases in later iterations, indicating room for improving task quality during iterative studying.

Importantly, despite these imperfections, STAGE already demonstrates meaningful learning benefits, improving Test-Challenge avg@4 from 27.9% to 43.2%. This suggests that trajectory-grounded exercises can provide useful learning signals even without explicit factual verification.

We believe that incorporating an additional quality-verification step, such as environment execution, rule-based validation, or targeted human review, could further improve the reliability of generated tasks and enable more robust studying, particularly across repeated iterations. We view this as a promising extension rather than a prerequisite for obtaining useful learning benefits from the current framework.


W2/Q3. Contributions of Learning Signals and Knowledge Discovery

We agree that obtaining more informative learning signals is an intended benefit of STAGE, as it transforms sparse trajectory-level feedback into focused, learnable exercises. Our results provide three supporting observations:

  • Solvability filtering (Table 2): Removing solvability filtering reduces ID avg@4 from 52.1% to 36.9% and OOD avg@4 from 43.2% to 39.7%, highlighting the importance of selecting learnable tasks.
  • Concept extraction (W1/Q1): Direct trajectory-to-QA generation improves OOD avg@4 from 27.9% to 37.5%, while explicit concept extraction further improves it to 43.2%, demonstrating the additional benefit of trajectory-driven knowledge discovery.
  • Sparse-feedback settings (Figure 3): STAGE achieves meaningful gains even from failure-only trajectories (OOD avg@4: 27.9% → 37.2%) and outperforms interaction-only RL when trained on Train-Hard, where successful rollouts are scarce.

Together, these findings suggest that STAGE not only provides informative learning signals but also discovers valuable learning opportunities from unsuccessful experiences. We will further illustrate its optimization behavior through additional training-dynamics plots in the revised appendix.


W3/Q4.

We thank the reviewer for this important concern. To distinguish environment-specific knowledge from generalizable decision-making skills, we conducted an LLM-based analysis classifying the extracted concepts into these two categories.

Environment-specific knowledgeGeneralizable decision-making skills
Iter 150.3%49.7%
Iter 244.3%55.7%
Iter 345.0%55.0%

Our findings highlight two complementary benefits of studying:

  • Environment-specific knowledge acquisition: Figure 6(c) shows that error types more frequently targeted by stepping-stone tasks exhibit larger reductions, suggesting that studying facilitates effective domain adaptation.
  • Transferable decision-making skills: Approximately half of the extracted concepts target generalizable skills. To examine their potential transfer, we directly evaluated the AppWorld-trained STAGE model on τ²-bench without any additional training or fine-tuning.
Tau2 (arline)Tau2 (retail)
Qwen3.5-4B0.5070.787
STAGE (training on AppWorld)0.5320.800

Despite being trained exclusively on AppWorld, STAGE improves performance on both unseen environments (+2.5 points on Airline and +1.3 points on Retail). These results suggest that studying can acquire capabilities that extend beyond memorizing AppWorld-specific API semantics, although isolating the contributions of individual concept categories requires further investigation.

We will incorporate these analyses and cross-environment transfer results into the camera-ready version.

Dear Reviewer d9ou,

Thank you again for your time and effort to provide your insightful feedback on our paper. If you have any remaining questions or require further clarification, we would be happy to address them before the time window closes.

Thank you so much for your time and valuable feedback.

Thank you for the detailed rebuttal and additional experiments. The new ablation study and τ²-bench results provide encouraging evidence for the contribution of Concept Extraction and its transferability. However, some concerns are not fully resolved, particularly regarding the lack of direct validation of QA correctness and a more detailed analysis of reward distributions.

Overall, the authors have addressed most of my concerns. I will maintain my positive rating

Reviewer_KoHeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new method to fine-tune agents by extracting key concepts from trajectories and generating questions based on these concepts. Instead of directly learning from sparse rewards or long trajectories, the agent learns from these generated questions. The method shows improvements over SFT and multi-turn RL baselines on AppWorld.

Critical strengths
  1. The motivation of this paper is clear. The authors identify some limitations of existing SFT and RL methods, such as sparse rewards and long trajectories. They propose to extract useful knowledge from trajectories and use it to fine-tune the agent. I think this is a reasonable motivation.
  2. The paper is well presented and easy to follow. The authors provide detailed explanations of their method, including implementation details, prompts, and hyperparameters. The experimental results also show improvements over several baselines.
Critical weaknesses

I have a few concerns about the novelty and evaluation of this paper:

  1. The main concepts of this paper, such as "studying" and "stepping-stones", are not clearly defined. This makes me confused about the actual novelty of this work. My question is: what is the difference between "studying" and normal learning? More importantly, how is learning from "stepping-stones" different from existing methods such as curriculum learning, skill learning, or fine-tuning with synthetic questions? I think the authors should explain these differences properly. Also, the mathematical formulation and the examples in Table 6 and Figure 10 do not clearly address this question.
  2. The evaluation is limited to only one benchmark, AppWorld. However, the proposed method seems to be quite general and should also work for other agent tasks. Why do the authors only evaluate it on AppWorld? I think the authors should provide more experiments on other benchmarks, such as ALFWorld, TextWorld, MATH, or SWE-bench, to demonstrate that the proposed method can generalize to different tasks. It is also unclear whether the proposed stepping-stone generation and selection strategy can work equally well in other domains.
Questions for authors

I've asked my main questions in the weaknesses. Please clarify the novelty of the proposed method compared to existing learning methods, and provide more discussion or experiments to support its generalization beyond AppWorld.

2 comments

Dear Reviewer KoHe,

We appreciate your comments and positive feedback on our work. We will address the concerns raised by the reviewer in the comments below:


W1.

Thank you for raising this important point. We clarify the definitions of studying and stepping-stone tasks, and how STAGE differs from existing approaches.

1. What are studying and stepping-stone tasks?

  • Studying is not a new optimization algorithm, but a dedicated learning phase that identifies what to learn from agent trajectories and provides focused opportunities to practice it.
  • Stepping-stone tasks are single-turn QA exercises that isolate specific knowledge or decision-making skills from trajectories, allowing agents to practice them with direct feedback without replaying the full interaction.

For example, when an agent accidentally deletes existing note content using the update_note API, STAGE identifies the underlying concept (the API replaces the entire note) and generates a QA exercise requiring the agent to preserve unchanged content.

2. How does STAGE differ from existing approaches?

We acknowledge the connections to prior work and do not claim that practicing individual skills or learning from QA is novel in itself.

  • Curriculum learning: While curriculum learning organizes or generates learning tasks, STAGE focuses on first discovering what needs to be learned from successful and failed agent trajectories.
  • Skill learning: Unlike context-based skill reuse methods, STAGE converts extracted knowledge and skills into targeted exercises for updating model parameters.
  • Synthetic QA fine-tuning: Rather than directly generating QA from trajectories, STAGE explicitly separates what to learn (concept extraction) from how to practice (concept-conditioned QA generation).

3. What is the key contribution?

STAGE introduces a structured learning process:

Trajectory → Concept (what to learn) → Stepping-stone task (how to practice) → Single-turn RL → Multi-turn RL

The central idea is that agents should not rely solely on trial-and-error interaction, but can also benefit from explicitly studying the knowledge and skills revealed by their experience. Single-turn RL develops these targeted capabilities, while subsequent multi-turn RL learns to apply and combine them in complete workflows.

To further validate the contribution of explicit concept extraction (i.e., exploration for learning), we conducted an ablation that directly generates QA tasks from the same trajectories, while keeping the generator, dataset size, solvability filtering, and RL training budget identical. As shown above, removing concept extraction reduces ID avg@4 from 52.1% to 42.4% and OOD avg@4 from 43.2% to 37.5%. While direct trajectory-to-QA generation itself improves performance over the base model, explicitly identifying what to learn before constructing how to practice provides substantial additional gains. These results provide direct empirical evidence that concept extraction contributes beyond simply generating synthetic QA tasks from trajectories, supporting our central motivation of exploration for learning.

ID testOOD test
pass@4avg@4pass@4avg@4
Qwen3.5-4B55.434.451.627.9
STAGE73.252.164.743.2
STAGE w/o concept extraction64.942.459.737.5

W2.

We thank the reviewer for highlighting the importance of evaluating STAGE beyond AppWorld. We initially focused on AppWorld because it provides complex, long-horizon workflows involving multiple applications and APIs. To further assess generalization, we conducted additional experiments on τ²-bench, following [1]. Specifically, we use 800 tasks from the Airline training split for training, the remaining Airline tasks for ID evaluation, and the Retail domain for OOD evaluation.

ID test (airline)OOD test (retail)
Qwen3.5-4B50.778.7
Multi-turn RL61.578.7
Studying only55.277.5
STAGE (Studying + Multi-turn RL)63.580.0

STAGE outperforms multi-turn RL on both ID (+2.0 points) and OOD (+1.3 points) evaluations. Although these gains are modest, they provide additional evidence that trajectory-driven studying can complement interaction-based learning beyond AppWorld. In particular, the OOD results suggest that capabilities acquired through Airline trajectories can benefit unseen tasks in the Retail domain. We will incorporate these results and the corresponding discussion into the camera-ready version.


[1] From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents, Gao et al., 2026

Dear Reviewer KoHe,

Thank you again for your time and effort to provide your insightful feedback on our paper. If you have any remaining questions or require further clarification, we would be happy to address them before the time window closes.

Thank you so much for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.