- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposed "STAGE", a method to automatically curate smaller/simpler training tasks from a target models' rollout, perform a round of RL to improve the model on these targeted capabilities, and finally perform standard multi-turn RL. The authors show that under a similar training budget this outperforms directly performing multi-turn RL, which indicates the importance of improving the model prior before RL. The authors mainly experimented on APPWorld with 126 training tasks and conducted numerous analysis such as learning from mostly failed trajectories or iterative using STAGE.
Critical strengths
- The authors showed positive result on APPWorld where STAGE(ours, self) outperforms simple multit-turn RL on Test-Challenge.
- Interesting analysis including training with mostly failed experiences (section 4.3, which I believe mimics the idea of offline RL) and running STAGE for multiple iterations (section 5).
Critical weaknesses
-
The overall idea seems relatively simple: the authors aim to give the agent a stronger prior (L84) to improve the effectiveness of standard multi-turn RL. A common approach to achieving this is to perform SFT on high-quality synthetic or human-annotated data that targets the missing or weak capabilities of the base model. The proposed "studying" method, which generates tasks targeting these weaknesses based on existing trajectories, seems conceptually very similar. It feels like the authors are introducing a new term or concept for an approach that is already common in LLM agent post-training.
-
The proposed method is evaluated only on AppWorld. It is therefore unclear whether the method generalizes to other benchmarks or domains, such as coding or web/computer-use.
-
Why does SFT degrade performance so substantially (Table 1), especially when the training data comes from the same benchmark? This is quite surprising and could indicate issues with the experimental setup, training data, or implementation in Section 4.
-
STAGE results in Table 1 seems to be unfair. Specifically, without accessing any stronger LLMs STAGE (ours, self) underperforms multi-turn RL in Test-Normal. It is only when STAGE uses a stronger model (Qwen3.8-27B) for data generation it outperforms multi-turn RL. I believe this is not an apple-to-apple comparison.
-
The training pool consists of only 126 tasks. I believe this is a relatively weak setup for RL (especially for multi-turn agent tasks used in this work), and the conclusions about the effectiveness of baseline methods such as direct GRPO/RL may not be reliable. With such a limited number of training tasks, it is possible that the gains of the proposed method (STAGE) over standard RL methods would diminish if more training data were available.
Questions for authors
please see weaknesses
6 comments
W3. SFT Experiments
We thank the reviewer for pointing out this issue. We used the standard SFT implementation from prime-rl, and the original setting (LR = 2e-5) substantially improved training performance but degraded test performance. This prompted us to investigate the sensitivity of SFT to the learning rate.
We conducted additional experiments with lower learning rates and found that SFT achieves substantial test improvements when properly tuned, as shown below.
| Train set | ID test | OOD test | ||||
|---|---|---|---|---|---|---|
| pass@4 | avg@4 | pass@4 | avg@4 | pass@4 | avg@4 | |
| Qwen-4B | 53.2 | 33.3 | 55.4 | 34.4 | 51.6 | 27.9 |
| SFT (lr: 2e-5) | 72.2 | 63.9 | 51.8 | 29.9 | 28.5 | 13.2 |
| SFT (lr: 1e-5) | 70.6 | 63.7 | 60.1 | 40.0 | 51.6 | 27.9 |
| SFT (lr: 5e-6) | 69.1 | 63.1 | 70.8 | 47.6 | 55.4 | 35.0 |
| STAGE (ours) | 60.3 | 49.8 | 73.2 | 52.1 | 64.7 | 43.2 |
These results suggest that the original degradation was primarily due to learning-rate sensitivity rather than an inherent limitation of SFT. Importantly, STAGE still outperforms the best-tuned SFT baseline by 4.5 and 8.2 percentage points in ID and OOD avg@4, respectively.
We acknowledge that our original SFT baseline was insufficiently tuned and that its results and corresponding discussion should be revised. We will update the manuscript accordingly. We appreciate the reviewer for identifying this important issue.
W4. Fairness of Comparison and Stronger Generator
We thank the reviewer for raising this concern. We acknowledge that generator capability affects STAGE's performance. Our primary goal is to investigate whether studying can complement trial-and-error learning, rather than claim that self-studying consistently outperforms multi-turn RL. Indeed, self-studying outperforms multi-turn RL on Test-Challenge (+5.0 avg@4), but underperforms on Test-Normal.
We use a stronger 27B generator to compensate for the 4B learner's limited task-generation capability. However, using a stronger model alone does not guarantee comparable improvements, as supported by the following comparisons:
- Table 1: SFT using 27B-generated demonstrations, as well as world-modeling and self-reflection ([1], [2]) baselines constructed using 27B-generated trajectories, all perform below STAGE.
- Table 8: Training on 27B-generated step-level actions from 4B trajectories also fails to achieve comparable downstream improvements.
These results suggest that how knowledge from a stronger model is transformed into learning opportunities matters, rather than simply having access to stronger supervision. Nevertheless, we acknowledge that these comparisons do not fully isolate the effect of teacher capability, and we will clarify this limitation in the revised manuscript.
[1] Agent Learning via Early Experience, Zhang et al., 2025
[2] Agentic Critical Training, Liu et al., 2026
Response
Thanks for the detailed response. I believe my main concerns are still not addressed. Since these concerns affects not only the novelty but also the main experiment results, I am inclined to keep my current score and suggest substantial revision.
W1. ...The proposed "studying" method, which generates tasks targeting these weaknesses based on existing trajectories, seems conceptually very similar. It feels like the authors are introducing a new term or concept for an approach that is already common in LLM agent post-training.
I remain confused what is the contribution. You acknowledged that "the connections to prior work and do not claim that practicing individual skills or learning from QA is novel in itself". In this case what is the novelty of "Trajectory → Concept (what to learn) → Stepping-stone task (how to practice) → Single-turn RL"? My understanding is its a common practice for researchers to analyze error patterns of a given model and then convert that to training data (for SFT or RL). I believe the proposed method cannot meaningfully differentiate itself from this common approach.
W2. The proposed method is evaluated only on AppWorld.
I believe this needs to be included in the main result, as evaluating on more than one benchmark is essential to ensure the proposed method can generalize. While the new results on tau2bench is positive, it (1) omits main baselines mentioned in the paper such as world modeling and self-reflection; and (2) it only outperforms direct RL by less than 2 points - are the results statistically significant?
W3. Why does SFT degrade performance so substantially (Table 1), especially when the training data comes from the same benchmark? This is quite surprising and could indicate issues with the experimental setup, training data, or implementation in Section 4.
In the updated result table the SFT baseline increased from 34.4 (as also in Table 1) to 47.6 in ID test score, an absolute increment of 13.3 points. I believe this already causes much concern (as I originally indicated) about the rest of the baselines included in Table 1. For example, world modeling and self-reflection is also SFT based method. Since the proposed method STAGE (w/ stronger generator) only outperform these baselines by less than 5 point, it is also possible that slightly tuning the LR for these baseline can also outperform STAGE.
Dear Reviewer ryAx,
We appreciate your thoughtful review and recognition of our efforts. We hope that the following responses help clarify the reviewer’s questions:
W1.
Thank you for raising this important point. We clarify the definitions of studying and stepping-stone tasks, and how STAGE differs from existing approaches.
1. What are studying and stepping-stone tasks?
For example, when an agent accidentally deletes existing note content using the
update_noteAPI, STAGE identifies the underlying concept (the API replaces the entire note) and generates a QA exercise requiring the agent to preserve unchanged content.2. How does STAGE differ from existing approaches?
We acknowledge the connections to prior work and do not claim that practicing individual skills or learning from QA is novel in itself.
3. What is the key contribution?
STAGE introduces a structured learning process:
The central idea is that agents should not rely solely on trial-and-error interaction, but can also benefit from explicitly studying the knowledge and skills revealed by their experience. Single-turn RL develops these targeted capabilities, while subsequent multi-turn RL learns to apply and combine them in complete workflows.
W2/W5. Generalization Beyond AppWorld and Larger-Scale Training
We thank the reviewer for raising concerns about generalization beyond AppWorld and the limited training pool of 126 tasks. To address both concerns, we conducted additional experiments on τ²-bench, following the experimental setup of AReaL-SEA [1]. Specifically, we use 800 tasks from the Airline training split for training, the remaining Airline tasks for ID evaluation, and the Retail domain for OOD evaluation.
STAGE consistently outperforms multi-turn RL, achieving improvements of +2.0 points on Airline and +1.3 points on Retail. Although the gains are modest, these results suggest that the benefits of combining studying with interaction extend beyond AppWorld and persist with a substantially larger training pool. We will incorporate these additional experimental results and the corresponding discussion into the camera-ready version to further strengthen the empirical evidence for STAGE's generalization and scalability.
[1] From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents, Gao et al., 2026