- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper identifies a selection bias in existing agent-training pipelines: tasks are often constructed retrospectively from executable tool-call trajectories or successful workflows, resulting in training data dominated by user requests that can be fulfilled as stated and underrepresenting requests that conflict with the environment’s state or constraints. To address this issue, the authors propose scenario co-design and instantiate it in ReAgent. Starting from conflicts grounded in programmatic environments and their valid resolutions, ReAgent jointly constructs task requests, simulated-user playbooks, and evaluation criteria. It then uses execution-based testing and repair to curate environment and reward code, followed by a two-stage reinforcement-learning curriculum that trains tool use before multi-turn interaction.
Critical strengths
The paper identifies a consequential training-data bias: generating tasks from successful workflows systematically underrepresents requests that conflict with environmental constraints. This deployment-relevant observation provides a strong motivation for the work.
Critical weaknesses
-
ReAgent is primarily an engineering framework that integrates familiar techniques into an effective pipeline, but offers limited methodological novelty.
-
The ablations support the effectiveness of the overall pipeline, but do not isolate the benefit of scenario co-design over separately generating and aligning tasks, users, and rewards.
Questions for authors
-
How does zeroing the reward for any failed hard criterion compare with a graded penalty, especially for partially resolvable requests? Is this approach sensitive to evaluator noise?
-
Does the multi-step to multi-turn curriculum outperform direct or mixed multi-turn training when compute and data exposure are matched?
5 comments
Question 1 & Question 2
Q1. Questions on Rewards
Q1.1 Hard Gating and Graded Rewards
Our current reward design has already combined hard gating with graded rewards (Eq. 5). A failed hard criterion sets the reward to zero, preventing critical violations, such as executing a strictly prohibited operation, from being offset by success on other subgoals. Once all hard criteria pass, the criterion outcomes are averaged, allowing satisfaction of non-hard criteria to provide graded feedback on different degrees of completion.
For partially resolvable requests, hard gating constrains unacceptable actions, while graded rewards reflect the completion of the feasible parts. Together, they allow the model to earn corresponding rewards for completing those parts while respecting the constraints. In Table 2, ReAgent-8B trained with this reward achieves 38.8 on PARTIAL requests, exceeding 33.9 for the strongest evaluated 8B baseline, supporting the applicability of this design to partially resolvable requests.
Q1.2 Sensitivity to Evaluator Noise
We evaluated the judgement eliability of the two reward types separately.
- Reward functions. These functions directly check the terminal environment state through code, giving deterministic, reproducible scores.
- Rubrics. Rubrics express environment facts and constraints as explicit criteria that can be checked individually, helping improve the accuracy of LLM-as-judge evaluations [4]. This is supported by Figure 5(b), which reports a rubric-level accuracy of approximately 90%. Considering that a single hard-rubric error can zero the reward for an entire trajectory, we also measured the trajectory-level false hard-zero rate in Appendix C.3, which is only 1.8%–5.6%. This indicates that rubric misjudgments rarely cause valid trajectories to lose their entire reward.
Q2. Multi-Step-to-Multi-Turn Curriculum
(1) Our stage order is motivated by early budget-matched experiments and their behavioral analysis. Starting from the same Qwen3-8B-Base, we compared (1) Multi-Step RL alone with (2) mixed Multi-Step and Multi-Turn RL, using the same training budget and data exposure. The results are Avg@4 percentages.
| Training setting | τ²-Bench Retail | τ²-Bench Airline |
|---|---|---|
| Multi-Step RL | 44.74 | 34.00 |
| Mixed MS + MT RL | 34.65 (-10.09) | 38.00 (+4.00) |
Mixed training improves Airline by 4.00 points, showing the value of Multi-Turn training, but reduces Retail by 10.09 points. Further analysis found that, on Retail, failures due to exceeding the interaction turn limit after repeatedly asking users for information they did not know increased from 16 to 80. In 27 of these trajectories, the agent had already retrieved an item list through tools but asked the user to identify the target item instead of using the appropriate tools to query and match it.
These observations suggest that introducing user communication before establishing tool-querying skills can encourage agents to replace available environment queries with questions to the user, increasing unnecessary questioning. They motivated us to first develop the agent's ability to query tools and use environment information through Multi-Step RL, then introduce user communication and conflict resolution through Multi-Turn RL. We therefore adopt this Multi-Step-to-Multi-Turn curriculum, rather than direct or mixed multi-turn training.
(2) Training efficiency also informs the stage order. Multi-Turn rollouts require additional user-simulator calls and generally involve more interaction turns (Figure 4). In synchronous batches, mixing Multi-Step and Multi-Turn tasks can increase waiting for slower rollouts. The staged curriculum postpones user-simulation overhead until the Multi-Turn stage and helps reduce waiting caused by mixing the two rollout types.
We will include the early comparison and error analysis alongside Table 3 to explain the empirical motivation and observed stage-wise gains of the curriculum.
References
[1] Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents.
[2] Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards.
[3] AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Use.
[4] Gunjal et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. ICLR 2026.
Follow-up on Rebuttal for Paper 27914
We sincerely thank you once again for your time and valuable feedback on Paper 27914. We have provided detailed, point-by-point responses to all of your comments. We kindly and respectfully ask that you reconsider our responses and reevaluate the score , especially in light of your positive remarks regarding the technical contribution and motivation.
If you have any remaining concerns, please do not hesitate to let us know — we would be happy to address them.
Weakness 1 & Weakness 2
We sincerely thank the reviewer for recognizing the consequential training-data bias, the effective pipeline, and the evidence provided by the overall ablations. We hope the clarifications and additional evidence below address your concerns.
W1. Methodological Contribution
Our key methodological contribution is the construction of training scenarios for conflict resolution, which addresses the underrepresentation of user–environment conflicts in existing construction pipelines. To the best of our knowledge, this is the first work to automatically synthesize scalable conflict scenarios covering execution, guidance toward valid revisions, and refusal in programmatic environments and use them for Agentic RL training. Established techniques such as user simulation serve as building blocks for implementing such construction, rather than being proposed as new foundational algorithms.
Following Reviewer MDP2's suggestion, we further clarify the novelty through comparisons with the related work.
Trajectory2Task [1] reuses the tools and environments of τ²-Bench, infers ambiguous or infeasible user requests from execution trajectories, and collects successful trajectories for SFT. Its focus is SFT data synthesis within existing environments, while ReAgent constructs scenarios for Agentic RL training.
SynthAgent [2] conducts Agentic RL in LLM-simulated environments. However, the paper also notes that inconsistent responses from such environments may affect training stability. In contrast, we synthesize programmatic environments whose tools execute through deterministic code, providing stable environment feedback for training.
AgenticQwen [3] synthesizes scenarios with adversarial users to train agents to resist attempts to induce rule violations. We instead focus on conflicting requests from real users who lack knowledge of environment states or constraints, training agents to guide users toward feasible revisions and appropriately refuse parts that cannot be fulfilled, enabling more flexible handling of real-world scenarios.
We have compared ReAgent-8B directly with AgenticQwen-8B, the closest of these baselines. In Table 1, the average scores of ReAgent-8B are significantly higher on all benchmarks. In Table 2, the overall score for PARTIAL requests, which require revision or partial completion, increases from 29.3 to 38.8. FEASIBLE and BLOCKED scores also improve. These results further support ReAgent's value in training agents to resolve conflicts between real user requests and environment states or constraints.
W2. Benefit of Scenario Co-Design
Thank you for asking us to isolate the benefit of scenario co-design over separately generating and aligning tasks, users, and rewards. ReAgent jointly constructs the task, user playbook, and oracle around the same conflict and valid resolution (§3.1, Eq. 3). The rationale is to consider interdependent conditions, including how the user cooperates and how the reward evaluates the resolution, together during generation. This aligns the interaction design with the evaluation criteria from the outset and aims to improve the training quality of the synthesized scenarios.
Following Reviewer QunN's concrete proposal, we implemented a matched comparison with separate generation and alignment. We first generated the conflict task, then independently generated user behavior and reward criteria from the environment and task, and finally applied the same consistency checks. We replaced only the co-designed conflict scenarios with an equal number of scenarios constructed through this separate route. The feasible scenarios, model initialization, training budget, data amount, and other training and evaluation settings were kept fixed. Both variants use the same Qwen3-8B backbone. The scores (%) are as follows.
Co-design achieves higher scores on all six metrics even when the separately generated scenarios undergo the same consistency checks. With the alignment procedure and subsequent training held fixed, the comparison supports the additional training value of jointly specifying user response conditions and evaluation requirements during scenario construction. These results are consistent with our rationale that considering the interdependent components together improves the training quality of conflict scenarios. We will include this matched construction baseline and its results in the revision.