acceptodds
Under review as a conference paper at ICLR 2027

ReAgent: Rethinking Agent Training through Scenario Co-Design

Abstract

Agentic reinforcement learning scales training by expanding programmatic environments and synthesizing diverse tasks. Yet this scaling can paradoxically narrow training coverage through how data are constructed. To make such synthesis tractable at scale, existing pipelines often derive tasks from executable paths or successful workflows, thereby inducing a construction-induced selection bias toward feasible requests. Real users, however, state goals without full visibility into the environment state or its constraints, so agents must identify conflicts, guide users toward feasible revisions, and refuse when necessary. Such resolution behaviors cannot be captured by task synthesis alone, as conflicts and valid resolutions are co-constructed by environment constraints, user intent and appropriate reward. This interdependence motivates us to rethink agent training by shifting the unit of construction from feasible tasks alone to \itshape the whole training scenario (i.e., task, user, and reward together). We call this principle scenario co-design and instantiate it through ReAgent. Starting from conflicts and resolutions grounded in the environment, ReAgent jointly constructs the task request, simulated user behavior, and reward criteria as a coherent conflict scenario, enabling scalable training with broader coverage of feasible, blocked, and partially resolvable requests. Empirically, augmenting training with these co-designed conflict scenarios not only directly improves Qwen3-8B and Qwen3-30B-A3B-Thinking-2507 on constraint-governed agent benchmarks such as -Bench but also even enables them to outperform competitive baselines across general agent benchmarks. These results suggest that broadening training coverage within environments through scenario co-design offers a promising direction for scaling agent training beyond simply expanding environments and feasible tasks.

Primary area:
reinforcement learning
Keywords:
Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gGkxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper identifies a selection bias in existing agent-training pipelines: tasks are often constructed retrospectively from executable tool-call trajectories or successful workflows, resulting in training data dominated by user requests that can be fulfilled as stated and underrepresenting requests that conflict with the environment’s state or constraints. To address this issue, the authors propose scenario co-design and instantiate it in ReAgent. Starting from conflicts grounded in programmatic environments and their valid resolutions, ReAgent jointly constructs task requests, simulated-user playbooks, and evaluation criteria. It then uses execution-based testing and repair to curate environment and reward code, followed by a two-stage reinforcement-learning curriculum that trains tool use before multi-turn interaction.

Critical strengths

The paper identifies a consequential training-data bias: generating tasks from successful workflows systematically underrepresents requests that conflict with environmental constraints. This deployment-relevant observation provides a strong motivation for the work.

Critical weaknesses
  1. ReAgent is primarily an engineering framework that integrates familiar techniques into an effective pipeline, but offers limited methodological novelty.

  2. The ablations support the effectiveness of the overall pipeline, but do not isolate the benefit of scenario co-design over separately generating and aligning tasks, users, and rewards.

Questions for authors
  1. How does zeroing the reward for any failed hard criterion compare with a graded penalty, especially for partially resolvable requests? Is this approach sensitive to evaluator noise?

  2. Does the multi-step to multi-turn curriculum outperform direct or mixed multi-turn training when compute and data exposure are matched?

5 comments
AuthorsOpenReview ↗

Weakness 1 & Weakness 2

We sincerely thank the reviewer for recognizing the consequential training-data bias, the effective pipeline, and the evidence provided by the overall ablations. We hope the clarifications and additional evidence below address your concerns.


W1. Methodological Contribution

Our key methodological contribution is the construction of training scenarios for conflict resolution, which addresses the underrepresentation of user–environment conflicts in existing construction pipelines. To the best of our knowledge, this is the first work to automatically synthesize scalable conflict scenarios covering execution, guidance toward valid revisions, and refusal in programmatic environments and use them for Agentic RL training. Established techniques such as user simulation serve as building blocks for implementing such construction, rather than being proposed as new foundational algorithms.

Following Reviewer MDP2's suggestion, we further clarify the novelty through comparisons with the related work.

  1. Trajectory2Task [1] reuses the tools and environments of τ²-Bench, infers ambiguous or infeasible user requests from execution trajectories, and collects successful trajectories for SFT. Its focus is SFT data synthesis within existing environments, while ReAgent constructs scenarios for Agentic RL training.

  2. SynthAgent [2] conducts Agentic RL in LLM-simulated environments. However, the paper also notes that inconsistent responses from such environments may affect training stability. In contrast, we synthesize programmatic environments whose tools execute through deterministic code, providing stable environment feedback for training.

  3. AgenticQwen [3] synthesizes scenarios with adversarial users to train agents to resist attempts to induce rule violations. We instead focus on conflicting requests from real users who lack knowledge of environment states or constraints, training agents to guide users toward feasible revisions and appropriately refuse parts that cannot be fulfilled, enabling more flexible handling of real-world scenarios.

We have compared ReAgent-8B directly with AgenticQwen-8B, the closest of these baselines. In Table 1, the average scores of ReAgent-8B are significantly higher on all benchmarks. In Table 2, the overall score for PARTIAL requests, which require revision or partial completion, increases from 29.3 to 38.8. FEASIBLE and BLOCKED scores also improve. These results further support ReAgent's value in training agents to resolve conflicts between real user requests and environment states or constraints.


W2. Benefit of Scenario Co-Design

Thank you for asking us to isolate the benefit of scenario co-design over separately generating and aligning tasks, users, and rewards. ReAgent jointly constructs the task, user playbook, and oracle around the same conflict and valid resolution (§3.1, Eq. 3). The rationale is to consider interdependent conditions, including how the user cooperates and how the reward evaluates the resolution, together during generation. This aligns the interaction design with the evaluation criteria from the outset and aims to improve the training quality of the synthesized scenarios.

Following Reviewer QunN's concrete proposal, we implemented a matched comparison with separate generation and alignment. We first generated the conflict task, then independently generated user behavior and reward criteria from the environment and task, and finally applied the same consistency checks. We replaced only the co-designed conflict scenarios with an equal number of scenarios constructed through this separate route. The feasible scenarios, model initialization, training budget, data amount, and other training and evaluation settings were kept fixed. Both variants use the same Qwen3-8B backbone. The scores (%) are as follows.

BenchmarkSeparate generation + alignmentReAgent (co-design)
ACEBench-Agent Multi-Step93.7596.30
ACEBench-Agent Multi-Turn61.6765.80
τ-Bench Retail59.7163.10
τ-Bench Airline38.0041.21
BFCL-V4 Multi-Turn50.0050.80
BFCL-V4 Memory26.0228.90

Co-design achieves higher scores on all six metrics even when the separately generated scenarios undergo the same consistency checks. With the alignment procedure and subsequent training held fixed, the comparison supports the additional training value of jointly specifying user response conditions and evaluation requirements during scenario construction. These results are consistent with our rationale that considering the interdependent components together improves the training quality of conflict scenarios. We will include this matched construction baseline and its results in the revision.

AuthorsOpenReview ↗

Question 1 & Question 2

Q1. Questions on Rewards

Q1.1 Hard Gating and Graded Rewards

Our current reward design has already combined hard gating with graded rewards (Eq. 5). A failed hard criterion sets the reward to zero, preventing critical violations, such as executing a strictly prohibited operation, from being offset by success on other subgoals. Once all hard criteria pass, the criterion outcomes are averaged, allowing satisfaction of non-hard criteria to provide graded feedback on different degrees of completion.

For partially resolvable requests, hard gating constrains unacceptable actions, while graded rewards reflect the completion of the feasible parts. Together, they allow the model to earn corresponding rewards for completing those parts while respecting the constraints. In Table 2, ReAgent-8B trained with this reward achieves 38.8 on PARTIAL requests, exceeding 33.9 for the strongest evaluated 8B baseline, supporting the applicability of this design to partially resolvable requests.

Q1.2 Sensitivity to Evaluator Noise

We evaluated the judgement eliability of the two reward types separately.

  1. Reward functions. These functions directly check the terminal environment state through code, giving deterministic, reproducible scores.
  2. Rubrics. Rubrics express environment facts and constraints as explicit criteria that can be checked individually, helping improve the accuracy of LLM-as-judge evaluations [4]. This is supported by Figure 5(b), which reports a rubric-level accuracy of approximately 90%. Considering that a single hard-rubric error can zero the reward for an entire trajectory, we also measured the trajectory-level false hard-zero rate in Appendix C.3, which is only 1.8%–5.6%. This indicates that rubric misjudgments rarely cause valid trajectories to lose their entire reward.

Q2. Multi-Step-to-Multi-Turn Curriculum

(1) Our stage order is motivated by early budget-matched experiments and their behavioral analysis. Starting from the same Qwen3-8B-Base, we compared (1) Multi-Step RL alone with (2) mixed Multi-Step and Multi-Turn RL, using the same training budget and data exposure. The results are Avg@4 percentages.

Training settingτ²-Bench Retailτ²-Bench Airline
Multi-Step RL44.7434.00
Mixed MS + MT RL34.65 (-10.09)38.00 (+4.00)

Mixed training improves Airline by 4.00 points, showing the value of Multi-Turn training, but reduces Retail by 10.09 points. Further analysis found that, on Retail, failures due to exceeding the interaction turn limit after repeatedly asking users for information they did not know increased from 16 to 80. In 27 of these trajectories, the agent had already retrieved an item list through tools but asked the user to identify the target item instead of using the appropriate tools to query and match it.

These observations suggest that introducing user communication before establishing tool-querying skills can encourage agents to replace available environment queries with questions to the user, increasing unnecessary questioning. They motivated us to first develop the agent's ability to query tools and use environment information through Multi-Step RL, then introduce user communication and conflict resolution through Multi-Turn RL. We therefore adopt this Multi-Step-to-Multi-Turn curriculum, rather than direct or mixed multi-turn training.

(2) Training efficiency also informs the stage order. Multi-Turn rollouts require additional user-simulator calls and generally involve more interaction turns (Figure 4). In synchronous batches, mixing Multi-Step and Multi-Turn tasks can increase waiting for slower rollouts. The staged curriculum postpones user-simulation overhead until the Multi-Turn stage and helps reduce waiting caused by mixing the two rollout types.

We will include the early comparison and error analysis alongside Table 3 to explain the empirical motivation and observed stage-wise gains of the curriculum.

References

[1] Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents.

[2] Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards.

[3] AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Use.

[4] Gunjal et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. ICLR 2026.

AuthorsOpenReview ↗

Follow-up on Rebuttal for Paper 27914

We sincerely thank you once again for your time and valuable feedback on Paper 27914. We have provided detailed, point-by-point responses to all of your comments. We kindly and respectfully ask that you reconsider our responses and reevaluate the score , especially in light of your positive remarks regarding the technical contribution and motivation.

If you have any remaining concerns, please do not hesitate to let us know — we would be happy to address them.

Reviewer_MDP2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReAgent, a framework for generating agentic RL training scenarios where user requests may conflict with environment states or constraints. The method jointly generates tasks, simulated user behaviors, and reward criteria. It also uses code verification to improve data quality. The generated data are used to train agents with a two-stage GRPO pipeline. Experiments on three benchmarks show improvements in both conflict resolution and general agent performance.

Critical strengths
  1. The paper addresses a practical problem in agentic RL. It considers cases where user requests cannot be completed due to environment constraints. Training agents to explain these limitations, suggest alternatives, or refuse invalid requests is useful for real-world applications.

  2. The paper provides a complete pipeline for constructing training scenarios. It jointly generates tasks, simulated users, and reward criteria. It also includes consistency checks and code verification to improve the quality of the generated data.

  3. The paper is well written and easy to follow. The motivation is clear, and the examples help readers understand the proposed method.

Critical weaknesses
  1. The novelty of the proposed data and reward design is not fully clear. SynthAgent [1] already uses simulated users and rubric-based rewards to evaluate user interactions and forbidden actions. AgenticQwen [2] also considers scenarios where the agent must reject invalid actions. The authors should better explain what is new in the proposed scenario co-design method.

  2. The evaluation results may depend on the user simulator and tool interface. Although the authors evaluate the baselines under the same settings, it would be helpful to test whether the improvements still hold with different user simulators or evaluation settings.

[1] Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards.

[2] AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Use.

Questions for authors
  1. Rewarding an agent for explaining constraints, suggesting valid alternatives, or refusing invalid requests seems like a natural approach. Previous works have also explored related ideas, such as rubric-based rewards in SynthAgent [1] and infeasible user requests in AgenticQwen [2] and Trajectory2Task [3]. I am curious about what the authors consider the key novelty of this work. What makes the proposed scenario co-design method different from these existing approaches?

  2. Could the authors provide more ablation studies on the key components of scenario co-design? For example, what happens if the task, user playbook, and oracle are generated separately, or if the consistency checks are removed? These experiments would help clarify which parts of the proposed method contribute to the improvements.

[1] Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards.

[2] AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Use.

[3] Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents.

6 comments
AuthorsOpenReview ↗

Weakness 1 & Weakness 2

We sincerely thank the reviewer for recognizing the practical importance of user–environment conflicts, the complete scenario-construction pipeline, and the clarity of the presentation. We hope the clarifications and additional results below may address your concerns and increase your confidence in our contribution.


W1. Distinctions from Related Work

ReAgent's key innovation lies in studying how to broaden scalable Agentic RL training beyond feasible requests to user–environment conflicts and their valid resolutions, which differs from existing approaches primarily in the problem settings.

  1. Trajectory2Task [1] reuses the tools and environments of τ²-Bench, infers ambiguous or infeasible user requests from execution trajectories, and collects successful trajectories for SFT. Its focus is SFT data synthesis within existing environments, while ReAgent constructs scenarios for Agentic RL training.

  2. SynthAgent [2] conducts Agentic RL in LLM-simulated environments. However, the paper also notes that inconsistent responses from such environments may affect training stability. In contrast, we synthesize programmatic environments whose tools execute through deterministic code, providing stable environment feedback for training.

  3. AgenticQwen [3] synthesizes scenarios with adversarial users to train agents to resist attempts to induce rule violations. We instead focus on conflicting requests from real users who lack knowledge of environment states or constraints, training agents to guide users toward feasible revisions and appropriately refuse parts that cannot be fulfilled, enabling more flexible handling of real-world scenarios.

To the best of our knowledge, this is the first work to automatically synthesize conflict scenarios covering execution, guidance toward valid revisions, and refusal in programmatic environments and use them for Agentic RL training.

Moreover, we have compared ReAgent-8B directly with AgenticQwen-8B, the closest of these baselines. In Table 1, the average scores of ReAgent-8B are significantly higher on all benchmarks. In Table 2, the overall score for PARTIAL requests, which require revision or partial completion, increases from 29.3 to 38.8. FEASIBLE and BLOCKED scores also improve. These results further support ReAgent's value in training agents to resolve conflicts between real user requests and environment states or constraints.


W2. Questions on Evaluations

(1) User simulators. We address this concern from two perspectives.

  • Different training and evaluation simulators. In our paper, training uses Qwen3.6-27B for user simulation, while evaluation uses Seed1.8 (Appendix B). The reported gains therefore already transfer to a user simulator different from the training simulator.
  • Robustness across evaluation simulators. We further directly evaluate models with GPT-4.1 as the different user agent on τ²-Bench Retail and Airline. We note that ReAgent-8B still outperforms all three baselines with significant improvement over the strongest baseline in each domain, showing that the relative performance advantage persists after changing the user simulator.
ModelRetailAirline
AgenticQwen-8B52.4030.50
EnvScaler-8B51.7531.50
EnvFactory-8B54.3826.50
ReAgent-8B64.0344.00

2. Evaluation settings. We follow each benchmark's evaluation protocol and use public evaluation implementations. These settings facilitate comparison with other work. In Table 7, we compare our results with independently obtained results reported in Agent-World, where The BFCL-V4 overall scores differ by no more than 1.6 percentage points for all three shared baselines, supporting the comparability of the evaluation results.

References

[1] Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents.

[2] Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards.

[3] AgenticQwen: Training Small Agentic Language Models with Dual Data Flywheels for Industrial-Scale Tool Use.

AuthorsOpenReview ↗

Question 1 & Question 2

Q. More Ablation Studies

Q1. Separate Component Generation

Thank you for suggesting separate component generation as an ablation. Compared with this procedure, ReAgent instead jointly constructs the task, user playbook, and oracle around the same conflict and valid resolution (§3.1, Eq. 3). The rationale is to consider interdependent conditions, including how the user cooperates and how the reward evaluates the resolution, together during generation. This aligns the interaction design with the evaluation criteria from the outset and aims to improve the training quality of the synthesized scenarios.

To directly compare scenario co-design with separate component generation, we implemented the concrete separate component generation pipeline suggested by Reviewer QunN. We first generated the conflict task, then independently generated user behavior and reward criteria from the environment and task, and finally applied the same consistency checks. We replaced only the co-designed conflict scenarios with an equal number of scenarios constructed through this separate route. The feasible scenarios, model initialization, training budget, data amount, and other training and evaluation settings were kept fixed. Both variants use the same Qwen3-8B backbone. The scores (%) are as follows.

BenchmarkSeparate constructionReAgent (co-design)
ACEBench-Agent Multi-Step93.7596.30
ACEBench-Agent Multi-Turn61.6765.80
τ-Bench Retail59.7163.10
τ-Bench Airline38.0041.21
BFCL-V4 Multi-Turn50.0050.80
BFCL-V4 Memory26.0228.90

Co-design achieves higher scores on all six metrics even when the separately generated scenarios undergo the same consistency checks. The comparison therefore supports the additional training value of jointly specifying user response conditions and evaluation requirements during scenario construction. These results are consistent with our rationale that considering the interdependent components together improves the training quality of conflict scenarios. We will include this matched construction baseline and its results in the revision.

Q2. Consistency Checks

The consistency checks compare the task request, the user playbook's response conditions, and the oracle's conflict facts and valid resolution boundaries against the same environment, and then revise inconsistent components. Removing this step could allow mismatched user response conditions and reward requirements to reach reward construction. However, a full ablation would require regenerating the training data and rerunning the whole Agentic RL training, which exceeds our available compute during the rebuttal period. So we instead provide a case study from Appendix D.2 to clarify what the check verifies.

In this case, the user asks to approve parent appeal APL002 immediately, but pending child appeal APL004 must be completed first. The playbook and oracle must therefore agree on how the agent should guide the user toward this valid resolution.

ComponentConditions that must agree
EnvironmentAPL002 is automatically approved once its child appeals are completed. APL004 is still pending.
User playbookAfter the agent explains the dependency and identifies APL004, the user accepts approving APL004 instead.
OracleThe agent should obtain user confirmation, approve APL004, and verify APL002's automatic update. Directly updating APL002 or refusing the entire request is inappropriate.

Removing the check would leave this agreement unverified. For example, the playbook could accept approval of APL004 while the oracle still requires directly updating APL002, potentially penalizing correct guidance. The check compares these conditions against the environment and jointly revises mismatches before reward construction.

AuthorsOpenReview ↗

Follow-up on Rebuttal for Paper 27914

Dear Reviewer MDP2,

Thank you very much for your time and thoughtful feedback on Paper 27914. We were especially encouraged by your comments on the practical importance of user–environment conflicts and the completeness of the scenario construction pipeline. In our rebuttal, we explain how ReAgent differs from the related work you highlighted. We also report results with GPT-4.1 as the user simulator and a comparison with the separate generation of scenario components, as you suggested.

If these additional results and clarifications address your concerns, we would be very grateful if you could reconsider your rating. We would be happy to clarify any remaining questions. Thank you once again for your support and kind consideration.

Reviewer_QunNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper primarily discusses the necessity, beyond simply increasing the number of environments and tasks, of incorporating scenarios where user requests conflict with environmental states or business rules during agent training. This enables the model to learn how to execute requests, guide users toward necessary modifications, partially fulfill requests, or provide reasonable refusals. The authors propose "ReAgent," a framework that jointly constructs tasks, simulates user behavior, and defines reward criteria, integrated with code-based verification and a two-stage reinforcement learning process. This approach ultimately yields improved performance.

Critical strengths
  1. The research problem is specific and holds practical value.

The observation that "training tasks are derived by working backward from successful workflows, and thus inherently lean toward requests that can be fulfilled" is clear and worthy of investigation.

  1. The proposed method is well-aligned with the problem.

The integration of tasks, users, and rewards is supported by a clear logical rationale.

  1. The experimental gains are significant, and the study includes a robust ablation analysis.

Performance improvements were observed across two model scales and three primary benchmarks, demonstrating the broad applicability of the approach.

Critical weaknesses
  1. While it has been demonstrated that "adversarial training is effective," there is no direct proof that "joint design" is superior to other more naive construction methods.

This, in my view, is the most critical limitation. The paper emphasizes the joint construction of the task, the playbook, and the oracle. Although the ablation studies confirm the utility of the individual components, they lack a direct comparison: for a given environment and a specific conflict, is the joint construction approach actually better than an alternative, such as first generating the task, then independently generating the user behavior and the reward function, and finally performing a consistency check? Taking the "parental appeal" example, a natural alternative workflow would be: 1) Generate the conflict task (e.g., "the user demands the immediate approval of a parental appeal"). 2) Generate user behaviors that accept valid modifications, based on the task. 3) Generate the reward function, based on the environment and the task. 4) Finally, verify the consistency among these three elements. This alternative approach could also potentially yield high-quality scenarios. The current experiments do not clarify what specific advantages ReAgent’s joint construction offers over such an approach.

  1. Conflicting data and richer reward signals are introduced simultaneously, making it difficult to fully disentangle their respective contributions.

For scenarios where a task can be successfully completed, the reward is primarily based on a final-state function; for conflicting scenarios, however, the reward incorporates LLM-based trajectory scoring and hard constraints. Consequently, when replacing conflicting scenarios, both the request type and the supervision method are altered. Some of the performance gains may stem from this richer process-based supervision. The authors conducted an experiment in which the LLM-based criteria were removed; while this provides some insight, it still fails to completely isolate the two factors.

If the authors can address my concerns, I might raise my score.

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer QunN

We sincerely thank the reviewer for recognizing the practical value of the research problem, the clear rationale for integrating tasks, users, and rewards, and the gains across model scales and benchmarks.


Q1. Comparison with Separate Scenario Construction

Thank you for proposing this specific separate-construction control. Compared with it, ReAgent jointly constructs the task, user playbook, and oracle around the same conflict and valid resolution (§3.1, Eq. 3). The rationale is to consider interdependent conditions, including how the user cooperates and how the reward evaluates the resolution, together during generation. This aligns the interaction design with the evaluation criteria from the outset and aims to improve the training quality of the synthesized scenarios.

We implemented the workflow you suggested. We first generated the conflict task, then independently generated user behavior and reward criteria from the environment and task, and finally applied the same consistency checks. We replaced only the co-designed conflict scenarios with an equal number of scenarios constructed through this separate route. The feasible scenarios, model initialization, training budget, data amount, and other training and evaluation settings were kept fixed. Both variants use the same Qwen3-8B backbone. The scores (%) are as follows.

BenchmarkSeparate constructionReAgent (co-design)
ACEBench-Agent Multi-Step93.7596.30
ACEBench-Agent Multi-Turn61.6765.80
τ-Bench Retail59.7163.10
τ-Bench Airline38.0041.21
BFCL-V4 Multi-Turn50.0050.80
BFCL-V4 Memory26.0228.90

Co-design achieves higher scores on all six metrics even when the separately generated scenarios undergo the same consistency checks. The comparison therefore supports the additional training value of jointly specifying user response conditions and evaluation requirements during scenario construction. These results are consistent with our rationale that considering the interdependent components together improves the training quality of conflict scenarios. We will include this matched construction baseline and its results in the revision.


Q2. Richer Reward Signals

Thanks for the suggestion on the contribution of reward designs. The different reward designs reflect the different requirements for evaluating agent behavior in feasible and conflict scenarios.

  1. Feasible scenarios focus on programmatically verifiable completion requirements. Appendix A.2.1 explicitly requires feasible tasks to have terminal-state conditions that can be verified programmatically. Following EnvScaler, we decompose the goal into these conditions and use the proportion satisfied as task-completion feedback. These checks are efficient and reproducible, and allow different valid execution paths. LLM rubrics evaluating the same completion conditions would repeat the existing checks, so we use direct programmatic verification for these tasks.

  2. Conflict scenarios further require evaluating how the agent handles the interaction. An appropriate refusal and a failure to respond may leave the same terminal environment state while differing in correctness. Terminal-state verification alone cannot distinguish these behaviors. We therefore use LLM rubrics to evaluate guidance, refusal, and compliance with non-negotiable requirements. In Table 4, removing LLM rubrics while retaining conflict scenarios lowers all three benchmark averages, supporting the need for this supervision when training on conflict scenarios.

We agree that reward design contributes to the gains. These gains, however, arise from providing the supervision required by each scenario type, rather than from adding reward components as standalone enhancements. For feasible scenarios, programmatic terminal-state checks suffice to evaluate task completion. Conflict scenarios additionally require trajectory rubrics to assess appropriate guidance and refusal, which terminal states alone cannot distinguish from ineffective responses. This claim is supported by Table 4, which shows that removing LLM rubrics while retaining conflict scenarios lowers all three benchmark averages. Taken together, the ablations and the scenario-specific analysis have covered all necessary combinations of scenario type and reward design.

AuthorsOpenReview ↗

Follow-up on Rebuttal for Paper 27914

Dear Reviewer QunN,

Thank you very much for your time and thoughtful feedback on Paper 27914. We were especially encouraged by your positive remarks regarding the technical contribution and overall performance of the method. We ran the experiment you suggested and report the results in Q1. In Q2, we address your question about how much of the gain can be attributed to richer reward supervision.

If the new results and clarifications address your concerns, we would be very grateful if you could reconsider your rating. We would be happy to clarify any remaining questions. Thank you once again for your support and kindly consideration.

AuthorsOpenReview ↗

Request for Follow-up Before the Discussion Closes

Dear reviewer QunN, with the rebuttal period ending soon, we'd like to kindly request you to read our rebuttal and reconsider your evaluation for our paper. We hope our responses have addressed your concerns.

Thanks very much for your effort and valuable time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.