- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper addresses the problem of synthesizing training data for tool-use agents operating in policy-constrained environments (such as airline, retail, and telecom customer service in -Bench, as well as in-vehicle assistants in CAR-Bench). Specifically, the authors extract Guarded Operators from the target environments and maintain a global ledger to progressively evolve task blueprints along both structural dimensions (e.g., causal dependencies, parallelism, and repetition) and interaction dimensions (e.g., ambiguous references and user pressure). To ensure data quality, the framework incorporates an "initialization–simulation–verification–diagnostic routing" closed loop that performs bidirectional refinement—modifying either the task specification or the solution side—for candidate instances that fail verification. Empirically, the authors synthesize approximately 1,000 instances each on -Bench and CAR-Bench to fine-tune the Qwen3 model family, and further construct "Frontier Exams" from deeply evolved tasks to benchmark frontier models including Opus 4.5 and GPT-5.2.
Critical strengths
- Soundness of Causal Structural Evolution: The approach of leveraging Guarded Operators to formalize precondition activation and blocking relationships among tool calls (e.g., first invoking a rebooking API to upgrade the cabin class in order to unlock business-class refund eligibility, and then executing the refund) is well-motivated. Expanding single-step operations into multi-step causal chains at the action-topology level provides a sensible way to construct structurally complex tasks.
- Identifying the Limitations of Single-Pass Rejection Sampling and End-to-End Empirical Evaluation: The paper pinpoints a practical limitation of single-pass rejection sampling—namely, its tendency to discard long-tail compositional tasks—and conducts end-to-end training experiments combining SFT and reinforcement learning (GRPO) across multiple tool-intensive subdomains.
Critical weaknesses
- The teaser figure (Figure 1) is visually cluttered, aesthetically unpolished, and uninformative, requiring a complete redesign: Figure 1 is crammed with numerous cartoon character illustrations irrelevant to the academic content, haphazardly piling visual elements together with chaotic information flow and ill-defined module boundaries. Readers cannot discern the core data-structure transformations or the algorithmic pipeline of the proposed method from Figure 1 at all. This severely degrades the presentation quality of the manuscript, and the figure needs to be completely redrawn from scratch.
- Synthesizing Interaction Data Without Prior Field Research is Methodologically Unsound: While structural evolution (Sec. 2.3) is logical, jumping directly into synthesizing interaction difficulty (; Alg. 1, Line 18) without first conducting field research on real user behavior is fundamentally unreasonable. Instead of grounding failure modes in empirical customer service studies, the authors rely on armchair heuristics (ambiguity, user pressure), such as mechanically scripting users to bundle invalid claims and "push back twice" (App. A.7). In reality, support friction stems from cognitive asymmetry between users and system policies (misunderstanding tiered rules), accidental misoperations needing rollback, or genuine intent shifts upon discovering hidden constraints, rather than deliberate ambiguity or scripted adversarial pestering. Generating data in a vacuum without field research inevitably distorts the task distribution, as confirmed by the authors' own human audit (App. A.9, Tables 11 and 12): 73.4% (Airline) and 53.9% (Retail) of evolved tasks are judged Unusual or Contrived (with only 26.6% and 46.2% rated Natural, and Airline Gwet's ).
- Flawed task-side recovery: unrealistic sandbox assumptions and unchecked task simplification. While using an LLM judge to trigger task re-initialization is acceptable, the mechanism has two flaws: (1) Recovering "contradictory states" is an artifact of an unguarded sandbox: In App. A.7 (L997, L1089), a "flown segment" task is wrongly grounded to a
cancelledflight (HAT277) yet passesREPLAY(ref, s0)(Alg. 1, L25) only because "the API does not enforce the flown-portion... requirements" (L944). Real-world production software enforces strict state-machine invariants at the API boundary, forbidding such buggy states outright. (2) Unchecked difficulty dilution during rewriting: Alg. 1 updates coverage ledger statically in Stage 2 (L820, L831) and never verifies in Stage 3 (L841–850) whether the rewritten(task, ref, s0)still covers the target branches . In App. A.7, recovery explicitly allows alteringrefto match alternative trajectories (L1098) and resolves disagreement (Table 7, L1105–1136) by appending exclusionary prompt constraints (banning basic-economy and connecting flights) until an open search collapses into a trivial rejection (agent_db_match=true, L1121).
Questions for authors
- Benchmark overlap & decontamination: What fraction of test tasks in -Bench and CAR-Bench share an identical state-changing tool sequence or
(policy, outcome)branch set with the ~1,000 synthesized training tasks? How much performance remains after removing all training examples that share the same tool-call graph or rule path with any test instance? - Preventing difficulty dilution in task recovery: Since Ledger is updated statically in Stage 2 (Alg. 1, L820, L831), why is there no check in Stage 3 (L841–850) verifying that a rewritten
(task, ref, s0)still covers the target branches ? Given that App. A.7 permits alteringrefto fit alternative trajectories (L1098) and resolves mismatches by appending exclusionary prompt constraints (Table 7, L1105–1136), how do you prevent systematic task simplification? Also, are the difficulty labels in Fig. 3(c) assigned statically from pre-rewrite blueprints or re-evaluated post-rewrite? - Realism of interaction variation: Are the interaction kernels grounded in any real customer-service logs or field research? How do you explain that 73.4% (Airline) and 53.9% (Retail) of evolved tasks are rated Unusual or Contrived by human auditors (Table 11), and why were realistic user issues (e.g., policy misunderstanding, accidental action rollback) omitted in favor of scripted nagging ("push back twice")?
- Updated frontier baselines & Figure 1: Can you evaluate the Frontier Exams on newer frontier models (e.g.,
GPT-6/GPT-5.4,Claude Opus 5.5,Kimi K3,Gemini 4) and redesign Figure 1 into a clean, readable technical diagram?
9 comments
Comment 2/6: On whether recovery dilutes difficulty (W3, Q2)
We thank the reviewer for raising this important point. We answer it with three findings, each backed by data.
A blueprint fixes the intended lesson; recovery repairs its instantiation. Stage 2 specifies the actions, causal dependencies, policy/outcome targets, and interaction requirements; its ledger tracks those blueprint-level targets. In Stage 3, our re-initialization prompt explicitly instructs the LLM to retain these requirements, re-grounding entities and state values or repairing a provisional reference rather than selecting an easier behavioral target. If an intended target cannot be instantiated, initializer declares the task infeasible.
This explains the replacement-flight subtask in Appendix A.7. Its target was refusal of an unavailable booking, but the first request inadvertently allowed connecting and basic-economy alternatives. Recovery made the intended direct-economy constraint and the user's rejection of alternatives explicit. The task still required the upgrade–cancel chain, exception-set cancellation, refusal for a genuinely flown segment, compensation-policy resistance, and a baggage-fee boundary. The change corrects a mismatch between wording and blueprint, rather than abandoning the target for an easier one. Respectively, scenarios that intend for a "accept" case (open search for connecting flights) are initialized and verified in other blueprints, managed by global ledger.
Structural preservation. We compared the accepted recovered tasks with their first realizations using ordered state-changing tool-call sequences:
| Cohort | Identical write-call sequence before and after recovery |
|---|---|
| Airline, Round 2 | 93.8% |
| Airline, Round 3 | 88.0% |
| CAR, Round 2 | 87.7% |
| CAR, Round 3 | 100.0% |
Thus, 87.7–100% retain the original operational sequence. This is observable evidence of preservation, complemented by an independent measure of the final task difficulty.
Recovery retains the hardest tasks instead of discarding them (Q2). The difficulty labels in the figure the reviewer refers to (Figure 3(c) in the paper) are neither assigned from the blueprints nor used by the pipeline; they are a post-hoc measurement on the final verified tasks, taken after recovery. Each task is run 8 times by the off-the-shelf student (Qwen3-30B-A3B-Thinking, before training) and labelled by its number of successes: Easy (8/8), Medium (5 to 7), Medium-Hard (2 to 4) or Hard (0 to 1). This measurement aligns steadily with evolution depth, from 0.87 at depth 0 to 0.17 at depth 4 (Figure 3(a)), indicating tasks at deeper evolution get harder. Harder tasks are also more likely to fail verification on their first realization, and therefore to need one or more rounds of recovery. Consequently, the tasks accepted in later rounds are markedly harder than those accepted initially (mean zero-shot success 0.31 and 0.27 vs. 0.60; Table R-difficulty). Recovery does not make these tasks hard; it keeps hard tasks that a pipeline without recovery would have discarded, and their measured difficulty after recovery confirms that it does not simplify them.
Table R-difficulty. Difficulty of verified airline tasks, measured after recovery, by the round in which they were accepted.
| Accepted in | Mean zero-shot success | Medium-Hard + Hard | Hard |
|---|---|---|---|
| Round 1 (initial) | 0.60 | 41% | 28% |
| Round 2 (recovered) | 0.31 | 75% | 49% |
| Round 3 (recovered) | 0.27 | 83% | 52% |
Comment 3/6: On field research, interaction patterns and the reviewer's suggested behaviours (W2, Q3)
We thank the reviewer and agree that field research on real user behaviour is valuable. GRADUS builds a capability-oriented curriculum over the behaviors permitted or prohibited by a fixed environment; it is not designed to reproduce the distribution of production customer-service requests.
Thus, our contribution is a plug-and-play framework. Any interaction pattern, whether it comes from field studies, domain data or expert knowledge, can be written as a kernel markdown file and composed with the evolution process, without changing the pipeline at all.
To demonstrate this, we introduced two behavior kernels specifically suggested by the reviewer, without changing the pipeline:
| New kernel | User behaviour | Expected agent behaviour |
|---|---|---|
| Rule misunderstanding | Sincerely misunderstands a tiered or conditional rule | Explain the rule as it applies to the user, then act on the revised request |
| Intent shift on a discovered constraint | Learns of an unknown constraint and genuinely changes the goal | Confirm the new goal before acting |
These kernels produced tasks that passed the same verification as our original data. We replaced 18.2% of airline and 13.1% of CAR training tasks with them, keeping the training-set size and evolution-depth distribution unchanged. With the same Opus 4.5 proposer and Qwen3-8B SFT recipe:
Table R-new-kernels-results. Training with tasks from the new kernels (same training-set size).
| Setting | Airline | CAR base |
|---|---|---|
| GRADUS 8B (original) | 0.71 | 0.76 |
| GRADUS 8B + new kernels | 0.70 | 0.74 |
The new kernels integrate without materially changing benchmark performance. To showcase the actual generated tasks, Appendix A.18 gives verified examples: misunderstanding the airline 24-hour cancellation rule and changing an in-car request after discovering that AC is on (please see latest PDF revision). These results demonstrate that empirical findings about user friction can be easily incorporated through writing additional kernels without any code change in GRADUS.
How our current kernels are designed. The original kernels were not mined from customer-service logs, but their role is to produce plausible, verifiable variations rather than claim that any one pattern is statistically typical. They cover two things: how users communicate their needs with varying levels of ambiguity, and what they do when their goal conflicts with the policy. The persistence behavior in Appendix A.7 is an instance of the latter: a user asks again for an exception after a refusal (see "Goal conflict with policy"). Our original taxonomy includes 13 sub-dimensions across indirect parameter values (indirect references, computed values, observed-state dependencies, policy-derived values), underspecified requests (infer from context, infer from stored preferences, need clarification, policy priority), and goal/policy conflicts (incorrect claims, appeals, inconsistent statements, mixed requests). Many address genuine informational asymmetry and ambiguity rather than only adversarial persistence.
In summary, the reviewer's suggestions show how the framework is meant to grow: findings about real user behavior can be readily added as kernels directly.
Comment 1/6: On the unguarded APIs and the A.7 example (W3)
We sincerely thank the reviewer for the detailed and thoughtful review, and for recognizing the motivation for guarded operators and for preserving long-tail tasks through recovery. We have addressed the concerns with targeted experiments and a revised figure:
Why the tools are loosely guarded. We agree that production systems enforce invariants at the API boundary. Leaving policy enforcement to the agent is, however, a design choice of the benchmarks rather than of GRADUS. Both τ²-Bench and CAR-Bench deliberately evaluate whether an agent follows a written policy, so their tools accept calls the policy forbids, and GRADUS generates training data for this setting. Many guards would also remain the agent's responsibility even behind strict APIs. They depend on the conversation, such as disclosing options and risks, obtaining explicit confirmation, or resisting a request to bend the policy. A refusal, moreover, issues no call for an API to reject.
The A.7 example shows how a defect in a generated task is recovered. The defect was in the initial task design: it intended a reservation with an already-flown segment (whose un-flown segment therefore cannot be cancelled), but was grounded on flight HAT277, a past flight that had been cancelled by the airline. A past flight being airline-cancelled is not intrinsically an inconsistent backend state; the inconsistency was between the provisional reference's assumption and the actual environment state. Our workflow uncovered and repaired this step by step:
Only the repaired, verified realization entered the corpus. We will revise A.7 to state which check each step passed.