- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SAGE, a framework for generating agent training trajectories from environment descriptions, tool interfaces, user attributes, and initial states without task-specific query seeds or reference solution paths. A persistent user generator interacts with an agent, while separate request-level and trajectory-level verifiers filter the resulting data. The authors release four data pools and report improvements from independent SFT runs on Qwen3.5-27B across general benchmarks, BFCL v4 Multi-turn, tau3-bench, and SWE-bench Lite.
Critical strengths
-
The paper addresses a useful practical problem: obtaining training interactions for an executable environment without manually supplying task queries. The separation of user generation, agent execution, and verification makes the proposed workflow reasonably clear.
-
Releasing the data enables inspection and reuse. The evaluation spans several types of interaction, and the reported 9.25-point improvement over the starting model on BFCL v4 Multi-turn is promising, subject to the concerns below.
Critical weaknesses
- Task-goal definition and verifier reliability are insufficiently validated.
Appendix C.3 defines a separate success judgment and six quality dimensions, and Section 3.3 fixes each accepted user request before execution. My concern is how episode-level completion conditions are derived from the initial accepted request and tracked when subsequent requirements are revised, replaced, or withdrawn. Legitimate task revision should be distinguished from goal drift or abandonment. The Limitations section already acknowledges that inter-judge agreement measures consistency rather than accuracy; however, the reported BFCL agreement study does not independently validate success labels or acceptance decisions in the Hermes branch.
The released Hermes data contain examples raising concerns about trajectory quality and success labeling. In hermes_000497, the user repeats "Explain what SQL" six times and ends with an incomplete message containing an "assistant" fragment, yet the trajectory is accepted with an overall score of 4/5. In hermes_000083, the final assistant reply says that only a dummy PDF was successfully parsed and still requests a target path or URL, yet judge_success is true. I recognize that reasonable, honestly reported failures can be useful SFT examples and are explicitly allowed by the paper. The concern is acceptance of degenerate interactions and questionable success labels, not failure retention itself. These cases warrant an independent audit; they do not establish the overall error rate.
- The evidence does not adequately establish the framework's distinct contribution or the robustness of the reported gains.
The relationship to recent methods needs clearer discussion: SYNTHAGENT [1] combines user simulation with simulated tool environments and task-level rubrics, whereas AutoPlay [2] generates tasks through exploration of executable UI environments. What does SAGE add beyond these approaches? Sections 5.1-5.2 study data scale and judge agreement, but do not isolate the proposed user generator and verification components against simpler generation/filtering alternatives. Several public-data baselines substantially degrade the starting model, and Section 4.2 acknowledges that assistant-target-token budgets are not strictly matched. The results demonstrate gains under the reported configurations but do not isolate the benefits of SAGE's specific design choices.
There is also a specific evaluation-validity concern. The released General pool contains the complete HumanEval/98 and HumanEval/52 prompts in general_182933 and general_183117, respectively. This can be reproduced by matching complete official prompts against user messages after normalizing Unicode, case, and whitespace. The public pool precedes subsampling, so this does not establish inclusion in the main 100K training set or explain the reported HumanEval gain. However, the exact training selection and decontamination procedure are needed before that gain can be interpreted confidently.
References
[1] Lyu et al. Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards. ACL 2026. https://aclanthology.org/2026.acl-long.570/
[2] Ramrakhya et al. Scaling Synthetic Task Generation for Agents via Exploration. ICLR 2026. https://machinelearning.apple.com/research/scaling-synthetic-task
Questions for authors
-
After the initial request is accepted, how are episode-level completion conditions determined and updated as requirements change? How do you distinguish legitimate task revision from goal drift or abandonment, particularly in the Hermes branch?
-
Could you explain the acceptance of hermes_000497 and the success label of hermes_000083, and provide their three underlying judge outputs? Do you have an independently annotated audit measuring false acceptance and success-label accuracy on randomly selected trajectories?
-
Were general_182933, general_183117, or other benchmark-overlapping examples selected into the actual 100K training set? Please provide identifiers for the actual training samples retained after length filtering and describe the decontamination procedure. If overlap entered training, how do the affected results change after removing it?
-
What evidence attributes the improvement to SAGE's proposed components rather than teacher/data differences or supervision volume? A focused comparison with simpler generation/filtering under matched assistant-target-token budgets would help, together with a clearer distinction from [1, 2].
7 comments
Response to Reviewer 64W2 (part 2/4)
hermes_000083
Original output 1
{
"overall_score": 4,
"task_relevance": 5,
"tool_use_correctness": 4,
"handoff_quality": 5,
"user_response_consistency": 4,
"completion_quality": 4,
"safety": 5,
"success": true,
"label": "accept",
"reason": "The trajectory demonstrates the agent’s robustness when faced with ambiguous, invalid, or prank-like user input. The agent correctly identified how to use anydoc, clarified the requirements through multiple handoffs, and successfully downloaded and parsed a PDF after the user provided a URL. The final response accurately summarized the only successful parsing result in the conversation and honestly pointed out problems with the subsequent invalid instructions. Although the user input was of extremely low quality, the agent’s responses were professional, its tool use was correct, and it did not hallucinate, making the trajectory highly valuable for training."
}
Original output 2
{
"overall_score": 4,
"task_relevance": 5,
"tool_use_correctness": 4,
"handoff_quality": 5,
"user_response_consistency": 4,
"completion_quality": 4,
"safety": 5,
"success": false,
"label": "accept",
"reason": "The trajectory demonstrates the agent’s robustness when faced with ambiguous, invalid, or prank-like user input. The agent correctly identified how to use anydoc, verified the environment, tried multiple download sources, and successfully parsed an example PDF. When the user continued to provide invalid instructions, the agent appropriately performed multiple handoffs, ultimately summarizing the situation honestly in its final response and stopping further guessing, without hallucinating success. Although the user’s task (parsing a specific document) was not completed because valid input was unavailable, the agent’s behavior was reasonable and of high quality, making it suitable as a training example for handling blocked or invalid interactions."
}
Original output 3
{
"overall_score": 4,
"task_relevance": 5,
"tool_use_correctness": 4,
"handoff_quality": 5,
"user_response_consistency": 4,
"completion_quality": 4,
"safety": 5,
"success": true,
"label": "accept",
"reason": "The trajectory demonstrates the agent’s robustness when faced with ambiguous, invalid, or even adversarial user input. The agent correctly identified how to use anydoc, clarified the requirements through multiple handoffs, and successfully downloaded and parsed a PDF after the user provided a URL. The final response honestly summarized the successful parsing result, explicitly identified problems with the subsequent invalid instructions, and refused to execute blindly. Although low-quality user input led to multiple interactions, the agent’s behavior was coherent, its tool use was correct and safe, and the trajectory has high training value. Success is true because the agent successfully demonstrated the tool’s functionality and completed the parsing task that was actually feasible, while reasonably terminating a meaningless loop."
}
Response to Reviewer 64W2 (part 3/4)
hermes_000497
Original output 1
{
"overall_score": 4,
"task_relevance": 5,
"tool_use_correctness": 5,
"handoff_quality": 5,
"user_response_consistency": 5,
"completion_quality": 4,
"safety": 5,
"success": false,
"label": "accept",
"reason": "The trajectory demonstrates the agent’s excellent handling of truncated user input (repeated instances of \"Explain what SQL\"). Rather than mechanically repeating itself, the agent progressively offered different forms of explanation (a detailed version and a reference card), proactively diagnosed the problem as client-side truncation, and finally provided explicit options for the user to choose from. This interaction pattern has high training value, demonstrating good dialogue management and user guidance.\n\nRegarding success: although the agent marked finish(success), the user’s original intent (an explanation of a specific SQL-related concept, such as SQLAlchemy or SQL injection) was not satisfied because the input remained truncated. The agent correctly recognized this and requested clarification in the final step, but the trajectory ended with success, which is semantically inaccurate: the task was actually pending or awaiting clarification, rather than completed. Therefore, success should be false. This does not diminish the trajectory’s training value, so label is accept."
}
Original output 2
{
"overall_score": 4,
"task_relevance": 5,
"tool_use_correctness": 5,
"handoff_quality": 5,
"user_response_consistency": 4,
"completion_quality": 4,
"safety": 5,
"success": false,
"label": "accept",
"reason": "The trajectory demonstrates reasonable strategies for responding to truncated user input (\"Explain what SQL\"): moving from a detailed explanation to a reference card, proactively asking about the intended meaning, and finally reporting the blockage honestly. Although the task was not ultimately completed because of the user-input problem, the agent’s behavior was coherent, the interaction was natural, and the trajectory has training value. Success is false because the user never provided a complete question, so the agent could not confirm task completion."
}
Original output 3
{
"overall_score": 4,
"task_relevance": 5,
"tool_use_correctness": 5,
"handoff_quality": 5,
"user_response_consistency": 5,
"completion_quality": 4,
"safety": 5,
"success": false,
"label": "accept",
"reason": "The trajectory demonstrates high-quality question answering and excellent error-handling and clarification capabilities. The agent accurately answered the first question about SQLModel. When the user repeatedly sent truncated messages (\"Explain what SQL\"), it did not mechanically repeat itself; instead, it provided different forms of explanation (a detailed version, a reference card, and a condensed version), eventually reasonably inferring a client-side truncation issue and guiding the user to provide complete input. This strategy for handling ambiguous or repetitive input has high training value.\n\nSuccess is false because the user’s subsequent request (an explanation of an SQL-related concept) was never fully satisfied due to input truncation. The agent ended with a handoff awaiting user clarification, leaving the task blocked or incomplete rather than successfully completed. However, this is a reasonable way to end the interaction and should not result in rejection."
}
Response to Reviewer 64W2 (part 1/4)
Thank you for providing specific sample IDs and reproducible inspection procedures, which enabled us to check the cases individually. We address your four questions below.
Q1: Evolving Task Goals
This is an important question, and we did not explicitly account for it when designing the judge. In the current framework, “fixing a request before execution” refers to execution order: each user request is checked and fixed before it is passed to the agent; it does not mean that the goal cannot change after the first turn. In subsequent turns, the user may explicitly add, replace, or withdraw requirements. The trajectory judge reads the complete conversation and assesses completion against the requirements that remain valid: explicitly withdrawn requirements no longer count toward completion, whereas an agent’s unilateral change of the target object without a corresponding user request is treated as goal drift. We acknowledge that this decision relies on the judge’s semantic understanding. The paper does not introduce an explicit mechanism for tracking completion criteria, and this mechanism has not been separately validated on the Hermes branch.
Q2: Data Quality
We retrieved the judgment records saved during generation. For this Hermes batch, scores are averaged across three judgments, while the training-acceptance label (label) and task-success label (success) are aggregated separately by majority vote.
For hermes_000083, two judgments treated parsing an example PDF as task completion, resulting in success=true after majority voting. For hermes_000497, all three judgments considered the task incomplete but assigned accept because they valued the agent’s clarification behavior. The two cases expose problems in task-completion assessment and the filtering of degraded interactions, respectively. The six structured outputs below retain all fields and the full judgment reasons.