acceptodds
Under review as a conference paper at ICLR 2027

SAGE: Seedless Agent Task Induction from Environments for Post-Training

Abstract

Training capable agents requires interaction trajectories grounded in executable environments, yet constructing such data for a new environment often requires manually designing tasks, organizing interactions, and verifying execution outcomes. We introduce SAGE, a framework for seedless construction of agent training data from environment specifications. SAGE generates requests from domain context, tool interfaces, user attributes, and initial states, without task-specific query seeds or reference solution paths. A persistent user generator and an agent engage in multi-turn interaction, while an isolated verifier checks candidate requests before execution and complete trajectories after execution. Rejected requests are removed and regenerated, and accepted trajectories are converted into supervised fine-tuning data. We use Qwen3.8-2.4T, Qwen3.8-Next-Flash, GLM5.2, and Kimi-K3 as teacher models to construct data for general tasks, tool interaction, and software engineering, and evaluate the resulting data against public-data baselines. Independent SFT runs on Qwen3.5-27B yield absolute score gains over the initial model of 1.11%, 9.25%, 0.86%, and 2.33% on General Avg, BFCL Multi-turn, Tau2Bench three-domain Overall, and SWE-bench Lite, respectively. The corresponding absolute gains over the best public-data baselines in our comparison are 10.82%, 19.75%, 7.50%, and 8.00%. These results show that SAGE can provide effective supervision for agent post-training across different environments.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Synthetic Data Generation, Trajectory Verification, Agent Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_64W2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAGE, a framework for generating agent training trajectories from environment descriptions, tool interfaces, user attributes, and initial states without task-specific query seeds or reference solution paths. A persistent user generator interacts with an agent, while separate request-level and trajectory-level verifiers filter the resulting data. The authors release four data pools and report improvements from independent SFT runs on Qwen3.5-27B across general benchmarks, BFCL v4 Multi-turn, tau3-bench, and SWE-bench Lite.

Critical strengths
  1. The paper addresses a useful practical problem: obtaining training interactions for an executable environment without manually supplying task queries. The separation of user generation, agent execution, and verification makes the proposed workflow reasonably clear.

  2. Releasing the data enables inspection and reuse. The evaluation spans several types of interaction, and the reported 9.25-point improvement over the starting model on BFCL v4 Multi-turn is promising, subject to the concerns below.

Critical weaknesses
  1. Task-goal definition and verifier reliability are insufficiently validated.

Appendix C.3 defines a separate success judgment and six quality dimensions, and Section 3.3 fixes each accepted user request before execution. My concern is how episode-level completion conditions are derived from the initial accepted request and tracked when subsequent requirements are revised, replaced, or withdrawn. Legitimate task revision should be distinguished from goal drift or abandonment. The Limitations section already acknowledges that inter-judge agreement measures consistency rather than accuracy; however, the reported BFCL agreement study does not independently validate success labels or acceptance decisions in the Hermes branch.

The released Hermes data contain examples raising concerns about trajectory quality and success labeling. In hermes_000497, the user repeats "Explain what SQL" six times and ends with an incomplete message containing an "assistant" fragment, yet the trajectory is accepted with an overall score of 4/5. In hermes_000083, the final assistant reply says that only a dummy PDF was successfully parsed and still requests a target path or URL, yet judge_success is true. I recognize that reasonable, honestly reported failures can be useful SFT examples and are explicitly allowed by the paper. The concern is acceptance of degenerate interactions and questionable success labels, not failure retention itself. These cases warrant an independent audit; they do not establish the overall error rate.

  1. The evidence does not adequately establish the framework's distinct contribution or the robustness of the reported gains.

The relationship to recent methods needs clearer discussion: SYNTHAGENT [1] combines user simulation with simulated tool environments and task-level rubrics, whereas AutoPlay [2] generates tasks through exploration of executable UI environments. What does SAGE add beyond these approaches? Sections 5.1-5.2 study data scale and judge agreement, but do not isolate the proposed user generator and verification components against simpler generation/filtering alternatives. Several public-data baselines substantially degrade the starting model, and Section 4.2 acknowledges that assistant-target-token budgets are not strictly matched. The results demonstrate gains under the reported configurations but do not isolate the benefits of SAGE's specific design choices.

There is also a specific evaluation-validity concern. The released General pool contains the complete HumanEval/98 and HumanEval/52 prompts in general_182933 and general_183117, respectively. This can be reproduced by matching complete official prompts against user messages after normalizing Unicode, case, and whitespace. The public pool precedes subsampling, so this does not establish inclusion in the main 100K training set or explain the reported HumanEval gain. However, the exact training selection and decontamination procedure are needed before that gain can be interpreted confidently.

References

[1] Lyu et al. Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards. ACL 2026. https://aclanthology.org/2026.acl-long.570/

[2] Ramrakhya et al. Scaling Synthetic Task Generation for Agents via Exploration. ICLR 2026. https://machinelearning.apple.com/research/scaling-synthetic-task

Questions for authors
  1. After the initial request is accepted, how are episode-level completion conditions determined and updated as requirements change? How do you distinguish legitimate task revision from goal drift or abandonment, particularly in the Hermes branch?

  2. Could you explain the acceptance of hermes_000497 and the success label of hermes_000083, and provide their three underlying judge outputs? Do you have an independently annotated audit measuring false acceptance and success-label accuracy on randomly selected trajectories?

  3. Were general_182933, general_183117, or other benchmark-overlapping examples selected into the actual 100K training set? Please provide identifiers for the actual training samples retained after length filtering and describe the decontamination procedure. If overlap entered training, how do the affected results change after removing it?

  4. What evidence attributes the improvement to SAGE's proposed components rather than teacher/data differences or supervision volume? A focused comparison with simpler generation/filtering under matched assistant-target-token budgets would help, together with a clearer distinction from [1, 2].

7 comments
AuthorsOpenReview ↗

Response to Reviewer 64W2 (part 1/4)

Thank you for providing specific sample IDs and reproducible inspection procedures, which enabled us to check the cases individually. We address your four questions below.

Q1: Evolving Task Goals

This is an important question, and we did not explicitly account for it when designing the judge. In the current framework, “fixing a request before execution” refers to execution order: each user request is checked and fixed before it is passed to the agent; it does not mean that the goal cannot change after the first turn. In subsequent turns, the user may explicitly add, replace, or withdraw requirements. The trajectory judge reads the complete conversation and assesses completion against the requirements that remain valid: explicitly withdrawn requirements no longer count toward completion, whereas an agent’s unilateral change of the target object without a corresponding user request is treated as goal drift. We acknowledge that this decision relies on the judge’s semantic understanding. The paper does not introduce an explicit mechanism for tracking completion criteria, and this mechanism has not been separately validated on the Hermes branch.

Q2: Data Quality

We retrieved the judgment records saved during generation. For this Hermes batch, scores are averaged across three judgments, while the training-acceptance label (label) and task-success label (success) are aggregated separately by majority vote.

Sample IDJudgmentoverall_scorelabelsuccess
hermes_00008314accepttrue
hermes_00008324acceptfalse
hermes_00008334accepttrue
hermes_00049714acceptfalse
hermes_00049724acceptfalse
hermes_00049734acceptfalse

For hermes_000083, two judgments treated parsing an example PDF as task completion, resulting in success=true after majority voting. For hermes_000497, all three judgments considered the task incomplete but assigned accept because they valued the agent’s clarification behavior. The two cases expose problems in task-completion assessment and the filtering of degraded interactions, respectively. The six structured outputs below retain all fields and the full judgment reasons.

AuthorsOpenReview ↗

Response to Reviewer 64W2 (part 2/4)

hermes_000083

Original output 1

{
  "overall_score": 4,
  "task_relevance": 5,
  "tool_use_correctness": 4,
  "handoff_quality": 5,
  "user_response_consistency": 4,
  "completion_quality": 4,
  "safety": 5,
  "success": true,
  "label": "accept",
  "reason": "The trajectory demonstrates the agent’s robustness when faced with ambiguous, invalid, or prank-like user input. The agent correctly identified how to use anydoc, clarified the requirements through multiple handoffs, and successfully downloaded and parsed a PDF after the user provided a URL. The final response accurately summarized the only successful parsing result in the conversation and honestly pointed out problems with the subsequent invalid instructions. Although the user input was of extremely low quality, the agent’s responses were professional, its tool use was correct, and it did not hallucinate, making the trajectory highly valuable for training."
}

Original output 2

{
  "overall_score": 4,
  "task_relevance": 5,
  "tool_use_correctness": 4,
  "handoff_quality": 5,
  "user_response_consistency": 4,
  "completion_quality": 4,
  "safety": 5,
  "success": false,
  "label": "accept",
  "reason": "The trajectory demonstrates the agent’s robustness when faced with ambiguous, invalid, or prank-like user input. The agent correctly identified how to use anydoc, verified the environment, tried multiple download sources, and successfully parsed an example PDF. When the user continued to provide invalid instructions, the agent appropriately performed multiple handoffs, ultimately summarizing the situation honestly in its final response and stopping further guessing, without hallucinating success. Although the user’s task (parsing a specific document) was not completed because valid input was unavailable, the agent’s behavior was reasonable and of high quality, making it suitable as a training example for handling blocked or invalid interactions."
}

Original output 3

{
  "overall_score": 4,
  "task_relevance": 5,
  "tool_use_correctness": 4,
  "handoff_quality": 5,
  "user_response_consistency": 4,
  "completion_quality": 4,
  "safety": 5,
  "success": true,
  "label": "accept",
  "reason": "The trajectory demonstrates the agent’s robustness when faced with ambiguous, invalid, or even adversarial user input. The agent correctly identified how to use anydoc, clarified the requirements through multiple handoffs, and successfully downloaded and parsed a PDF after the user provided a URL. The final response honestly summarized the successful parsing result, explicitly identified problems with the subsequent invalid instructions, and refused to execute blindly. Although low-quality user input led to multiple interactions, the agent’s behavior was coherent, its tool use was correct and safe, and the trajectory has high training value. Success is true because the agent successfully demonstrated the tool’s functionality and completed the parsing task that was actually feasible, while reasonably terminating a meaningless loop."
}
AuthorsOpenReview ↗

Response to Reviewer 64W2 (part 3/4)

hermes_000497

Original output 1

{
  "overall_score": 4,
  "task_relevance": 5,
  "tool_use_correctness": 5,
  "handoff_quality": 5,
  "user_response_consistency": 5,
  "completion_quality": 4,
  "safety": 5,
  "success": false,
  "label": "accept",
  "reason": "The trajectory demonstrates the agent’s excellent handling of truncated user input (repeated instances of \"Explain what SQL\"). Rather than mechanically repeating itself, the agent progressively offered different forms of explanation (a detailed version and a reference card), proactively diagnosed the problem as client-side truncation, and finally provided explicit options for the user to choose from. This interaction pattern has high training value, demonstrating good dialogue management and user guidance.\n\nRegarding success: although the agent marked finish(success), the user’s original intent (an explanation of a specific SQL-related concept, such as SQLAlchemy or SQL injection) was not satisfied because the input remained truncated. The agent correctly recognized this and requested clarification in the final step, but the trajectory ended with success, which is semantically inaccurate: the task was actually pending or awaiting clarification, rather than completed. Therefore, success should be false. This does not diminish the trajectory’s training value, so label is accept."
}

Original output 2

{
  "overall_score": 4,
  "task_relevance": 5,
  "tool_use_correctness": 5,
  "handoff_quality": 5,
  "user_response_consistency": 4,
  "completion_quality": 4,
  "safety": 5,
  "success": false,
  "label": "accept",
  "reason": "The trajectory demonstrates reasonable strategies for responding to truncated user input (\"Explain what SQL\"): moving from a detailed explanation to a reference card, proactively asking about the intended meaning, and finally reporting the blockage honestly. Although the task was not ultimately completed because of the user-input problem, the agent’s behavior was coherent, the interaction was natural, and the trajectory has training value. Success is false because the user never provided a complete question, so the agent could not confirm task completion."
}

Original output 3

{
  "overall_score": 4,
  "task_relevance": 5,
  "tool_use_correctness": 5,
  "handoff_quality": 5,
  "user_response_consistency": 5,
  "completion_quality": 4,
  "safety": 5,
  "success": false,
  "label": "accept",
  "reason": "The trajectory demonstrates high-quality question answering and excellent error-handling and clarification capabilities. The agent accurately answered the first question about SQLModel. When the user repeatedly sent truncated messages (\"Explain what SQL\"), it did not mechanically repeat itself; instead, it provided different forms of explanation (a detailed version, a reference card, and a condensed version), eventually reasonably inferring a client-side truncation issue and guiding the user to provide complete input. This strategy for handling ambiguous or repetitive input has high training value.\n\nSuccess is false because the user’s subsequent request (an explanation of an SQL-related concept) was never fully satisfied due to input truncation. The agent ended with a handoff awaiting user clarification, leaving the task blocked or incomplete rather than successfully completed. However, this is a reasonable way to end the interaction and should not result in rejection."
}
Reviewer_F3kmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SAGE, a framework for constructing agent training data directly from environment specifications, without relying on task-specific query seeds or reference solution trajectories. SAGE coordinates a user generator, an execution agent, and a verifier, each with distinct information access, to generate requests, carry out multi-turn interactions, and filter the resulting trajectories. Using this framework, the authors construct approximately 270,000 examples spanning general dialogue, tool use, transactional interactions, and software engineering, and separately fine-tune Qwen3.5-27B on each corresponding data subset. The resulting models outperform the initial checkpoint on their respective evaluations.

Critical strengths

The paper addresses a practically important problem. Executable environments often expose tools and state transitions but lack the tasks and demonstrations required for agent training. Generating both requests and interaction trajectories directly from environment specifications offers a promising way to reduce manual task design when developing agents for new environments.

Critical weaknesses
  1. The empirical contributions of the proposed design choices are not sufficiently isolated. The experiments do not establish the incremental benefits of SAGE’s specific mechanisms, particularly role-specific information access, request-level verification and regeneration, and trajectory filtering.

  2. I wonder why the public-data baseline in Table 1 underperforms the initial checkpoint. The most striking example is xLAM, which reduces BFCL performance from 54.13 to 0.25 and τ³-bench performance from 82.90 to 1.44.

  3. The scope and robustness of the reported improvements remain insufficiently characterized. The tool-training data are constructed using BFCL and τ³-bench interfaces and environments, so the results primarily demonstrate adaptation to the target environments. The paper should clarify how overlap between training and evaluation is controlled and include explicit evaluations on held-out tools, states, or repositories.

Questions for authors

Please refer to paper weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer F3km (part 1/2)

Thank you for recognizing the practical importance of this problem. We address the three weaknesses below.

Q1. Component Contributions

Our contribution comprises both the construction pipeline and the data it produces and makes publicly available. We first discuss the data contribution, whose value stands independently of component ablations, and then address the components.

  1. Data contribution. SAGE publicly releases 269,703 records across four branches: 235,292 General records, 19,297 BFCL records, 14,616 τ³-bench transactional interactions, and 498 Hermes long trajectories involving software repositories. Each record in the tool branches is a complete interaction obtained through execution in an executable environment. We compared these data with the three public baselines in Table 1 using the same measurement criteria:
Training dataContains tool responsesContains ≥2 user turnsEnds with a natural-language responseMean user turnsMean tool calls
xLAM 10K0%0%0%1.001.66
ToolACE 10K7.4%4.7%20.6%1.111.68
APIGen-MT 4,80199.5%99.9%98.1%4.784.28
SAGE-BFCL 10K99.4%49.5%100%1.662.91
SAGE-Tau 10K99.3%92.0%100%4.264.37

This table corroborates the xLAM diagnosis below: data lacking tool responses and subsequent interaction cause models to collapse on multi-turn tasks. SAGE provides this complete interaction structure. In terms of training utility, 10K BFCL samples improve 4B and 27B by 11.37 and 9.25 percentage points, respectively, and the data-scale curves increase with sample count at both model sizes.

  1. Component ablation. We added an experiment using 10K BFCL samples without trajectory-level filtering and report it alongside the filtered 10K results in the main paper. The metric is BFCL v4 Multi-turn accuracy (%). This demonstrates the effectiveness of our final judge.
Student modelOriginal modelMain paper: filtered 10KAdditional: unfiltered 10K
Qwen3.5-4B36.3847.7535.38
Qwen3.5-27B54.1363.3856.25
AuthorsOpenReview ↗

Response to Reviewer F3km (part 2/2)

Q2. Baseline Degradation

As discussed in our response to Reviewer cyuC, we additionally evaluated and analyzed the general capabilities of the models trained on xLAM. Their single-turn tool-calling and general capabilities retain relatively high accuracy, whereas multi-turn failures concentrate on interaction termination and error recovery. This is precisely the problem our work seeks to address. We checked the saved chat templates and examined the evaluation records individually. The results are as follows:

Diagnostic / metricxLAM → 4BxLAM → 27B
BFCL NON_LIVE84.88%87.23%
BFCL LIVE80.46%82.75%
BFCL Multi-turn accuracy0.13% (1/800)0.25% (2/800)
BFCL multi-turn: forced termination at the interaction limit781/800 (97.62%)770/800 (96.25%)
τ³-bench success rate0.00% (0/278)1.44% (4/278)
τ³-bench: termination due to consecutive tool errors256/278 (92.09%)236/278 (84.89%)
General mean (change from original model)67.68 (−4.99)82.25 (−2.20)

Both model sizes can produce valid tool calls on the BFCL single-turn subsets, but more than 96% of multi-turn tasks terminate upon reaching the interaction limit. Consecutive tool errors are also the predominant termination cause on τ³-bench. All 10K xLAM samples used in training supervise a single user request followed by tool calls; a sample may contain multiple calls. The proportions containing tool responses, subsequent user messages, or a natural-language closing response are all 0%. This supervision structure lacks the complete process of calling a tool, observing its result, continuing the interaction, and concluding the task, consistent with the repeated calls and termination failures observed in evaluation.

Degradation on the other public baselines is strongly related to the student model and training recipe; please see Q1 of our response to Reviewer cyuC.

Q3. Evaluation Scope and Overlap

We agree that the tool-branch results primarily demonstrate adaptation within the target environments. To clarify the scope of this adaptation, we examined the relationship between training data and test sets record by record:

AspectBFCL v4 Multi-turnτ³-bench (Airline / Retail)
Test questions and reference answersNot exposed to the generatorTest-task instructions and evaluation criteria are not exposed to the generator
Text overlap between training messages and test questions16,555 user messages in the 10K training set vs. 3,136 test turns: only 4 exact matches, all short sentences restating credentials from the initial stateMaximum fragment overlap with 164 test-task descriptions: 12.5%; no near-duplicates
Initial states / databaseShared: initial states from all 800 test cases were used for generationShared: the same database; all users involved in test tasks appear in training (approximately 6.4% of training conversations)

Thus, there is no task-level leakage between training and evaluation, although the environments and initial states are shared. We will include this table in the revision to clarify the distinction.

Moreover, APIGen-MT uses the same τ-bench Airline and Retail environments as SAGE-Tau and has similarly complete interaction structures. With the same environments and comparable interaction structures, SAGE exceeds APIGen-MT by 10.07 points in the two-domain mean, with higher scores in both Airline and Retail. Please see Q2 of our response to Reviewer cyuC for details.

The evaluation-set overlap audit and decontamination procedure for the General branch are discussed in our response to Reviewer 64W2.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Response

Thank you again for your thoughtful feedback. We have responded to each of your comments and will incorporate the corresponding changes into the revised manuscript. As the discussion period is drawing to a close, we would be grateful if you could let us know whether our response addresses your concerns, and we are happy to clarify any remaining points before the discussion ends.

Reviewer_cyuCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAGE constructs agent SFT data without task-specific query seeds or reference solutions. It induces requests from environment context, separates user, agent, and verifier information, filters role-confused requests before execution, and assesses complete trajectories afterward. Independent fine-tuning experiments cover general dialogue, BFCL, transactional interaction, and software engineering. The clearest improvement over the initial 27B model is 9.25 percentage points on BFCL; gains on the other aggregate metrics are smaller.

Critical strengths
  • The workflow handles important details of interactive supervision: persistent user-visible history, pre-execution request rejection without state rollback, explicit handoffs, and removal of control metadata from SFT targets. The appendix provides prompts, filtering thresholds, hyperparameters, and complete illustrative traces, making the intended procedure reasonably concrete.

  • The BFCL improvement is substantial and appears at both 4B and 27B across multiple data sizes. The other branches also improve their aggregate target metrics over Base. Reporting length retention, rejected-data handling, and measured generation/judging time helps readers assess practical usability.

Critical weaknesses
  • Every public-data baseline in Table 1 underperforms Base, sometimes catastrophically: xLAM falls from 54.13 to 0.25 on BFCL. The shared 1e-4 learning rate is a legitimate fixed-recipe comparison, but these unexplained regressions limit what the large baseline gaps establish about dataset quality. Target-token budgets are explicitly unmatched, and SAGE tool data are generated in the evaluated environments. Teacher quality, environment alignment, and optimization sensitivity remain alternative explanations. The smaller General, tau, and SWE-Lite gains also lack uncertainty estimates. These limitations matter when judging the breadth of the empirical contribution.

  • Neither the scale experiment nor inter-judge agreement isolates SAGE's distinctive components. There is no downstream comparison removing the request Guard, information isolation, or trajectory filtering, nor an independent correctness audit of retained trajectories. The 87.76% judge agreement measures consistency. The authors acknowledge these limitations and defer component ablations; I nevertheless find the remaining evidence insufficient to establish the framework's incremental contribution over simpler environment-conditioned synthesis. This is a limitation of the demonstrated contribution, rather than a contradiction of the stated fixed-configuration results.

Questions for authors
  1. Do the baseline gaps persist after checking serialization/evaluation compatibility, tuning learning rates, and matching assistant-target-token budgets? Please provide uncertainty estimates for the smaller gains.

  2. Can you provide matched component ablations and an independently checked sample of accepted/rejected trajectories? What benchmark-instance separation and held-out-environment results support transfer beyond environment-specific adaptation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cyuC (part 1/2)

Thank you for your careful and balanced review. We address your comments below.

Q1: Baseline Performance

We agree that the large gaps in Table 1 cannot be interpreted directly as differences in data quality, and we will narrow this claim in the revision. To address your question, we conducted the following analyses.

We additionally evaluated and analyzed the general capabilities of the models trained on xLAM. Their single-turn tool-calling and general capabilities retain relatively high accuracy, whereas multi-turn failures concentrate on interaction termination and error recovery. This is precisely the problem our work seeks to address. We checked the saved chat templates and examined the evaluation records individually. The results are as follows:

Diagnostic / metricxLAM → 4BxLAM → 27B
BFCL NON_LIVE84.88%87.23%
BFCL LIVE80.46%82.75%
BFCL Multi-turn accuracy0.13% (1/800)0.25% (2/800)
BFCL multi-turn: forced termination at the interaction limit781/800 (97.62%)770/800 (96.25%)
τ³-bench success rate0.00% (0/278)1.44% (4/278)
τ³-bench: termination due to consecutive tool errors256/278 (92.09%)236/278 (84.89%)
General mean (change from original model)67.68 (−4.99)82.25 (−2.20)

Both model sizes can produce valid tool calls on the BFCL single-turn subsets, but more than 96% of multi-turn tasks terminate upon reaching the interaction limit. Consecutive tool errors are also the predominant termination cause on τ³-bench. These records locate the main failure behaviors in the use of feedback, error recovery, and timely termination during sustained interaction. All 10K xLAM samples used in training supervise a single user request followed by tool calls; a sample may contain multiple calls. The proportions containing tool responses, subsequent user messages, or a natural-language closing response are all 0%. This supervision structure lacks the complete process of calling a tool, observing its result, continuing the interaction, and concluding the task, consistent with the repeated calls and termination failures observed in evaluation.

Learning rate and training recipe. We retrained the General branch on Qwen2.5-3B using the same recipe. The following results report the average over 11 General benchmarks. The extent of baseline degradation depends strongly on the student model and training recipe.

Qwen2.5-3B training dataGeneral meanChange from original model
Original model56.63—
Magpie-Pro-MT 100K54.89−1.74
UltraChat 100K51.03−5.60
Infinity-Instruct 100K55.72−0.91
SAGE-General 100K56.24−0.39

On the earlier-generation 3B model, the three public-data baselines show performance drops of only 0.91–5.60 points; in contrast, on the 27B model, their performance drops by 13.17, 27.01, and 9.71 points, respectively. We also report that SAGE-General does not outperform the original model on 3B.

We have not performed a learning-rate sweep or fully matched the supervised token budgets across data sources. As this is a dataset paper, we regard these investigations as subsequent work on SFT using the released data.

AuthorsOpenReview ↗

Response to Reviewer cyuC (part 2/2)

Q2: Ablations, Audit, and Transfer

Our contribution has two parts: the construction pipeline and the four publicly released data branches produced with it. The value of the data contribution stands independently of component ablations.

We address your question through four analyses using consistent measurement criteria.

  1. Analysis of the tool-interaction data:
Training dataContains tool responsesContains ≥2 user turnsEnds with a natural-language responseMean user turnsMean tool calls
xLAM 10K0%0%0%1.001.66
ToolACE 10K7.4%4.7%20.6%1.111.68
APIGen-MT 4,80199.5%99.9%98.1%4.784.28
SAGE-BFCL 10K99.4%49.5%100%1.662.91
SAGE-Tau 10K99.3%92.0%100%4.264.37

xLAM and ToolACE provide very little complete supervision covering tool calls, observation of results, continued interaction, and task closure. This is the reason for xLAM’s collapse on multi-turn tasks (see Q1). SAGE supplies this structure through actual execution in the target environments.

  1. Component ablation. We added an experiment using 10K BFCL samples without trajectory-level filtering and report it alongside the filtered 10K results in the main paper. The metric is BFCL v4 Multi-turn accuracy (%). This demonstrates the effectiveness of our final judge.
Student modelOriginal modelMain paper: filtered 10KAdditional: unfiltered 10K
Qwen3.5-4B36.3847.7535.38
Qwen3.5-27B54.1363.3856.25

The unfiltered group was randomly sampled from the same pool of 29,285 raw trajectories. Its 10K samples comprise 6,673 originally accepted, 3,310 originally rejected, and 17 unjudged trajectories; only records that could not provide assistant supervision targets were excluded. The request-level Guard used during generation remained unchanged. Both model sizes used the same fixed sample, with 9,825 records remaining for training after the 16K length filter.

  1. Human audit. We randomly sampled 25 records from each of the public BFCL, τ³, Hermes, and General branches, for a total of 100, and conducted a human audit. Of these, 57% were reasonable and 33% had flaws, which we consider a relatively favorable result. Hermes tasks are substantially more complex, and their generation and judging costs are higher; consequently, the proportions classified as reasonable or flawed but acceptable are lower.
Data branchSamplesA: ReasonableB: FlawedC: Should be rejectedClear success mislabels
BFCL2520411
τ³ (Airline / Retail)25101230
Hermes2591061
General251870—
Total1005733102
  1. Generalization. APIGen-MT provides a useful comparison: its data also originate from the Airline and Retail environments of τ-bench and have similarly complete interaction structures. The difference is that APIGen-MT relies on task blueprints with reference action sequences, whereas SAGE uses neither task seeds nor reference solutions. The domain-level results are shown below. Airline and Retail are the training domains, while training on SAGE data also yields transfer to Telecom, beyond adaptation to the specific training environments.
Qwen3.5-27B, τ³-benchAirlineRetailTelecomAirline / Retail mean
Original model62.7982.1491.2372.47
APIGen-MT (4,801 samples)50.0072.8191.2361.41
SAGE-Tau (10K)64.0078.9597.3571.48

With the same environments and similar interaction structures, SAGE exceeds APIGen-MT by 10.07 points in the two-domain mean and achieves higher scores in both Airline and Retail. This indicates that the gains cannot be explained solely by generation within the same environment. We also note that the two datasets differ in sample count and teacher model, and that SAGE-Tau does not outperform the original model on the training-domain mean.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Response

Thank you again for your thoughtful feedback. We have responded to each of your comments and will incorporate the corresponding changes into the revised manuscript. As the discussion period is drawing to a close, we would be grateful if you could let us know whether our response addresses your concerns, and we are happy to clarify any remaining points before the discussion ends.

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive comments. We are glad the reviewers found value in SAGE's ability to turn executable environments into agent training data without human-written tasks or reference solutions, and in the released dataset of nearly 270,000 records. The shared concerns were the contribution of filtering, the weak xLAM baseline, verifier reliability, and generalization and contamination. We summarize the new results here; details are in the individual responses.

  1. Effect of filtering. At equal training-set size, training on verifier-filtered interactions rather than unfiltered ones raises BFCL multi-turn accuracy by 12.4 points for the 4B model and 7.1 points for the 27B model.
  2. Why the xLAM baseline scores low. Models trained on xLAM make valid individual tool calls, but more than 96% of their multi-turn episodes hit the turn limit without completing the task. The gap therefore comes from xLAM consisting of single-turn data, not from a failure of the evaluation.
  3. Verifier reliability. In a human audit of 100 sampled records, 57 were clean, 33 had minor flaws but remained usable, and 10 should have been rejected. The verifier is useful but imperfect, and we will state this as a limitation.
  4. Transfer to a held-out domain. On Telecom, which is excluded from SAGE-Tau training, task success rises from 91.23% for the base model to 97.35% after training.
  5. Benchmark overlap. We found that 221 of the ~270,000 records in the public General pool overlap with benchmark questions. We will remove them from the release.

We will include all of these results, together with the limitations noted above, in the revision.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.