acceptodds
Under review as a conference paper at ICLR 2027

ACE-Sampler: Hierarchical Stochastic Data Selection for Tool-Using Agents

Abstract

Training tool-using language models increasingly relies on heterogeneous collections of agent trajectories, yet larger data pools can introduce redundancy, imbalance, and unreliable supervision. Selecting useful agentic data requires capturing state-dependent behaviors and accounting for the uneven learning value of turns within a trajectory. We introduce ACE-Sampler, a hierarchical stochastic sampler that jointly considers Accuracy, Complexity, and divErsity under a fixed turn-level training budget. At its core is a factorized behavioral representation that captures interaction patterns and action dependencies beyond surface semantics. Using this representation, ACE-Sampler first samples trajectories for behavioral coverage, then prioritizes important assistant turns and samples them for novelty. A stochastic continuation rule balances coverage across trajectories with supervision within each trajectory, while validity checks and learner-aware complexity estimation favor valid examples of appropriate complexity. Across three tool-use benchmarks and three model scales, ACE-Sampler consistently outperforms the compared selectors with a budget of 5,000 training turns, improving aggregate accuracy over random selection by more than 4 percentage points on average. Ablations and scaling experiments examine the effects of behavioral representations, validity filtering, learner-aware complexity, and training data budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Data Selection, Tool Use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_uwkAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to select high value training data from trajectories across different tasks and proposes ACE-Sampler, a hierarchical random sampling framework. The method measures data diversity through modeling at both the trajectory and turn levels. It then combines importance ranking, validity filtering, and difficulty estimation to select training samples in a hierarchical manner. Experimental results show that ACE-Sampler achieves better overall performance than random sampling across three tool use benchmarks under a fixed budget of training iterations.

Critical strengths
  1. The paper addresses the problem of data sampling across different tasks and has practical significance. Unlike traditional sampling methods based on semantic similarity, the proposed approach also considers the distinctive characteristics of tool calls. This makes it better suited to agent tasks involving multiple turns.

  2. The paper proposes a hierarchical stochastic sampling framework that combines behavioral diversity at the trajectory level with importance at the turn level. It also incorporates sample validity filtering and model aware difficulty estimation to account for multiple key factors in data selection. Through hierarchical sampling, the framework can capture diverse behavioral patterns across trajectories while prioritizing turns with greater learning value within each trajectory.

  3. The paper evaluates ACE-Sampler on three benchmarks using three different models. It also compares ACE-Sampler with various data sampling methods based on quality, difficulty, and diversity. Overall, the experimental evaluation is relatively comprehensive.

Critical weaknesses
  1. The main contribution of the paper is the hierarchical stochastic sampling framework, which performs data sampling at both the trajectory and turn levels. However, the paper lacks ablation studies that examine the effectiveness of this mechanism. For example, it does not evaluate the performance of using only turn level sampling or only trajectory level sampling.

  2. Table 5 shows that the final 5,000 selected training turns are heavily concentrated in a few data sources. This raises the question of whether the performance gains mainly result from the sampler favoring data sources with higher quality or better alignment with the target benchmarks. Could these gains be driven by such a preference rather than by the proposed behavioral diversity modeling and hierarchical sampling strategy?

  3. ACE-Sampler shows only marginal improvements over the strongest baselines on Qwen3-1.7B and Qwen3-8B, with almost negligible performance gains.

Questions for authors
  1. The paper emphasizes that its behavioral representation captures action ordering and dependencies, but computes similarity using Jaccard similarity between factor sets. Could the authors clarify how action ordering and dependency information are encoded in these sets?

  2. In Algorithm 1, the variable k is incremented at both lines 19 and 24, which seems inconsistent with its definition in the main text as the number of selected turns from the current trajectory.

3 comments

We thank the reviewer for the constructed reviews! We address the corresponding questions below and all suggested additional experiments have been added in the revision, highlighted with blue color.

W1: The paper lacks ablation studies that examine the effectiveness of hierarchical sampling.

A: Thank you for pointing this out. We explored several method variants during the development of our final approach. Due to space limitations, we did not include all of them in the main paper, instead focusing on the key ablations concerning accuracy, complexity, and diversity. Here, we present the additional ablation on the hierarchical sampling mechanism. The results are as follows:

MethodModelBFCL Multi-Turnτ2\tau^2-Retail
ACE-Sampler (Full)Qwen3-4B42.8843.86
ACE-Sampler (Only Trajectory-level)Qwen3-4B34.8839.47
ACE-Sampler (Only Turn-level)Qwen3-4B41.0040.64
ACE-Sampler (Full)Qwen3-8B46.8846.49
ACE-Sampler (Only Trajectory-level)Qwen3-8B43.6240.64
ACE-Sampler (Only Turn-level)Qwen3-8B46.5044.44

Here, "Only Trajectory-level" selects trajectories using the same trajectory-level sampling method, but randomly samples at most three turns within each selected trajectory. "Only Turn-level" directly selects turns from all candidate turns based on turn diversity, importance ordering, and complexity scores.

As can be seen, both trajectory-level-only and turn-level-only sampling yield lower performance than the full method, particularly trajectory-level-only sampling, which performs substantially worse due to random turn selection. The results validate the effectiveness of hierarchical sampling.

W2: There is a question of whether the performance gains mainly result from the sampler favoring data sources with higher quality or better alignment with the target benchmarks.

A: Thank you for raising this important question. To address this concern, we conducted additional experiments. According to Table 5, EnvFactory accounts for the largest proportion of the selected data. Therefore, we conducted two additional experiments by applying our method and random sampling exclusively to the EnvFactory dataset. To further examine the effect of data source composition, we constructed another dataset with exactly the same source-wise sampling quotas as ACE-Sampler in Table 5, but randomly selected turns within each source quota. The results are shown below (using Qwen3-4B as the base model):

MethodBFCL Multi-Turnτ2\tau^2-Retail
ACE-Sampler (Full)42.8843.86
ACE-Sampler (Only EnvFactory)40.0042.54
Random Sampling (Fixed source quotas as ACE-Sampler)35.8839.47
Random Sampling (All)33.9441.23
Random Sampling (Only EnvFactory)39.1241.52

Based on these results, we draw the following observations:

  1. EnvFactory appears to have higher average data quality, as random sampling exclusively from EnvFactory outperforms random sampling across all sources.

  2. The improvements of our method cannot be explained solely by its preference for EnvFactory or its source composition. Both applying our method exclusively to EnvFactory and random sampling with identical source quotas yield lower performance than the full ACE-Sampler. These results suggest that our method effectively selects informative samples across the seven datasets, beyond simply favoring particular data sources.

  3. Regarding potential alignment with the target benchmarks, our sampling procedure does not explicitly use any information from the target benchmarks to guide data selection. Therefore, the observed improvements are more likely attributable to the effectiveness of the proposed sampling mechanism rather than explicit benchmark-specific alignment.

W3: ACE-Sampler shows only marginal improvements over the strongest baselines on Qwen3-1.7B and Qwen3-8B.

A: The comparison should be evaluated across model scales and benchmarks rather than by the strongest result of an individual baseline at one scale. A single-scale advantage does not demonstrate that a selector generalizes across learners or evaluation settings. For example, ATLaS is close to our method on Qwen3-1.7B, but its performance is substantially weaker on Qwen3-4B and Qwen3-8B. Similarly, D3 is particularly strong on Qwen3-8B, while its results are weaker at the other two model scales. Neither baseline exhibits a consistent advantage across the benchmarks.

In contrast, our method provides the most stable cross-scale behavior. It achieves the best aggregate Overall score for all three base models, and it leads (or the second best) the average results for each benchmark at every model scale. Thus, the meaningful result is not a marginal improvement at one model size, but the consistent top-ranked aggregate performance across all model scales and the three benchmarks.

Q1: Could the authors clarify how action ordering and dependency information are encoded in these sets?

A: Each turn in a trajectory is annotated with two factors: turn function, which describes the type of action performed, and interaction relation, which captures its dependency on previous interactions. For example, turn-function labels include "acquire_information" and "request_clarification", while interaction-relation labels include "uses_earlier_tool_result" and "uses_user_followup". The complete vocabulary is provided in Appendix A.3 (Annotation Prompt).

At the trajectory level, the turn-function and interaction-relation labels are aggregated across all turns to form the turn-function pattern and dependency pattern, respectively. These trajectory-level factors capture the overall action and dependency patterns within a trajectory, while the corresponding turn-level annotations preserve the information associated with individual actions.

Q2: In Algorithm 1, the variable k is incremented at both lines 19 and 24.

A: Thanks for pointing that out. It is a typo and line 24 should be removed. We have revised it in the revision.

Dear Reviewer,

As the discussion period is approaching its end, we kindly invite you to review our responses and let us know whether your concerns have been adequately addressed. If so, we would greatly appreciate your consideration of updating your score. Otherwise, we would be happy to continue the discussion and clarify any remaining questions.

Thank you for your time and valuable feedback!

Reviewer_NVPBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies data selection for tool-using agents. Open-source agentic datasets are heterogeneous, large, and of uneven quality, and the question is how to pick the most valuable training samples under a fixed budget. The paper proposes a hierarchical stochastic data selection framework that jointly considers sample validity, learner-aware difficulty, and behavioral diversity, allocating a turn-level training budget without access to the original execution environments, and demonstrates gains over random selection and multiple representative selection baselines across three tool-use benchmarks and three model scales.

Critical strengths
  1. The motivation is clear. The paper highlights that textual similarity is a poor proxy for behavioral diversity.
  2. The technical design is reasonable. Accuracy, learner-aware complexity, and diversity are handled jointly, and each component is validated by its own ablation.
  3. The experiments are comprehensive. The experiments on three benchmarks demonstrated the competitiveness of the proposed method.
Critical weaknesses
  1. The completeness of the factor system is unjustified. The 10 factor groups and their closed vocabularies are the foundation of the entire similarity computation, yet the paper gives no argument for why these groups are the right ones: no discussion of potentially missing behavioral dimensions, and no account of how the vocabularies were constructed or their coverage validated.
  2. No annotation quality check is reported, especially to the turn-importance labels.
  3. The ablation of the core mechanism is insufficient. No experiment isolates the hierarchy. There is no comparison against a flat, turn-level-only selector with the same gates. No test of importance-ordered vs random processing order.
  4. As shown in Table 4, the factor representation’s advantage over embeddings is inconclusive, and statistical support is weak.
  5. The difficulty acceptance probability is defined differently in Eq. (9) and in Algorithm 1 (line 15). Eq. (9) references a configurable difficulty center c, while the algorithm fixes it at 0.5 and introduces an additional exponent q that is not defined in the paper.
  6. Typos:
  • “traning” in Line 81 should be “training”.
  • “the selector favor” in Line 474 should be “the selector favors”.
Questions for authors
  1. Why is medium difficulty optimal for tool-use tasks? Is it that partially-correct parameter predictions provide effective learning signal, or something else?
  2. What is the annotation quality? Please report agreement with human labels.
  3. How were the factor groups and closed vocabularies constructed? Was any coverage or completeness analysis performed?
  4. Does the 4B-selected set transfer to 8B without re-selection?
  5. Why is trajectory similarity aggregated with max over the selected set, rather than average or other ways?
  6. Please report per-source (Table 5) gate statistics. For each source, how many candidate turns are removed by the validity filter, rejected by the diversity gate, suppressed by the difficulty gate.
4 comments

We thank the reviewer for the constructed reviews! We address the corresponding questions below and all suggested additional experiments have been added in the revised paper, highlighted with blue color.

W1&Q3: Completeness analysis of the factor system. How were the factor groups and closed vocabularies constructed?

A: The factor system is designed around two complementary notions of coverage. Trajectory-level factors describe the task context and global interaction structure: scenario, task type, tool families, dependency pattern, and turn-function pattern. Turn-level factors describe the behavior supervised at a particular state: turn function, interaction relation, target tools, call mode, and tool schema. This separation prevents a large trajectory from being represented only by its topic while preserving the local action patterns that the learner must reproduce.

The vocabulary is deliberately closed for semantic labels (including scenario, task type, turn function, and interaction relation), while the remaining factors can be directly inferred from the sample content or combinations of existing labels. We started from the behavioral categories observed across the seven source datasets, merged synonymous labels, and retained categories corresponding to distinct supervision patterns. Scenario labels follow the predefined domain taxonomy in the paper, while task and interaction labels describe agent operations rather than surface topics.

The annotation prompt also allows an "other" value when the evidence is insufficient or no matching label exists in the closed vocabulary. Therefore, the vocabulary can be considered exhaustive in the sense that any uncovered category falls under "other". Meanwhile, the closed vocabulary has been iteratively refined through a pre-annotation process. The final version ensures that fewer than 1% of cases are labeled as "other" for each category.

Nevertheless, we do not claim that the annotated factors cover all possible cases, as the annotation candidates were derived from the seven datasets used in this study. However, we do not consider this limitation critical to the main conclusions of the paper. Our experiments demonstrate that the current factor system already yields meaningful improvements and outperforms embedding-matching-based methods, supporting the effectiveness of our approach.

W2&Q2: Report annotation quality check, and agreement with human labels.

A: To examine the annotation quality, we randomly sampled 100 complete trajectories for independent human annotation. Each sample was annotated independently by two human experts, and later any disagreements were discussed until consensus was reached. The table below reports the agreement between the LLM annotations and the final consensus annotations by human experts, measured using Cohen's κ\kappa for each dimension.

Annotation dimensionResult
Scenario0.967
Task type0.938
Turn function0.843
Interaction relation0.691
Importance tier0.856
Importance rank0.696

As shown, 4 of the 6 dimensions achieve high Cohen's κ\kappa scores (> 0.8). For interaction relation, the main disagreements arise from turns that rely on both earlier tool results and user follow-ups, where either "uses_user_followup" or "uses_earlier_tool_result" can be considered appropriate. For importance rank, the annotation model (DeepSeek-V4-Flash) tends to assign a rank of "1" to many turns, indicating that they are all important, whereas human experts tend to assign this rank to only a small fraction of turns within each trajectory. Although these two dimensions show relatively lower agreement, we do not consider this a critical limitation for the subsequent sampling process, as we can still conduct meaningful selection.

W3: The ablation of the core mechanism is insufficient.

A: Thank you for pointing this out. We explored several method variants during the development of our final approach. Due to space limitations, we did not include all of them in the main paper, instead focusing on the key ablations concerning accuracy, complexity, and diversity. Here, we present two additional ablations: the hierarchical sampling mechanism and importance ordering. The results are as follows:

MethodModelBFCL Multi-Turnτ2\tau^2-Retail
ACE-Sampler (Full)Qwen3-4B42.8843.86
ACE-Sampler (Only Trajectory-level)Qwen3-4B34.8839.47
ACE-Sampler (Only Turn-level)Qwen3-4B41.0040.64
ACE-Sampler (Random Ordered Turns)Qwen3-4B41.6242.40
ACE-Sampler (Full)Qwen3-8B46.8846.49
ACE-Sampler (Only Trajectory-level)Qwen3-8B43.6240.64
ACE-Sampler (Only Turn-level)Qwen3-8B46.5044.44
ACE-Sampler (Random Ordered Turns)Qwen3-8B45.0044.74

Here, "Only Trajectory-level" selects trajectories using the same trajectory-level sampling method, but randomly samples at most three turns within each selected trajectory. "Only Turn-level" directly selects turns from all candidate turns based on turn diversity, importance ordering, and complexity scores. "Random Ordered Turns" replaces importance ordering with random ordering while keeping all other components unchanged.

As can be seen, both trajectory-level-only and turn-level-only sampling yield lower performance than the full method, particularly trajectory-level-only sampling, which performs substantially worse due to random turn selection. Comparing importance ordering (our full method) with random ordering, the former achieves better performance in all evaluation, validating the effectiveness of both our importance annotations and the ordering strategy used during selection.

W4: The factor representation’s advantage over embeddings is inconclusive, and statistical support is weak.

A: The comparison in Table 4 supports the intended role of factorized behavioral representations. Under our full selector, replacing factors with embeddings reduces BFCL-MT from 42.88 to 42.25 and τ2\tau^2-Retail from 43.86 to 42.40, corresponding to improvements of 0.63 and 1.46 points from the factor representation. The same pattern appears for D3 on τ2\tau^2-Retail, where factors improve performance from 37.72 to 39.18 points.

The embedding representation is slightly stronger for MIG when used as the only selection signal. This is expected because whole-text embeddings capture broad semantic and lexical information beyond diversity. Once accuracy and complexity are controlled, as in D3 and our method, the factor representation provides more useful agent-specific distinctions, especially for multi-step tool-use evaluation. Across the two benchmarks, the factor version of our method improves the average score by approximately 1.05 points over its embedding counterpart.

W5: The difficulty acceptance probability is defined differently in Eq. (9) and in Algorithm 1 (line 15).

A: Sorry for the confusion. This is a typo in the algorithm. The final method uses the probability in Eq. (9), with configurable center cc and no additional exponent qq. Algorithm 1 has been corrected to match the equation in the revision, and the default experimental setting is c=0.5c=0.5.

W6: Two typos.

A: Thanks for pointing this out. We have corrected both occurrences in the revision.

Q1: Why is medium difficulty optimal for tool-use tasks? Is it that partially-correct parameter predictions provide effective learning signal, or something else?

A: The difficulty score measures agreement with the target tool call at the parameter level. Samples with very high agreement scores are often already mastered by the current model and thus provide limited corrective signals. In contrast, samples with very low scores may reflect substantial mismatches in tool selection or argument structure, making them difficult to learn from a single supervised turn. Another possibility is that some of these samples contain issues that were not detected by the validity checks, making it difficult for the model to produce predictions with high agreement. Therefore, excluding such samples can benefit training under a fixed sample budget. Overall, partially correct predictions may provide more effective learning signals, but this remains a selection hypothesis rather than a universal rule. Accordingly, the difficulty center is configurable, and we have reported an ablation study on its selection in Figure 2.

Q4: Does the 4B-selected set transfer to 8B without re-selection?

A: The validity checks and behavioral factors are model-independent, while the difficulty gate is learner-dependent. Therefore, a dataset selected using a different model should remain valid and behaviorally diverse, but its difficulty distribution may not be optimal for the target model. To examine this, we conducted two additional experiments: training Qwen3-4B using the dataset selected based on Qwen3-8B difficulty scores, and training Qwen3-8B using the dataset selected based on Qwen3-4B difficulty scores. The results are shown below:

Selection setQwen3-4B BFCL MTQwen3-4B τ2\tau^2-RetailQwen3-8B BFCL MTQwen3-8B τ2\tau^2-Retail
Random33.9441.2343.5040.94
Selected with 4B difficulty score42.8843.8645.8845.18
Selected with 8B difficulty score40.8840.7946.8846.49

As can be seen, both models achieve their best performance when trained on datasets selected using their own difficulty scores. This supports our claim that the difficulty distribution of selected samples should be aligned with the target learner. Meanwhile, cross-model selection still achieves better average performance than random sampling, suggesting that the model-independent validity checks and diversity mechanisms remain beneficial even without learner-specific re-selection.

Q5: Why is trajectory similarity aggregated with max over the selected set, rather than average or other ways?

A: The selector is a greedy coverage procedure. A candidate is redundant if it is highly similar to any already represented trajectory, so the relevant quantity is its nearest selected neighbor, implemented as the maximum similarity. Averaging would allow a candidate that duplicates one selected trajectory to appear novel because it is dissimilar to many unrelated trajectories. The max therefore directly penalizes worst-case redundancy and gives a simple marginal-coverage interpretation.

Q6: Please report per-source (Table 5) gate statistics.

A: We have added a per-source accounting table with the number of candidate turns removed by the validity or length (we make the max length of 16k) filters, the number of trajectories rejected by the trajectory-level diversity gate, the number of turns rejected by the turn-level diversity gate, the number rejected by the difficulty gate, and the number finally selected. Below are the statistics on Qwen3-4B selected dataset (we have included statistics for 1.7B and 8B in the Appendix A.4 of our revision version):

SourceCandidate turnsValidity filteredLength filteredTraj. div. rejectedTurn div. rejectedDifficulty rejectedSelected
APIGen-MT-5k26,76133271,03616,49969594
Nemotron14,06838303909,2662,387677
EnvFactory66,9813,4147,29372421,4073,6461,853
EnvScaler54,0004,521065222,1706,2221,539
ToolACE3,7397904882,202964482
ToolMind12,91012,6970411423710
Toucan8,273880377644,982818345

These statistics reveal distinct characteristics of the seven data sources. For example, APIGen-MT-5k has the highest rejection rate at the trajectory-level diversity gate, likely because it covers only two domains. ToolACE has the highest rejection rate at the difficulty gate (964 out of 3,739 candidate turns, approximately 26%), suggesting that its samples are relatively easier, as most involve single-turn interactions. Further analysis is provided in Appendix A.4 of our revision.

Dear Reviewer,

As the discussion period is approaching its end, we kindly invite you to review our responses and let us know whether your concerns have been adequately addressed. If so, we would greatly appreciate your consideration of updating your score. Otherwise, we would be happy to continue the discussion and clarify any remaining questions.

Thank you for your time and valuable feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

Dear AC and Reviewers,

We have carefully addressed the concerns of the reviewers and revised accordingly, with all changes highlighted in blue.

The revisions include:

  1. Additional ablations: Validated the effectiveness of hierarchical sampling and importance-based turn ordering.
  2. Data source bias analysis: Demonstrated that the improvements are not solely attributable to source composition through controlled experiments.
  3. Annotation quality: Clarified factor construction and coverage, and reported human annotation agreement on 100 trajectories.
  4. Cross-model transfer: Evaluated dataset transferability between Qwen3-4B and Qwen3-8B, confirming the benefits of learner-aware selection.
  5. Additional analyses and clarifications: Added per-source filtering statistics across model scales, and corrected algorithmic inconsistencies.

Overall, these revisions provide stronger empirical evidence for the effectiveness and robustness of ACE-Sampler. We hope our responses and additional experiments have adequately addressed the reviewers' concerns. Thank you for the time!

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.