acceptodds
Withdrawn from ICLR 2027

SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Today, tool-calling agents are commonly evaluated or tested on static datasets of execution traces, including input commands, agent responses, and associated tool calls. However, internal production datasets are often insufficient or unusable for testing; for example, they may contain sensitive or proprietary data, or they may be too sparse to support comprehensive testing (especially pre-deployment). In these settings, practitioners are increasingly replacing or augmenting real datasets with synthetic ones for evaluation purposes. A key challenge is quantifying the relation between these synthetic datasets and the real data. We introduce SynAE, an evaluation framework for assessing how well synthetic benchmarks for multi-turn, tool-calling agents replicate and augment the characteristics of real data trajectories. SynAE assesses the validity, fidelity, and diversity of synthetic data across four metric categories: (i) task instructions and intermediate responses, (ii) tool calls, (iii) final outputs, and (iv) downstream evaluation. We evaluate SynAE using recent agent benchmarks and test common synthetic data failure modes via realistic and controlled generation schemes. SynAE detects fine-grained variations in data validity, fidelity and diversity, and shows that no single metric is sufficient to fully characterize synthetic data quality, motivating a multi-axis evaluation of synthetic data for agent testing. SynAE is available at https://synae-2026-synae-demo.static.hf.space/index.html, with code at https://github.com/synae-cais-2026/synae-code.

Primary area:
datasets and benchmarks
Keywords:
Meta-evaluation, Agentic Evaluation, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gNKtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SynAE. It is a framework for evaluating synthetic datasets used to test tool-calling agents. The framework measures validity, fidelity and diversity across instructions, tool calls, final outputs and downstream agent performance.

SynAE combines existing distributional and embedding metrics with measures of tool usage and tool call transitions. It also uses an LLM judge to assess validity and compare generated agent traces with reference traces. Downstream evaluation measures differences in task success and whether model rankings are preserved.

The experiments use subsets of T1, BFCL and ACP as reference datasets. Synthetic datasets are created through controlled modifications and generation pipelines such as NeMo. The paper shows that different metrics capture different changes in the data. It also presents a case study where metric feedback guides successive changes to a synthetic data generator.

Critical strengths
  1. Validity, fidelity and diversity answer different questions. A dataset can match the reference distribution while containing incorrect tool calls. It can also be diverse while failing to represent the intended tasks. SynAE makes these differences explicit and evaluates several parts of an agent trajectory. The modular design is useful because different datasets expose different components and may have different validation requirements.

  2. The paper studies controlled changes in duplication, masking and invalidity. It also evaluates NeMo-generated data and includes a comparison with APIGen-MT. The appendix reports human agreement for the validity judge and examines sensitivity to sample size, embeddings and metric aggregation. The template-versus-slot diversity experiment is particularly useful. It shows why variation in wording should not automatically be treated as broader semantic coverage.

Critical weaknesses
  1. The main text and robustness appendix name different default embedding models without explaining the mapping. The k-step planning formula weights distances by raw prefix counts but does not show a normalization term. As written, its scale changes with dataset size. The aggregate scores also need an explicit normalization rule and reference range. Since the paper is intended to compare candidate datasets, the same candidate should not receive a different interpretation just because the comparison set or implementation choices changed.

  2. Vendi requires a positive semidefinite similarity matrix. However, the proposed normalized Levenshtein similarity does not guarantee this. Can the authors specify a valid kernel or explain how negative eigenvalues are handled and whether this changes the results?

  3. The human evaluation covers 100 synthetic T1 examples. The default Mistral judge has an F1 score of 0.86 and κ\kappa of 0.61. The paper also reports 21 disagreements with human annotations. These include cross-turn inconsistencies and tool-use errors. The stronger-judge comparison is useful but remains on the same T1 sample. There is no corresponding human validation for BFCL or ACP. The downstream equivalence judgments are not separately validated either. This leaves uncertainty about whether the reported differences reflect dataset quality or evaluator errors.

  4. The framework allows the agents used by SynAE to differ from the agent that will eventually be evaluated. However, the experiments only use three agents with backbones ranging from 1B to 8B. They do not test whether benchmark selection using these agents transfers to held-out agents. Ranking correlation over only three models is also a coarse signal. Preserving this ordering does not establish that a synthetic dataset will preserve meaningful performance differences for other agents.

  5. In Table 13, does aggregate fidelity, FID alone and KNN-Precision plus KNN-Recall all select the same dataset? The random baseline is only one draw. This does not show that the larger metric suite leads to better decisions than simpler alternatives. The selected relabeling dataset also has validity of only 0.82. Its low TDD therefore does not by itself establish that it is a suitable evaluation benchmark. A stronger test would evaluate decisions that jointly account for validity and coverage against independently verified outcomes.

Questions for authors

Can the authors please clarify the following:

  1. Which exact observations and correlation estimator were used for Table 12? Please explain why the results cannot be reproduced from the five NeMo-Nemotron settings in Table 8.

  2. How are negative eigenvalues handled in the tool-call Vendi calculation? Does the implementation modify the similarity matrix or its spectrum? Does this occur in the evaluated datasets?

  3. How well do the validity and downstream equivalence judges agree with human or native benchmark checks on BFCL and ACP? Does using the stronger judge change the main dataset comparisons?

  4. Can the framework select synthetic benchmarks using one set of agents and preserve evaluation behavior for held-out agents? This would directly test the proposed use of proxy agents.

  5. Does a selection rule using validity, fidelity and diversity outperform simpler rules on independently verified benchmark quality? Please compare against repeated random selection rather than a single draw.

  6. Which embedding model is used for each metric? Please also specify the normalization for k-step planning and the aggregate scores.

Reviewer_MZXYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a practical problem: researchers often replace or augment a real agent benchmark with synthetic trajectories, for privacy reasons or because the real set is too small, but there is no good way to check whether the synthetic data is a good substitute.

The paper proposes SynAE, a framework that measures synthetic multi-turn tool-calling trajectories from three sides: validity (whether the tool calls and outputs actually fulfill the instructions, judged by an LLM by default), fidelity (how similar the synthetic data is to the real data, over instructions/responses, tool calls, outputs, and downstream agent behavior), and diversity (Vendi score plus an attribute-based entropy measure). It is instantiated on three benchmarks (T1, BFCL multi-turn, ACP) and tested with four controlled corruption methods (blank filling, oversampling, in-context generation, invalidation) plus NVIDIA NeMo generation with different backbones and temperatures. The main findings are that the metrics track these controlled changes in the expected directions, and that no single metric is enough to describe synthetic data quality. A case study shows how a data owner could use SynAE to diagnose and repair a skewed dataset.

Critical strengths
  1. The problem is real and, as far as I know from the literature, not well addressed before. Previous checks on synthetic agent benchmarks are mostly single-turn, only look at the instructions, or are qualitative checklists. This paper scores the whole trajectory (instructions, tool calls, outputs) with quantitative metrics, and also compares downstream agent behavior on real vs. synthetic data. Two results I found useful: the fidelity-diversity trade-off (Fig. 6), and the observation that simple surface metrics keep improving with more in-context examples while the actual quality does not (Fig. 7a, Table 20). These support the paper's message that a single score is not enough.
  2. The authors are honest about their own framework. They check the judge against 100 human labels and report the types of disagreement, they test sensitivity to sample size and embedding model, they release code and prompts, and they state limitations plainly. The case study in Sec. 3.1.3 is written in a way that a benchmark owner could actually follow.
Critical weaknesses
  1. The contribution is mostly putting existing metrics together (Vendi, FID, KNN precision/recall, TV and Wasserstein distances, Spearman correlation). The new part is the organization into validity/fidelity/diversity over the trajectory components, which is useful but conceptually limited. More importantly, the claim I care about most — that intrinsic metrics can predict downstream behavior, so candidates can be screened before running expensive downstream evaluation — is supported by a correlation over about five NeMo temperature variants (Table 12), a rank correlation over only three agents, and a selection demo with four candidates (Table 13). From what is shown I cannot tell whether this association is a robust property or an artifact of one generator family.
  2. Most of the validation checks things the authors designed to move. For example the validity rate drops with slope −1 under invalidation because that is exactly the injected rate; this shows the pipeline works, but not that the metrics measure what they claim. The external checks are few: 100 human labels with κ = 0.61 for the validity pillar, and I did not see any independently produced synthetic dataset with known downstream results used to validate the fidelity metrics. Also, in Table 1 most metric cells are N/A for BFCL and ACP, so in practice the framework is mainly demonstrated on T1. Overall I see SynAE as a useful diagnostic tool for relative comparisons inside one setup, but I am not convinced yet that its scores can be used to accept or reject a synthetic benchmark, and I am not sure whether practitioners will actually adopt it in their pipelines.
Questions for authors
  1. Can you add more data points to the intrinsic-downstream correlation—more generators or domains, per-task scores, or confidence intervals over agents and tasks? From the current evidence I cannot tell whether the association in Table 12 is a robust property or specific to the NeMo temperature variants, and this is the part of the paper I would most like to see strengthened.
  2. Is there any synthetic dataset produced outside your own pipelines—for example one published by another group with reported downstream results—on which SynAE's scores anticipated the outcome? Even a single such case would help my confidence more than additional ablations.
  3. In the case study, validity 0.82 is called "poor" and 0.98 "strong", but the default judge agrees with humans at κ = 0.61 and about half of the disagreements are parsing errors. How many annotators labeled the 100 examples in Appendix B.4, and do the Table 2 verdicts hold under the stronger judges in Table 3?
Reviewer_8jwfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SynAE is a framework for evaluating synthetic datasets used to test multi-turn, tool-calling agents. It assesses synthetic trajectories along three dimensions: validity, fidelity to a reference dataset, and diversity. Its metrics cover instructions and responses, tool calls, final outputs, and downstream agent performance.

Experiments on T1, BFCL, and ACP show that SynAE can detect controlled problems such as invalid tool calls, distribution shifts, and duplicated samples. The results also demonstrate trade-offs: increasing diversity can reduce fidelity, and preserving model rankings does not necessarily preserve tool-use patterns. A case study illustrates how the framework can help diagnose a biased dataset and compare alternative augmentation strategies.

Critical strengths

The paper assesses a consequential scope including validity, fidelity, and diversity across instructions, tool calls, outputs, and downstream agent performance.

Controlled perturbations and the augmentation case study show that metrics can reveal different failure modes and trade-offs.

Critical weaknesses

The reported tool-planning experiments focus on short action sequences, and the paper does not directly test whether the full framework reliably detects errors involving dependencies across distant steps. This leaves uncertainty about its ability to assess the fidelity of longer trajectories.

The downstream analysis uses only three agents. This small agent set limits confidence that the findings extend to a broader range of model capabilities and behaviors.

Synthetic perturbations such as masking, duplication, and answer replacement represent predefined defects. It remains unclear whether the metrics can reliably identify the more varied and subtle errors that arise in practical synthetic-data generation.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.