- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SynAE. It is a framework for evaluating synthetic datasets used to test tool-calling agents. The framework measures validity, fidelity and diversity across instructions, tool calls, final outputs and downstream agent performance.
SynAE combines existing distributional and embedding metrics with measures of tool usage and tool call transitions. It also uses an LLM judge to assess validity and compare generated agent traces with reference traces. Downstream evaluation measures differences in task success and whether model rankings are preserved.
The experiments use subsets of T1, BFCL and ACP as reference datasets. Synthetic datasets are created through controlled modifications and generation pipelines such as NeMo. The paper shows that different metrics capture different changes in the data. It also presents a case study where metric feedback guides successive changes to a synthetic data generator.
Critical strengths
-
Validity, fidelity and diversity answer different questions. A dataset can match the reference distribution while containing incorrect tool calls. It can also be diverse while failing to represent the intended tasks. SynAE makes these differences explicit and evaluates several parts of an agent trajectory. The modular design is useful because different datasets expose different components and may have different validation requirements.
-
The paper studies controlled changes in duplication, masking and invalidity. It also evaluates NeMo-generated data and includes a comparison with APIGen-MT. The appendix reports human agreement for the validity judge and examines sensitivity to sample size, embeddings and metric aggregation. The template-versus-slot diversity experiment is particularly useful. It shows why variation in wording should not automatically be treated as broader semantic coverage.
Critical weaknesses
-
The main text and robustness appendix name different default embedding models without explaining the mapping. The k-step planning formula weights distances by raw prefix counts but does not show a normalization term. As written, its scale changes with dataset size. The aggregate scores also need an explicit normalization rule and reference range. Since the paper is intended to compare candidate datasets, the same candidate should not receive a different interpretation just because the comparison set or implementation choices changed.
-
Vendi requires a positive semidefinite similarity matrix. However, the proposed normalized Levenshtein similarity does not guarantee this. Can the authors specify a valid kernel or explain how negative eigenvalues are handled and whether this changes the results?
-
The human evaluation covers 100 synthetic T1 examples. The default Mistral judge has an F1 score of 0.86 and of 0.61. The paper also reports 21 disagreements with human annotations. These include cross-turn inconsistencies and tool-use errors. The stronger-judge comparison is useful but remains on the same T1 sample. There is no corresponding human validation for BFCL or ACP. The downstream equivalence judgments are not separately validated either. This leaves uncertainty about whether the reported differences reflect dataset quality or evaluator errors.
-
The framework allows the agents used by SynAE to differ from the agent that will eventually be evaluated. However, the experiments only use three agents with backbones ranging from 1B to 8B. They do not test whether benchmark selection using these agents transfers to held-out agents. Ranking correlation over only three models is also a coarse signal. Preserving this ordering does not establish that a synthetic dataset will preserve meaningful performance differences for other agents.
-
In Table 13, does aggregate fidelity, FID alone and KNN-Precision plus KNN-Recall all select the same dataset? The random baseline is only one draw. This does not show that the larger metric suite leads to better decisions than simpler alternatives. The selected relabeling dataset also has validity of only 0.82. Its low TDD therefore does not by itself establish that it is a suitable evaluation benchmark. A stronger test would evaluate decisions that jointly account for validity and coverage against independently verified outcomes.
Questions for authors
Can the authors please clarify the following:
-
Which exact observations and correlation estimator were used for Table 12? Please explain why the results cannot be reproduced from the five NeMo-Nemotron settings in Table 8.
-
How are negative eigenvalues handled in the tool-call Vendi calculation? Does the implementation modify the similarity matrix or its spectrum? Does this occur in the evaluated datasets?
-
How well do the validity and downstream equivalence judges agree with human or native benchmark checks on BFCL and ACP? Does using the stronger judge change the main dataset comparisons?
-
Can the framework select synthetic benchmarks using one set of agents and preserve evaluation behavior for held-out agents? This would directly test the proposed use of proxy agents.
-
Does a selection rule using validity, fidelity and diversity outperform simpler rules on independently verified benchmark quality? Please compare against repeated random selection rather than a single draw.
-
Which embedding model is used for each metric? Please also specify the normalization for k-step planning and the aggregate scores.