- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes STED, a type-aware agreement score for hierarchical structured outputs. It retains SLOT’s exact terminal-path correspondence and soft-F1 aggregation while introducing graded comparisons for numeric values, literals, and text. The score is evaluated using human reference-relative comparisons, numerical-equivalence tests, BFCL-accepted argument alternatives, and external judgments and execution labels.
The fixed score is then applied to approximately 2.1 million repeated generations from 18 models. The application separates conditional consistency from validity and prompt coverage and examines temperature and schema-enforcement effects.
Critical strengths
1.Separating consistency, validity, and prompt coverage is practically useful. The paper demonstrates how changes in coverage can reverse an aggregate trend and how higher conditional consistency can coexist with lower validity. This improves the interpretation of repeated-output evaluations.
2.The submission provides extensive controlled comparisons and transparently reports mixed results. Matched-text SLOT controls, rule removals, stricter acceptance budgets, and external tests help identify where individual rules help and where STED does not outperform existing alternatives.
Critical weaknesses
1.The strongest human-preference evidence is not independent validation. All three annotators are co-authors, know the research hypothesis, and supplied labels used in configuration selection. The supplement was additionally selected for disagreements between an earlier STED configuration and SLOT. The paper discloses these facts, but bootstrap intervals and multiple-comparison correction do not remove configuration-selection bias. Frozen external model judgments establish no significant STED–SLOT advantage and do not replace an independent human test. Consequently, the generalization of the human-agreement improvements remains unproven.
2.The incremental methodological benefit is narrow and target-dependent. STED inherits its correspondence and aggregation from SLOT; the strongest independent gain concerns numerical-string equivalence, which direct canonicalization can solve. Other tests are mixed: CPython family-macro AUC is .694 versus .722 for matched-text SLOT, and BFCL corpus-negative pooled AUC is .655 versus .851 for index-free SLOT. On the hard-negative set at the 0.1% calibration budget, STED accepts 85 positives versus 210 for index-free SLOT, with three versus four false accepts. The large generation corpus supplies no independent pair-equivalence labels and uses heterogeneous serving paths. Thus its scale does not establish that the added comparison rules provide a generally better measure of structured-output consistency.
Questions for authors
1.What independent evidence supports the near-miss and graded callee-name rules on unseen structured-output comparisons, beyond the development judgments? These rules account for important human-agreement gains but lack a demonstrated external preference advantage.
2.Can the numerical-equivalence results include a direct numeric-canonicalization baseline under the same path correspondence and aggregation? This would clarify the incremental benefit of STED over a simpler type-aware extension.
3.Are the principal model-ordering and enforcement conclusions stable when the same outputs are rescored using matched-text SLOT or its numeric-canonicalization extension? Reporting these sensitivities would clarify how much of the application depends on STED-specific rules.