acceptodds
Withdrawn from ICLR 2027

STED: MEASURING CONSISTENCY IN LLM STRUCTURED OUTPUTS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Structured responses are common in industrial applications of large language models (LLMs), with tool-calling agents a prominent use case. Yet even at temperature , repeated prompts can produce inconsistent tool choices and argument values despite parseable responses. We propose STED (Semantic Tree Edit Distance), a type-aware score for comparing hierarchical structured outputs. It matches terminal values at corresponding tree paths and aggregates their graded agreement using soft precision and recall. On 208 structural human comparisons used during development, STED makes 88.0% correct decisions versus 79.8% for a field-path similarity baseline, counting metric ties as unsuccessful decisions. Frozen numerical-equivalence tests isolate a benefit of interpreting numerical literals, with within-value-group AUCs of .999–1.000 versus .612–.619 for a control with matched text processing. Textual relatedness and external model judgments show no consistent improvement. We apply the fixed score to 2.1M repeated generations from 18 LLMs, measuring consistency separately from validity and prompt coverage. The observed rankings and model-dependent schema-enforcement effects show why repeatability adds information to model selection alongside task quality.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, structured output, tool calling, similarity metrics, tree edit distance, evaluation methodology, execution-based validation, adversarial robustness, consistency benchmark, threshold soundness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UsT7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STED, a type-aware agreement score for hierarchical structured outputs. It retains SLOT’s exact terminal-path correspondence and soft-F1 aggregation while introducing graded comparisons for numeric values, literals, and text. The score is evaluated using human reference-relative comparisons, numerical-equivalence tests, BFCL-accepted argument alternatives, and external judgments and execution labels.

The fixed score is then applied to approximately 2.1 million repeated generations from 18 models. The application separates conditional consistency from validity and prompt coverage and examines temperature and schema-enforcement effects.

Critical strengths

1.Separating consistency, validity, and prompt coverage is practically useful. The paper demonstrates how changes in coverage can reverse an aggregate trend and how higher conditional consistency can coexist with lower validity. This improves the interpretation of repeated-output evaluations.

2.The submission provides extensive controlled comparisons and transparently reports mixed results. Matched-text SLOT controls, rule removals, stricter acceptance budgets, and external tests help identify where individual rules help and where STED does not outperform existing alternatives.

Critical weaknesses

1.The strongest human-preference evidence is not independent validation. All three annotators are co-authors, know the research hypothesis, and supplied labels used in configuration selection. The supplement was additionally selected for disagreements between an earlier STED configuration and SLOT. The paper discloses these facts, but bootstrap intervals and multiple-comparison correction do not remove configuration-selection bias. Frozen external model judgments establish no significant STED–SLOT advantage and do not replace an independent human test. Consequently, the generalization of the human-agreement improvements remains unproven.

2.The incremental methodological benefit is narrow and target-dependent. STED inherits its correspondence and aggregation from SLOT; the strongest independent gain concerns numerical-string equivalence, which direct canonicalization can solve. Other tests are mixed: CPython family-macro AUC is .694 versus .722 for matched-text SLOT, and BFCL corpus-negative pooled AUC is .655 versus .851 for index-free SLOT. On the hard-negative set at the 0.1% calibration budget, STED accepts 85 positives versus 210 for index-free SLOT, with three versus four false accepts. The large generation corpus supplies no independent pair-equivalence labels and uses heterogeneous serving paths. Thus its scale does not establish that the added comparison rules provide a generally better measure of structured-output consistency.

Questions for authors

1.What independent evidence supports the near-miss and graded callee-name rules on unseen structured-output comparisons, beyond the development judgments? These rules account for important human-agreement gains but lack a demonstrated external preference advantage.

2.Can the numerical-equivalence results include a direct numeric-canonicalization baseline under the same path correspondence and aggregation? This would clarify the incremental benefit of STED over a simpler type-aware extension.

3.Are the principal model-ordering and enforcement conclusions stable when the same outputs are rescored using matched-text SLOT or its numeric-canonicalization extension? Reporting these sensitivities would clarify how much of the application depends on STED-specific rules.

Reviewer_qcjROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes STED - improvement of SLOT, score used to determine the similarity between 2 objects in JSON format or similar objects with tree-structured objects. Authors build on SLOT changing some type-specific similarity measures

Critical strengths

Authors develop an important topic, as models’ structured outputs are widely used nowadays.

Authors implement type-specific similarity scores.

Critical weaknesses

Insufficient novelty: Main improvement being type-specific similarity function for terminal objects, the functions themselves are not that innovative

The exact method explanation is incomplete.

Concerns about the generality of the experimental sets: I didn't fully understand the choices of such experimental sets.

Questions for authors
  1. Please elaborate on novelty: why do you consider this change as significant novelty?

  2. Please provide detailed algorithm for computing S_sted

  3. Please explain experimental sets choice

  4. Which parameters of baselines like BERTScore, TED, DeepDiff were used?

Reviewer_aykxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper tackles the problem of evaluating consistency of LLM outputs by defining a tree-based similarity metric over the structured objects.

Critical strengths

Evaluating the consistency of LLM structured outputs is a real problem.

Critical weaknesses
  • The motivation of nondeterminism in LLM inference is a bit strange - seems like we're tackling a very narrow problem. There is the general problem of determininig whether two structured outputs are equivalent (which usually is ground truth versus model prediction).
  • It would be nice to have a concrete running example showing an example of the metric being computed to help the readers understand what regime we're in.
  • Is human comparisons the best way to evaluate structured outputs? Presumably structured outputs tend to be handled programmatically (e.g., by an agent), so it's not clear that a human staring at JSON output makes sense.
  • More motivation for the underlying problem setting would be helpful here. It seems like BFCL is the main evaluation dataset, which is quite limiting.
Questions for authors
  • Are there cases where the strictly recursive approach doesn't make sense?
  • The embedding e is doing a lot of the work; any guidance on what this should be set to?
  • How would your method compare to simply feeding both outputs into a frontier model and ask it to evaluate similarity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.