acceptodds
Under review as a conference paper at ICLR 2027

When Metrics Mislead: Auditing the Measurement Layer of LLM Agent Evaluation

Abstract

LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are useful, but they are measurements produced by parsers, behavioral detectors, and harness rules. We study what happens when those measurements do not fully capture the trace's behavior. We use GSM8K, a benchmark of grade-school arithmetic word problems, in a tool-use setting where agents can call a calculator whose outputs are occasionally corrupted. This gives us a clear ground truth for the expected tool result and allows us to compare what the calculator returned, what the model later used, and what the automated detectors reported. We audit three trace-level detectors across six production model families and 30 problems per model. We find that final correctness, visible re-verification, and fault propagation can disagree in systematic ways. An important failure mode is the silent tool-output override. In these cases, the calculator first returns a corrupted value, but the model later uses a different, often correct, value without calling the calculator again. The bad value may therefore stop propagating even though a verification detector that looks for explicit re-execution records no visible re-check. Human review of 90 traces shows perfect agreement with the reviewed comparable cases for final correctness (67/67), fault recovery/impact (19/19), and fault relevance (18/18), but only 67/81 agreement for verification. All 14 verification disagreements fall into four recurring patterns: repeated calculator calls from stuck loops, verbal recognition of an incorrect tool result without recomputation, repeated expressions within a single reasoning step, and quiet correction after an incorrect value is observed. We also find that execution limits affect the reported result. In an ablation study across two models, one model ranges from 21/30 to 27/30 correct as step and token budgets change, while another remains between 27/30 and 29/30. These findings motivate measurement transparency: an automated label should be accompanied by trace evidence supporting it, whether the detector's question applied to that trace, whether the run ended early due to an execution limit, and any human corrections to the label. Our trace viewer makes this information inspectable alongside the reported results.

Primary area:
interpretability and explainable AI
Keywords:
LLM agents, agent evaluation, measurement validity, trace-level evaluation, detector reliability, tool-use agents, fault injection, human adjudication, evaluation transparency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_E1doOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper audits the measurement layer of LLM agent evaluation using GSM8K problems and a calculator with injected numerical faults. Across six model families and 30 questions per model, it compares final-answer correctness, explicit calculator re-checking, and fault propagation, and examines silent tool-output overrides and execution-budget effects. It proposes reporting detector labels together with evidence, applicability, censoring, and human corrections, and presents a trace-inspection viewer. A human audit covers 90 of the 180 baseline traces.

Critical strengths

None.

Critical weaknesses
  1. Unclear methodological contribution. I do not see a sufficiently specified and independently validated method for accurately measuring agent behavior. The main proposal is to attach evidence, applicability, censoring, and human corrections to detector labels. But it does not demonstrate a reusable improvement in measurement accuracy.

  2. Narrow empirical scope. All experiments use GSM8K arithmetic problems with a calculator, with only 30 questions per model. Arithmetic provides useful external ground truth, but six models on the same task and tool do not establish generality across agent workflows. There is no validation on another dataset or tool type, such as retrieval, code execution, or file manipulation.

  3. Insufficient operational detail for reproduction. The paper does not adequately specify the exact fault-injection probability and sampling rules, the selected question list, or the parser and detector algorithms.

  4. The figures do not clarify the method sufficiently. Figure 1 is a rasterized JPEG workflow diagram rather than a crisp vector schematic, and it names the detector stages without explaining their decision rules. Figure 2 consists of dense viewer screenshots with very small text.

Questions for authors

Same weaknesses.

Reviewer_666YOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper questions the measurement layer of agentic language model evaluation. In some sense a meta evaluation of how detectors miss things by reducing traces to final answer correctness, tool verification and fault propagation. That parsers and rule detectors miss some information that is available in the trace but this behaviour is not detected. They use GSM8k because of its simplicity and the fact that it uses a tool that can be controlled well that of a calculator which they corrupt with some probability. They use multiple models and evaluate them on 30 GSM8k traces with this faulty calculator. The major "failure" mode they detect is silent tool output override, where the calculator provides a wrong calculation but the model silently uses the correct output (not always) later, this is done without calling the tool again.

Their contributions can be summed up as :

  1. Suggest each detector label should come with its evidence, whether it applied, whether the run was cut off, and any human correction (C1),
  2. an audit of three detectors across six models on 30 problems each, plus a step and token budget ablation on two models (C2), and
  3. a trace viewer that shows labels next to the trace evidence (C3).

For validation, they also do a human review of these audits and report agreement with the model on correctness, recovery, and fault relevance for the comparable cases, and 67/81 for verification, with all 14 disagreements falling into recurring patterns (which they elaborate on).

Critical strengths
  1. The controlled setup is a good way of measuring something that is known about in the agentic evaluation space but not studied in a systematic manner. Given that arithmetic has ground truth at every step, the authors can separate what the tool should have returned, what it actually returned, what the model used, and what the detector said, which is difficult to do in most agentic benchmarks.
  2. The other finding that these measurement gaps show up across several model families, and that the verification disagreements fall into a small number of recognizable patterns is also interesting. The recurring patterns namely being (copied from the paper): (a) repeated calculator calls caused by a stuck loop, (b) Verbal recognition of an incorrect tool result without an explicit recomputation (c) repeated expressions within a single reasoning step. (d) Quiet correction after an incorrect value is observed.
  3. Silent tool call correction/change has been observed to exist across different models.
Critical weaknesses
  1. The choice of GSM8K. GSM8K is saturated for most available models, as the paper’s own results show: claude-sonnet-5 gets 30/30, claude-opus-4.8 and gpt-oss-120b get 28/30, and nemotron gets every problem it finishes right. The arithmetic is easy enough that the model can redo it in its head, so silently fixing a corrupted calculator output is the expected behavior here. It would be interesting to observe this in more difficult problems where such a correction might not be possible and the model uses a different value which is also wrong, or there are corruptions from multiple tools. Even using MATH (Hendrycks et al.) would have been a good choice, given the ability to use different tools for like symbolic or code computation. The authors do acknowledge the arithmetic limitation, but since the paper’s main argument is about agentic evaluation, i think this is more central. MATH is also saturated, but offers a bigger surface area for observing this silent repair.
  2. Ultimately each model is evaluated on 30 problems with a single tool, which shows these failure modes exist but does not say much about how often they happen when multiple tools exist or how that might influence performance if the repair is beyond the models ability or how models compare.
Questions for authors
  1. Given (W1), could the authors produce some examples of MATH traces or any other agentic evaluation where they observe this behaviour? It does not need to be rigorous large study, and some examples would be fine. The main things I would want to see are whether the silent override rate drops and whether the gap between D-verify and D-impact still appears when the model can’t silently repair the tool output.
  2. Given (W2), do the authors expect different model families to have different behaviours in general agentic evals?
  3. (does not impact the score) How do you expect this to generalize in the case of agentic evals in coding/web based evaluation? What is the impact present in those cases, how would it be interesting to detect these errors in the traces?
  4. Could you shed more light on the interactive trace inspection platform? What are your learnings from building it and what is important to get right when making such a tool?
Reviewer_yogcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper frames agent evaluation as a measurement process and audits the labels that trace detectors assign. Agents solve 30 GSM8K problems per model in a ReAct-style loop with a calculator; on a configured fraction of calculator calls, a fault injector returns a plausible wrong number (for example a small offset or a digit transposition). Three detectors are audited: D-correct (parsed final answer against the gold answer), D-verify (explicit calculator re-check versus only a claim of checking) and D-impact (whether a corrupted value appears in later arithmetic). Six production model families are evaluated under a baseline of steps=50 and tokens=1024 (Sec. 3.2, Table 3).

The motivating phenomenon is silent tool-output override: the calculator returns a corrupted value and the model later continues with a different, often correct, value without a new tool call, so D-impact may record non-propagation while D-verify records no visible re-check (Sec. 4.1). The authors hand-review 90 of 180 traces and report agreement of 67/67 for D-correct, 19/19 for D-impact and 18/18 for fault relevance, but 67/81 for D-verify (Table 5). The 14 D-verify disagreements are grouped into four patterns, and 10 of them come from nemotron-3-ultra (Table 6). A harness ablation over step limits {25, 50, 150} and token limits {1024, 4096} on two models (Table 4) shows nemotron-3-ultra ranging from 21/30 to 27/30 correct while gpt-oss-120b stays between 27/30 and 29/30. The paper proposes a measurement-transparency reporting principle (C1), presents the audit itself (C2) and a trace viewer that keeps automatic labels next to human corrections (C3).

Critical strengths

The paper is candid about scope: it does not claim to discover tool-memory conflict (Sec. 1, Sec. 2), and it discloses that the detector rules were revised after manual review (Sec. 4.3, Sec. 5.1). The arithmetic setting gives real ground truth for the tool value, so it can cleanly separate what the tool returned, what the model used, and what the final answer was. That is a good design for auditing trace detectors. Automatic labels are kept next to human corrections instead of being overwritten (Sec. 3.5). Censored runs are reported apart from wrong answers (Table 3). The paper also notes that per-call fault injection gives more fault exposure to models that call the tool more often (Sec. 2, Sec. 5.1). The four D-verify disagreement patterns (Sec. 4.4) are concrete and useful to practitioners.

Critical weaknesses
  1. The human-agreement evidence is measured on the tuning data. One primary reviewer (Sec. 5.1) labeled traces, found coverage gaps, revised the detector rules, reran them, and re-checked the same traces (Sec. 4.3). The abstract's "perfect agreement" (67/67, 19/19, 18/18) is therefore measured on the same traces used to revise the detectors. The paper does not say how the 90 of 180 traces were chosen. There is no before/after-revision agreement, no second annotator and no kappa. "Fault relevance" appears in Table 5 and the abstract but is not defined as a detector in Table 1. Even at face value, 0 errors in 19 or 18 cases only bounds the error rate below about 18 to 19% (upper end of the exact two-sided 95% interval). The disagreements are also not split by direction. Patterns 1 and 3 in Sec. 4.4 are cases where the detector over-credits verification, while the motivating override story is about recovery that goes uncredited. Without that split, it is unclear whether the audit supports the motivating story.

  2. The verification findings are dominated by one loop-prone model, and clustered counts are treated as independent. 10 of the 14 D-verify disagreements come from nemotron-3-ultra (10/17 reviewed, Table 6). If the Faults denominators in Table 4 are calculator calls, nemotron makes about 29 to 147 calls per trace (e.g. 3627/30 = 121 at steps=50, tokens=4096), versus about 2 per trace for gpt-oss-120b (66/30). That looks like stuck-loop behavior, which is also one of the four disagreement patterns. The authors note the exposure confound (Sec. 5.1), but the analysis does not adjust for it. Counts such as 0/460 propagated come from at most 30 traces, not 460 independent events. As it stands, the audit mainly shows that a loop-prone model confuses a re-execution detector. That is narrower than the title and abstract suggest.

  3. The ablation and cross-model numbers have no uncertainty estimates, which is the paper's own failure class F6 ("A single batch percentage is treated as a stable rate without uncertainty"). Each cell is one run on n=30, and the binomial SE of an accuracy near 0.75 is about 2.4 questions. The baseline configuration also gives different numbers in Tables 3 and 4. nemotron at steps=50, tokens=1024 is 20/30 with 155 fault events in Table 3 but 21/30 with 174 in Table 4. gpt-oss-120b is 28/30 (verify 8/30) in Table 3 and 27/30 (verify 7/30) in Table 4. Run-to-run variation is therefore similar in size to many of the reported effects, and accuracy is not monotone in steps (23, 21, 24 at tokens=1024). Table 4 also does not separate censored runs from completed-wrong runs, so the budget effect cannot be tied to the censoring mechanism (F7) the paper highlights. The silent-override counts that motivate the paper are never validated against human labels (Table 5 has no override row).

Notes on the Presentation, Informativeness and Soundness scores

Presentation: The paper is clearly written and easy to follow, the setup is described in plain terms, and the limitations section (Sec. 5.1) is candid about the single reviewer, the small sample and the detector revision.

A few presentation issues reduce how much the reader can take from it. Table 2 lists seven failure classes, but F2 (parser loss), F4 (formatting sensitivity) and F5 (instrument assumption) are not counted or discussed in the results (C2 mentions "formatting failures" without numbers), and F4 has only the illustrative "184-145" versus "184 -145" example in the table itself. "Fault relevance" appears in Table 5 and the abstract but is not one of the three detectors in Table 1 and is not defined. Units are mixed in Table 3 (the caveat for one of the models gives a per-fault-event figure, 5 of 15, next to per-trace counts). Table 1 describes D-verify as an explicit re-check "or only a claim of checking", but the "+3 claimed" entries in Table 3 are not explained, so the claimed-only criterion is not operationalised. The abstract's "perfect agreement" goes further than the evidence supports (see Weakness 1).

Minor: Sec. 5 has a stray parenthesis after "Figure 1".

Informativeness: The setting is well chosen for the question asked: because the correct tool value is known, the paper can separate what the tool returned, what the model used and what the detector reported.

The most useful finding is that one loop-prone model (nemotron-3-ultra) accounts for most of the verification disagreements (10 of 14), and that step and token budgets move that model's reported accuracy by several questions. The core observation, that a detector looking for explicit re-execution will not credit recovery that happens without re-execution, follows from the detector's definition, and the authors acknowledge this in Sec. 4.1 ("working as defined"). Because the audited detectors are three rules from the authors' own pipeline rather than detectors used in established harnesses, it is not yet clear how much the findings tell us about LLM agent evaluation as practised more broadly.

Soundness: The study is carefully described and its reported arithmetic is internally consistent (Table 3 outcomes sum to 30 per model, Table 6 sums to 14 disagreements over 81 reviewed traces, and 67/81 = 82.7%).

The concerns are fixable with additional experiments, but they bear directly on the headline claims: every cell is a single run on 30 questions with no uncertainty estimate (the paper's own failure class F6); the human-agreement figures come from one reviewer on the same traces used to revise the detectors; the silent-override counts that motivate the paper have no human validation; and fault-level counts are pooled across traces as if independent. In addition, the same baseline configuration yields different numbers in Table 3 and Table 4, and backend errors are handled inconsistently: Table 3 records eight Mistral-Large-3-675B backend-error traces as "not completed", a category that also holds nemotron-3-ultra's step-limit hits, while Table 4 excludes one backend-error trace from the denominator (27/29).

Questions for authors

Clarification requests

  1. What is the per-call fault probability? (Table 4's ratios suggest roughly 13-15%, 18% in one cell.) What sampling temperature was used, which fault seed was used for the Table 3 baseline, and how were the 30 questions chosen? Are they the same for every model?

  2. Why do Table 3 and Table 4 differ for the same baseline configuration (nemotron 20/30 vs 21/30 and 155 vs 174 faults; gpt-oss-120b 28/30 vs 27/30)? Different seed, or a separate run?

  3. How were the 90 of 180 traces selected? Why are only 67 comparable for D-correct, when about 22 of 180 traces did not complete?

  4. How many rules changed after the first human pass, and what was agreement before the revision?

  5. Is the Faults denominator in Table 4 the number of calculator calls? Are inconclusive D-impact cases excluded from the 19/19?

  6. How is "fault relevance" defined and computed?

  7. Of the 14 D-verify disagreements, how many are the detector over-crediting verification and how many are it missing verification?

  8. In Table 4, how many failures per cell are censored (step or token limit) and how many are completed-wrong?

Suggestions (any of these could change my score)

  1. Report precision and recall of the override detector against human labels, and the share of override cases where the later value is correct rather than another wrong value.

  2. Repeat the baseline and the ablation over several fault seeds and report paired CIs (McNemar or bootstrap by question). Use per-trace (cluster-aware) counts, and cap or separately report looping traces.

  3. Add a second annotator, blind to detector output, on traces sampled before any rule revision, and report kappa and before/after agreement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.