-
The human-agreement evidence is measured on the tuning data. One primary reviewer (Sec. 5.1) labeled traces, found coverage gaps, revised the detector rules, reran them, and re-checked the same traces (Sec. 4.3). The abstract's "perfect agreement" (67/67, 19/19, 18/18) is therefore measured on the same traces used to revise the detectors. The paper does not say how the 90 of 180 traces were chosen. There is no before/after-revision agreement, no second annotator and no kappa. "Fault relevance" appears in Table 5 and the abstract but is not defined as a detector in Table 1. Even at face value, 0 errors in 19 or 18 cases only bounds the error rate below about 18 to 19% (upper end of the exact two-sided 95% interval). The disagreements are also not split by direction. Patterns 1 and 3 in Sec. 4.4 are cases where the detector over-credits verification, while the motivating override story is about recovery that goes uncredited. Without that split, it is unclear whether the audit supports the motivating story.
-
The verification findings are dominated by one loop-prone model, and clustered counts are treated as independent. 10 of the 14 D-verify disagreements come from nemotron-3-ultra (10/17 reviewed, Table 6). If the Faults denominators in Table 4 are calculator calls, nemotron makes about 29 to 147 calls per trace (e.g. 3627/30 = 121 at steps=50, tokens=4096), versus about 2 per trace for gpt-oss-120b (66/30). That looks like stuck-loop behavior, which is also one of the four disagreement patterns. The authors note the exposure confound (Sec. 5.1), but the analysis does not adjust for it. Counts such as 0/460 propagated come from at most 30 traces, not 460 independent events. As it stands, the audit mainly shows that a loop-prone model confuses a re-execution detector. That is narrower than the title and abstract suggest.
-
The ablation and cross-model numbers have no uncertainty estimates, which is the paper's own failure class F6 ("A single batch percentage is treated as a stable rate without uncertainty"). Each cell is one run on n=30, and the binomial SE of an accuracy near 0.75 is about 2.4 questions. The baseline configuration also gives different numbers in Tables 3 and 4. nemotron at steps=50, tokens=1024 is 20/30 with 155 fault events in Table 3 but 21/30 with 174 in Table 4. gpt-oss-120b is 28/30 (verify 8/30) in Table 3 and 27/30 (verify 7/30) in Table 4. Run-to-run variation is therefore similar in size to many of the reported effects, and accuracy is not monotone in steps (23, 21, 24 at tokens=1024). Table 4 also does not separate censored runs from completed-wrong runs, so the budget effect cannot be tied to the censoring mechanism (F7) the paper highlights. The silent-override counts that motivate the paper are never validated against human labels (Table 5 has no override row).
Notes on the Presentation, Informativeness and Soundness scores
Presentation: The paper is clearly written and easy to follow, the setup is described in plain terms, and the limitations section (Sec. 5.1) is candid about the single reviewer, the small sample and the detector revision.
A few presentation issues reduce how much the reader can take from it. Table 2 lists seven failure classes, but F2 (parser loss), F4 (formatting sensitivity) and F5 (instrument assumption) are not counted or discussed in the results (C2 mentions "formatting failures" without numbers), and F4 has only the illustrative "184-145" versus "184 -145" example in the table itself. "Fault relevance" appears in Table 5 and the abstract but is not one of the three detectors in Table 1 and is not defined. Units are mixed in Table 3 (the caveat for one of the models gives a per-fault-event figure, 5 of 15, next to per-trace counts). Table 1 describes D-verify as an explicit re-check "or only a claim of checking", but the "+3 claimed" entries in Table 3 are not explained, so the claimed-only criterion is not operationalised. The abstract's "perfect agreement" goes further than the evidence supports (see Weakness 1).
Minor: Sec. 5 has a stray parenthesis after "Figure 1".
Informativeness: The setting is well chosen for the question asked: because the correct tool value is known, the paper can separate what the tool returned, what the model used and what the detector reported.
The most useful finding is that one loop-prone model (nemotron-3-ultra) accounts for most of the verification disagreements (10 of 14), and that step and token budgets move that model's reported accuracy by several questions. The core observation, that a detector looking for explicit re-execution will not credit recovery that happens without re-execution, follows from the detector's definition, and the authors acknowledge this in Sec. 4.1 ("working as defined"). Because the audited detectors are three rules from the authors' own pipeline rather than detectors used in established harnesses, it is not yet clear how much the findings tell us about LLM agent evaluation as practised more broadly.
Soundness: The study is carefully described and its reported arithmetic is internally consistent (Table 3 outcomes sum to 30 per model, Table 6 sums to 14 disagreements over 81 reviewed traces, and 67/81 = 82.7%).
The concerns are fixable with additional experiments, but they bear directly on the headline claims: every cell is a single run on 30 questions with no uncertainty estimate (the paper's own failure class F6); the human-agreement figures come from one reviewer on the same traces used to revise the detectors; the silent-override counts that motivate the paper have no human validation; and fault-level counts are pooled across traces as if independent. In addition, the same baseline configuration yields different numbers in Table 3 and Table 4, and backend errors are handled inconsistently: Table 3 records eight Mistral-Large-3-675B backend-error traces as "not completed", a category that also holds nemotron-3-ultra's step-limit hits, while Table 4 excludes one backend-error trace from the denominator (27/29).