acceptodds
Under review as a conference paper at ICLR 2027

Where the Thread Breaks: Evaluating LLM Auditors in Long-Horizon Scientific Workflows

Abstract

Advances in large language models (LLMs) have enabled autonomous agents to perform complex, long-horizon tasks. These tasks require reliable output verification, as errors can propagate across steps and compromise the final results. LLM-based auditing provides a practical approach to verifying the output of each step. However, its performance in long-horizon workflows, where later steps depend on earlier outcomes, remains underexplored. We introduce AuditSpan, a benchmark for evaluating LLM-based auditing in long-horizon scientific workflows. Specifically, we (1) examine whether and how the benefits and limitations of auditing individual outputs carry over to long-horizon workflows and (2) investigate challenges arising from interactions among workflow steps, including difficulties in assessing overall scientific validity and tracking error propagation and correction. Our results show that, when auditing extends from individual outputs to long-horizon workflows, appropriate abstention decreases and overconfidence increases. Beyond these amplified limitations, auditors may approve scientifically flawed workflows when most individual steps are correct and may treat errors as resolved even when their effects persist in later outputs. These findings can guide the development of more reliable LLM-based auditors and support progress toward fully automated long-horizon workflows.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Verification, Auditing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_P6qMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AuditSpan, a benchmark for evaluating the reliability of LLM-based auditors in long-horizon scientific workflows. AuditSpan investigates whether auditors can assess scientific validity across interdependent steps, identify propagated errors, and recognize incomplete corrections. The benchmark consists of 120 workflow instances derived from 10 synthetic scientific workflows across five domains, with controlled error injection, explicit step dependencies, and independently established reference labels. The authors propose evaluation metrics covering step-level and workflow-level correctness, abstention, confidence calibration, resistance to pressure, scientific validity, error tracing, and correction tracking. Experiments involving 12 LLMs, with and without tool access, reveal that auditing reliability deteriorates across extended workflows and that models frequently fail to identify downstream consequences of upstream errors or incorrectly consider partially repaired workflows fully resolved.

Critical strengths
  1. Well-motivated research problem and clear contributions. The paper addresses an important and relatively underexplored problem: evaluating the reliability of LLM auditors in long-horizon workflows where errors can propagate through dependent steps.

  2. Comprehensive and well-structured evaluation metrics. A major strength is the evaluation methodology, which goes beyond conventional step-level accuracy to examine workflow-level reliability and dependency-aware reasoning.

  3. Controlled benchmark construction and informative empirical analysis. The benchmark uses explicit workflow dependencies, controlled error perturbations, and independently established reference labels, enabling systematic analysis of auditing failures.

Critical weaknesses
  • Limited realism and generalizability of the constructed workflows. Although the benchmark contains 120 workflow instances, these are derived from only 10 synthetic base workflows, each containing 4–10 auditable steps. Moreover, the error injection mechanism modifies only one output field at a single step, limiting the evaluation of more complex propagation patterns involving multiple interacting errors, branching dependencies, or error amplification.

  • This paper provides a meaningful end-to-end measure, but does not clearly distinguish failures in downstream error detection from failures in source attribution. Furthermore, the analysis does not sufficiently characterize how propagation depth, the number of affected downstream steps, or dependency complexity influence auditing performance.

  • Although some constructed workflows include scientific validation steps, the benchmark does not explicitly model autonomous verification and self-correction loops, which may exhibit different error propagation and auditing pattern.

Questions for authors
  • How does AuditSpan distinguish local computational correctness from global scientific validity when assigning reference labels to downstream outputs? In particular, when a downstream step correctly processes an erroneous upstream result, is it labeled as correct or invalid?
Reviewer_8p9wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AuditSpan, a benchmark for evaluating LLM auditors in long-horizon scientific workflows. It constructs synthetic workflows with explicit dependencies, controlled errors, and missing evidence. The evaluation examines step-level correctness, abstention, calibration, resistance to pressure, scientific validity, error propagation, and repair recognition. Experiments across twelve LLMs reveal difficulties in auditing complete workflows, tracking upstream errors, and recognizing incomplete corrections.

Critical strengths
  1. The paper addresses an important problem in evaluating LLM-based auditors. Examining error propagation and incomplete repairs extends beyond conventional step-level verification and is relevant to autonomous research agents.
  2. The evaluation considers multiple auditing behaviors across a range of models and tool settings. The controlled perturbations and explicit reference labels provide a useful basis for analyzing auditing failures.
Critical weaknesses
  1. Limited benchmark representativeness and contribution. Despite containing 120 workflow instances, AuditSpan is built from only ten synthetic base workflows with relatively short and mostly sequential dependencies. The controlled errors and predefined verification rules simplify many challenges encountered in real scientific research. Consequently, the benchmark provides limited evidence about auditing realistic long-horizon workflows, which substantially weakens its main contribution.
  2. Core findings are not sufficiently isolated from evaluation artifacts. The decline in whole-workflow success partly reflects the expected accumulation of step-level errors rather than a distinct failure of long-horizon reasoning. Although the paper includes some controlled comparisons, the evidence for additional degradation remains limited and varies across settings. Furthermore, strict response-format requirements substantially distort several model results. For example, Claude Opus 4.5 with tools achieves only 0.6% accuracy under strict scoring but 83.6% after JSON extraction. These issues make it difficult to draw reliable conclusions about the relative auditing capabilities of different models.
  3. The paper is overly text-heavy, with no figures illustrating the benchmark construction, workflow dependencies, or error propagation and repair scenarios. A representative workflow diagram and clearer visualizations of the experimental findings would substantially improve readability.
Questions for authors
  1. How do the main findings and model comparisons change when judgments are evaluated independently of response-format compliance?
  2. Can the authors more clearly isolate failures caused by long-range dependencies from the accumulation of independent step-level errors?
  3. Can the benchmark be validated on real scientific workflows with more complex dependencies and naturally occurring errors?
Reviewer_UdBxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

AuditSpan tests whether LLM auditors can detect errors, recognize missing evidence, trace errors through scientific workflows, and distinguish partial from complete repairs. It uses ten synthetic workflow families with explicit dependencies and reference checks. Twelve models are tested with and without tools, with further comparisons of history, workflow structure, and follow-up interactions.

Critical strengths
  1. The matched evidence tests expose a blind spot that accuracy alone misses: auditors can judge sufficient evidence correctly yet fail to recognize when it is missing.

  2. The paper shows why detecting an error is insufficient for reliable workflow auditing: auditors can still mistake a local repair for a complete correction while downstream errors remain.

Critical weaknesses
  1. The main comparisons mix formatting failures with scientific errors: extracting unchanged valid JSON raises Opus-with-tools accuracy from 0.6% to 83.6% and reverses Sonnet's tool-budget trend (Table 10(c–d)). Pressure and repair tests also require a correct initial rejection under strict scoring, so JSON extraction alone cannot recover missing follow-up responses (§3.2).

  2. Harmful flips do not measure the net effect of pressure: for GPT-5 mini with tools, the 14.8% rate counts only neutral-Reject/pressure-Accept pairs, while accuracy changes from 77.4% to 73.7% (Table 9(a)). Repeated sampling alone can produce such flips, and the neutral-repeat controls do not cover this model (Table 9(b)).

Questions for authors

Questions

Please address the weaknesses above.

Minor comments

  1. The abstract's claim that appropriate abstention decreases with longer workflows is too broad. GPT-5.5's no-tools prefix recall rises from 0% to 6.7%; GPT-5.4 with tools rises from 0% to 10% in the controlled length experiment (Tables 5(a), 7(a)).

  2. Discuss how AuditSpan's evidence-withholding and partial-repair tests relate to AgentProp-Bench's evaluator-reliability and error-propagation tests and Who&When Pro's controlled failure attribution.

  3. Table 5(b) reports zero harmful-change rates for Opus and Sonnet without tools, while Tables 1 and 9(a) report N/A. E.2.4 requires N/A when no schema-valid reply pair exists.

  4. Table 5(b) reports ECE and coverage, not the confidence-minus-accuracy gaps of 5.2 and 34.2 percentage points quoted in §4.1, so those values cannot be checked from the cited table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.