- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AuditSpan, a benchmark for evaluating the reliability of LLM-based auditors in long-horizon scientific workflows. AuditSpan investigates whether auditors can assess scientific validity across interdependent steps, identify propagated errors, and recognize incomplete corrections. The benchmark consists of 120 workflow instances derived from 10 synthetic scientific workflows across five domains, with controlled error injection, explicit step dependencies, and independently established reference labels. The authors propose evaluation metrics covering step-level and workflow-level correctness, abstention, confidence calibration, resistance to pressure, scientific validity, error tracing, and correction tracking. Experiments involving 12 LLMs, with and without tool access, reveal that auditing reliability deteriorates across extended workflows and that models frequently fail to identify downstream consequences of upstream errors or incorrectly consider partially repaired workflows fully resolved.
Critical strengths
-
Well-motivated research problem and clear contributions. The paper addresses an important and relatively underexplored problem: evaluating the reliability of LLM auditors in long-horizon workflows where errors can propagate through dependent steps.
-
Comprehensive and well-structured evaluation metrics. A major strength is the evaluation methodology, which goes beyond conventional step-level accuracy to examine workflow-level reliability and dependency-aware reasoning.
-
Controlled benchmark construction and informative empirical analysis. The benchmark uses explicit workflow dependencies, controlled error perturbations, and independently established reference labels, enabling systematic analysis of auditing failures.
Critical weaknesses
-
Limited realism and generalizability of the constructed workflows. Although the benchmark contains 120 workflow instances, these are derived from only 10 synthetic base workflows, each containing 4–10 auditable steps. Moreover, the error injection mechanism modifies only one output field at a single step, limiting the evaluation of more complex propagation patterns involving multiple interacting errors, branching dependencies, or error amplification.
-
This paper provides a meaningful end-to-end measure, but does not clearly distinguish failures in downstream error detection from failures in source attribution. Furthermore, the analysis does not sufficiently characterize how propagation depth, the number of affected downstream steps, or dependency complexity influence auditing performance.
-
Although some constructed workflows include scientific validation steps, the benchmark does not explicitly model autonomous verification and self-correction loops, which may exhibit different error propagation and auditing pattern.
Questions for authors
- How does AuditSpan distinguish local computational correctness from global scientific validity when assigning reference labels to downstream outputs? In particular, when a downstream step correctly processes an erroneous upstream result, is it labeled as correct or invalid?