acceptodds
Under review as a conference paper at ICLR 2027

Assurance, Not Just Accuracy: Auditing LLM Reasoning with Audit-Risk Control

Abstract

Benchmarks report how accurate a large language model (LLM) is on average; users of a single answer need to know how much they can trust this output. We transplant the institutional design of financial auditing into LLM quality control and study the resulting system, Ara (Audit-Risk control for Assurance). Ara converts a target misstatement risk into per-claim detection-risk budgets via the audit-risk model (inherent, control, and detection risk), purchases evidence under a token budget by risk-weighted knapsack, ranks evidence by independence (CPU recomputation over isolated LLM procedures over non-isolated ones), and issues per-output audit opinions. The deliverable is an empirical dual-caliber certificate, conditional on the realized certified subset of the measured distribution: a one-sided Clopper–Pearson 95% upper bound on residual error over the extracted claim universe, at a stated coverage; it is not a per-answer or future-sample guarantee. is a target, not a promise, and we falsify it openly: across twelve trackdomain settings is falsified in three, while 0.05 and 0.10 hold wherever the pass is not vacuous. On GSM8K with Qwen2.5-7B, Ara certifies 89.0% of answers at 3.8% residual error (raw: 10.9%). Auditing the frontier API model DeepSeek V4 Pro certifies 93.4% of GSM8K-slice answers at 2.1% risk (CP95 3.6%) and 57.7% of stress answers at zero error as-run; a deterministic rejudging recovers coverage to 96.2%/78.2%. Root-causing attributes 98.7% of adverse evidence to our own extraction layer, dominated by a single Unicode minus-sign artifact, not arithmetic inability. A sealed reproduction on a second, weaker deployment (Tencent Hunyuan Hy3, reasoning disabled, ¥15.43) re-confirms the certificate on GSM8K, does not confirm it on stress, and finds evidence-channel value generator-dependent, not universal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM auditing, selective prediction, risk control, statistical guarantees, audit opinions, evidence hierarchy, conformal prediction, reliable deployment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BMuFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARA (Audit-Risk Control for Assurance), a framework for evaluating the reliability of LLM reasoning by adapting principles from financial auditing. ARA decomposes generated responses into verifiable claims, estimates their error risks, and allocates verification resources under a token budget using risk-weighted knapsack optimization. The framework combines deterministic recomputation and LLM-based verification to issue audit opinions and provides empirical certificates reporting coverage, residual error rates, and confidence bounds. Experiments on Qwen2.5-7B, DeepSeek V4 Pro, and Tencent Hunyuan Hy3 show that ARA can reduce the error rate among certified answers while also revealing limitations in evidence extraction and risk control.

Critical strengths
  1. The paper investigates how to assess the reliability of LLM-generated reasoning through structured verification and evidence-based auditing. This is relevant to practical applications where identifying unreliable model outputs is important.
  2. The paper includes multiple baselines, ablation studies, statistical tests, and evaluations across different model deployments. The authors also report negative results and methodological limitations, improving the transparency of the evaluation.
  3. The paper investigates errors introduced by the evidence extraction process, particularly the Unicode normalization issue. This reveals how verification pipelines can introduce failures independently of the models being audited and provides useful insights into the reliability of LLM auditing systems.
Critical weaknesses
  1. The proposed framework primarily integrates existing techniques, including claim-level verification, selective prediction, classical statistical confidence bounds, and budget-constrained verification allocation. Although the financial auditing perspective is interesting, the paper does not sufficiently clarify what substantive technical contributions this integration provides beyond existing verification and risk-control approaches.
  2. ARA's statistical certificate relies on ground-truth correctness labels for certified answers, with residual risk and confidence bounds computed on evaluated data. Although the system can issue an audit opinion for an individual answer, the certificate does not provide a correctness guarantee for a new individual response. This leaves a gap between the paper's motivation of per-answer assurance and the statistical guarantees it actually provides.
  3. The CPU-only evidence setting achieves 99.0% and 98.8% coverage on DeepSeek GSM8K and Stress, respectively, while maintaining relatively low error rates. In comparison, the full ARA framework achieves lower coverage in some settings. Furthermore, risk-weighted knapsack allocation does not consistently outperform uniform evidence purchasing. These results raise questions about the additional value of the risk model, LLM auditors, and multi-level evidence hierarchy.
Questions for authors
  1. Compared with existing selective prediction and risk-control approaches, what specific technical capabilities or practical advantages does the financial audit-risk formulation provide beyond integrating established verification components into a unified framework?
  2. For new problems without ground-truth answers in real-world deployment, how should users interpret and utilize the reported certificate to assess the reliability of a particular response? Could the authors further clarify the relationship between per-answer audit opinions and the statistical guarantees over the certified subset?
  3. Could the authors provide a more direct comparison between CPU-only auditing and the full ARA framework under comparable coverage, risk, and computational cost, and identify which additional components contribute meaningful improvements?
Reviewer_jKT6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper transplants the financial-auditing risk model into LLM reasoning quality control, the target misstatement risk is converted into a per-claim detection-risk budget, and a knapsack allocator "purchases" evidence under a token budget according to an evidence-independence hierarchy.

Critical strengths
  1. The authors commit to testing each alpha cell in both directions, report three falsifications at alpha equals 0.01 and one vacuous pass, and keep a repair batch at a non-significant p-value rather than hiding it. This is consequential because the central claim of the paper is that assurance is only meaningful if it can fail in public, and the authors hold themselves to that standard rather than merely asserting it.
  2. The root-cause analysis of adverse evidence, which attributes the overwhelming majority of adverse-driving claims to a single Unicode normalization artifact in the authors' own extractor. This is consequential because it converts a potential weakness into the paper's most persuasive demonstration that an assurance system must audit the interface as much as the model, and it is the kind of finding that only a system actually run end-to-end could produce.
Critical weaknesses
  1. The narrow domain of validation relative to the generality of the framing. The certificate is demonstrated only on arithmetic reasoning where claims are exactly CPU-recomputable, and the authors concede that without machine-checkable claims the adverse tier is unreachable. Since the paper positions itself as a general framework for LLM quality control, the gap between the institutional framing and the arithmetic-only evidence is substantial, and the out-of-domain probes are explicitly not certificate-issuing.
  2. The extraction layer, which the authors correctly identify as the dominant error source, has recall bounded at roughly 0.40 even on human-cleaned text, and answers with zero extracted claims receive an unqualified opinion by default. The authors flag this as a design option rather than enabling the conservative variant, which means the reported certified subsets may include answers that were never substantively audited at the claim level. This is a material concern for the interpretation of the certified risk numbers, even if the validity argument in terms of gold-label counting is technically correct.
Questions for authors
  1. Could the authors report the certified risk and coverage under the conservative variant in which zero-claim answers are routed to an inconclusive flag rather than an unqualified opinion, at least for the Track A and Track B working points, so that readers can see how much of the certified subset is substantively audited at the claim level?

  2. How should the dual-caliber certificate be interpreted in a deployment where the extractor recall is known to be low and the claim universe is a small visible subset, given that the numerator counts wrong certified answers against gold but the audit's flagging power is bounded by what the extractor sees?

  3. The allocator is reported to tie or lose to uniform purchasing on every workload, and the authors attribute this to floors and coarse procedure costs. Is there a workload regime the authors can identify, even analytically, in which the knapsack allocation would be expected to dominate uniform purchasing, and if not, what does the budget equation contribute beyond accounting?

  4. Given that certificate-level error events did not transfer across serving stacks at ninety-seven percent answer agreement, what is the intended deployment protocol for re-issuing certificates, and how frequently would re-measurement need to occur for the assurance to remain meaningful?

Reviewer_AJfoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes to not only use the benchmarks' accuracy to identify the performance and the robustness of an LLM. The paper claim that the reliability of the answers from LLMs can be judged by the Audit-Risk control method. To achieve this, when a LLM is generating one response, the method can break down the generation into several segments or intermediate steps, given the results of each step, the paper design ARA to automatically detect whether it is reliable enough to move to the next steps.

Critical strengths

The paper approaches LLM reliability from the perspective of financial auditing, offering a novel angle.

Critical weaknesses

The paper reads as if it were almost entirely generated by an AI agent. The writing is extremely difficult to follow and does not adhere to common writing styles in computer science research. Instead of using established terminology and clearly defining technical concepts, the paper introduces an excessive number of unusual, seemingly invented terms, such as "dual-caliber guarantee," "sealed reproduction," "honesty protocol," and "independence as the currency." These terms make the paper unnecessarily complicated without clearly communicating the underlying technical contributions.

The section titles also have a distinctly AI-generated style, with examples such as "The Guarantee, Purchased for Real Money," "Anatomy: Where the Errors Actually Live," and "Root Cause: The Audit Caught the Interface, Not the Arithmetic." The paper has too many sections and subsections, making it difficult to identify a coherent technical narrative. There is no sufficiently clear or systematic presentation of the proposed method or experimental results. Although the paper contains tables and figures, they are poorly organized and do not provide a straightforward overview of the methodology, experimental setup, and key findings. The unconventional formatting further reduces readability.

Much of the paper consists of grandiose, novelty-sounding claims and excessively defensive writing rather than concrete technical explanations. The authors repeatedly emphasize their honesty, rigor, and willingness to report negative results, often using unnecessarily rhetorical language. Overall, the paper feels more like a collection of AI-generated terminology, claims, and justifications than a carefully structured scientific manuscript. In its current form, it is extremely difficult to understand what the actual technical contributions are and how the experimental evidence supports them.

Questions for authors

I have no questions for the authors. The paper is incoherently organized that I find it difficult to meaningfully assess its technical contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.