- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ARA (Audit-Risk Control for Assurance), a framework for evaluating the reliability of LLM reasoning by adapting principles from financial auditing. ARA decomposes generated responses into verifiable claims, estimates their error risks, and allocates verification resources under a token budget using risk-weighted knapsack optimization. The framework combines deterministic recomputation and LLM-based verification to issue audit opinions and provides empirical certificates reporting coverage, residual error rates, and confidence bounds. Experiments on Qwen2.5-7B, DeepSeek V4 Pro, and Tencent Hunyuan Hy3 show that ARA can reduce the error rate among certified answers while also revealing limitations in evidence extraction and risk control.
Critical strengths
- The paper investigates how to assess the reliability of LLM-generated reasoning through structured verification and evidence-based auditing. This is relevant to practical applications where identifying unreliable model outputs is important.
- The paper includes multiple baselines, ablation studies, statistical tests, and evaluations across different model deployments. The authors also report negative results and methodological limitations, improving the transparency of the evaluation.
- The paper investigates errors introduced by the evidence extraction process, particularly the Unicode normalization issue. This reveals how verification pipelines can introduce failures independently of the models being audited and provides useful insights into the reliability of LLM auditing systems.
Critical weaknesses
- The proposed framework primarily integrates existing techniques, including claim-level verification, selective prediction, classical statistical confidence bounds, and budget-constrained verification allocation. Although the financial auditing perspective is interesting, the paper does not sufficiently clarify what substantive technical contributions this integration provides beyond existing verification and risk-control approaches.
- ARA's statistical certificate relies on ground-truth correctness labels for certified answers, with residual risk and confidence bounds computed on evaluated data. Although the system can issue an audit opinion for an individual answer, the certificate does not provide a correctness guarantee for a new individual response. This leaves a gap between the paper's motivation of per-answer assurance and the statistical guarantees it actually provides.
- The CPU-only evidence setting achieves 99.0% and 98.8% coverage on DeepSeek GSM8K and Stress, respectively, while maintaining relatively low error rates. In comparison, the full ARA framework achieves lower coverage in some settings. Furthermore, risk-weighted knapsack allocation does not consistently outperform uniform evidence purchasing. These results raise questions about the additional value of the risk model, LLM auditors, and multi-level evidence hierarchy.
Questions for authors
- Compared with existing selective prediction and risk-control approaches, what specific technical capabilities or practical advantages does the financial audit-risk formulation provide beyond integrating established verification components into a unified framework?
- For new problems without ground-truth answers in real-world deployment, how should users interpret and utilize the reported certificate to assess the reliability of a particular response? Could the authors further clarify the relationship between per-answer audit opinions and the statistical guarantees over the certified subset?
- Could the authors provide a more direct comparison between CPU-only auditing and the full ARA framework under comparable coverage, risk, and computational cost, and identify which additional components contribute meaningful improvements?