acceptodds
Under review as a conference paper at ICLR 2027

SurveyAudit: Evidence-Grounded Detection of Reviewer Concerns in Literature Surveys

Abstract

The rapid advancement of large language models has made automated survey generation increasingly practical, yet evaluation of survey quality remains unreliable: existing methods rely on opaque LLM verdicts or heuristic metrics that lack empirical grounding and evidence transparency. We propose SurveyAudit, a verifiable framework focusing on the detection of reviewer concerns in literature surveys. We first introduce ConTax, an empirically grounded taxonomy of 11 recurring quality concerns derived from 519 real peer-review records on 168 survey papers. SurveyAudit then operationalizes these concerns through category-specific detection procedures and an evidence-tracing protocol: each detected concern must be supported by a verifiable source span, a bounded absence certificate, or a fully specified and reproducible computation path. Experiments on human-written, AI-generated, and perturbation-controlled surveys show that SurveyAudit substantially achieves higher precision and recall than LLM- and tool-augmented judges on reviewer-relevant concern detection.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Survey Evaluation, LLM Evaluation, AI for Science, Peer Review, Concern Detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qUWeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SurveyAudit, an automated framework for detecting reviewer concerns in literature surveys and providing traceable supporting evidence. The authors first construct ConTax, a taxonomy of 11 reviewer concern categories derived from real-world survey reviews. The framework then designs detection procedures for each category by combining LLM-based semantic judgments and explicit verification mechanisms. Experiments on human-written surveys and controlled perturbations show that SurveyAudit achieves higher detected concern precision and concern coverage. Ablation studies further investigate the contributions of the framework's key components.

Critical strengths
  1. The paper is well-written. The authors provide detailed descriptions of the construction of the taxonomy and the evaluation benchmark, and the implementation of SurveyAudit, supporting transparency and reproducibility. The taxonomy and framework are also well-motivated and supported in empirical evidence.

  2. The experimental evaluation is comprehensive. The authors evaluate the framework on human-written surveys, controlled perturbations, and AI-generated surveys to demonstrate its practical value. Ablation studies and detailed analyses provide further insights into the effectiveness of the framework and its individual components.

Critical weaknesses
  1. Although the framework provides traceable evidence for detected concerns, traceability does not necessarily guarantee that the evidence meaningfully supports the reported concerns. While the human evaluation assesses whether concerns are acceptable, it does not explicitly evaluate the relevance or sufficiency of the supporting evidence. A separate evaluation of evidence quality would help validate this central aspect of the proposed framework.
Questions for authors
  1. Could the authors provide the domain distribution of the surveys used to construct ConTax? If the dataset is concentrated in certain research domains, could the authors further analyze or discuss the extent to which the resulting 11 concern categories generalize to reviewer concerns in other domains?

  2. The denominator of RCC includes all observed reviewer concerns, including those outside the ConTax taxonomy. Therefore, this metric conflates taxonomy coverage with the framework's detection capability within the defined taxonomy. Could the authors also report and discuss the in-scope RCC for the 11 concern categories?

  3. Could the authors provide a computational cost comparison between SurveyAudit and the baselines?

2 comments
AuthorsOpenReview ↗

Response to reviewer qUWe

We thank all reviewers for their careful reading and constructive feedback. We have conducted additional analyses and revised our explanations to clarify the scope, evidence guarantees, reproducibility, and practical requirements of \textsc{SurveyAudit}. Below, we respond to each comment in detail.

W1. Evidence relevance and sufficiency

Thanks for pointing out. As clarified in our response to Reviewer o3AQ (W1), our verification mechanism establishes evidence provenance and enables readers to inspect the evidence and reasoning behind each finding. It does not guarantee the semantic correctness of every judgment. We will clarify this distinction in the revised manuscript and use more precise terminology to characterize the framework as auditable and evidence-traceable, rather than implying that all judgments are independently verified.

Q1. Domain distribution of the 118 papers used to construct \textsc{ConTax}

We will add a domain-level breakdown of the 118 development papers. Under our coarse-grained classification, the distribution is:

DomainNumber of papersPercentage
NLP2420.3%
Computer Vision1311.0%
Core ML3025.4%
Extended ML2924.6%
AI Applications2218.6%
Total118100%

This distribution shows that the development corpus is not concentrated in a single narrow technical subfield. The 11 \textsc{ConTax} categories are intended to characterize recurring forms of survey-quality concerns—such as inconsistency, missing evidence, comparison, synthesis, and literature coverage—rather than domain-specific technical errors. These concern structures are therefore expected to be relevant across multiple AI/ML subfields.

We nevertheless acknowledge two limitations. First, all 118 development papers are from TMLR, so the development corpus is venue-concentrated. Second, the domain distribution alone does not establish detector performance in every domain. The held-out evaluation on 20 non-TMLR surveys provides additional cross-venue evidence, but we will avoid making a universal cross-domain generalization claim. We will clarify these limitations and provide the domain-classification criteria in the supplementary material.

Q2. In-scope RCC for the 11 \textsc{ConTax} categories

We agree that the overall RCC denominator includes concerns outside the 11 supported categories and therefore combines taxonomy coverage with detection performance within the intended scope. We now report in-scope RCC using only the 322 golden reviewer concerns assigned to C1–C11:

MethodIn-scope RCC (%)
Conventional LLM Judge16.77 (54/322)
Tool-augmented LLM Judge13.35 (43/322)
\textsc{SurveyAudit}56.52 (182/322)

This analysis makes the distinction explicit: overall RCC measures recovery over all observed reviewer concerns, whereas in-scope RCC measures recovery within the 11 operational categories. \textsc{SurveyAudit} substantially outperforms both baselines on the latter. We will add this result and discuss the two metrics separately.

Q3. Computational-cost comparison

We agree that readers need operational information to assess the practical cost of \textsc{SurveyAudit}. For human-authored surveys, the conventional LLM judge uses one model call and approximately 43,114 tokens (39,779 input / 3,335 output) per survey, with a runtime of about 4 minutes. The tool-augmented judge takes approximately 8 minutes, although detailed model-call and token logs are not available. \textsc{SurveyAudit} takes approximately 25 minutes per human-authored survey, with an average of 807.10 LLM calls, approximately 806,139 tokens (725,415 input / 80,724 output), and 298.04 academic-search calls.

These figures show that \textsc{SurveyAudit} is more resource-intensive than the no-tool baseline. This is expected because it retrieves and processes cited literature and performs category-specific evidence checks. Our contribution is not a cost-minimization method. It is a framework for detecting a broader set of reviewer-relevant concerns with explicit evidence traces.

We further evaluated \textsc{SurveyAudit} with GPT-OSS-120B, an alternative open-weight model, and obtained 84.15% DCP, 32.16% RCC, and 9.56 ACpS. The results provide preliminary evidence that \textsc{SurveyAudit} is not tied to a single proprietary model backbone. In particular, using an open-weight model that can be self-hosted offers an alternative to relying exclusively on externally hosted proprietary APIs. This may provide users with greater flexibility in model deployment and API expenditure, depending on their available hardware and serving infrastructure.

We will report the observable operational statistics and clarify that we cannot provide GPU-hour or monetary infrastructure-cost estimates because the experiments used an internally deployed model service and the corresponding hardware-usage logs were unavailable to us.

Thank you for the detailed responses and additional analyses. I appreciate the authors' efforts in addressing my concerns and will maintain my positive rating.

Reviewer_QNhZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SurveyAudit, a framework for identifying concrete problems in literature surveys and providing supporting evidence for each finding. The authors derive 11 concern categories from real peer reviews and develop category-specific detection procedures combining LLM judgments with retrieval and explicit verification. Experiments on human-written surveys and controlled perturbations show that SurveyAudit produces more valid concerns and recovers more reviewer-identified issues than general-purpose LLM judges. The paper also applies the framework to AI-generated surveys to characterize their shortcomings.

Critical strengths
  1. A strength of the paper is the connection between real reviewer concerns and the framework’s design. The authors organize concerns from human reviews into a taxonomy and translate them into concrete rules, prompts, and detection algorithms. This gives the framework a clear basis in what reviewers actually care about and makes its evaluation criteria explicit and inspectable.
Critical weaknesses
  1. The main comparison may overstate the precision gains attributable to the specialized pipeline. The –Algorithm variant, which retains ConTax and the evidence requirements, achieves nearly identical DCP to the full system (86.40% vs. 86.55%). Presenting this variant alongside the main baselines would clarify that the specialized detectors primarily improve concern coverage and yield, while the precision gains can largely be achieved without them.

  2. All experiments rely on a single model (DeepSeek-V4-Flash-Preview), so it is unclear whether the gains are specific to that model. Adding 1–2 other LLMs would help validate that the framework’s effectiveness does not depend on a specific model.

Questions for authors

n/a

2 comments
AuthorsOpenReview ↗

Response to reviewer QNhZ

We thank all reviewers for their careful reading and constructive feedback. We have conducted additional analyses and revised our explanations to clarify the scope, evidence guarantees, reproducibility, and practical requirements of \textsc{SurveyAudit}. Below, we respond to each comment in detail.

W1. Contribution of the specialized detectors and the –Algorithm ablation

Thanks for pointing out. We agree that the –Algorithm ablation provides an important distinction between the contributions of evidence-constrained evaluation and category-specific detector design. In Table 6, removing the category-specific detector procedures changes DCP only slightly, but reduces RCC from 34.50% to 23.54% and ACpS from 9.84 to 5.36. This indicates that the specialized detectors primarily improve concern discovery, reviewer-concern coverage, and the yield of accepted concerns, rather than DCP itself. The high DCP can largely be retained by the taxonomy and evidence requirements, whereas the specialized detector procedures enable the system to identify a broader and more diverse set of concrete concerns.

W2: Evaluation with an additional model backbone

Thanks for pointing out. We thank the reviewer for suggesting an evaluation with an additional model backbone. To examine whether the results are specific to DeepSeek-V4-Flash-Preview, we additionally evaluated \textsc{SurveyAudit} using GPT-OSS-120B on the full test set. GPT-OSS-120B achieved a DCP of 84.15%, an RCC of 32.16%, and an ACpS of 9.56, compared with the results for DeepSeek-V4-Flash-Preview. These results provide additional evidence regarding the portability of \textsc{SurveyAudit} across model backbones. We will include the experimental setup and results in the revised manuscript.

Thank you for your responses. My major concerns have been addressed and I will maintain my positive rating.

Reviewer_o3AQOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SurveyAudit, a system for identifying potential reviewer concerns in literature surveys with supporting textual evidence. The authors derive 11 verifiable concern categories (ConTax) from 519 reviews of 168 surveys on OpenReview. The system uses category-specific detectors and restricts LLMs to targeted extraction and evidence-based judgments. Each reported concern requires a source quotation, whose presence is programmatically verified.

On 50 human-written surveys, SurveyAudit achieves higher DCP (86.6% vs. 52.0%/56.5%) and RCC (34.5% vs. 20.8%/16.2%) than general-purpose LLM reviewers. Additional experiments examine 500 surveys with injected defects and AI-generated surveys.

Critical strengths
  • Requiring textual evidence for each concern is a meaningful alternative to unconstrained LLM-generated criticism. The resulting reports are easier for humans to inspect.
  • Deriving concern categories from real peer reviews and organizing them by verifiability provides a potentially valuable resource for survey evaluation.
  • The paper reports category-level accuracy, coverage, error types, and relevant prompts.
Critical weaknesses
  • The programmatic verification only checks whether a quoted passage exists in the source document. It does not verify whether the passage actually supports the identified concern or whether the LLM's interpretation is correct. Thus, the central judgments remain unverified LLM outputs. The terms "verifiable" and "evidence-grounded" suggest stronger guarantees than the system provides.

  • Approximately 57% of annotated reviewer concerns (702 external-normative and 1,232 evaluative concerns) fall outside the 11 supported categories. This substantially limits the system's ability to capture reviewer concerns in general. Furthermore, the taxonomy and detectors were developed entirely from TMLR reviews, raising questions about generalizability.

  • All systems appear to be evaluated using a single run, without confidence intervals or statistical significance tests. Correctness judgments rely on only three PhD students (two annotators and one adjudicator). The paper does not report whether evaluations were blinded or provide chance-corrected inter-annotator agreement. These limitations make the reported performance differences difficult to assess reliably.

  • The detectors also contain numerous manually selected rules and constants tuned on the development set. The paper does not report LLM calls per survey, literature retrieval requirements, computational cost, or runtime, making practical reproducibility difficult to evaluate.

Questions for authors
  • The tool-based baseline DCP is 56.46% in Table 3, but recalculating from Tables 12 and 14 gives approximately 46.4%. Which value is correct? Additionally, the descriptions of SurveyGen-I and Claude Code defects on page 8 appear reversed relative to Table 11.

  • What are the DCP and RCC results when all systems are evaluated exclusively on the same 11 categories? Is the −Algorithm ablation equivalent to an LLM reviewer explicitly informed of the taxonomy?

  • Were the injected defects manually constructed or LLM-generated? Can the method be evaluated on defects not designed around ConTax categories?

  • How much variance arises across repeated runs? Were human evaluators blinded to system identities?

6 comments
AuthorsOpenReview ↗

Response to reviewer o3AQ

We thank all reviewers for their careful reading and constructive feedback. We have conducted additional analyses and revised our explanations to clarify the scope, evidence guarantees, reproducibility, and practical requirements of \textsc{SurveyAudit}. Below, we respond to each comment in detail.

W1. Evidence verification and the meaning of “verifiable” and “evidence-grounded”

Thanks for pointing out. We agree that programmatic source-span verification alone does not establish that a quoted passage semantically supports the associated concern, nor does it guarantee that the LLM's interpretation is correct. Our current implementation checks the correspondence between a returned evidence span and its source, and it does not independently prove semantic entailment or the correctness of the underlying reasoning. Our intended meaning of evidence-grounded is that each detected concern is explicitly associated with a source span, a bounded absence certificate, or a reproducible verification trace, and the intended meaning for verifiable is that a reader can check and verify the correctness of a detected concern directly from the presented evidence. This requirement makes the basis of each judgment inspectable, but it should not be interpreted as a guarantee that every judgment is correct. In particular, the evidence trace allows readers to examine the evidence and the conditions under which a concern was reported, and thereby identify unsupported or erroneous interpretations. We will revise the abstract and introduction to remove the overly strong description of \textsc{SurveyAudit} as a “verifiable framework” and define the evidence-grounding requirement more precisely. We will clarify this distinction in the camera-ready version.

To further examine whether the evidence meaningfully supports the reported concerns, we conducted an additional manual audit of 120 sampled evidence instances. We stratified the sample into 60 instances from each of the two evidence-span precision groups. The observed support rates were 85.0% for the precise group and 61.7% for the imprecise group. Because the audit sample was stratified equally rather than sampled proportionally, we estimated the overall support rate by reweighting these two rates according to their observed proportions in the full evidence pool, obtaining 82.3% [insert the two actual stratum weights]. This additional audit directly examines evidence relevance to the associated concern, beyond checking whether a quoted span appears in its source.

W2. Coverage of reviewer concerns outside the 11 categories and generalizability beyond TMLR

Thanks for pointing out. We agree that the 11 operational categories do not cover every type of reviewer concern. We do not intend to claim that \textsc{SurveyAudit} automatically captures all reviewer concerns.

\textsc{ConTax} deliberately focuses on concerns for which an objective or reproducible verification criterion can be specified. External-normative concerns often depend on context-specific conventions or open-ended normative judgments, including conference requirements and graphics presentation concerns. Evaluative concerns encompass heterogeneous assessments of adequacy or quality that cannot generally be established through the same type of reproducible evidence check. We therefore exclude them from the directly supported detector categories rather than treating unconstrained LLM judgments as verified findings.

Importantly, our evaluation of reviewer-concern coverage does not remove these concerns from the reference set. The denominator of RCC includes all observed reviewer concerns, including External Norms and Evaluative concerns. Table 13 further shows that \textsc{SurveyAudit} can recover some concerns originally expressed in evaluative terms by identifying more concrete, operationalizable failures. For example, an evaluative concern about insufficient comparison can be addressed by identifying a specific missing comparison. However, we acknowledge that this does not amount to comprehensive coverage of Evaluative concerns: \textsc{SurveyAudit} covers 48/232 such concerns, compared with 60/232 for the conventional judge and 45/232 for the tool-augmented judge. Its coverage of External Norms is limited. We will make these boundaries explicit rather than implying that the excluded categories are fully addressed.

Regarding generalizability, the 118-paper development set is indeed composed entirely of TMLR papers. However, the held-out test set contains 50 papers, including 20 non-TMLR papers from ICLR, EMNLP, COLM, and NeurIPS. Table 14 reports the results separately. On the non-TMLR subset, \textsc{SurveyAudit} achieves 81.14% DCP, 40.74% RCC, and 7.10 ACpS, compared with 44.20%, 31.22%, and 4.00 for the conventional judge, and 43.78%, 23.81%, and 4.05 for the tool-augmented judge. These results provide evidence that the observed advantage is not restricted to TMLR surveys.

AuthorsOpenReview ↗

Response to reviewer o3AQ (2)

W3. Repeated runs, confidence intervals, and annotation reliability

Thanks for pointing out. We repeat all systems five times on the same representative subset of [N] surveys under identical configurations. We report the mean, standard deviation, and 95% confidence intervals across the five runs for DCP, RCC, and ACpS. The results show that \textsc{SurveyAudit} consistently achieves higher DCP, RCC, and ACpS than both baselines across repeated executions. We will include the repeated-run results in the camera-ready version.

MethodDCP (%)95% CIRCC (%)95% CIACpS95% CI
Conventional LLM Judge51.85 ± 3.15[45.60, 57.94]20.97 ± 2.36[16.61, 25.86]4.60 ± 0.30[4.04, 5.22]
Tool-augmented LLM Judge46.58 ± 3.73[39.21, 53.83]16.42 ± 2.38[12.18, 21.50]4.19 ± 0.33[3.53, 4.84]
\textsc{SurveyAudit}86.46 ± 1.27[84.14, 89.11]34.53 ± 1.01[32.54, 36.48]9.82 ± 0.15[9.53, 10.11]

Regarding annotation reliability, two annotators independently labeled each item, and a third Ph.D. student adjudicated disagreements. Annotators were blinded to the identity of the system that generated each audit report. We additionally computed Cohen's κ from the two independent annotations before adjudication, obtaining κ = 0.545 for DCP and κ = 0.619 for RCC. The corresponding raw agreement rates were 76.3% and 80.2%, respectively. We will add these details to the human annotation section.

W4. Manually specified rules and constants, and missing computational-cost information

Thanks for pointing out. We appreciate the concern regarding the manually specified rules and parameters in our detectors. We would like to clarify that the development set supported the construction of \textsc{ConTax} and the design of the category-specific detectors, but the numerical parameter values were manually specified rather than obtained through hyperparameter search or optimization against development-set performance. These parameters primarily define candidate-selection budgets, thresholds, and computational limits. The corresponding detection procedures, rules, and parameter configurations are disclosed in the paper and supplementary material.

We agree that the sensitivity of manually specified parameters deserves further investigation. In the camera-ready version, we will conduct a controlled sensitivity analysis on a representative subset of surveys. Rather than exhaustively searching the parameter space, we will vary each parameter independently while keeping the others fixed, covering candidate discovery, evidence verification, and finding consolidation. We will report the resulting changes in DCP, RCC, ACpS, finding count, and operational cost. The planned parameters and values are summarized below.

ParameterDescriptionValues to be TestedExpected Impact
Candidate output budget KoutK_{\text{out}}Maximum number of candidate findings retained by a detector{3,5,10}\{3, 5, 10\}Candidate coverage and finding yield
Evidence retrieval budget KverifyK_{\text{verify}}Maximum number of evidence candidates retrieved or examined for each finding{5,10,20}\{5, 10, 20\}Evidence support, verification quality, and computational cost
Category-specific decision weight α\alphaWeight balancing the two components in the category-specific decision function, e.g., I(o)=αD(o)+(1−α)G(o∣A)I(o)=\alpha D(o)+(1-\alpha)G(o\mid A){0.25,0.50,0.75}\{0.25, 0.50, 0.75\}Finding quality and category-specific detection behavior
Duplicate detection threshold τdup\tau_{\text{dup}}Similarity threshold used to identify and consolidate duplicate findings{0.70,0.80,0.90}\{0.70, 0.80, 0.90\}Finding redundancy and final finding count
Candidate expansion budget KgraphK_{\text{graph}}Number of graph-based candidates explored during candidate expansion (if applicable){50,100,200}\{50, 100, 200\}Candidate recall and retrieval cost
AuthorsOpenReview ↗

Response to reviewer o3AQ (3)

(Continue for W4)

We have also collected additional operational statistics. Since the evaluation workload depends strongly on survey length and bibliography size, we report the average cost per survey grouped by survey source:

Survey sourceLLM callsAcademic-search callsRuntimeReferencesSectionsParagraphsAtomic facts
Claude Code870.43283.73~23 min73.1746.5779.07233.20
SurveyG1,349.23199.70~49 min60.4725.47158.93317.93
SurveyGen-I2,932.231,510.43~2.6 h625.67115.50241.331,698.00
Human-authored surveys807.10298.04~25 min107.8039.84138.06136.12

These are per-survey averages over [insert sample sizes for each group]. Runtime is end-to-end wall-clock time under the released implementation configuration. It depends on the model-serving interface, the number of referenced papers, document availability, and parallelism. In particular, \textsc{SurveyAudit} retrieves and processes the full text of cited papers, so bibliography size is a major determinant of runtime. The laboratory-deployed model backend used in our experiments is slower than some externally provided inference services; therefore, these runtimes should be interpreted as measurements under our experimental deployment rather than universal runtime estimates.

For a direct cost comparison on human-authored surveys, the conventional LLM judge uses one model call and approximately 50,000 tokens per survey, with an average runtime of about four minutes. The tool-augmented judge takes about eight minutes, but its detailed model-call and token logs are unavailable. \textsc{SurveyAudit} takes approximately 25 minutes per human-authored survey, with an average of 807.10 LLM calls, approximately one million tokens, and 298.04 academic-search calls. We acknowledge that \textsc{SurveyAudit} is substantially more resource-intensive than a no-tool LLM judge. It is not designed to minimize inference cost; rather, it trades additional retrieval and evidence-processing work for broader reviewer-concern coverage and evidence-linked findings.

We cannot provide GPU-hour, energy, or monetary infrastructure-cost estimates because the experiments used an internally deployed model service and we did not have access to the corresponding hardware-usage and infrastructure-accounting logs. The internal model API did not incur per-request billing, and academic retrieval was performed within the available OpenAlex free quota. We will clarify that the absence of direct API charges does not imply zero computational cost.

Q1. Numerical correction

We thank the reviewer for identifying the numerical error. We will correct the affected value in Table 3 and check all related text and tables to ensure consistency. In particular, the Tool-augmented LLM Judge DCP value will be corrected from 56.46% to 46.55%, and the results of Citation-claim mismatch and Fabricated reference should be swapped in Table 11.

Q2. Results restricted to the 11 supported categories

We agree that reporting in-scope performance separately makes the distinction between taxonomy coverage and detector performance clearer. We aggregate the per-category DCP and RCC counts in Tables 12 and 13 over C1–C11:

MethodIn-scope DCP (%)In-scope RCC (%)ACpS
Conventional LLM Judge72.53 (66/91)16.77 (54/322)1.32
Tool-augmented LLM Judge78.08 (57/73)13.35 (43/322)1.14
\textsc{SurveyAudit}86.62 (492/568)56.52 (182/322)9.84

The in-scope RCC denominator is 322, corresponding to the reviewer concerns in C1–C11. The results show that \textsc{SurveyAudit} recovers a substantially larger fraction of concerns within its intended scope. The in-scope DCP of the two general-prompt baselines is higher than their overall DCP because their overall outputs include more External Norms and Evaluative concerns, which are less frequently accepted by annotators. We will report this analysis and explicitly distinguish in-scope detection performance from overall reviewer-concern coverage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.