- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SurveyAudit, an automated framework for detecting reviewer concerns in literature surveys and providing traceable supporting evidence. The authors first construct ConTax, a taxonomy of 11 reviewer concern categories derived from real-world survey reviews. The framework then designs detection procedures for each category by combining LLM-based semantic judgments and explicit verification mechanisms. Experiments on human-written surveys and controlled perturbations show that SurveyAudit achieves higher detected concern precision and concern coverage. Ablation studies further investigate the contributions of the framework's key components.
Critical strengths
-
The paper is well-written. The authors provide detailed descriptions of the construction of the taxonomy and the evaluation benchmark, and the implementation of SurveyAudit, supporting transparency and reproducibility. The taxonomy and framework are also well-motivated and supported in empirical evidence.
-
The experimental evaluation is comprehensive. The authors evaluate the framework on human-written surveys, controlled perturbations, and AI-generated surveys to demonstrate its practical value. Ablation studies and detailed analyses provide further insights into the effectiveness of the framework and its individual components.
Critical weaknesses
- Although the framework provides traceable evidence for detected concerns, traceability does not necessarily guarantee that the evidence meaningfully supports the reported concerns. While the human evaluation assesses whether concerns are acceptable, it does not explicitly evaluate the relevance or sufficiency of the supporting evidence. A separate evaluation of evidence quality would help validate this central aspect of the proposed framework.
Questions for authors
-
Could the authors provide the domain distribution of the surveys used to construct ConTax? If the dataset is concentrated in certain research domains, could the authors further analyze or discuss the extent to which the resulting 11 concern categories generalize to reviewer concerns in other domains?
-
The denominator of RCC includes all observed reviewer concerns, including those outside the ConTax taxonomy. Therefore, this metric conflates taxonomy coverage with the framework's detection capability within the defined taxonomy. Could the authors also report and discuss the in-scope RCC for the 11 concern categories?
-
Could the authors provide a computational cost comparison between SurveyAudit and the baselines?
2 comments
Thank you for the detailed responses and additional analyses. I appreciate the authors' efforts in addressing my concerns and will maintain my positive rating.
Response to reviewer qUWe
We thank all reviewers for their careful reading and constructive feedback. We have conducted additional analyses and revised our explanations to clarify the scope, evidence guarantees, reproducibility, and practical requirements of \textsc{SurveyAudit}. Below, we respond to each comment in detail.
W1. Evidence relevance and sufficiency
Thanks for pointing out. As clarified in our response to Reviewer o3AQ (W1), our verification mechanism establishes evidence provenance and enables readers to inspect the evidence and reasoning behind each finding. It does not guarantee the semantic correctness of every judgment. We will clarify this distinction in the revised manuscript and use more precise terminology to characterize the framework as auditable and evidence-traceable, rather than implying that all judgments are independently verified.
Q1. Domain distribution of the 118 papers used to construct \textsc{ConTax}
We will add a domain-level breakdown of the 118 development papers. Under our coarse-grained classification, the distribution is:
This distribution shows that the development corpus is not concentrated in a single narrow technical subfield. The 11 \textsc{ConTax} categories are intended to characterize recurring forms of survey-quality concerns—such as inconsistency, missing evidence, comparison, synthesis, and literature coverage—rather than domain-specific technical errors. These concern structures are therefore expected to be relevant across multiple AI/ML subfields.
We nevertheless acknowledge two limitations. First, all 118 development papers are from TMLR, so the development corpus is venue-concentrated. Second, the domain distribution alone does not establish detector performance in every domain. The held-out evaluation on 20 non-TMLR surveys provides additional cross-venue evidence, but we will avoid making a universal cross-domain generalization claim. We will clarify these limitations and provide the domain-classification criteria in the supplementary material.
Q2. In-scope RCC for the 11 \textsc{ConTax} categories
We agree that the overall RCC denominator includes concerns outside the 11 supported categories and therefore combines taxonomy coverage with detection performance within the intended scope. We now report in-scope RCC using only the 322 golden reviewer concerns assigned to C1–C11:
This analysis makes the distinction explicit: overall RCC measures recovery over all observed reviewer concerns, whereas in-scope RCC measures recovery within the 11 operational categories. \textsc{SurveyAudit} substantially outperforms both baselines on the latter. We will add this result and discuss the two metrics separately.
Q3. Computational-cost comparison
We agree that readers need operational information to assess the practical cost of \textsc{SurveyAudit}. For human-authored surveys, the conventional LLM judge uses one model call and approximately 43,114 tokens (39,779 input / 3,335 output) per survey, with a runtime of about 4 minutes. The tool-augmented judge takes approximately 8 minutes, although detailed model-call and token logs are not available. \textsc{SurveyAudit} takes approximately 25 minutes per human-authored survey, with an average of 807.10 LLM calls, approximately 806,139 tokens (725,415 input / 80,724 output), and 298.04 academic-search calls.
These figures show that \textsc{SurveyAudit} is more resource-intensive than the no-tool baseline. This is expected because it retrieves and processes cited literature and performs category-specific evidence checks. Our contribution is not a cost-minimization method. It is a framework for detecting a broader set of reviewer-relevant concerns with explicit evidence traces.
We further evaluated \textsc{SurveyAudit} with GPT-OSS-120B, an alternative open-weight model, and obtained 84.15% DCP, 32.16% RCC, and 9.56 ACpS. The results provide preliminary evidence that \textsc{SurveyAudit} is not tied to a single proprietary model backbone. In particular, using an open-weight model that can be self-hosted offers an alternative to relying exclusively on externally hosted proprietary APIs. This may provide users with greater flexibility in model deployment and API expenditure, depending on their available hardware and serving infrastructure.
We will report the observable operational statistics and clarify that we cannot provide GPU-hour or monetary infrastructure-cost estimates because the experiments used an internally deployed model service and the corresponding hardware-usage logs were unavailable to us.