- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents DeepSurvey, an agent-oriented framework for automated survey generation that emphasizes analytical depth and citation reliability. The framework integrates graph-based retrieval, full-text analysis, cross-paper relation modeling, and code-level analysis with evidence-constrained writing and multi-granularity refinement. Extensive experiments demonstrate improvements in content quality and citation reliability across multiple domains, as well as promising results on downstream research tasks. Although its methodological novelty is somewhat limited, the framework is comprehensive and supported by substantial empirical evaluation.
Critical strengths
-
Comprehensive literature analysis. DeepSurvey constructs a structured knowledge base at the paper, cluster, and survey levels. Full-text analysis, cross-paper relation graphs, comparison tables, and guided Q&A help capture technical connections across papers and improve survey depth.
-
Useful code-level analysis. The Code Agent Subsystem examines open-source repositories to recover implementation details that are insufficiently described in papers. This is particularly valuable for computer science surveys.
-
Citation reliability throughout the pipeline. Citation graph expansion, citation anchoring, evidence-constrained writing, and multi-agent revision collectively address citation reliability. Comparisons with SurveyForge support the effectiveness of this design.
-
Extensive evaluation. The experiments compare against four representative methods across five domains, with additional cross-domain evaluation and ablation studies. The paper also examines the reliability of its evaluation methodology.
-
Practical downstream value. The PINNacle experiments evaluate whether generated surveys can support autonomous research agents. The positive results provide encouraging evidence of their practical utility.
Critical weaknesses
-
Limited methodological novelty. The main contribution lies in integrating established techniques into a comprehensive system. While this integration is valuable, there is a lot of existing deepsurvey research. So the paper could more clearly distinguish its novel contributions from existing approaches.
-
Best-of-three reporting. Table 2 reports the best result across three runs rather than the average. Although Appendix C.5 explains this choice and applies it consistently, the results do not adequately characterize typical performance or variability. Reporting means and standard deviations would strengthen the evaluation.
-
Small-scale expert evaluation. The reported win rates against human-written surveys—83.3% for overall quality and 100% for content depth—are encouraging. However, the evaluation covers only six topics with four annotators each, yielding 24 paired judgments per dimension. This scale limits the generalizability of the conclusions, particularly claims of superiority over human-written surveys.
Minor issues:
- In Section 2.3, “(details in Appendix A.1.” is missing a closing parenthesis.
- Capitalization is inconsistent in Figure 3.
Questions for authors
- Could you further explain which components or interactions in DeepSurvey constitute the main methodological advances over existing survey generation frameworks?
- Could you report the mean and standard deviation across the three runs in Table 2?
- Given the limited scale of the expert evaluation, how robust are the reported win rates across topics and annotators? Could you provide more evidence supporting its robustness?
6 comments
Response to Reviewer 6Aes (Part 2 of 3)
2. Complete three-run results support DeepSurvey's overall advantage (-> weakness 2, question 2)
We thank the reviewer for this suggestion. We report the case-by-case results for all three runs, followed by their mean and standard deviation (Lines 1558–1609 in the revised manuscript).
2.1 Case by case and mean&std data
Table 1. Case-by-case PINNacle results for all three runs of each condition. Errors are relative (lower is better); X-factor, Novelty, and Insightfulness are higher-is-better.
| Condition | Run | Burgers-1D-C | Heat-2D-CG | Heat-2D-MS | Wave-1D-C | Overall | X-factor | Novelty | Insightfulness |
|---|---|---|---|---|---|---|---|---|---|
| DeepSurvey | 1 | 1.36e-2 | 2.64e-2 | 1.18e-2 | 9.54e-3 | 1.53e-2 | 11.18× | 5.67 | 6.00 |
| DeepSurvey | 2 | 1.46e-2 | 2.74e-2 | 2.63e-2 | 9.54e-3 | 1.95e-2 | 8.81× | 5.00 | 5.00 |
| DeepSurvey | 3 | 1.88e-2 | 2.60e-2 | 3.12e-5 | 5.10e-1 | 1.39e-1 | 1.24× | 5.00 | 5.00 |
| Human-written survey | 1 | 2.20e-2 | 2.59e-2 | 4.80e-2 | 1.63e-1 | 6.48e-2 | 2.65× | 2.67 | 2.00 |
| Human-written survey | 2 | 5.12e-2 | 1.63e-1 | 1.23e-1 | 1.68e-1 | 1.26e-1 | 1.36× | 4.33 | 4.00 |
| Human-written survey | 3 | 1.56e-2 | 5.66e-2 | 7.70e-2 | 3.72e-1 | 1.30e-1 | 1.32× | 4.67 | 3.00 |
| SurveyForge | 1 | 1.88e-2 | 2.60e-2 | 9.73e-2 | 5.10e-1 | 1.63e-1 | 1.05× | 1.00 | 1.00 |
| SurveyForge | 2 | 1.88e-2 | 2.60e-2 | 9.73e-2 | 5.10e-1 | 1.63e-1 | 1.05× | 1.00 | 1.00 |
| SurveyForge | 3 | 1.42e-2 | 6.77e-2 | 6.64e-2 | 5.41e-1 | 1.72e-1 | 1.00× | 4.33 | 4.67 |
| No survey | 1 | 1.33e-2 | 1.78e-2 | 1.85e-2 | 4.68e-1 | 1.29e-1 | 1.33× | 4.00 | 2.67 |
| No survey | 2 | 1.88e-2 | 2.60e-2 | 9.73e-2 | 5.10e-1 | 1.63e-1 | 1.05× | 1.00 | 1.00 |
| No survey | 3 | 1.34e-2 | 1.66e-2 | 1.30e-2 | 6.22e-1 | 1.66e-1 | 1.03× | 4.67 | 3.67 |
The best run for each condition, selected by the lowest Overall error, is summarized below.
Table 2. Best-run PINNacle result per condition, where the best run is selected by the lowest Overall error.
| Condition | Burgers-1D-C | Heat-2D-CG | Heat-2D-MS | Wave-1D-C | Overall | X-factor | Novelty | Insightfulness |
|---|---|---|---|---|---|---|---|---|
| DeepSurvey | 1.36e-2 | 2.64e-2 | 1.18e-2 | 9.54e-3 | 1.53e-2 | 11.18× | 5.67 | 6.00 |
| Human-written survey | 2.20e-2 | 2.59e-2 | 4.80e-2 | 1.63e-1 | 6.48e-2 | 2.65× | 2.67 | 2.00 |
| SurveyForge | 1.88e-2 | 2.60e-2 | 9.73e-2 | 5.10e-1 | 1.63e-1 | 1.05× | 1.00 | 1.00 |
| No survey | 1.33e-2 | 1.78e-2 | 1.85e-2 | 4.68e-1 | 1.29e-1 | 1.33× | 4.00 | 2.67 |
Table 3. Three-run mean ± standard deviation of the PINNacle results per condition.
| Condition | Burgers-1D-C | Heat-2D-CG | Heat-2D-MS | Wave-1D-C | Overall | X-factor | Novelty | Insightfulness |
|---|---|---|---|---|---|---|---|---|
| DeepSurvey | 1.567e-2 ± 2.253e-3 | 2.660e-2 ± 5.888e-4 | 1.271e-2 ± 1.074e-2 | 1.764e-1 ± 2.359e-1 | 5.793e-2 ± 5.735e-2 | 7.077× | 5.223 ± 0.316 | 5.333 ± 0.471 |
| Human-written survey | 2.960e-2 ± 1.550e-2 | 8.183e-2 ± 5.875e-2 | 8.267e-2 ± 3.088e-2 | 2.343e-1 ± 9.737e-2 | 1.069e-1 ± 2.984e-2 | 1.777× | 3.890 ± 0.874 | 3.000 ± 0.816 |
| SurveyForge | 1.727e-2 ± 2.168e-3 | 3.990e-2 ± 1.966e-2 | 8.700e-2 ± 1.457e-2 | 5.203e-1 ± 1.461e-2 | 1.660e-1 ± 4.243e-3 | 1.033× | 2.110 ± 1.570 | 2.223 ± 1.730 |
| No survey | 1.517e-2 ± 2.569e-3 | 2.013e-2 ± 4.177e-3 | 4.293e-2 ± 3.851e-2 | 5.333e-1 ± 6.500e-2 | 1.527e-1 ± 1.678e-2 | 1.137× | 3.223 ± 1.596 | 2.447 ± 1.101 |
Response to Reviewer 6Aes (Part 3 of 3)
Values are reported as mean ± standard deviation over three runs. X-factor is the arithmetic mean of the three run-level factors,
2.2 Brief analyses on the case by case and mean&std data in PINN experiment
where each run-level factor is computed as the baseline error divided by that run's Overall error. We also found that the original Table 2 in the manuscript combined algorithmic novelty with diagnostic insightfulness; we therefore separated these dimensions, re-evaluated all runs, and updated Table 2 of the manuscript and Appendix C accordingly (Lines 357–389, 1358–1440, and 1558–1609).
The advantage is not limited to the selected best run. DeepSurvey achieves the lowest three-run mean Overall error (5.793e-2), 45.3% below the next-best condition, together with the highest mean X-factor, Novelty, and Insightfulness. The case-wise results also show a clear advantage on Heat-2D-MS and Wave-1D-C while remaining competitive on Burgers-1D-C and Heat-2D-CG, yielding the strongest aggregate performance across cases.
Note that some runs or individual cases have identical errors because unsuccessful modifications leave the vanilla implementation unchanged.
3. Internally consistent expert judgments and appropriately scoped conclusions (-> weakness 3, question 3)
We thank the reviewer for highlighting this limitation. We agree that six topics and 24 paired judgments per dimension are insufficient to establish broad superiority over human-written surveys across topics and domains. Our study was intended to provide initial high-quality evidence through blinded pairwise evaluation by researchers with relevant domain expertise, rather than large-scale crowdsourced evaluation. Expanding it reliably would require recruiting additional experts with research and publication experience in each new topic, which is difficult to complete within the rebuttal period. Nevertheless, our meta-evaluation shows substantial agreement among human experts: mean pairwise Cohen’s κ is 0.6958, and the all-human panel achieves Fleiss’ κ of 0.6578 (Section 3.6 and Appendix B.3). Although these agreement results do not increase the evaluation scale, they indicate that the expert judgments are internally consistent rather than driven by annotator noise. We will explicitly acknowledge the scale limitation and consider a broader expert evaluation in future work.
The current results therefore provide consistent but deliberately scoped evidence: DeepSurvey received 83.3% preference for overall quality and was preferred in all 24 judgments for Content Depth. Importantly, this human evaluation is supplementary; our primary evidence for DeepSurvey’s advantage in agent-oriented settings comes from the downstream agent experiments (Lines 348–410), which directly measure how effectively each survey supports research agents. To ensure academic rigor and appropriately scope our conclusion, we will revise the general claim “superior to human-written surveys” to the more precise statement “preferred by domain experts in our six-topic evaluation.” (Lines 421–422 in the revised manuscript)
4. Minor issues corrected
We thank the reviewer for the careful reading and for identifying these issues. We have corrected them in the revised manuscript.
Response to Reviewer 6Aes (Part 1 of 3)
Your careful and constructive comments have helped us improve the clarity and empirical presentation of our work, and we sincerely thank you for them. The revised manuscript has been uploaded, with the main revisions marked in blue.
1. Methodological advances beyond existing survey-generation frameworks (-> weakness 1, question 1)
We thank the reviewer for raising this important point. We first emphasize that our primary contribution and core motivation lie in being the first work to propose agent-oriented survey generation. Rather than focusing on the isolated construction of any single module, our paper centers on realizing agent-oriented survey generation around analytical depth and citation reliability—an objective thoroughly validated by both our rubric-based evaluations and downstream experiments. As automated research and recursive self-improvement (RSI) become increasingly important, we believe this problem formulation is both timely and novel, and that DeepSurvey provides useful infrastructure for subsequent research in these areas.
Beyond this novel problem formulation, DeepSurvey introduces methodological advances at both the component and system levels. We agree that the original manuscript did not explain these distinctions from prior methods in sufficient detail and will clarify them in the revision (Lines 935–963 in the revised manuscript).
System-level architectural designs and interactions.
Cross-domain framework design. To the best of our knowledge, DeepSurvey is the first survey-generation framework explicitly designed for cross-domain generalization, whereas most prior systems focus on a single domain. It integrates multi-domain reference sources and open-ended full-text extraction across retrieval, analysis, and generation, supporting cross-domain RSI and automated research.
Pipeline-wide citation-reliability control. Prior systems typically check citations through post-hoc refinement. In contrast, DeepSurvey coordinates source attribution and citation verification across analysis, outline construction, paragraph-level writing, and multi-granularity refinement, preventing citation errors from accumulating between stages.
Component-level innovations.
Citation-graph-based retrieval. Prior work mostly uses embedding-based querying or re-ranking, or employs graphs only for analysis. To the best of our knowledge, DeepSurvey is the first survey-generation framework to use citation-graph expansion as an integral literature-retrieval mechanism. As discussed in Lines 156-161, citation graphs inherently encode inter-paper relationships, making the retrieved literature better suited to downstream graph construction and clustering.
Code-analysis integration. To our knowledge, DeepSurvey is the first survey-generation system to introduce a dedicated code-agent subsystem. It examines official repositories and extracts implementation-level knowledge that is often omitted or condensed in papers, thereby providing concrete guidance for downstream research. The experiments (Lines 471–480) and case study (Lines 2741–2753) further demonstrate the effectiveness of this subsystem.
Tiered evidence-scoped writing. DeepSurvey organizes the evidence available to each writing unit into two tiers: a compact set of gold papers that provides the primary evidential basis and a broader set of relevant papers that supplies complementary context when needed. This design keeps generation anchored in a small number of high-value sources while retaining access to broader literature for synthesis and comparison, thereby reducing irrelevant context and hallucination risk[1][2].
The ablation study (Lines 426–470 and Appendix E) empirically confirms the contribution of each of these novel components to overall system performance.
[1] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
[2] Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, et al. Context length alone hurts LLM performance despite perfect retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 23281–23298, 2025.